Paranoid Penguin - The Future of Linux Security

Just because Linux can be more secure than other systems doesn't mean that your Linux system is. How can developers and distributors help the sysadmins of the future?
Mandatory Access Controls

I've alluded to the fact that access controls or file permissions on Linux, UNIX in general and Windows are discretionary, and that this is a weak security model. Well, what about SELinux? Doesn't that use RBACs and type enforcement (TE), both of which are examples of mandatory access controls? Yes, indeed, it does. But I'm afraid that this probably isn't the future of Linux security, for the same reasons that SELinux isn't a huge part of present Linux security.

RBACs restrict users' behavior and access to system resources based on carefully defined roles that are analogous to but more far-reaching than the conventional UNIX groups mechanism. Similarly, type enforcement restricts processes' activities based on their predefined domains of operation. The net effect of RBAC and TE is to create segregated silos (my term) in which users and processes operate, with strictly limited interaction being permitted between silos.

This is an elegant and effective security model. However, for most people, RBAC, TE and other mandatory access controls are too complicated and involve too much administrative overhead. This, in many people's view, dooms SELinux and similar operating systems to the realm of niche solutions: OSes that are useful to people with specific needs and capabilities but not destined for widespread adoption. Despite admiring SELinux's security architecture and being a fan of the concept of RBAC in general, I do not think that mandatory access controls by themselves are likely to revolutionize Linux security.

Hypervisors and Virtual Machines

If RBAC and TE do in fact prove too unwieldy to compartmentalize security breaches at the OS level, hypervisors and virtual machines (VMs) may achieve this at a higher level. We're already familiar with virtual machines in two different contexts: runtime virtual environments, such as those use by Java programs, and virtual platforms, such as VMware, plex86 and VirtualPC, that allow you to run entire operating systems in a virtualized hardware environment.

The Java Virtual Machine was designed with particular security features, most notably the Java sandbox. In general, though, Java security comes from the fact that Java applets run isolated from raw or real system resources; everything is mediated by the Java Virtual Machine. Besides being a good security model, it's also relatively simple to use safely, both for programmers and end users. Java also is, for many reasons, already ubiquitous.

Virtual platforms take this concept a step further by mediating not only individual programs but the operating systems on which they run. Security architecture in this scenario, however, isn't as mature as with the Java Virtual Machine. For the most part, security is left to the guest operating systems running in the virtual environment. A SUSE Linux virtual machine running on VMware, therefore, is no more or less secure than a real SUSE system running on its own hardware.

Hypervisor technology addresses the need to isolate virtual machines running on the same hardware from one another, restrict their interactions and prevent a security breach on one virtual machine from affecting others. IBM has created a security architecture called sHype for hypervisors. An open-source hypervisor/virtual-machine project called Xen also is available.

Although the driving purpose of a hypervisor is to prevent any one virtual machine from interfering with other virtual machines running on the same hardware—for example, by monopolizing shared hardware resources—the idea of having some sort of intelligence managing systems at this level is powerful. It may even have the potential to overshadow or, at the very least, significantly augment traditional intrusion detection systems (IDSes) as a means of detecting and containing system compromises.

Mandatory access controls and hypervisors/virtual machines aren't mutually exclusive. On the one hand, I am of the opinion, strongly influenced by my friend and fellow security analyst Tony Stieber, that hypervisors have much greater potential to shape the future of Linux security than do MACs. But on the other hand, the two can be used together. Imagine a large, powerful server system running several virtual machines controlled by a hypervisor. One VM could be running a general-purpose OS, such as Linux, serving as a Web server. Another VM, serving as a database for sensitive information, could run a MAC-based OS such as SELinux. Both VMs would benefit from security controls enforced by the hypervisor, with SELinux providing extra levels of security of its own.

Anomaly-Based Intrusion Detection and Antivirus

One additional technology, like MACs and hypervisors, already exists today but potentially will have a much bigger impact on the future: the anomaly-based intrusion detection system. The idea of anomaly-based IDS is simple: it involves creating a baseline of normal network or system activity and sending an alert any time unexpected or anomalous behavior is detected.

If the idea is simple and the technology already exists, why isn't this approach commonly used? Because it isn't nearly as mature or easy to use as signature-matching. We're all familiar with signature-based IDSes; they maintain databases of attack signatures, against which observed network packets or series of packets are compared. If a given packet matches one in the attack database, the packet is judged to be part of an attack, and an alert is sent.

The strengths of this approach are that it's easy to use and typically involves few false positives or false alarms. The fatal weakness of signature-based systems is if an attack is too new or too complicated for there to be a corresponding signature in your IDS' signature database, it is not detected.

With anomaly-based IDS, in contrast, any new attack that sufficiently differs from normal behavior is detected. The trade-off is the IDS administrator must train and periodically re-train the IDS system in order to create the normal-behavior baseline. This results in a period of frequent false positives, until the baseline has been fine-tuned.

I attended a lecture by Marcus Ranum in 1999 or so in which he described anomaly-based systems as the future of IDS. Obviously, we're not there yet. Such products are available from vendors such as Lancope and Arbor Networks. But I remain hopeful that someone will figure out how to do this sort of thing in ways that are cheaper and easier to use than current systems. Potentially, this could lead to a sort of network hypervisor that lends the same sort of intelligence to networks, whether comprised of virtual or real machines, that hypervisors lend to virtual platforms.

By the way, virus scanners need and can benefit from anomaly detection technology as much as IDSes do. This point is illustrated amply by the fact that the vast majority of organizations that use modern virus scanners, which rely almost exclusively on virus-signature matching, nonetheless suffer from major virus/trojan/worm outbreaks. Current signature-based antivirus tools clearly are not effective enough.