Hardening Linux for the Net

[ Site Index] [ Linux Index] [ Feedback ]

If you use a Linux system that never connects to the internet, you can ignore this feature: you're safe -- as long as nobody else ever logs on to your machine, you never install any new software, and you're certain that nobody has sneaked a modem or ethernet card into your PC.

But what if you have a network card connected to a cable modem? Or an ADSL line? Or a modem, for that matter, or a server co-located in a hosting facility's racks?

Linux is not particularly insecure; but like all large modern operating systems it harbours a number of undetected bugs and security flaws, some of which can be exploited by attackers. The Honeynet project ran a study last year, exposing machine with standard out-of-the-box operating systems to the internet. Their findings were alarming: one machine was completely compromised within seven hours of going online, while on average the exposed systems came under attack within minutes of being connected and most of them succumbed within a week.

In this feature, we're going to look at ways and means of "hardening" a Linux system that's going to be exposed to the risks of the public internet. There's nothing magical about this: it's just a matter of knowing how network services are controlled, how to switch them on or off and secure them, and how to keep track of current security alerts and install upgrades to prevent attacks. All of this is standard good practice for a professional system administrator -- but as Linux becomes more widespread, the availability of people who know how to deal with security issues effectively will become a problem.

Scoping out the threat

There are as many as twenty million PCs running Linux out there today. Because Linux is usually installed by a user because they're interested in it, it follows that most of them are run by people with a greater-than-normal interest in their PC. As a back-of-the-envelope estimate, if only one in a thousand of them is inclined to try cracking security on other people's systems, that gives us a community of twenty thousand script kiddies, any one of whom could be scanning for a back door that will let them into your machine.

The fundamental problem of network security in the internet age is that it is possible to write a tool that can scan hundreds of IP addresses per second, looking for a known vulnerability: and having found a machine that has such a weakness, to exploit it automatically. Few people have the nous to carefully unearth new bugs or vulnerabilities, and most of them are good guys who see their work in terms of fixing weaknesses. But a few of them take the security holes they discover and package them as automated attack toolkits; and there's a community of script kiddies out there who will use these toolkits to scan random machines, looking for one that will succumb. The result is that the internet is a massive force-multiplier for the attackers -- that figure of one in a thousand bad guys translates into tens of attacks per day on every system on the net.

What sort of weaknesses and security holes exist, and what are the consequences of a successful attack?

The purpose of an attack on a computer is to gain access to its resources -- disk storage space, ability to run processes, its network address and network services, and so on. For example, if your system is running an FTP server that permits anonymous uploads, attackers may create subdirectories on your machine and publicize its whereabouts for exchange of bootleg software, music, or other files. This will hoover up disk space at a rate of knots, and monopolize your bandwidth as hordes of warez d00dz log in and grab files or donate their own items. Again, if you're running an old copy of sendmail that is configured to operate as a mail relay, spammers will eventually find it and start relaying millions of MAKE MONEY FAST!!! adverts to their unwitting and angry victims. This will gobble up your bandwidth, and when the recipients start complaining to your upstream service provider you may find your connectivity is yanked or your cable modem account is suspended.

You'll note that both these scenarios assume that you're running a server (be it an FTP -- file transfer protocol -- server, or an SMTP -- email -- server). Most Linux systems run potloads of server processes; unlike Windows NT or 2000, where a server machine is typically dedicated to providing email or file storage or some other individual service, Linux is able to happily juggle a couple of hundred different services concurrently without major problems. In fact, providing network services is the task Linux is best suited for. Unfortunately, you need to know how to control this activity and secure the servers -- otherwise they'll render your system vulnerable. And almost every Linux distribution comes with a ton of servers configured and running, by default.

There's a second and more serious type of assault that you can experience; the so-called root exploit. A root exploit is one that gives the attacker access to an interactive shell (such as bash) running with root permissions. The special user "root" is privileged: root can do anything root feels like to your machine. Ordinary user accounts are limited to working on files and processes that they have permission to access, but root can freely trample across access permissions, run any program, or delete any file. A root shell gives one the ability to use this privilege level effectively. If an attacker uses some security hole to get themselves a root shell on your machine, you have effectively lost control of it: they can do anything they want, up to and including installing a different operating system on it.

Probably the commonest use of a root exploit today is to install a "root kit" -- a set of snooping tools designed to give the attacker the root password to other machines on the same network by monitoring the activities of users on the compromised machine. This is closely followed by the installation of Distributed Denial of Service software -- ownership of a large fleet of "zombies" (compromised machines that are running DDoS tools) gives a script kiddie the ability to shut down virtually any server on the internet.

Root exploits are usually more obscure than the simple attacks on unsecured servers described earlier. However, many of the Linux internet servers run with the root user id; in some cases they can be tricked into crashing in such a way that they execute some arbitrary code (uploaded by the attacker) such as a shell with root permissions. Feed such a shell the right script and you can do things like add a guest account to the password list with full root permissions.

Here's an important distinction: attacks on network services are typically possible because the services are incorrectly secured, but attacks aimed at gaining root access via bugs in the server software are typically possible because whoever owns the machine hasn't installed an essential bugfix. (You would be very unlucky indeed to find yourself the first victim of a wholly new type of root exploit for which no patch is available.)

Finally, there's a third type of attack that sounds almost silly; logging in via telnet or ssh and typing a password at random. Or rather, it only sounds silly until you realise that studies have shown that 50% of servers have one or more users whose password is their spouse's name, a child's name, their birthday, a religious term ("god", "jesus", "allah" ...), their pet dog's name, or worst of all, "aaaaaa" or "        " or something similar. Password-guessing tools have been around for years and are trivially easy to deploy; while a human being would get bored, an attack script can hammer at your front door for weeks on end until it finally guesses the right password for root (or your own user ID).

Virtually all UNIX and Linux systems have options to throw out "bad" passwords (predictable ones or ones based on a dictionary word) and to "age" passwords so that users are forced to change them after a few days or weeks; you need to switch this feature on when deploying a public server.

inetd, /etc/init.d, and friends: defeating attacks on services

How do you go about securing network services on Linux (or related UNIX-type operating systems)?

Network services are provided by programs called daemons that run continuously, waiting for an incoming request from a client system connecting over the internet. In general, there are two types of server daemon: standalone servers and servers controlled by inetd, the master internet daemon.

Standalone servers run continuously until a connection comes in; they then spawn a child copy of themselves to deal with the connection, while the parent process goes back to listening for more requests. A common standalone server is Apache, the standard web server found on most Linux systems. Other common standalone servers include BIND (the Berkeley Internet Name Daemon, which handles domain name lookup requests), sendmail (the heavy-duty SMTP mail server), inn (a usenet news server), Zope (an object-oriented web application server), and MySQL or PostgreSQL (two common free relational database servers).

Standalone servers are usually started and stopped by an rc (runcom) shell script, which is executed by the init (system control) process when the Linux system changes run level -- for example, when it first brings up networking and switches to multi-user mode while booting.

So what are the rc scripts, and what can we do with them?

Linux (and non-BSD UNIX systems) operate in one of a number of 'run levels' that indicate the system's availability. Run level zero means 'switched off'; run level 1 is single-user mode, in which networking is disabled, filesystems are unmounted, and only the system administrator can log in via the console (for example, to repair damaged filesystems or carry out heavy-duty maintenance). Run level 3 is standard multi-user with networking support, and in run level 5 the X-based login server Xdm may be run (giving you a graphical login screen on a workstation). Switch to run level 6, and the system will shut down and reboot; switch to run level 0 and it will shut down and halt.

Switching runlevels is controlled by the init process, and a special program 'telinit' can be used to tell init to change run level. When you change run level, init reads the file /etc/inittab, and executes apropriate programs specified in it.

One program is executed whenever you change run level -- /etc/init.d/rc (or /etc/init.d/rc.d/rc on Redhat). This in turn looks in the appropriate directory for the new run level (for example, if switching to run level 3 this would be /etc/init.d/rc.d/rc3.d on SuSE) and executes scripts it finds there. These have names beginning with the letters S or K (for Start or Kill, depending on the mode they're executed in when entering the run level) then a two-digit number; the number dictates the order they're executed in, and the 'S' or 'K' prefix determines whether they're run with the 'start' or 'stop' parameter. Each script controls a system service, such as a standalone web server or the PCMCIA card interface on a laptop.

Using this system it's possible to specify the order in which services start and stop with extreme precision, at the cost of simplicity. (However, in practice the S and K scripts in each run level are really symbolic links pointing to a single 'master' script for that service, which lives in /etc/init.d. Thus, editing the file /etc/init.d/rc3.d/S21apache actually modifies the file /etc/init.d/apache, which may appear in other runlevels.)

By default, most Linux distributions (for example, SuSE or Redhat) enable the Apache web server as a standalone server. You can disable this server by either deleting the 'S'-prefix links that point to the rc script /etc/init.d/apache, or by deleting or renaming the rc script completely; and by renumbering the script or creating new links you can control when and how it is started up. The important point here is to identify what run level your system boots into by default (there's a line in /etc/inittab that begins 'initdefault:' and specifies this run level) and then examine the services that are started or stopped in that level.

A green cross code for Linux servers

Individual servers that are designed to operate as standalone tools often have their own filter mechanism, separate from tcpd. However, there's no single common format for defining how servers behave on Linux; while most of them keep their configuration files in /etc, a vanilla installation of Apache may install them in /usr/local/etc (for no very obvious reason!), and so on. But in general a few simple rules make life safer:

These rules of thumb only address ways of hardening servers; you may also have to look into IP packet filtering -- the topic of a future Shopper feature.

Keeping up with security

Every Linux distribution has security holes. A typical heavyweight distribution may have 20 million lines of code; the odds are that somewhere in there, there are unknown or unquantifiable bugs that some cracker can exploit to gain control of your system.

If you run a publicly-connected machine, you need to take a couple of routine measures. Firstly, your distributor almost certainly runs a web site with software updates and security patches. You should keep an eye on this, or use the distribution's automatic update feature to automatically install cryptographically signed security patches from the distributor.

However, Linux distributors aren't always on the ball in spotting new problems and producing patches. Your early warning radar is essential; if a problem is identified you need to disable the service at once, and worry about installing a patch later. So keeping your eyes open for new security announcements is essential.

For starters, subscribe to the BUGTRAQ mailing list. BUGTRAQ is used for announcements of new security holes; there's a corresponding MS-BUGTRAQ for Microsoft-related incidents, but the original BUGTRAQ list, and the associated Linux-SecNews list, is where UNIX security holes surface first. (Forget CERT, the Computer Emergency Response Taskforce; they always seem to be last.) You can find details of the BugTraq mailing list here; please read the FAQ about the list before you subscribe.

If you're running Red Hat, you should either keep a routine eye on Red Hat Support's security alert page, and install the upgrades and updates; or use the Red Hat Network to automatically install patches for your system.

If you're using SuSE Linux, you can automatically update your system online, using the YAST2 system administration tool; while running X, type "yast2" and enter the root password when prompted. When the Control Center runs, look under "Software" for "Online Update". (Warning: this is best done over a broadband connection such as a cable modem or ADSL line if you select the "automatic" option -- if you use a modem for dialup access to the internet, you should manually pick and choose the updates to install and focus on security patches. Otherwise, pulling in the whole lot will take forever.) See also SuSE's security announcements.

Debian linux users will of course be sniggering up their sleeves by this point; the apt-get (advanced package tool) command can be configured to grab packages over the network via a variety of protocols and from a number of servers, and automagically keeps all package versions in sync; but as Debian tends to be used by experts, this advice is redundant.

In addition to these tips, almost all major distributions have scripts available for "hardening" them -- for example, Bastille Linux is a system that hardens Red Hat (or Mandrake) distributions. SuSE also has a hardening script, accessed via the YAST2 control center. But you can't rely on hardening scripts to do everything for you: they only work against attacks the authors knew about, and by definition new attacks may get around them.

The best advice is, ultimately, to read up on security and to become security-minded: you could do worse than start with "Practical UNIX & Internet Security, 2nd Edition", by Simson Garfinkel & Gene Spafford (pub. O'Reilly and Associates; ISBN 1-56592-148-8).

Remember, security is a process, not a goal that you can reach. Just keep your ears open for new exploits and patches, and you'll probably be safe. But if you just connect a server to the net and leave it for six months, by the time you go back to it the thing will probably be crawling with strangers.

A quick tour of inetd

One important piece of software you need to know about is inetd, the internet daemon.

Inetd is a special daemon that acts as a despatcher for a wide range of internet services. Inetd listens for incoming connections on a wide range of sockets; when one arrives, it consults a table of service types and spawns an appropriate process to handle the particular connection. Thus, inetd may be configured to handle web (HTTP), FTP, SMTP (email) and other request types -- even though these can also be handled by standalone servers. (But in general, heavily-used services are configured as standalone servers rather than going through inetd.)

Inetd-spawned servers are started and stopped automatically by inetd, which in turn is a standalone server (like the others), switched on or off by an rc script during system startup and shutdown.

The core configuration files for inetd are all stored in /etc; they consist of /etc/protocols, /etc/services, and /etc/inetd.conf. In addition, tcpd (the TCP filtering system) is invoked from inetd; more on this below.

The files /etc/services and /etc/protocols contain information that defines services that the system recognizes -- specifying the packet types (UDP or TCP), the IP port number the service runs over, and so on. You shouldn't ever need to edit these; they're specified by IANA, the Internet Assigned Numbers Authority, and define how clients and servers can communicate. However, in /etc/services we see that each service is given a human-readable name -- for example, "smtp" for the TCP or UDP services running on port 25 (which are used to transfer mail).

These names are used in the inetd configuration file, /etc/inetd.conf, which tells inetd what ports to listen on and what to do when it receives a connection on such a port. Each line in /etc/inetd.conf that doesn't start with a hash sign (i.e. that isn't a comment) starts with a service name and then a series of fields that specify how to deal with a connection on the port defined for that service name in /etc/services. If a line starting with "smtp" occurs in /etc/inetd.conf, then as long as inetd is running, it will listen on port 25 (defined as belonging to smtp in /etc/services) and handle it in accordance with the record in this configuration file.

Note that inetd reads its configuration file only at start-up; if you edit it, you need to send a HANGUP signal to inetd (for example, by restarting it using the '/etc/init.d/inetd restart' command, or by running 'kill -HUP inetd'). This will force it to update its actions.

Most of the details of inetd.conf aren't relevant; but the last fields specify a command to execute (along with possible command-line options). Inetd executes this command and connects its input and output to the client; for example, a line like this:

smtp stream tcp nowait root /usr/sbin/sendmail sendmail -bs

means that when an SMTP connection comes in, inetd should execute the program /usr/sbin/sendmail with the command line 'sendmail -bs'.

You can disable unwanted services in /etc/inetd.conf by simply commenting out the line they're on (put a '#' symbol in front of it) and then restarting inetd. Or you can control access to a service using tcpd, the TCP Wrappers system.

The TCP Wrapper (/usr/sbin/tcpd) is designed to look at a network connection and decide whether or not to accept it, on the basis of two 'rules' files. The commonest use of this is to restrict a service to local or private machines, or to deny access to some external offender (for example, to a spammer who is trying to assault your mail or news server).

For example, if we want to provide the telnet service but restrict access using wrappers, we need a line like this in /etc/inetd.conf:

telnet stream tcp nowait root /usr/sbin/tcpd in.telnetd

This means that /usr/sbin/tcpd is to be executed, with the parameter 'in.telnetd' (which is the name of the telnet server program).

tcpd runs, decides whether or not the connection is legal, and either closes it immediately or fires up a telnetd process to handle it.

The tcpd rules files are /etc/hosts.allow and /etc/hosts.deny. These two files use a pattern-matching language; tcpd looks for lines in the two configuration files that tell it how to deal with requests for the named daemon (in this example, in.telnetd), and then looks for patterns that match the client-side connection. Rules in /etc/hosts.allow specify connections that are allowed to proceed; it then scans /etc/hosts.deny to look for rules that deny connections (and override previously granted access). A common pattern you'll see is something like this:

in /etc/hosts.allow:

in.telnetd: ALLOW FROM ALL

in /etc/hosts.deny:

in.telnetd: DENY FROM ALL EXCEPT LOCAL, 192.168.1.

What this means is that essentially all connections to in.telnetd are blocked, except ones from LOCAL (a shortcut for localhost or hostnames that contain no dot characters) and 192.168.1., an IP address corresponding to a (standard) private class C network.

When you try and telnet to a server with this inetd setup and these tcpd rules, the first thing that happens is that inetd, which is listening on the port defined for the telnet service in /etc/services, will pick up the phone. Seeing that it's a known service, inetd will spawn a copy of tcpd. This will read its /etc/hosts.allow and /etc/hosts.deny files to decide whether to reject the connection. If it is rejected, tcpd will close the socket; otherwise, it will in turn spawn a copy of in.telnetd and hand over control to it.

And in summary, you can switch off a service completely by commenting it out of (or deleting it from) your /etc/inetd.conf file, and you can switch it off selectively (still permitting access for local users, for example) using the tcpd control files.

One important point to note is that inetd is somewhat elderly. When handing a protocol like SMTP, which is heavily used, a single connection can entail running three new child processes, whether or not the connection is permitted; this makes it too inefficient for running high-volume services, and consequently heavy-duty servers such as sendmail and Apache invariably have standalone modes. In general, inetd is used for managing less-frequently-used services: you should give serious thought to switching off anything that you aren't sure is necessary. For example, by default SuSE and Redhat distributions ship with an inetd.conf file that handles telnet and rlogin. It is a good idea to switch these off and use OpenSSH instead for remote terminal connections -- they're deeply insecure.

Recent attempts to replace inetd include the newer high-performance server xinetd; if you want to experiment with one of these, you'll need to work your way through the documentation with care.

[ Site Index] [ Linux Index] [ Feedback ]