[ Site Index] [ Linux Index] [ Feedback ]
A Grab-bag of Grief and Gaiety
Redhat loses it?
Experiences you don't want to undergo:
It's eight in the morning and the cat's woken you up by trying to french-kiss you (it's not love: she just wants feeding). You get out of bed and realise that the nuclear A-bomb test series is inside your head, not outside it -- you've got the mother of all hangovers. There's no point trying to go back to sleep with a headache and an amorous cat lurking on the pillow, so, being a born-again geek, you pour yourself your morning mug (or glass) of tea (or coffee or Mountain Dew) and stumble bleary-eyed into the study or office or den, there to log in and check your email.
Bad news: a worm is doing the rounds. This worm, called Lion, is basically a self-propelled root kit. It looks for Linux boxes running BIND, the Berkeley Internet Name Daemon -- specifically version 8.2.2 or earlier, because it knows about a vulnerability in BIND that lets it get root access. Once it's into a Linux box it installs various trapdoors, mails a report to a dead letter box in China, and starts scanning for more victims.
Guess what: you manage a Linux server. And it's running BIND 8.
So, being hung-over but diligent, you log into your server via SSH and simultaneously check out the CERT warning notices. Your server is about ten months old; it was configured with Redhat 6.1, and you've been applying security patches as they come out, but you can't take it down for a full upgrade because it's at the other end of a leased line, sitting in a bombproof bunker surrounded by barbed wire and attack dogs and searchlights specifically to keep posessive would-be system administrators from fondling their kit.
A cursory check indicates that the worm-spoor isn't present: you have not been hit by it yet. So there's time for a fix. You look at the vendor-specific fixes and follow the link to redhat.com, there to discover your first nasty shock: Redhat only provide fixes for versions 6.2 and later.
Huh? "Redhat 6.1 is less than eighteen months old! What kind of nonsense is this?" You think. Don't they realise the importance of having a world-beating uptime record?
You drink your tea/coffee/Mountain Dew, whimper faintly as the H-bomb tests are replaced by a platoon of dwarves with road drills, and feel yourself break out into a hot sweat and the shakes. (Whether it's the hangover or the idea of an 8am system upgrade on a Saturday morning I leave to your tender imagination. )
Well, you suppose that the BIND 8.2.3 release for 6.2 should run on 6.1. To be on the safe side, you'll download the source RPM package and rebuild the binary -- just type "rpm --rebuild bind.src.rpm" and it'll prep one specific to your machine and dump it into /usr/src/redhat/RPMS/i386 ready for you to install.
So you try installing the sources, and your server helpfully tells you:
only packages with major numbers <= 3 are supported by this version of RPM error: bind-8.2.3-0.6.x.src.rpm cannot be installedWhimpering is for wimps; you're a born-again sysadmin, so you do the sysadminly thing and FTP into your local Redhat FTP mirror and hunt feverishly for a new copy of RPM. For the horrible fact is, the Redhat Package Manager on Redhat 6.1 is version 3, but the update package (for Redhat 6.2) is in RPM version 4 format.At this point, your blood pressure begins to soar. Because Red Hat don't seem to have maintained the old subdirectory "rpm", where they kept the source code for RPM. There's a new RPM for rpm itself in the updates directory for 6.1, but it turns out to be one you've already installed. You look in Redhat 6.2's download area and pull in the RPM for the new version -- but it turns out to be another damned RPM version 4 file. Finally you find a source RPM for rpm version 3 -- the latest one you can actually install -- and it rebuilds okay up to a point. Then it complains about popt, which turns out to be a library for parsing command line options. It seems that RPM relies on some spiffy up-to-date version of popt -- and the only RPM you can find for it is in version 4 format.
You are hung over, you are lost in a twisty little maze of software updates, and you can feel the worm getting closer to your precious machine with every mistyped command and failed build.
Finally, you despair and phone your co-sysadmin. Who yawns and says "go back to bed, I already installed BIND 8.2.3 weeks ago -- by hand, from the original tar file".
Thus, our story ends more or less happily (save for a couple of hours of entirely evitable anguish on the part of a hung-over sysadmin) -- but there's a sting in the tail. Because you know that if you used Red Hat's up2date utility you wouldn't suffer this way: you'd run it nightly and let it automagically download and install new RPMs whenever Red Hat publishes them.
Personally, I have some issues with up2date: for example, it assumes that the way Red Hat set up a machine is the way you want it, and it happily installs upgrades you don't want on top of packages you've painstakingly customized. But at least it exists -- and for people who don't know what they're doing it's a life-saver. I mean that. A colocated box run by someone humble enough to admit that an upgrade robot would make their life easier is a happy colocated box. Mea culpa.
At least, up2date was a good idea until last week, when Red Hat announced that they're going to charge for the service, leaving users who rely on it for the patch-upgrade cycle swinging in the wind next time a worm comes calling. This is a really sensible policy -- not! In fact, if Red Hat want to develop a reputation for being worm-friendly, this is about the best way to do it: pull the rug from under inexperienced administrators who don't know how to compile BIND from the original sources and who run into difficulties applying security patches to an old system by hand.
It doesn't matter that the administrators of these machines could get the up2date service if they paid a little money: the annually published surveys showing whose operating system succumbed to the highest number of hacking attacks don't bother asking whether or not they were paying for remote updates. All they'll show is that Red Hat is second only to Microsoft in the who's-been-a-bad-boy-now stakes.
Another way of developing a bad reputation is to pull support from products that are less than two years old. It's downright silly: there are tons of machines out there that still run Redhat 5.3! Red Hat the company lives or dies by the reputation of its products, and if old versions of Red Hat the distribution acquire security vulnerabilities they need to be fixed as long as people are out there and using them. Saying "it's two years old, nobody serious can still be using it" is nonsensical -- it's exactly the most important mission critical systems that are at risk, because their owners are unable to shut them down for an upgrade every few months simply because Red Hat feels like issuing a new distribution. The odds are high that the remaining Red Hat 5.2 systems out there are stable production servers with major databases or e-commerce applications running on them -- not toy systems run by students who can afford to spend a day upgrading whenever the latest and greatest new distribution comes out. It is precisely these bet-the-company boxes that Red Hat's policies on support are neglecting.
A third way of getting a bad rep is by pulling the same sort of silly tricks that gained Microsoft their satanic image in the open source world. One such trick is to revise your key file formats frequently, so that people who don't have the latest version of the software have to keep buying upgrades. Microsoft do this with Office; if you don't have Office 2001, you need to buy it so you can read those files people keep emailing you. It now looks suspiciously as if Red Hat have done this with RPM; they've brought out a new, improved version of the file format, so that users of SuSE or Caldera or other RPM 3.0-based distributions can't install and compile Red Hat RPMs, and to make it hard for people to catch up they've gone and hidden the source to the new version so that you have to be running their system before you can use RPM 4.0.
I really hope this isn't happening, but the software industry is a jungle: and a company that's mushroomed over the past couple of years -- and is full of people who're there to work for a living, not because they're open source zealots -- might make some dubious decisions.
Red Hat made a bit of a blunder with their release of Red Hat 7.0: they shipped a version of Gcc that was so bleeding edge it couldn't even compile the kernel. Now they've compounded it by dropping support for recent products, and they're shooting themselves in both feet with the decision to charge for up2date support. If you're in the Linux distribution business it is vital that you focus on producing the best damn distribution you can -- screwing up the security and upgradability is simply not on. Red Hat rose to ascendency because their RPM system guaranteed straightforward system upgrades: if even that has been broken (whether deliberately or by neglect), why stick with them?
SuSE me
The good news is that if Red Hat have lost track of the ball, SuSE have picked it up and run with it.
My first exposure to SuSE -- version 5.3 -- left me with a very negative impression. The YAST system administration tool seemed to be conspiring to get in my way and stop me running my own system. Important system configuration files were hidden in weird places, YAST had swallowed a lot of configurable stuff, and all in all it just seemed like too much effort.
But SuSE 7.0 showed a sudden improvement in the usability stakes. They've been making a concerted effort to adhere to the FHS (Filesystem Hierarchy Standard), the open specification for how and where programs and system files should live on a Linux system. The rather annoying YAST 1 was been by YAST 2, a graphical configuration tool that was easier to use and less intrusive. All the weird configuration stuff had migrated into one file, /etc/rc.config, which was well-commented and easy to hack on by hand.
SuSE 7.1 shipped in late February, and it's the distribution that Red Hat 7 should have been (but isn't). This one comes with Kernel 2.4. They've finally switched back to the One True Location for init scripts (/etc/rc.d). YAST has been beefed up and gained ADSL support, ALSA sound support and a whole lot of other items: it's actually easy to use and doesn't get in the way any more. The new X11 configuration tool (Sax2) picked up my laptop's weird Neomagic chipset -- and so did their sound configuration tool (subject to one glitch, which is Neomagic's fault for using the video RAM as a sound buffer). I actually had a complete working Laptop install, with Firewire and USB support, PCMCIA ethernet and modem, sound, printer drivers, and X11, up and running in about four hours -- and the only reason it took that long was because I insisted on manually dinking with the package selection in vast detail. (SuSE 7.1 occupies all of seven CDROMs, or a single DVD.)
Ideological purists give SuSE a hard time because YAST isn't released under a license that's fully compliant with the open source definition. This is true: you aren't allowed to sell works derived from YAST for a profit without getting SuSE's permission. Other than that, the licensing situation looks increasingly open to me -- and I'm not sure which is worse: a distribution that contains some not-100%-free software, or a distribution that is 100%-guaranteed-free but that uses an ASP business model to milk the users for money in return for vital security updates, and which uses Microsoft-style change-the-file-format games to keep people on an continuous upgrade treadmill.
(Although to be fair to Microsoft, that one was invented by IBM -- whose mainframe operating systems have, for many years, been charged for on the basis of usage, and supported on the basis of a new release every six months and no support for systems more than three releases behind the cutting edge.)
All in all, the only thing I had to add to SuSE 7.1 to make it perfect was KDE 2.1 (it ships with 2.0, because 2.1 came out after the release was frozen), and the RPM's from ftp.kde.org installed just fine.
(Did I say RPM's? Yes, SuSE 7.1 still uses RPM version 3.0.)
RPM; behind the scenes
RPM, the Red Hat Package Manager, is a tool on which Red Hat has built an empire. Prior to its development, you installed software on a Linux system by unpacking a compressed tar archive in your root filesystem and hoping nothing broke. (Experts didn't do that: experts compiled and installed each package from scratch. But most users are not experts.) RPM is basically a cpio archive (cpio is a more efficient but more obscure UNIX archiver tool) with some additional meta-information: lists of contents, lists of dependencies, scripts to execute before and after installation and de- installation. When an RPM is installed, the rpm tool checks a set of databases (in /var/lib/rpm) to ensure that all the necessary pre-requisites are there and that installing the package won't nuke an existing subsystem. When it has been installed, its details are added to the database; and by calling rpm -e <packagename> you can erase the package from your system cleanly.
RPM isn't the only package manager; the Debian project developed their own, the dpkg system, which is arguably superior in some ways. For example, if you try to install a Debian package that needs some other packages before it will run, the Debian tools will offer to install all the necessary prerequisites for you -- and do so in the right order, without trashing anything. (Debian is a non-commercial distribution; commercial variants based on it include Corel Linux, the now-defunct Stormix, and a handful of others.)
RPM was released as open source by Red Hat; if you hunt around you will eventually find its home at www.rpm.org, although there are hardly any links to this site on www.redhat.com. (Red Hat used to maintain a publicly accessible directory, /pub/redhat/rpm, on their servers, containing the latest source code to RPM. This has now been moved to ftp://ftp.redhat.com/pub/redhat/code/rpm/, but it was last updated in 1998, and there's a README saying go look at "www.rpm.org". (A search of Redhat's website didn't reveal any links to that site in the first fifty or so search results.)
Some more hunting to figure out what was going on with the RPM version 4 update finally revealed a security advisory which states: "A common version of rpm for all Red Hat distributions is being released. This version of rpm understands legacy version 3 packaging used in Red Hat 6.x/5.x distributions as well as version 4 packaging used in Red Hat 7.x. In addition, rpm-4.0.2 has support for both the legacy db1 format used in Red Hat 6.x/5.x databases as well as support for the db3 format database used in Red Hat 7.x".
Actually, a lot of the changes in RPM 4 show up from 3.0.4 onwards: the changes include the ability to use GnuPG to sign packages (so you know who they came from), the ability to use bzip2 compression, and changes to the internal database structure. Nothing earth-shattering, and certainly it rather begs the question of why they broke backward compatability and felt the need to add major changes to a minor-versioned release. Maybe it's just a case of creeping feature bloat running ahead of their release scheme: or maybe it was a security panic -- someone was releasing fake Official Red Hat RPMs and they needed to add digital signatures fast, before fakes containing trojans or worms flooded their user base.
Whichever, if you're running an older version of Red Hat you really need to look at RHSA 2001-016 and update your RPM system -- before you find yourself dealing with a hangover and a worm attack at eight o'clock on a Saturday morning. More importantly, if you're running a Red Hat release that ends with a .0 or .1, you want to consider upgrading to the final point release in a series -- because they provide some support for the final releases (5.3 or 6.2), but not, it seems, for earlier versions in a series.`
A stab in the dark
I don't like the stuff that's been coming out of Red Hat lately, but that doesn't mean they've been transformed into the Great Satan of Software. Similarly, while I cordially detested early SuSE releases, their produce has been growing on me recently. Regardless of whether you think I'm right or wrong about these companies, one thing is sure: the big Linux distributors are hunting around for new ways to make money. Both are emphasizing corporate services and trying to grab a lucrative enterprise market. Both are trying to sell differentiated (personal or professional) products. It's likely that the errors and enhancements stem from the same source: a desperate urge to find new ways to grow.
Times are hard in the software industry -- that much is obvious. Stormix have gone bust, Corel are backing out of Linux distribution territory, and I expect some other distributors to file for bankruptcy or merge in the next few months. Volunteer efforts (from Debian to Mastodon) will continue regardless, but the big guys are going to intensify their efforts to extract money from their customers. And if the customers don't keep their eyes open and vote with their wallets, there will be unpleasant long-term prospects.
[ Site Index] [ Linux Index] [ Feedback ]