oreilly.comSafari Books Online.Conferences.


Linux in the Enterprise It's a Cycle of Life Thing: Managing Linux Releases

by David HM Spector

Doing the Penguin Dance

One of the things I used look forward to about twice a year was a new release of Red Hat (RH) Linux. I'm pretty conservative on my main desktop system, where I write my articles and do my main development, and tend to run a pretty out-of-the-box configuration. I leave my hotrodding for my experimental system, where it doesn't matter if I accidentally trash a disk. Up until recently (say, Red Hat 7.2), upgrading my system wasn't a major hassle, because the overall rate of change of packages was relatively small and the improvements in system performance, as well as the look and feel of the user interface/experience, got to be more fun with each new release.

The release of the 2.4 series kernel made a lot more functionality available to developers, and the Linux community has taken advantage of it with wild abandon. With the release of Red Hat 7.3 (and SuSE 8.0, and most other Linux distributions from about mid-2001), I noticed a sudden bump in the number of applications available and a radical change in the dependencies in any given distribution, release after release.

For many users, upgrading a system or two is not a major hassle; they put in the CD-ROMs and go. However, the picture gets a bit more interesting if you're an enterprise using Linux for development or, dare I say it, as the corporate desktop standard for your users. A small change to a critical library (like glibc or gtk) can cause applications to break. A change in application functionality sprung on unwary users can cause support headaches galore. One of the great things about Linux is how quickly new software and services are becoming available--but it's also turning into a major headache for enterprise IT support people who have to keep track of all of these changes and software versions. Let's face it, the system itself (and its OS) is in reality only a small part of the Total Cost of Ownership (TCO) and one reason why end-users would rather fight than switch when it comes to operating systems and applications.

Also in Linux in the Enterprise:

Unfinished Business Part 2: Closing the Circle

Unfinished Business: The One Missing Piece

Linux in the Enterprise at LWE 2003

CASE Tools: Large System Development

Ship in a Bottle

One of the selling points and competitive levers for commercial Linux distributions is how many applications they bring to the table for their various constituencies. For power programmers, it's all about the depth of the development environment. For those deploying web services, it's the degree of support for Apache and related technologies like Tomcat. Most recently, for the desktop user, it's been the ever-growing number of applications available on the desktop, like StarOffice and

It's interesting to note that in the RH7.2 release there were perhaps 75 desktop GUI-based packages and utilities available. Under RH7.3, and now 8.0, a default, standard desktop system includes over 200 (yes, 200!) unique applications installed and available for the GNOME and KDE desktop environments. All of these programs have some level of dependence on other parts of the system, and as the depth of applications grow, so does the level of dependence.

What's Really in a Linux Distribution?

Before Red Hat developed the first serious package manager--the Redhat Package Manager, or "RPM"--for Linux, there was a list of packages that you needed to install in order to make a bootable system. This list consisted of the Linux Kernel, glibc, various "userland" utilities (e.g., the GNU binutils), networking tools like ftp and telnet, and a few shells. Everything else was optional. Fast-forward to 2003: the smallest possible automated installations (in terms of packages) from the major Linux vendors look like this:

Red Hat8.0

SuSE 8.1

Debian 3.0 (woody)

acl, anacron, apmd, aspell, at, attr, authconfig, authconfig, autofs, bash, bc, bdflush, bind-utils, bzip2, cpio, crontabs, cyrus-sasl-plain, dhclient, diffutils, dos2unix, dosfstools, dump, e2fsprogs, ed, eject, ethtool, fbset, file, filesystem, finger, ftp, gash, glibc, gpm, grub, hdparm, hotplug, initscripts, iproute, iputils, irda-utils, jfsutils, kbd, kbdconfig, kernel, kernel-pcmcia-cs, ksymoops, kudzu, kudzu, lftp, lha, libgcc, libtermcap, logrotate, logwatch, lokkit, losetup, lsof, mailcap, man, man-pages, mkbootdisk, mouseconfig, mt-st, mtools, mtr, net-snmp-utils, netconfig, nfs-utils, nss_ldap, ntsysv, openssh-clients, openssh-server, pam_krb5, pam_smb, parted, passwd, pax, pciutils, pinfo, procps, quota, raidtools, rdate, rdist, readline, reiserfs-utils, rp-pppoe, rpm, rsh, rsync, sendmail, setserial, setup, setuptool, slocate, specspo, star, stunnel, sudo, sysklogd, SysVinit, talk, tcp_wrappers, tcpdump, tcsh, telnet, termcap, time, timeconfig, tmpwatch, traceroute, unix2dos, unzip, up2date, utempter, util-linuxvim-minimal, vim-common, vixie-cron, wget, whois, wireless-tools, ypbind, zip

acl, ash, at, attr, autoyast2-installation, bash, bc, bzip2, cpio, cpp, cracklib, cron, curl, cyrus-sasl, db, devs, dhcpcd, dialog, diffutils, e2fsprogs, ed, efibootmgr, eject, elilo, fbset, file, filesystem, fileutilsfillup, findutilsfinger, gawkgdbm, glibc, gpg, gpm, grep, gruff, grub, gzip, hdparm, heimdal-lib, hotplug, hwinfo, ibmsis, initviocons, iproute2, iptables, iputils, isapnp, jfsutils, kbd, ksymoops, lcs, less, libgcc, libstdc++, libxcrypt, libxml2, liby2util, lilo, logrotate, lsof, lukemftp, lvm, m4, mailx, man, mdadm, mktemp, modutils, ncurses, net-tools, netcat, netcfg, openldap2-client, openssh, openssl, pam, pam-modules, parted, pciutils, pcre, pdisk, perl, permissions, popt, portmap, postfix. Procmail, providers, ps, raidtools, readline, recode, reiserfs, rpm, rsh, s390-tools, sash, sed, sh-utils, shadow, silo, sitar, src_vipa, suse-build-key, SuSEfirewall2, Sysconfig, Syslogd, Sysvinit, Tar, Tcsh, telnet, terminfo, textutils, timezone, unitedlinux-release, usbutils, utempter, util-linux, vim, w3m, xfsprogs, yast2, yast2-backup, yast2-bootloader, yast2-core, yast2-country, yast2-firewall, yast2-inetd, yast2-installation, yast2-ldap-client, yast2-mail, yast2-mouse, yast2-ncurses, yast2-network, yast2-nfs-client, yast2-nfs-server, yast2-nis-client, yast2-nis-server, yast2-nisplus-client, yast2-online-update, yast2-packagemanager, yast2-packager, yast2-pam, yast2-restore, yast2-runlevel, yast2-security, yast2-storage, yast2-support, yast2-sysconfig, yast2-theme-SuSELinux, yast2-transfer, yast2-tune, yast2-update, yast2-users, yast2-xml, zipl, zlib

adduser, apt, apt-utils, at, base-config, base-files, base-passwd, bash, bsdmainutils, bsdutils, console-common, console-data, console-tools, console-tools-libs, cpio, cron, debconf, debianutils, dhcp-client, diff, dpkg, e2fsprogs, ed, exim, fdutils, fileutils, findutils, gettext-base, grep, groff-base, gzip, hostname, ifupdown, info, ipchains, iptables, kernel, klogd, libc6, libcap1, libdb2, libdb3, libgdbmg1, libident, libldap2, liblockfile1, libncurses5, libnewt0, libpam-modules, libpam-runtime, libpam0g, libpcap0, libpcre3, libpopt0, libreadline, libsasl, libstdc++, libwrap0, lilo, login, logrotate, mailx, makedev, man-db, manpages, mawk, mbr, modconf, modutils, mount, nano, ncurses-base, ncurses-bin, net-tools, netbase, netkit-inetd, netkit-ping, nvi, passwd, pciutils, pcmcia-cs, perl-base, ppp, pppconfig, pppoe, pppoeconf, procps, psmisc, sed, setserial, shellutils, slang1, sysklogd, syslinux, sysvinit, tar, tasksel, tcpd, telnet, textutils, util-linux, whiptail

Wow, that's a lot of stuff isn't it? That doesn't include X11, or any user-level applications like or even freecell. If you count just the basic packages that make up X11, window managers like GNOME or KDE, support libraries, and user-level productivity tools, you can add up to 100 more packages to this list. This isn't a bad thing, but it's clear that a minimum usable, networked system comprises a lot of software.

Related Reading

Learning Red Hat Linux
By Bill McCarty

Imagine that you run a large enterprise and support not only the base operating system across hundreds or thousands of machines, but also end-user applications ranging from office productivity to CRM, or order management, or customer-service applications. It also makes for a lot of packages for an IT manager to support.

What do Enterprises Need?

One of the hardest things for any enterprise to deal with is change. Back in the days of the Internet bubble when we thought (wrongly) that old paradigms didn't matter, no amount of disruption to the workflow of an organization was too radical. Now, we understand that change is good, but unexpected change can damage the ability of people to work together and get things done. In deploying IT solutions, the two hardest parts of any deployment are 1) consistent equipment configuration and 2) training the people who will use it in the software packages they need to get their jobs done.

Some people might suggest that this is a non-issue and that services such as the Red Hat Network allow IT managers to tame this complexity with automatic updating. This is true, provided you always want Red Hat's suggestions. What if installing a new version of or some other application would mean that your staff needs to be retrained, or a critical feature changes in some way that negatively affects your operations? What happens then?

The ability to upgrade systems in a logical way that minimizes system-level changes (except for bug fixes) and allows the enterprise to keep using older/existing versions of applications to help manage user expectations and constrain retraining is extremely important.

In the days of mainframes and mini-computers, upgrades were a lot more logical and well-contained because an upgrade applied to a system shared by a large number of users. With workstations and PCs, even, a small upgrade can cause issues far beyond the scope of a single package, as anyone who has ever had the pleasure of maintaining even a small gaggle of Microsoft Windows-based systems can attest. One of the things that made the non-PC world manageable was that, for the most part, hardware vendors made a very clean separation between the operating system and the applications.

O'Reilly Events at LinuxWorld

Check out what went on at the O'Reilly booth and find out who the drawing winners are.

Back in January at the NY LinuxWorld Expo, it was interesting to hear a developer from Dell opine onstage that the current "problem" with Linux distributions was all about the number of releases per year and that "things were settling down to a nice 18-month release cycle." I contend that the release cycle of distributions as a whole is less of a problem than the lack of a clean interface for layering products onto Linux systems. There can be no argument that the speed of Linux development--especially from a security perspective--is a good thing; a cleaner mechanism is needed in order to support multiple versions of software to help IT managers keep end-user-visible changes and disruptions under control.

Chuck Yerkes, a consultant specializing in Unix and sendmail, suggests that the Linux world could benefit from the experiences of the BSD world, where almost all non-core functionality is installed in a non-root tree. Instead of a package like OpenOffice living under /usr/bin, it would live in /packages or a similar directory that allows multiple versions of packages to be installed easily. Of course, this "re-rooting" can be done now with RPM and other package managers, but it has to be done manually.

One way to look at this is that, in the old parlance of Digital's VAX/VMS, the core of the operating system itself is a separate product from the "layered products" used by end users. It should be possible to install or upgrade the core of the Linux operating system without forcing the upgrade of the layered packages. In fact, it should be possible for a Linux system to support multiple versions of a package along with whatever underlying libraries are needed, without causing application dependencies to break. This might seem to add major complications to Linux distributions, since vendors would have to track and understand more details about every package they include in their distributions, but it would make the adoption of Linux by large and medium-sized organizations a lot easier by making their support mechanisms more clear-cut and predicable.

A Call to Distribution Vendors and the LSB

It seems to me that it's time for Linux distribution vendors, backed by the Linux Standards Base (LSB), to break the core OS out from the rest of the "layered products" that make up the bulk of their offerings. Make all of the end-user applications into a separately-selectable set of installable packages and develop a way to allow for multiple versions of packages to coexist on the same system. In the old days, under mainframes and minicomputers, this was done with "logical names" under VMS and virtual directories under VM and other OSes. The BSD world seems to have navigated this shoal; it shouldn't be to hard for Linux world to do so, either.


Deep thanks to Steve Jones of Crash Computing, Chuck Yerkes of, Stephen Degler, Andrew "Bobcat" Templin, and Richard Rognlie, "Sr. Lackey" for Gamerz.NET Enterprises, for helping me sift through bits and pieces of various Linux distributions and offering suggestions on how maintaining large Linux installations could be better handled.

David HM Spector is President & CEO of Really Fast Systems, LLC, an infrastructure consulting and product development company based in New York

Read more Linux in the Enterprise columns.

Return to the Linux DevCenter.

Linux Online Certification

Linux/Unix System Administration Certificate Series
Linux/Unix System Administration Certificate Series — This course series targets both beginning and intermediate Linux/Unix users who want to acquire advanced system administration skills, and to back those skills up with a Certificate from the University of Illinois Office of Continuing Education.

Enroll today!

Linux Resources
  • Linux Online
  • The Linux FAQ
  • Linux Kernel Archives
  • Kernel Traffic

  • Sponsored by: