ONLamp.com    
 Published on ONLamp.com (http://www.onlamp.com/)
 See this if you're having trouble printing code examples


Making Packager-Friendly Software, Part 2

by Julio M. Merino Vidal
04/28/2005

My previous article, Making Packager-Friendly Software, explains why software packaging is sometimes problematic due to real problems in the mainstream sources. It also discusses many issues that affect the distribution files and the configuration scripts (the most visible items when trying out a new program). This part explores the problems found in the build infrastructure and the code itself.

If you haven't yet read Part 1, it may be a good idea to do so now. It introduces many concepts that you need to know to understand completely the following explanations.

Recursive Dependencies

If your program has any runtime dependency, such as a shared library or a helper utility, be careful not to introduce other direct dependencies involuntarily. If you do, your program will end up depending on more packages than you expected. That will make it more difficult to issue an update of an installed package.

To get an idea of the real problem, consider a situation I faced a while ago. GNOME VFS can use optional OpenSSL support. When building it with this feature enabled, the linker flags stored in the pkgconfig file receive an extra -lssl argument. When any other program requests the flags it needs to link against GNOME VFS, it will see something like -lgnomevfs-2 -lssl. What happens here? The program that requested GNOME VFS support now directly depends on SSL too, although it shouldn't (because it does not care about the internal behavior of GNOME VFS). This has introduced a hidden dependency.

"Yeah, well, what's the problem with that?" you may ask. "Everything you've said seems harmless." Suppose that the GNOME VFS package has a future modification to use GnuTLS instead of OpenSSL. (This happened in pkgsrc a while ago.) You build the new version of the GNOME VFS package and update the one installed on your system, doing an in-place update, without removing any package using it. Assuming nothing else depends on OpenSSL after replacing the old GNOME VFS package, you remove it. Oops! All those programs that previously used GNOME VFS with OpenSSL support are now broken, because you've removed one of their dependencies. (Remember? They linked against OpenSSL by using the -lssl flag). The removal was possible because the packaging system did not record that dependency (and it shouldn't); this is another example of the hidden dependencies explained earlier.

You may argue that the packages that use GNOME VFS need the -lssl flag because some platforms do not support the recursive loading of shared libraries. OK, fine, but this is something (as far as I know) that GNU Libtool handles through the dependency_libs variable in .la files. Leave that job to a tool that really knows what's going on.

Related Reading

Managing Projects with GNU Make
By Robert Mecklenburg

Handling Configuration Files

Installing configuration files isn't easy, especially if you want to please all packaging systems and users. Some of them don't care at all about how you install these files, while others want to do the entire task for themselves to control what happens.

Before continuing, let me explain how pkgsrc handles them, so that you can see the big picture from a different perspective. I'll simplify and omit some details to make the explanation easier.

Pkgsrc installs the configuration files provided by a package in an examples directory, namely ${PREFIX}/share/examples/package-name/. Every installation of the respective package always updates these files. The administrator never modifies them. They are just examples.

When the package is finally installed, pkgsrc looks in the examples/ directory. Each time it finds a file there, it checks whether that file exists in the system configuration directory (usually but not always /usr/pkg/etc). If it doesn't exist, pkgsrc copies it verbatim. However, if it does exist, pkgsrc checks it for local modifications and overrides it with the new version only if there are no changes. Finally, the administrator may choose to install the configuration files for a specific package in a separate directory, out of the generic system configuration directory.

As you can see, this particular packaging system wants to do all the installation by itself. The advantage of this is that a generic and consistent framework handles all configuration files. It also works perfectly with binary packages. The disadvantage is that unfortunately, almost all third-party utilities need patches to make them work within this framework. I am sure that other packaging systems also have their own structure to manage configuration files, and that they need to patch programs to work according to their rules.

"Why do these programs need patching?" you ask. Some of the reasons:

Here are some better ways to handle configuration files, in case you need them. I know that some people will disagree with the last ones; in that case, consider following these rules only in --enable-packager-mode.

Unprivileged Builds

Many packaging systems (including pkgsrc) let you build packages as a regular user and require only superuser privileges to install them (to have the right permissions, ownerships, setuid flags, and so on). Therefore, you should make sure that your program builds correctly without superuser privileges to ease the packaging task. I can't think of an example in which a program requires full privileges to build.

Furthermore, the installation stage should not trigger any rebuild rule (causing the creation of new files) and must not leave temporary files in the source tree.

The rationale behind this is the following: the only stage that happens as a privileged user is the installation. Other stages, such as the build or the cleaning, happen as a regular user. The one most affected by incorrect permissions is the cleaning that happens after an installation, because the regular user will not be able to remove some of the files owned by the superuser (especially if he owns a directory in the source tree).

The Make Utility

It is very common to find makefiles that are GNU Make-specific and fail to build with other tools. This can manifest in two ways: either the other make utility produces a syntax error and aborts, or it shows strange errors such as missing rules. The former is easy to diagnose and fix; the latter can lead to headaches, especially if you're a novice packager.

When developing your program, you should try to build it with at least GNU Make and BSD's make. If it builds only with GNU's, you can:

In my opinion you should try to fix your makefiles first, because many times the incompatibilities come from minor details. If that's impossible, go for the configure check--really. Please note that depending on GNU Make by itself is not a problem, because good packaging systems can use it when needed. The annoyance comes from having to check manually whether the package will build with another tool, because some packaging systems don't use GNU Make by default. Can you imagine building an enormous program such as Mozilla, only to realize after hours of compilation that it failed because it requires GNU Make?

Another problem related to the make utility is that it may be present with a name other than make; common names are gmake for GNU Make, and bmake and pmake for BSD's make. If you call the make utility within your makefiles using the make name, it may pick up a different version than the one you need, causing further problems. To solve this issue, simply avoid hard-coding make in makefiles; use ${MAKE} instead.

Still, be aware that some old make utilities do not set the ${MAKE} variable while running. In such cases, you can use GNU Autoconf's @SET_MAKE@ macro to define it.

Build Infrastructure Oddities

The way your program is built also affects the packaging process. The following list describes the most important issues, most of which are related to your makefiles or the configuration script:

Code Portability

Writing portable code is a difficult quest. Nowadays, the most popular Unix-like system is Linux; thus almost all software development happens under it. This drives to several portability issues for less than careful code. The most common ones are:

There is much, much more. Analyzing portability issues in detail could be worth another (very long) article.

How does this affect the packaging process? While writing your code, you may realize that a part of it will not be portable, especially if it has to deal with system intrinsics or hardware devices. You can either try to fix it, which may be difficult if you do not have other systems around, or you can clearly mark such code as not portable. The latter is possible by using independent source files for each system (for example, cdrom-linux.c and cdrom-netbsd.c) or with conditional compilation.

For instance, suppose you have a chunk of code that builds only if Linux's CD access interface is available; you aren't aware of how to do the same thing on non-Linux systems. The only right way to fix this is to detect the presence of the required feature from the configuration script (but avoid testing macros such as __linux__ whenever possible; see below). Once you've checked for it, you should do something like the following in your code:

#ifdef HAVE_LINUX_CD_INTERFACE
    /* Code that only works under Linux systems... */
#else
#  error Sorry, this part of the code is not yet ported \
         to non-Linux systems.
#endif

First of all, there is a clear separation of the code in OS-specific chunks, which eases the porting process. However, note the #error line in the #else clause, which is important. This is to make the compilation fail on non-Linux systems with an appropriate error message. The packager will quickly see where the error is, and he may attempt to patch your code to support another platform. (If he's able to do that, you will receive a patch to improve your program!) The rule to remember is that it's better to keep errors controlled than to let your code fail in unexpected places.

As I said two paragraphs ago, you shouldn't use macros like __linux__ or __i386__ and friends. Why not? These macros fail to detect changes across versions of the same OS. For example, you may be using a Linux-specific feature, but unfortunately it is available only with a 2.6 kernel (and you are not aware of this fact). Therefore, you use __linux__ to separate your code, but then it will mysteriously fail in older versions.

OSes have evolved so much, as has their support for different architectures, that relying on these definitions is bad. Check for the specific feature you need from the configuration script, and use the results. This means that your code will be not only more portable, but also easier to maintain. Examples of programs that rely on these definitions are Perl, OpenSSL, and Crypto++; they are a real nightmare to port and maintain.

As we are talking about conditional compilation definitions, let me mention some standards. Don't expect _XOPEN_SOURCE or _POSIX_SOURCE (and similar macros) to do anything useful across multiple platforms. Don't define them unless absolutely necessary; they likely will restrict the namespace further rather than provide portability. If you need to use a specific function, first verify whether it is available in the most common systems, and then add a check in your configuration script.

Another solution, which may be better in some situations, is to use hardware or system abstraction libraries. These often deal with portability details and remove that burden from you. Furthermore, they have wide testing and already have packages for other operating systems, so you have a better chance that your program will work properly. Examples include Glib, libgtop, and Qt.

The End

I have tried to describe many of the problems that have bothered packagers over time as well as possible; let me know if something is hard to understand and I will try to explain further.

I also may have left out other important problems. I can't be aware of them all, especially because something that another developer sees as a problem may look harmless to me.

Anyway, I (I think I can safely say we) hope you have found this interesting and that you will try to make your developments more packager friendly. Thanks in advance!

Julio M. Merino Vidal studies computer science at the FIB faculty in Barcelona, Spain.


Return to ONLamp.com.

Copyright © 2009 O'Reilly Media, Inc.