Distributed Cfengine

In this article we are going to take the script we wrote in Introducing Cfengine and distribute it to all of our servers using cfengine. As an added bonus, we're going to pull both our cfengine configuration and the sudoers file directly out of a versioning system. It's a simple additional step — something you should do with all centralized configuration files — and provides a convenient control point for modifying and auditing your configurations.

The goal here is to allow us to commit changes to our sudoers files in one place and have those changes automatically propagated to all of our servers. You check the file out into your CVS sandbox, make your changes, and commit them, and the system will handle publishing those changes. We'll modify and distribute our cfengine configuration via the same mechanisms, which will allow us to expand the cfengine configuration as necessary without worrying about how to distribute the enhanced configuration.

Copying Our Configuration

There are two sides to every distribution mechanism: the client and server. The cfagent binary is the client side of cfengine and cfservd functions as the server. Cfengine has multiple layers of security checks, which results in a somewhat onerous transport setup. Unfortunately (but not surprisingly, given the difficulty of doing so), cfengine also does not always provide relevant error codes when something goes wrong.

To distribute our configurations we have to configure both the client and the server. For cfengine to run automatically we have to configure cfexecd, a simple wrapper for cfagent that sends output to system administrators via email.

Before we start that, though, let's put all of our files into CVS, since it's easy and important. It's probably possible to centralize configurations without a version control system, but I certainly would not want to try it. Since most versioning systems were created for developers, we aren't likely to use all of their features, which makes this task even easier.

I use CVS here because it's a de facto standard and is easy to use; the example should work with little modification with any other versioning system. This article assumes a working knowledge of CVS and a functional CVS repository. If you do not have these, CVS Home can likely provide any necessary help.

Version-Controlling Your Configuration

We're going to start with the configuration for cfengine itself, then add the sudoers file once we have cfengine working. It's unlikely that the examples here will mesh perfectly with an existing repository, so expect to modify them if you are already versioning files. Given that, let's jump right in.

Create a temporary directory for your cfengine files, and create three files in it:

~ $ mkdir /tmp/cfinit/inputs
~ $ cd /tmp/cfinit/inputs
/tmp/cfinit/inputs $ touch cfagent.conf cfservd.conf update.conf
/tmp/cfinit/inputs $ cp /tmp/sudo.cf .

Notice we've included the sudo.cf configuration we created in the previous article. We're creating these files in an inputs subdirectory; as our configuration grows more sophisticated, we'll add other cfengine files, such as modules and data files, and we need to support them now so we don't have to change our system later.

Import those files into an appropriate CVS module:

~ $ cd /tmp/cfinit
/tmp/cfinit $ cvs import -m "new cfengine configuration" \
config/cfengine LAK yay

The last two arguments to cvs import are generally not useful to system administrators. I use my initials for the second argument and a somewhat random term for the third argument; I have not yet used either of these arguments for anything meaningful once I have imported something.

Finally, put our sudoers file into CVS:

~ $ mkdir /tmp/sudo
~ $ cp /etc/sudoers /tmp/sudo
~ $ cd /tmp/sudo
/tmp/sudo $ cvs import -m "distributing" config/sudo LAK init

We've put it into the same tree as cfengine, so it will be easy to update both with a single CVS command.

Now decide where you want to store those files so that cfservd can distribute them. I generally pick a location appropriate for the given site and then link it back to /cfengine, which makes it easy to move things around if necessary.

~ $ mkdir /export/cfengine
~ $ cd /export/cfengine
/export/cfengine $ cvs checkout config
/export/cfengine $ ln -s /export/cfengine /cfengine

We've now laid the groundwork for distribution, in that we have all of the files centralized in a versioning system and we have them checked out into a central location from which clients can retrieve them. Our versioning system functions as a kind of funnel here, with all changes passing from people's sandboxes through the CVS repository and into the checked out store on the server. This eliminates most problems with two people modifying the same file and provides auditing and historical functions.

Since the cfengine configuration files that we've checked into CVS are empty, we need to check them out in a CVS sandbox to allow us to edit them:

~ $ mkdir cvs
~ $ cd ~/cvs
~/cvs $ cvs checkout config

A Note About Keys

Related Reading CVS Pocket Reference By Gregor N. Purdy Table of Contents Index

Cfengine uses public/private key pairs similar to SSH's key pairs. Unlike SSH, which usually only has a key pair for the server, cfengine requires that both sides of a connection trust the other side. This means that for a cfengine connection to work, each side of the connection must have the public key for the other side. Also unlike SSH, cfengine does not normally run interactively, which means that there's no prompt for whether you want to accept a public key from a new host. There are several ways to work around this problem, but they mostly come down to one of two solutions: copying public keys manually, or explicitly trust the IP addresses of the clients, servers, or both.

It is possible to use another cfengine tool, cfrun, to retrieve a public key for a host manually. This can be an acceptable method for some cases, but it requires extra setup and, being an interactive utility, does not solve the automation problem. Hopefully we will discuss cfrun in a later article.

One important note about explicitly trusting an IP address is that it is only ever used on the first connection. Once you've retrieved the public key for a machine through a trusted connection, trust is never used again. This means that if a machine's keys change, you must manually update the keys; you cannot use trust to do so.

I prefer solving the problem of key management with a combination of trust and manual intervention. Because my server's public key should never change, I distribute that key with my cfengine package. Use cfkey to create the key pair. Once you've run it on your server, distribute /var/cfengine/ppkeys/localhost.pub from your server to your clients as /var/cfengine/ppkeys/root-192.168.0.2.pub, assuming 192.168.0.2 is your server's IP address.

Add this public key to your CVS repository:

~/cvs/config/cfengine $ mkdir ppkeys
~/cvs/config/cfengine $ cvs add ppkeys
~/cvs/config/cfengine $ sudo cp /var/cfengine/ppkeys/localhost.pub \
    ppkeys/root-192.168.0.2.pub
~/cvs/config/cfengine $ sudo chown $LOGNAME ppkeys/root-*
~/cvs/config/cfengine $ cvs add ppkeys/root-*

This will distribute our server's public key with our configuration, and sets an example for distributing other servers' public keys. Ironically, this is a mostly pointless exercise because I can't pull this file down unless trust is already set up and I can't trust without this key. It makes me feel better, though, so I do it anyway.

Now that our client can verify the authenticity of our server, we just need to tell our server about the client. We're going to use trust for this end of the connection, because this is a simple example. This trust relationship is configured using the TrustKeysFrom directive in cfservd.conf. In most real-world configurations, I collect the client key as part of my cfengine bootstrap process. I hope to delineate this process in a later article focusing on bootstrapping cfengine.

Configuring the Distribution

OK, now we have the files and they're all version-controlled, and we have some confidence that our key exchange will work. It's time to configure the files for distribution.

Let's start with the server, since it's a bit easier.

# cfservd.conf
groups:
    # the name of our server is 'server'
    cfengine_server = ( server )

control:
    cfengine_server::
        # tcp_wrappers-like access control
        AllowConnectionsFrom = (
            192.168.0.0/24
        )

        TrustKeysFrom = (
            192.168.0.0/24
        )

admit:
    /var/cfengine/ppkeys/localhost.pub *.domain.com

    cfengine_server::
        /cfengine    *.domain.com

This is our first experience with cfengine's classes. You can think of them as Boolean (true or false values). Any incidence of a class in a configuration is essentially an if statement. This statement lasts until the action ends or until the introduction of another class.

Cfengine has many classes that it automatically sets, including the host name. This is why we're able to set the cfengine_server class based on our server name. Cfengine also provides a special action, groups, for setting classes. classes is an alias for that action, but since I usually use it to delineate groups of machines, I usually use the groups moniker.

We set the cfengine_server class only on our (wait for it...) cfengine server, creatively named server here. We could instead use the hostname throughout the configuration, but then it would be difficult to change servers, and as the configuration becomes more complex and servers take on multiple roles, it can become difficult to determine why a certain server has a certain trait. Using this class, it is always obvious the role of the server on which a rule operates.

There is not much more to it. We use our server class to trust and grant connectivity to a range of IP addresses — you can only trust or grant connectivity to IP addresses, not hostnames — and configure which files those clients can see. In addition to the main configuration tree, I've added an extra file, the cfengine public key, and have provided unrestricted access to it. We won't use that in this configuration, but it's a nice way of giving administrators the ability to collect a host's public key manually if there's a problem with the key exchange (which is common).

The AllowConnectionsFrom and TrustKeysFrom variables only work within cfservd. The admit action similarly only works within cfservd.

Commit these changes into CVS:

~/cvs/config/cfengine/inputs $ cvs commit cfservd.conf

Now, on to the client. There are two important tasks the client must perform before it can run normally. It must update its configuration, and it must make sure cfagent is capable of running. Both of those tasks take place within update.conf, which executes separately from the rest of the cfengine configuration. Let's deal with the functional aspects first. This configuration assumes that you have run cfkey to create the key pair and that cfagent is installed in /usr/local/sbin, which is the default.

# update.conf
groups:
	# the name of our server is 'server'
	cfengine_server = ( server )

control:
    actionsequence = ( directories links )
directories:
    /var/cfengine/bin
links:
    /var/cfengine/bin/cfagent -> /usr/local/sbin/cfagent

Cfengine was developed to operate well in an environment where machines automount binaries from a server. If you automount /usr/local, you may want to perform a copy instead of a link, so that cfengine will still work if the automount fails, but a link should suffice for most installations. Only cfexecd uses the link; it'd be nice to just configure cfexecd not to require it, but I don't know of a way to do so.

We could get more complicated if we wanted, but this is at least the minimum required to make sure cfengine works. Let's copy the configuration now:

# update.conf, take 2
control:
    actionsequence = ( copy directories links )

    domain = ( ExecResult(/bin/domainname) )

    TrustKeysFrom = ( 192.168.0.2 ) # server.domain.com
    !cfengine_server::
        SplayTime = ( 5 )

    any::
        workdir = ( /var/cfengine )
        configroot = ( /cfengine )
        server = ( server.domain.com )

copy:
    ${configroot}/config/cfengine    dest=${workdir}
                                     recurse=inf
                                     ignore=CVS
                                     server=${server}

directories:
    /var/cfengine/bin

links:
    ${workdir}/bin/cfagent -> /usr/local/sbin/cfagent

This is where the configuration becomes a little confusing, because we've encountered two frustrating aspects of cfengine. There is no indication of whether we are dealing with a system variable (like domain) or a user-defined variable (like server), and some variables are case-sensitive (e.g., SplayTime) while others are not. Using the wrong case on a case-sensitive variable can be very confusing because you will receive neither a warning nor the behavior you expect.

Before we go through the new aspects of this configuration, we have to discuss the configuration of the domain variable. Cfengine relies heavily (a bit too heavily, sometimes) on the domain of the machines it runs on. It is absolutely imperative that both the cfengine client and server agree on the domain of the client. It doesn't matter if that agreement reflects reality, it only matters that both ends of the pipe agree. The client configures the domain through the domain variable and the server finds the domain by performing gethostbyaddr on the IP address of the client. As important as this variable is, though, be warned that cfengine almost always considers this the source of any problems related to trust, which can be confusing when the real problem is something like incorrect keys.

Thus, setting domain is our first task. If all of your hosts have the same domain, it's probably easier to set the domain via a static string, but if you use multiple subdomains, you need some means of retrieving the domain automatically. The example uses /bin/domainname, but you could just as easily pull the domain out of /etc/resolv.conf. This can result in a Catch-22 situation if you hope to use cfengine to manage domain configuration — you must have the domain set correctly to run cfengine, but you want to use cfengine to set the domain. The only answer I've found for that situation is to use a one-liner that attempts to collect a domain and sets a default if it fails.

The next variable we set is SplayTime. It is especially critical to set if you have many clients. This variable causes cfagent to sleep for a random amount of time up to a specified maximum; we set our SplayTime to 5, so our clients will sleep up to 5 minutes before contacting our server. This is a simplistic but usually sufficient form of load balancing; it should spread the client connections evenly over 5 minutes.

Note that only the clients have a SplayTime set; we want the server to run immediately, so it can update any necessary files and dole out the most recent versions when clients connect. Also, note the capitalization of SplayTime. Cfengine seems to be somewhat random in its case sensitivity, and many configuration parameters aren't case-sensitive. SplayTime is.

We also set some simple helper variables: the base directory of our local cfengine configuration, the base directory of the configuration on the server, and the name of our server. Then we define a simple copy statement. It's pretty self-explanatory, but we'll go through it just for clarity. Notice our use of the any class here; this is a special class that always matches, so it removes the effects of the previous class test.

Copy the Configuration

Like the files action, a copy statement needs a filename or directory. This filename is the source of the copy, usually on the remote server. You must at least specify a destination for the copy. Our example copies /cfengine/config/cfengine to /var/cfengine. Currently, the only directory in there is the inputs directory. /var/cfengine/inputs is the default location for the cfengine configuration; the cfengine binaries look in that directory for their respective configurations.

To tell cfengine to perform a remote copy, we specify the server to copy the files from. We further specify that cfengine should recursively copy the contents of the directory, so we'll have all of the subdirectories and their contents. Lastly we tell cfengine to ignore any files or directories named CVS, so as to avoid copying the CVS control directories. Although copying the CVS directories would not be a problem in this case, there are cases where it can be. Either way it's a waste of processing power and time, and you might eventually be ignoring enough CVS directories that it would make a difference.

Actually Doing Something

Whew! Now we should be successfully distributing our empty cfagent.conf, so we can move on to doing something within this configuration. Remember that the update.conf file is purely for updating the cfengine configuration, so we must start within cfagent.conf for anything else.

We've already added our sudo.cf file to CVS, so now let's import that file into our configuration:

# cfagent.conf
groups:
	# the name of our server is 'server'
	cfengine_server = ( server )

control:
    domain = ( ExecResult(/bin/domainname) )
    workdir = ( /var/cfengine )
    configroot = ( /cfengine )
    server = ( server.domain.com )

import:
    cfengine_server::
        cvs.cf
    any::
        sudo.cf

Hmmm, that's annoying, we seem to have duplicate definitions of variables here. update.conf is completely separate from the rest of the cfengine configuration. This is intentional; if you break any other aspect of your cfengine configuration, you can fix it by updating from the central copy, but if you break update.conf, you've broken the update process itself. Keep this file as simple as possible; all information collected in this file is expunged before the normal configuration is executed. You may not consider this a feature, but the author of cfengine certainly does.

When this script runs on the cfengine server, it imports two files. For the clients all we do initially is import our sudo configuration. We need to modify our sudo.cf file to make it copy the sudoers file from the central server, rather than just enforcing permissions. This is the whole purpose of our article.

# sudo.cf
control:
    actionsequence = ( files copy )

files:
    /usr/local/bin/sudo owner=root group=root mode=4111
        checksum=md5 action=fixall

copy:
    ${configroot}/config/sudo/sudoers    dest=/etc/sudoers
                                         server=${server}
                                         owner=root
                                         group=root
                                         mode=0440

This is almost exactly the sudo.cf file we built in the first article in this series, but our files action has become a copy action. Now instead of verifying only the permissions of the sudoers file, we're updating it from a central, version-controlled location. This is not much different, but is much more functional (and requires just a bit more setup) than our original version. We still verify permissions on both the binary and the configuration file, but we now have the ability to commit modifications to our sudoers file into CVS and have those changes distributed to all of our clients.

Note that you can also configure CVS to verify the syntax of the sudoers file so that you never accidentally distribute an invalid file. This is done entirely within CVS, though, so it's left as an exercise for the reader. You could also use the above script to distribute the sudo binary itself, but the assumption here is that you've already installed the package when the system was built. Copying binaries quickly gets complicated if you're dealing with multiple platforms.

Updating CVS

Now that we have our complete sudo configuration and we are successfully updating it from the checked out copy on the server, it's time to see how the server gets the most recent version of the file. As your configurations get more complicated, this simple setup will likely not suffice, but this works well for getting started:

# cvs.cf
control:
    actionsequence = ( shellcommands )

shellcommands:
    "/bin/sh -c 'cd /cfengine; cvs update -d >/dev/null 2>/dev/null'"

This file, imported by the cfengine server from within cfagent.conf, introduces the shellcommands action. As you can see, this is a very simple action. There are some other options you can use, but this is how shellcommands instances usually look. Notice that we had to use an explicit subshell to use cd; when cfengine runs a shell for you, it never interprets shell metacharacters, so if you want the shell to interpret characters such as >, ;, or |, you have to launch a subshell explicitly, as above.

Figure 1 shows how data travels from your CVS sandbox to remote servers. Assuming that cfengine runs every 30 minutes, the potential delays mean that it can take up to 90 minutes for a CVS change to propagate completely.

Figure 1. How data travels from your CVS sandbox to remote servers.

This is a simple method of having cfengine use cvs to update its files. A more sophisticated and less error prone form of this would use a cfengine module that did error checking. As this stands, any errors go to /dev/null (because cvs produces output on STDOUT and STDERR) which means you are not likely to notice a problem quickly. If you did not redirect the output of cvs here, you would get an email every time cfengine ran, which would quickly cause you to ignore all cfengine emails.

Finishing Touches

While that may have seemed like a lot of work, we now have a solid groundwork for using cfengine as the automation harness for the rest of our network tasks. It's now as simple as modifying files and committing them to CVS. Cfengine might not be able to do everything we need it to, but it can at least function as the logic and initiation engine for most everything else. If you have recreated this configuration on your own site, you should now be able to use cfengine to distribute newly committed versions of the sudoers file and to verify that sudo is always set up correctly. All you need to do for this to work all the time is add a cron job to execute it periodically.

Although cfengine is perfectly capable of adding that cron job for you, there are some subtle and complicated issues in doing so, which makes it the perfect topic for the next article in this series. Next time we'll cover inline editing of files using cfengine and how to monitor and restart processes based on changes that cfengine makes. We'll also introduce the use of cfexecd to wrap our call to cfagent in order to get some handling of cfagent output. In the meantime, you can find these examples in CVS at cvs.madstop.com, and you can find multiple examples on how to add a cron job using cfengine at the Cfengine Homepage.

Luke A. Kanies is an independent consultant and researcher specializing in Unix automation and configuration management.

Return to ONLamp.com.

Copyright © 2007 O'Reilly Media, Inc.