Synchronizing Networks with NTPby Glenn Graham
Is Your Network Running On Time?
Most system administrators don't consider the importance of
accurate network timing. Under Unix, log daemons such as
syslogd write their information to the central log files
tcpdump writes information in the
00:00:00.000000 format. Cron jobs depend on an accurate
system software clock to execute processes. Apache timestamps its
Why Do We Need Accuracy?
If your server doesn't keep accurate time, your log files are useless in the event of an incident that requires log-dependent information, including security breaches. E-mail servers and other clients depend on accurate time to relay, send, and receive data. What good is the date stamp contained in an e-mail if the server that passed that information is inaccurate? These programs all must be timed precisely to within 1/100 of a second.
Last year our network suffered a power outage. Until I started reviewing the data in the log files, the exact time wasn't clear. I was pleased to find accurate information that pinpointed the exact time of failure on each separate A/C line and UPS in the facility for each server.
Around the same time, another network I administered was
breached. In this case, each server reported to a remote log server
syslogd that was timed to a Network Time
Server. After the system was attacked, the perpetrator attempted to
cover his tracks by altering the local logfile on the compromised
hosts. What he didn't realize was that another log existed -- "the
real McCoy" on the remote monitor, containing unaltered accurate
information and synchronized via NTP. The information I obtained
proved invaluable and helped to find and prosecute the intruder. When
presenting evidence to law enforcement, the first question they asked
was, "How accurate is the time on your logs?" I was quickly able to
prove the accuracy of the system's time.
Many organizations use a time standardazation protocol called NTP (Network Time Protocol). A team of professionals at the University of Delaware maintains it.
The Time Protocols
NTP is based on its own unique protocol. Today, NTP has become the de facto standard as it has evolved through various RFCs.
The most recent release is a complete implementation of version 4. It retains compatibility with version 3, as defined by RFC 1305, and versions 1 and 2, defined by RFC 1059 and RFC 1119, respectively. NTP v.4 implements asymmetric encryption, to prevent malicious users from spoofing time packets over port 123. There is no formal RFC for version 4 yet, though there is a draft for using public key cryptography with NTP.
The NTP Chain of Commands
An NTP primary server, or stratum 1, is connected to a high precision reference clock and equipped with NTP software. Other computers (stratum 2s), equipped with similar software automatically query the primary server to synchronize their system clocks. The stratum 2 computers can synchronize other computers, called stratum 3s, and so on, to 16 stratums. The further a computer is from stratum 1, the less accurate its timing information. In this example, each computer can be a client of a server in a higher stratum and a server for computers in a lower stratum. A single primary server can indirectly serve an unlimited number of clients.
To increase reliability, each client may query multiple servers in the upper stratum. In this case, the NTP software continuously monitors the stability and accuracy of all configured servers, switching dynamically to the server with the best figures.
Where does a stratum 1 get its initial data? A stratum 1 requires a radio or satellite receiver. Modern systems retrieve their data from the NIST radio broadcasts or GPS data.
NFS, the Network File System, has long been known to have remote exploits and vulnerabilities. Even secure NFS has weaknesses. However, NFS is also a very reliable means of copying data to a central backup server. Several systems I work with implement NFS for that very reason. Here's where the NTP comes in handy. Each server runs a cron job at a specified time to execute its backup. The backup job has the ability to enable and to disable the NFS server. Although I incorporate NFS, I don't allow it to be on all the time - only during automated backups.
Here's the order of operations. First, the cron job fires on the client. It starts NFS. The server knows when this should happen, so it mounts the client share. The server backs up the files on the share and unmounts it. Finally, the client stops NFS.
Once the backup server and clients have synchronized, the script executes on each machine. This is all based on extremely accurate, split-second timing. After the backup is complete, the scripts disable NFS. If one machine is out of sync, that machine won't be able to perform a backup to the central server.
The Importance of Choosing an Accurate NTP
Three years ago, I had several servers timed to a single NTP server located at a university. Unbeknownst to me, the administrators were checking for the Unix 2038 time overflow bug. During their test, they advanced their real time clock to the year 2038. Suddenly, each of my log files started writing the year 2038 instead of 1999. This caused massive problems for systems across the network. The lesson learned was that most NTP clients have the ability to query multiple stratums, writing the average of difference to the software clock. Where the difference is significantly out of sync, the client will either compensate to the clock closest to the last known value or exit, leaving the local clock(s) unchanged.
Choosing an NTP Architecture
Arguments abound on how to implement NTP on a local network. Some say that a (single) local server should be designated to poll the upstream stratums, synchronize, then broadcast to the local network. Others believe the local area hosts should (directly) poll the upstream stratums. Most modern systems use the first approach.
The advantage of an internal stratum is so that all machines on your LAN may be timed to the local network. Only one machine needs to be timed to the outside. The disadvantage is that downstream systems may be affected if your local stratum goes off sync.
We'll use a Unix host running
ntpd to connect to a stratum 2
across the Internet. Our local machines will each synchronize to the local
ntpd daemon can operate as a server, a client, or a relay
from a set of servers to a dependent client population on a local net.
ntp.conf man page is very long, listing every
option is outside the scope of this article. However, useful
configuration commands include
manycastclient, and options
Here is the basic
ntp.conf for "timekeeper", a stratum
3 on our internal network. Note each server listed is an official
# File /etc/ntp.conf # Localhost timekeeper.foo.com server tick.uh.edu server time.nist.gov server tick.usno.navy.mil driftfile /etc/ntp.drift
The next configuration configures our stratum 4 to check timekeeper, then a stratum 2 machine. It will write the difference to the drift file and the system clock.
# File /etc/ntp.conf # Localhost yogi.foo.com server timekeeper.foo.com server time.nist.gov driftfile /etc/ntp.drift
In each example, you must touch the file
contains the offset differences of each upstream stratum. Before attempting to
retrieve data, wait until
ntpd has had time to synchronize.
ntpd by running it from the command line
without any arguments. Be sure to add the daemon to your local start
script. Once the server has been started, check the offset by tailing
your messages log. You should see a line resembling:
Nov 13 03:16:52 striker ntpd: time set -0.096919 s
If you are running in a production environment, I suggest reading
man section pertaining to restricted access and
ntpd runs on port 123. Be sure other
applications such as PortSentry are not
bound to this port.
Running NTP Clients
I recommend running NTP software hourly from your cron. Most
ntpdate. (Some Linux distributions
provide a program called
netdate, but it does not support
version 4 of the protocol.) Both commands follow the same syntax:
$ ntpdate <primary time server> <secondary time server>
For a Windows server or workstation, you can download and install a program called Dimension 4. This program provides several options for checking and correcting the time, and has an excellent, up-to-date list of time servers.
Recent versions of Mac OS X provide a simple checkbox under the
Network Time tab of
Date & Time settings.
Time servers are a science in themselves. As we speak, scientists are working on improvements. If you take time synchronization for granted, then I suggest you read more about this fascinating technology. For those of you worried about the 2038 bug, relax. In 36 years most Unixes as we know them will have been improved; many of us will be retired on a beach in Bermuda or so we hope.
Glenn Graham has been working with telecommunications since 1977.
Return to the Linux DevCenter.