Synchronizing Networks with NTP
by Glenn Graham01/02/2003
Is Your Network Running On Time?
Most system administrators don't consider the importance of
accurate network timing. Under Unix, log daemons such as
syslogd write their information to the central log files
in 00:00:00 (hour:minute:second) format.
tcpdump writes information in the
00:00:00.000000 format. Cron jobs depend on an accurate
system software clock to execute processes. Apache timestamps its
logs.
Why Do We Need Accuracy?
If your server doesn't keep accurate time, your log files are useless in the event of an incident that requires log-dependent information, including security breaches. E-mail servers and other clients depend on accurate time to relay, send, and receive data. What good is the date stamp contained in an e-mail if the server that passed that information is inaccurate? These programs all must be timed precisely to within 1/100 of a second.
Last year our network suffered a power outage. Until I started reviewing the data in the log files, the exact time wasn't clear. I was pleased to find accurate information that pinpointed the exact time of failure on each separate A/C line and UPS in the facility for each server.
Around the same time, another network I administered was
breached. In this case, each server reported to a remote log server
via syslogd that was timed to a Network Time
Server. After the system was attacked, the perpetrator attempted to
cover his tracks by altering the local logfile on the compromised
hosts. What he didn't realize was that another log existed -- "the
real McCoy" on the remote monitor, containing unaltered accurate
information and synchronized via NTP. The information I obtained
proved invaluable and helped to find and prosecute the intruder. When
presenting evidence to law enforcement, the first question they asked
was, "How accurate is the time on your logs?" I was quickly able to
prove the accuracy of the system's time.
Many organizations use a time standardazation protocol called NTP (Network Time Protocol). A team of professionals at the University of Delaware maintains it.
|
Related Reading 
Essential System Administration Pocket Reference |
The Time Protocols
NTP is based on its own unique protocol. Today, NTP has become the de facto standard as it has evolved through various RFCs.
The most recent release is a complete implementation of version 4. It retains compatibility with version 3, as defined by RFC 1305, and versions 1 and 2, defined by RFC 1059 and RFC 1119, respectively. NTP v.4 implements asymmetric encryption, to prevent malicious users from spoofing time packets over port 123. There is no formal RFC for version 4 yet, though there is a draft for using public key cryptography with NTP.
The NTP Chain of Commands
An NTP primary server, or stratum 1, is connected to a high precision reference clock and equipped with NTP software. Other computers (stratum 2s), equipped with similar software automatically query the primary server to synchronize their system clocks. The stratum 2 computers can synchronize other computers, called stratum 3s, and so on, to 16 stratums. The further a computer is from stratum 1, the less accurate its timing information. In this example, each computer can be a client of a server in a higher stratum and a server for computers in a lower stratum. A single primary server can indirectly serve an unlimited number of clients.
To increase reliability, each client may query multiple servers in the upper stratum. In this case, the NTP software continuously monitors the stability and accuracy of all configured servers, switching dynamically to the server with the best figures.
Where does a stratum 1 get its initial data? A stratum 1 requires a radio or satellite receiver. Modern systems retrieve their data from the NIST radio broadcasts or GPS data.
Implementation
NFS, the Network File System, has long been known to have remote exploits and vulnerabilities. Even secure NFS has weaknesses. However, NFS is also a very reliable means of copying data to a central backup server. Several systems I work with implement NFS for that very reason. Here's where the NTP comes in handy. Each server runs a cron job at a specified time to execute its backup. The backup job has the ability to enable and to disable the NFS server. Although I incorporate NFS, I don't allow it to be on all the time - only during automated backups.
Here's the order of operations. First, the cron job fires on the client. It starts NFS. The server knows when this should happen, so it mounts the client share. The server backs up the files on the share and unmounts it. Finally, the client stops NFS.
Once the backup server and clients have synchronized, the script executes on each machine. This is all based on extremely accurate, split-second timing. After the backup is complete, the scripts disable NFS. If one machine is out of sync, that machine won't be able to perform a backup to the central server.
|
Related Reading 
|
The Importance of Choosing an Accurate NTP
Three years ago, I had several servers timed to a single NTP server located at a university. Unbeknownst to me, the administrators were checking for the Unix 2038 time overflow bug. During their test, they advanced their real time clock to the year 2038. Suddenly, each of my log files started writing the year 2038 instead of 1999. This caused massive problems for systems across the network. The lesson learned was that most NTP clients have the ability to query multiple stratums, writing the average of difference to the software clock. Where the difference is significantly out of sync, the client will either compensate to the clock closest to the last known value or exit, leaving the local clock(s) unchanged.
Choosing an NTP Architecture
Arguments abound on how to implement NTP on a local network. Some say that a (single) local server should be designated to poll the upstream stratums, synchronize, then broadcast to the local network. Others believe the local area hosts should (directly) poll the upstream stratums. Most modern systems use the first approach.
The advantage of an internal stratum is so that all machines on your LAN may be timed to the local network. Only one machine needs to be timed to the outside. The disadvantage is that downstream systems may be affected if your local stratum goes off sync.
We'll use a Unix host running ntpd to connect to a stratum 2
across the Internet. Our local machines will each synchronize to the local
stratum.
The ntpd Server
The ntpd daemon can operate as a server, a client, or a relay
from a set of servers to a dependent client population on a local net.
To configure ntpd, edit /etc/ntp.conf.
As the ntp.conf man page is very long, listing every
option is outside the scope of this article. However, useful
configuration commands include server, peer,
broadcast, manycastclient, and options
include autokey, burst, iburst,
minpoll and maxpoll.
Here is the basic ntp.conf for "timekeeper", a stratum
3 on our internal network. Note each server listed is an official
stratum 2.
# File /etc/ntp.conf
# Localhost timekeeper.foo.com
server tick.uh.edu
server time.nist.gov
server tick.usno.navy.mil
driftfile /etc/ntp.driftThe next configuration configures our stratum 4 to check timekeeper, then a stratum 2 machine. It will write the difference to the drift file and the system clock.
# File /etc/ntp.conf
# Localhost yogi.foo.com
server timekeeper.foo.com
server time.nist.gov
driftfile /etc/ntp.driftIn each example, you must touch the file /etc/ntp.drift, which
contains the offset differences of each upstream stratum. Before attempting to
retrieve data, wait until ntpd has had time to synchronize.
Start ntpd by running it from the command line
without any arguments. Be sure to add the daemon to your local start
script. Once the server has been started, check the offset by tailing
your messages log. You should see a line resembling:
Nov 13 03:16:52 striker ntpd[31962]: time set -0.096919 s If you are running in a production environment, I suggest reading
the man section pertaining to restricted access and
encryption. ntpd runs on port 123. Be sure other
applications such as PortSentry are not
bound to this port.
Running NTP Clients
I recommend running NTP software hourly from your cron. Most
Unixes provide ntpdate. (Some Linux distributions
provide a program called netdate, but it does not support
version 4 of the protocol.) Both commands follow the same syntax:
$ ntpdate <primary time server> <secondary time server>For a Windows server or workstation, you can download and install a program called Dimension 4. This program provides several options for checking and correcting the time, and has an excellent, up-to-date list of time servers.
Recent versions of Mac OS X provide a simple checkbox under the
Network Time tab of Date & Time settings.
Summary
Time servers are a science in themselves. As we speak, scientists are working on improvements. If you take time synchronization for granted, then I suggest you read more about this fascinating technology. For those of you worried about the 2038 bug, relax. In 36 years most Unixes as we know them will have been improved; many of us will be retired on a beach in Bermuda or so we hope.
Glenn Graham has been working with telecommunications since 1977. He runs inTEXT Communications.
Return to the Linux DevCenter.
Showing messages 1 through 16 of 16.
-
you can roll your own stratum 1 server quite cheaply
2003-01-16 09:14:22 anonymous2 [Reply | View]
I bought a Garmin GPS unit that has Pulse-per-second (PPS) output and hooked it up to the serial port of my overloaded 233MHz Pentium MMX laptop running RedHat 7.3. I then installed the PPSkit patch from ftp://ftp.kernel.org/pub/linux/daemons/ntp/PPS/ and rebuilt my kernel (now 2.4.18-5customNANO though later patch is available for later kernels). After compiling ntpd and configuring it with GPS_NMEA and PPS here are the results:
ntpdc>
ntpdc> kerninfo
pll offset: 4.817e-06 s
pll frequency: 95.529 ppm
maximum error: 0.002434 s
estimated error: 1.5e-05 s
status: 2107 pll ppsfreq ppstime ppssignal nano
pll time constant: 6
precision: 2.576e-06 s
frequency tolerance: 496 ppm
pps frequency: 95.529 ppm
pps stability: 0.020 ppm
pps jitter: 3.572e-06 s
calibration interval: 256 s
calibration cycles: 7714
jitter exceeded: 10997
stability exceeded: 334
calibration errors: 7
I could make these numbers better if I moved the GPS unit to another location with a better view of the sky (to see more satellites). But this is a huge improvement over its previous accuracy and good enough for my purposes. I'm sure a faster less loaded machine would help too. I recently rebooted the laptop after an uptime of 9 months(!) so the patch has been quite stable for me. Too bad nanosecond granularity isn't in the kernel yet...
- A happy reader
-
setting up clients -- ntpdate may be deprecated
2003-01-09 08:16:19 openhacker [Reply | View]
ntpdate may be replaced by ntpd -q (one shot).
But I find ntpdate only useful on bootup -- then
run ntpd -- it may upset programs to have
their clocks going backwards. Read the man page...
(from ntpdate man page)
ntpdate can be run manually as necessary to set the host clock, or it can be run from the host startup script to set the clock at boot time. This is useful in
some cases to set the clock initially before starting the NTP daemon ntpd. It is also possible to run ntpdate from a cron script. However, it is important to
note that ntpdate with contrived cron scripts is no substitute for the NTP daemon, which uses sophisticated algorithms to maximize accuracy and reliability
while minimizing resource use. Finally, since ntpdate does not discipline the host clock frequency as does ntpd, the accuracy using ntpdate is limited.
-
For intermitent net connections use chrony
2003-01-08 13:00:20 pelliott [Reply | View]
In cases where you are connected to the network only part time, use chrony. It does a better job "disiplining the clocks" when not connected.
-
Great !!
2003-01-07 23:55:59 anonymous2 [Reply | View]
This is really cool. After presenting the idea to my boss, I did an installation on our company network. Easily understood documentation. It works well
Thanks for the idea.
Joe Quan
Monrovia, CA
-
here is ntpd port for NT4/W2K
2003-01-07 07:26:11 anonymous2 [Reply | View]
http://www.five-ten-sg.com/util/ntp4171.zip
-
here is ntpd port for NT4/W2K
2005-05-02 04:40:12 HeikoFromMeinberg [Reply | View]
A Windows port of the original NTP implementation from www.ntp.org can be downloaded from Meinberg Radio Clocks. Please go to our NTP Download Page and get the latest version, which comes with a nice installer and helps you to create your first NTP configuration file.
If you want to build your own stratum 1 time server, you can choose a Meinberg reference clock or a complete Time Server which includes a reference clock (e.g. a GPS receiver) and a Linux based server and runs out of the box.
-
2 more free Windows NTP clients
2003-01-06 23:41:02 anonymous2 [Reply | View]
The article mentioned Dimension4, but it seems a bit old (NT 4 is the most recent OS it claims to support). The one I use is called 'Automachron' (http://oneguycoding.com) and provides a geekfull amount of information. Another one worth mentioning is WorldTime (http://pawprint.net), which shows clocks around the world and has NTP sync ability as well. If you need to see the time in NYC, Singapore and Liverpool at the same time, that's the app for you.
Enjoy,
Amir Katz
-
Your Windows box may already have NTP...
2003-01-06 20:41:30 anonymous2 [Reply | View]
Microsoft shipped TIMESERV.EXE on the NT 4.0 Resource kit.
On NT derived products, Win2K and WinXP, this has
been replaced by a default installation of W32TIME.
The annoying gotcha for most laptop or "non domain" users
is that the thing is set to synchonize via NTP to your
Primary Domain Controller (PDC). This can be
overriden. See:
http://support.microsoft.com/default.aspx?scid=kb;EN-US;q258059
-
What about securing the NTP config???
2003-01-06 19:49:42 anonymous2 [Reply | View]
Is there some reson there was no mention of securing the NTP service under *nix?
Actions such as setting the default access level
'restrict default ignore'
Alowing access from loopback:
restrict 127.0.0.1
The Client Network Access levels:
restrict 10.10.10.0 mask 255.255.255.0 notrust nomodify notrap
Then define Timeservers by first defining access levels and then defining the timeserver:
restrict 192.43.244.18 mask 255.255.255.255 nomodify notrap noquery
server 192.43.244.18
restrict 18.145.0.30 mask 255.255.255.255 nomodify notrap noquery
server 18.145.0.30
restrict 192.5.41.40 mask 255.255.255.255 nomodify notrap noquery
Just a few suggestions....
-
Free Client/Serv for Windows
2003-01-06 09:58:54 anonymous2 [Reply | View]
I see people sharing urls for payware, here is a free one for Windows
www.analogx.com/contents/download/network/ats.htm
As for the article, it's a tad on a lame side, read man ntpd, and ntpq -p is your friend. And don't even try to run ntp server on Windows.
-
Suggestions
2003-01-05 11:39:33 anonymous2 [Reply | View]
You may want to add one more 'server' to your ntp.conf file: localhost.
Adding the following two lines will add the system clock to ntpd(8)'s list of time servers. This way, if the network goes down you'll still get some kind of synchronization:
server 127.127.1.1
fudge 127.127.1.1 stratum 10
Also, it's generally not a good idea to use ntpdate; the sudden "popping" of time can cause havoc with certain programs tha rely on file date/time consistancy: make(1) is one very good example. It's best to run the ntpd(8) on the "clients" as well so the popping of time.
P.S. Can yoy guys add a "Preview" button so we can review our posts before they're submitted?
-
Tardis
2003-01-05 01:27:08 austinblues [Reply | View]
Tardis is a nice time client and server for Windows. It handles several different protocols and comes with a list of time servers. The one drawback is that it only syncs to a single server. There is also a client-only package call K9.
-
Intellisoft's TimeSync for WinNT/2k
2003-01-04 22:18:39 anonymous2 [Reply | View]
Yes, NTP rules... but it usually requires some tunning to work nicely. Read the doc.
To answer the question, I operate several stratum 3. My ISP has one/some 'private' stratum 1 to keep its Juniper routers on time... but these happily respond to NTP queries, offering me top-quality stratum 2 servers to sync onto. :)
What I mean by 'high quality' is low latency and low jitter. AltQ (alternate queuing, a QoS mechanism) is used on our gateway to help guarantee this for NTP packets.
In any case, it sounds like a good idea to check how close the NTP server(s) you want to rely on are (via traceroute, or 'ntpq -c peers' after ntpd has been running for a while).
Afaik, a nearby and properly operating stratum 3 server will serve you better than some remote and heavily loaded stratum 1.
Instead of waiting for clients' requests, our main NTP server broadcasts the time to all listening hosts. These run ntpd as well, as 'broadcastclients'.
ntpdate is therefore used during boot only (actually 'ntpdate -d').
For those still stuck with Windows, an (IMHO way better) alternative to D4 for NT/2k (and probably XP too, not tested) is Intellisoft's TimeSync.
Unlike D4, it works as a service and can receive NTP broadcasts, among other nice features. Details:
http://www.intsoft.com/products/timesync/instructions.html
Old3n <om_oreillyntp031@olden.ch>
-
Intellisoft's TimeSync for WinNT/2k
2006-11-06 08:08:21 ashinton [Reply | View]
We have a number of stratum 1 NTP time servers installed at our location. We use the free Windows NTP Server monitoring software from NTP Server (http://www.TimeTools.co.uk) to monitor and synchronise our windows clients.
http://www.locutuscodeware.com/swatch.htm
The NTP server runs great on Linux though, I have both Linux machines update then have my XP boxes update from one of them. Very Slick.
I also liked the first posters suggestion of using gps to roll your own stratum 1. I may have to look into that.