Are you sure ntpd wasn't running when the box rebooted? Check your syslog'ed boot logs and see whether it started when the box booted.
Also, remember that if the time is off by too much, ntpd will assume there's a possible problem with your time source, and will not change the time. By default, that happens if the time reported from the external time server is more than 1000 seconds off from the local time. There's a couple of things I've run into over the years (such as CMOS battery dying) that can cause the local machine time to skew greatly at a reboot.
Double check your /etc/ntp.conf config, make sure it's accurate (time servers are reachable, etc). Then try restarting ntpd, and watch the logs while you're doing it. See if everything looks good there. Then go back through your system logs and find all the ntpd related logs. Look for errors, problems, etc. Make sure it's starting on bootup.
In cases like this, logs are usually going to be your best source of information for troubleshooting.
If the time is off by more than 1000 seconds, and you need to reset it, you'll want to first stop ntpd, then run ntpdate to reset the time, then restart ntpd. The following commands will do it:
sudo /etc/init.d/ntp stop
sudo ntpdate 0.north-america.pool.ntp.org
sudo /etc/init.d/ntp start
If you are having problems where the time is skewing badly on every reboot, it may be worth scripting ntpdate to run before ntpd is started on each boot. That should help get the time reset to something manageable before ntpd handles it.
NTP should work fine. Look at some of the options for fast synchronization on start-up. Look at the burst and iburst options for the system B. Look at the true option for the GPS clock source.
Consider using the hardware clock as a backup time source on both systems. Set a higher stratum system B. Something like the following should work:
server 127.127.1.0
fudge 127.127.1.0 stratum 8
Watch the output of ntpq -c peers to see when you get a trusted clock source. Normally ntp wants a number of responses from a trusted time source before it trusts it. This is indicated by the first character on each line.
While NTP likes more sources, any odd number of time sources within one stratum level should work well. As you only have two servers and a GPS clock the priority (stratum) of the sources should increase from GPS, clock on server A, clock on server B. Increasing the stratum between each by three or four levels will ensure priorities are respected.
EDIT: If you have the busybox NTP server on server A, it may be worthwhile installing the full ntp server package. Understanding what is happening with server A should go a long way to solving your problem. You will need at least one trusted time source there before server B should trust it. If ntpq -c peers doesn't work, then you can try ntpdc peers. Both these commands allow you to query other hosts. A peerstats log could also be useful.
On server B use ntpclient as documented the busybox ntp howto to log what is happening on it
The clocks should be reasonably close to the correct time if the servers haven't been down for long. If you need to sync the two systems, that should be sufficient. The GPS will bring the time into sync with the real world eventually.
'ntpd -q' synchronizes quickly, but exits (ntpdate behaviour). It needs to be followed by an ntpd command without the quit option to have continuous synchronization.
EDIT2: I check my server and found one of the servers was off by a second. While fixing this I played with the settings. iburst gets a server trusted very quickly. true ensured the clock driver was trusted if there weren't multiple other trusted sources. The clock took a little more than a minute before it was locally trusted and could be trusted remotely.
When testing you should be able to restart the ntpd process once it is synchronized and test how fast settings work. In the above case Server B may need to be restarted to test how fast it synchronizes. When monitoring ntpd changes I use a line like:
while ntpq -c peers localhost; do sleep 10; done
The hostname and sleep time are adjusted as require. In some cases I chain two or more ntpq command lines in the loop. When doing so I use an echo and/or date command to provide an indication of where sets of data change.
Best Answer
I provided a detailed answer, some time back, of how we do exactly what you're looking to do in our data centers. Not much has changed since I made that post.
Link to the post:
Using NTP to sync a group of linux servers to a common time source