You can enable logging in ntpd by adding this to ntp.conf:
logfile /var/log/ntpd.log
Source: ntp manual
If you turn off ntpd, can you update the clock by command line? If you run the ntpdate command and receive an error like so:
# ntpdate ntp.ubuntu.com
10 Jan 23:47:57 ntpdate[26284]: Can't adjust the time of day: Operation not permitted
This means that you are probably on a VPS, and in that case you can not modify the system clock - this can only be done on the host machine.
some dude from data center accidentally modify one of the web server
date/time
This is your first problem. It is most likely caused by a combination of:
- 'dude[s] from [the] data center' with insufficient training and
- Overly high privileges
Changing system time requires administrative privileges. Changing the time manually on a system that not only has the correct time, but whose time is being managed using NTP is a sign of insufficient training. Solve this problem first, because until you solve it, accurate system time is probably the most visible of your problems. What else are they doing on this system, and why?
My managers ... said we shouldn't use timestamp to check expiry in the
first place
If there is a viable alternative option that has been proposed, I'd at least consider it. Somehow I suspect that isn't the case.
Network Time Protocol is implemented, because of data centers are
spread across different continents so we have one NTP server in each
data center.
I'd recommend two in each data center. And I'd have them each reference a different set of external NTP servers as well as referencing each other. This is going to result in more stable time and make you much more robust to single failures. I'm also paranoid and over-engineer things, so there's that. Still, NTP servers require roughly nil in terms of resources so run them wherever.
The servers within the data center will have cron jobs to check
against the time with their NTP server from the same data center. If
time is out of sync it will auto update the server date/time.
This is a bad plan. Cron has no place changing the time in an NTP system. The servers should run real NTP clients. These clients should each reference the (two) local NTP servers.
If you want to use cron, use cron on each server to verify that the server is successfully synchronized with both local NTP servers. You can do this by examining the output of the ntpq command. You should learn about the output of the ntpq command; it is your friend.
To address the questions you report as having been raised:
But then with our managers not happy with it, and think it could still
easily causes the same problem. e.g. what if someone accidentally
modify the NTP date/time? what if all the NTP servers are out of sync
with each other? which NTP servers we can really trust? and blah
blah..
The first question isn't insane. A bit paranoid if taken to the extreme, but fine. Answers are:
- Use more than one independent reference clock. (a single error will be ignored in preference to stable time from other sources)
- Use a trustworthy reference clock (e.g. GPS) (If your ops guys can modify the time on a GPS satellite accidentally, you have more serious problems than web server clocks.)
- Use cryptographic keys to ensure that the reference clock you are communicating with is the one you trust.
The second is addressed by configuring the NTP servers to reference each other. They will tend to pull together, all other things being equal. Also by using independent trustworthy reference clocks.
- If one of three lower stratum reference clocks goes out of sync, it will be ignored.
- If two go wildly out of sync, they will be ignored.
- If all three clocks go wildly out of sync, NTP will ignore all three of them and do the best it can (still pretty good, especially if there is a equal-stratum clock it can reference.)
- You pretty much only have to worry about a malicious attack here.
It can get complex to describe these cases, but NTP is about stable first, and accurate if it has an accurate source.
As far as trust, most people who run a public NTP server have no reason to interfere with your time. Many of them have a reason to provide accurate time. In terms of level of interest in providing accurate time, I'd suggest:
- GPS satellites.
- NIST NTP servers.
- Any well known stratum 1 provider.
- Any well known stratum 2 provider.
- Your datacenter (assuming you purchase hosting) should probably have an NTP server or three of their own, for their own use if no other.
Also, and this is important: The NTP protocol is designed to synchronize time to within milliseconds. Not seconds. If you use cron + ntpdate, your time can be off by multiple seconds (thank you variable latency!). NTP will keep your clocks much more stable and accurate under similar circumstances.
Best Answer
Unix and Linux servers as well as any network devices are best set to use UTC for their timezone. This is a standard practice to avoid time change related problems for regions that experience daylight savings.
Ensure that the timezone for the Cisco switch is configured for usage with NTP, and the hardware clock ("system calendar") is configured for UTC as well.
I would almost guess that the Cisco IOS or its NTP server defaults or is configured for EST/EDT (New York, USA) by your problem description.
Using UTC as the timezone everywhere helps prevents issues like these, including consolidating log files with external agencies or ISPes in the event of a malicious user (hacker/cracker) attack.