some dude from data center accidentally modify one of the web server
date/time
This is your first problem. It is most likely caused by a combination of:
- 'dude[s] from [the] data center' with insufficient training and
- Overly high privileges
Changing system time requires administrative privileges. Changing the time manually on a system that not only has the correct time, but whose time is being managed using NTP is a sign of insufficient training. Solve this problem first, because until you solve it, accurate system time is probably the most visible of your problems. What else are they doing on this system, and why?
My managers ... said we shouldn't use timestamp to check expiry in the
first place
If there is a viable alternative option that has been proposed, I'd at least consider it. Somehow I suspect that isn't the case.
Network Time Protocol is implemented, because of data centers are
spread across different continents so we have one NTP server in each
data center.
I'd recommend two in each data center. And I'd have them each reference a different set of external NTP servers as well as referencing each other. This is going to result in more stable time and make you much more robust to single failures. I'm also paranoid and over-engineer things, so there's that. Still, NTP servers require roughly nil in terms of resources so run them wherever.
The servers within the data center will have cron jobs to check
against the time with their NTP server from the same data center. If
time is out of sync it will auto update the server date/time.
This is a bad plan. Cron has no place changing the time in an NTP system. The servers should run real NTP clients. These clients should each reference the (two) local NTP servers.
If you want to use cron, use cron on each server to verify that the server is successfully synchronized with both local NTP servers. You can do this by examining the output of the ntpq command. You should learn about the output of the ntpq command; it is your friend.
To address the questions you report as having been raised:
But then with our managers not happy with it, and think it could still
easily causes the same problem. e.g. what if someone accidentally
modify the NTP date/time? what if all the NTP servers are out of sync
with each other? which NTP servers we can really trust? and blah
blah..
The first question isn't insane. A bit paranoid if taken to the extreme, but fine. Answers are:
- Use more than one independent reference clock. (a single error will be ignored in preference to stable time from other sources)
- Use a trustworthy reference clock (e.g. GPS) (If your ops guys can modify the time on a GPS satellite accidentally, you have more serious problems than web server clocks.)
- Use cryptographic keys to ensure that the reference clock you are communicating with is the one you trust.
The second is addressed by configuring the NTP servers to reference each other. They will tend to pull together, all other things being equal. Also by using independent trustworthy reference clocks.
- If one of three lower stratum reference clocks goes out of sync, it will be ignored.
- If two go wildly out of sync, they will be ignored.
- If all three clocks go wildly out of sync, NTP will ignore all three of them and do the best it can (still pretty good, especially if there is a equal-stratum clock it can reference.)
- You pretty much only have to worry about a malicious attack here.
It can get complex to describe these cases, but NTP is about stable first, and accurate if it has an accurate source.
As far as trust, most people who run a public NTP server have no reason to interfere with your time. Many of them have a reason to provide accurate time. In terms of level of interest in providing accurate time, I'd suggest:
- GPS satellites.
- NIST NTP servers.
- Any well known stratum 1 provider.
- Any well known stratum 2 provider.
- Your datacenter (assuming you purchase hosting) should probably have an NTP server or three of their own, for their own use if no other.
Also, and this is important: The NTP protocol is designed to synchronize time to within milliseconds. Not seconds. If you use cron + ntpdate, your time can be off by multiple seconds (thank you variable latency!). NTP will keep your clocks much more stable and accurate under similar circumstances.
Now it seems you're not communicating with your peers. Either they're not up, not reachable, or a firewall or similar device is blocking UDP port 123.
207.32.191.59 appears to be a working stratum 2 server to me, so if the machine otherwise has working Internet access, I'd bet on a firewall.
Best Answer
ntp.org provides a list of primary and secondary time servers. I think it's reasonable to assume these will generally be static addresses, especially compared to the NTP pool which intentionally rotates every hour.
Please note that not all are open/public access and abide by the rules of engagement, which suggest using secondary servers over primary among other things.