Linux – Server clock running too slow

In VMWare at least the hardware clock that the VM's OS sees is no more accurate than its own software clock as that clock is based on counting interrupts and ticks and such from the virtual hardware so is susceptible to skew when the VM or its host are under load.

The VMWare tools clock-sync does not sync against the hardware clock as presented as the clock in the virtual hardware - it talks over the VM/host boundary to the host parts of VMWare (which sync it with the OS clock on the host, not the hardware clock).

Other virtualisation solutions will work differently, of course. I have not used Xen enough to know specifically how it behaves in this area. Some VM solutions (including user-mode schemes like UML and arrangements that partition the existing kernel rather than actually virtualising the hardware) will use the host OS clock as the VM's clock directly so all you need to do is keep the host clock in sync.

In general, even on physical hardware, I would not have NTP use the hardware clock as a reference.

Also, if you are using NTP inside a VM make sure you have "tinker panic 0" at the top of your ntp.conf so that the ntp daemon doesn't give up and stop trying to slide the clock back to the right place when it sees a large clock skew caused by a recent glut of load on the VM/host.

Edit: missed a bit... Also, if using Linux in your VMs make sure you use a modern "tickless" kernel - these are said to be far less susceptible to clock skew problems. Most modern Linux releases use this kernel option by default anyway, as it can reduce CPU power consumption.

Are there any ntpd messages in /var/log/messages? As I was dealing with my EC2 NTP issues I noticed that NTP didn't like being too far out of sync and wanted a manual update. Perhaps you're too far out for it to decide it will update for you.