RTC time is going totally wrong after a few minutes (Ubuntu 16.04 + Hyper-V)

What does

/usr/sbin/ntpq -p

show?

Try doing

sntp -P no -r pool.ntp.org

to set the time right and do afterwards a

hwclock --systohc

to write the time to the hardware clock.

The most reliable method of maintaining time in a virtualized domain environment is to keep your PDC physical, increase the sync frequency and drift tolerance on all your virtualized DC and domain members, and disable host-guest time synchronization for all VMs in question.

Here's a pseudo-explanation of the problem (it's been a long time since I looked at this):

When a machine (workstation, server, VM) starts up, it reads the time from the RTC (battery backed clock chip on the motherboard/BIOS), and works out the duration of each CPU tick. The OS then counts the number of clock ticks that have occurred since the initial reading was taken, and adds the time from that to the original time reading taken at boot. This gives you the current time.

Problem is, hosts obfuscate the true clock cycles occurring from the VMs. A VM may have seen 100 clock cycles when 500 clock cycles have actually occurred on the host. So this method of calculating time breaks down, and time drifts out of whack on the VM.

Host-Guest time synchronization via the installed vm tools/enhancements packages on vSphere and Hyper-V go some way to curing this, but they're not perfect (in some setups they can pull a VM forwards if it's drifted behind realtime, but they don't jump a VM forwards if it's drifted ahead of realtime).

This complicated further by the way clock cycles are counted on multi-core setups (the timing counter is essentially emulated on each core) and on setups that can change clock speeds on the fly (I have no freaking idea how this is maintained). Factor in the idea that a VM can execute one clock cycle on one core, then jump to a different core on a different CPU for the next cycle, and it gets really horrible.

So back to the original point: Domain time by default starts at the PDC, then trickles down to the other DCs, then out to the member servers and workstations from there. So if you ensure your PDC is a truly reliable time source (by keeping it physical), and disable host-guest sync on all other domain members, you'll ensure a stable and relatively accurate time infrastructure.

Note that running your PDC as a physical server then enabling Hyper-V on that server and adding some guests to it is probably not a suitable fix either, as I believe that when you enable Hyper-V the 'base' OS actually becomes a virtualised OS as well (silently). So keep one physical server as a PDC, and keep Hyper-V off the box.

Interesting side point to note: Microsoft's official stance on Windows Time Synchronization, even using the compliant NTP service that's been built into Windows since XP/2003, is 15 seconds. In practice, you can get it down to sub-100ms, but all they'll support is synchronization to within 15 seconds. Kinda makes sense, the only time-sensitive key component at the core of most MS environments is Kerberos, and by default that'll operate happily as long as you're within a 5 minute tolerance.