Linux – Weird fluctuating time on a XEN linux guest

I have a weird problem with some servers here at work. We have a few XEN guests who's current time fluctuates.

# date;date;date;date;date;date;date
Thu Feb 25 16:00:40 PHT 2010
Thu Feb 25 16:00:48 PHT 2010
Thu Feb 25 16:00:40 PHT 2010
Thu Feb 25 16:00:48 PHT 2010
Thu Feb 25 16:00:40 PHT 2010
Thu Feb 25 16:00:48 PHT 2010
Thu Feb 25 16:00:40 PHT 2010

As seen above, the time fluctuates between 16:00:48 and 16:00:40, which is problematic for us since computing for time differences in some of our scripts becomes inaccurate (ex. what should be a few ms differences becomes some few second differences, and even sometimes, negative differences).

The problematic servers are linux guests on a XEN host. The time fluctuates on the guest systems, but it is okay in the host itself. I've ruled out ntpd since this happens irregardless of whether ntpd is running or not on the guest systems.

Guest is on full virtualisation. The time on both the host and the guest does match except that the time in the guest fluctuates at about a few seconds from the host's time, and the host time does not fluctuate.

/proc/sys/xen/independent_wallclock is 0 in the host and does not exist in the guest. Ntpd service was stopped and disabled. Setting independent_wallclock to 1 in the host has no effect (that is, time still fluctuates in the guest). Though I was not able to restart the guest as it is a production server. Might be able to do that over the weekend.

Any ideas on what to check and how to resolve this problem?

After a long bit of search and testing, the kernel parameters that worked perfectly are:

divider=10 clocksource=acpi_pm

I used this page to determine which parameters were best for the OS we were using. The information applied to VMware, but amazingly also applies to XEN. We also had ntpd re-enabled so we have a working time-sync.