locked
Time Sync issue with a virtualized RedHat Enterprise server RRS feed

  • Question

  • I am running a Hyper-V cluster with 40 virtual servers. About 10 of them are running RedHat Enterprise 5.5.

    When we first noticed the time sync issue, or should I rather say a time drift issue, we started looking at tricks like regularly doing an NTP update but as the load on the server increased the drift became bigger and bigger. I have read many different articles and I am confident in saying that the drift in time (or how quickly the time goes out of sync and by how much) is directly related to bith CPU usage and Network usage. It would appear that the harder the server works, the faster it goes out of sying and the bigger the gap gets. For example, on an idle server we noticed a drift forward in time by about 5 sec per hour but when the load increased it drifted forward by 20-30 sec per hour. As this is a billing server, that is a big problem.

    Finally I was extremely happy to hear about the Linux Integration services. I hastely installed it on one server and it looked fine. I could enable synthetic network adapters and so on but after a day I found that my time was out by two minutes and when we started increasing the load the time went haywire. What am I to do now? I can't find anyone  with the same problem nevermind someone with a solution.

    By the way, the mouse and keyboard integration doesn't work either.

     

    Any help?

    Thursday, January 27, 2011 3:24 PM

Answers

  • Are you using 32 bit or 64 bit RHEL 5.5?

    If you're using 64-bit, then you will need to load the adjtimex package to assist in keeping drift under control, as Red Hat chose to not backport the pluggable time source infrastructure to the 64-bit versions of RHEL.


    This posting is provided "AS IS" with no warranties, and confers no rights.
    Friday, February 18, 2011 5:26 AM

All replies

  • Are you using 32 bit or 64 bit RHEL 5.5?

    If you're using 64-bit, then you will need to load the adjtimex package to assist in keeping drift under control, as Red Hat chose to not backport the pluggable time source infrastructure to the 64-bit versions of RHEL.


    This posting is provided "AS IS" with no warranties, and confers no rights.
    Friday, February 18, 2011 5:26 AM
  • Is there a way to tell if a Linux guest is actually using the "pluggable time source infrastructure"?  I experience time drift on 32 bit RHEL 5.6 guests as well, even with Integration Services 2.1 installed and "Time synchronization" being offered from the Hyper-V host.  The drift is minor but I don't understand why it would occur at all.

    Friday, March 11, 2011 3:44 PM
  • The pluggable time source infrastructure generally keeps the VM clock close to the host clock. However, if the VM is under extreme load, the possibility of drift exists. To mitigate the drift, using NTP will help.

    -M


    This posting is provided "AS IS" with no warranties, and confers no rights.
    Friday, March 11, 2011 4:20 PM
  • Hello all,

     

    I just thought I'd bump this thread.

    I am having the exact same issue, with 64 bit RH 5.5 under Hyper V.

    Have tried pretty much everything listed here and on every other forum I could find.

    The 'clock=' kernel parameters trick seems to be very out of date (and wrong), and in any case I can't reboot to try it. Not yet anyway.

    I have an interim solution which is brutal - run ntp with minpoll and maxpoll both set to 4, but the clock still drifts way too fast.

    Local clock is disabled.

    The Hyper V cluster is communicating with the guest via the installed Tools, with Time Sync set to On, but this seems to not convince NTP of the validity of the local clock anyway.

    If anyone solves this, I'd like to add it to an FAQ somewhere as its a show-stopper for many things.   We are thinking of pushing the install to our VMWare setup instead, or a XenServer setup we have been playing with, but want to avoid that just for now.

     

    Tuesday, March 22, 2011 4:37 AM
  • Are you using 32 bit or 64 bit RHEL 5.5?

    If you're using 64-bit, then you will need to load the adjtimex package to assist in keeping drift under control, as Red Hat chose to not backport the pluggable time source infrastructure to the 64-bit versions of RHEL.


    This posting is provided "AS IS" with no warranties, and confers no rights.


    I am having trouble finding instructions for adjtimex, and adjtimex seems quite a low level tool - ie. you really need to know what you are doing with it and I certainly don't.

    Any possibility of a cheat-sheet somewhere for it?

     

    Regs

    Tuesday, March 22, 2011 4:40 AM
  • I am having trouble finding instructions for adjtimex, and adjtimex seems quite a low level tool - ie. you really need to know what you are doing with it and I certainly don't.

    Any possibility of a cheat-sheet somewhere for it?

    Generally, no configuration is necessary - as long as it's installed and running (which it is after installation), it should just work.
    This posting is provided "AS IS" with no warranties, and confers no rights.
    Wednesday, March 23, 2011 6:39 PM
  • What is the load in the VM that is causing the drift to be as extreme as it is?
    This posting is provided "AS IS" with no warranties, and confers no rights.
    Wednesday, March 23, 2011 6:41 PM
  • I am having trouble finding instructions for adjtimex, and adjtimex seems quite a low level tool - ie. you really need to know what you are doing with it and I certainly don't.

    Any possibility of a cheat-sheet somewhere for it?

    Generally, no configuration is necessary - as long as it's installed and running (which it is after installation), it should just work.
    This posting is provided "AS IS" with no warranties, and confers no rights.


    Thanks but the bizarre thing is that it seems to install just as a utility, not as a daemon in init.d, contrary to other forums I've been reading.  Dunno what the story is there, this is fully licensed RH 5.5 64 and yum searched.

    In the meantime, I got somewhere (thanks to coworker hammering the web in parallel with me) by the use of tickadj, which is part of the ntp suite.

    See this url:

    http://www.redhat.com/archives/rhl-list/2005-February/msg02987.html

    tickadj is virtually undocumented, didn't come with a man page, and has no inspection mode (though I found a util that shows all the clock variables but now I can't find it again, will post when I do).

    Currently a value of:

    tickadj 9650

    -does the trick, the clock now runs slow enough that ntpd can network sync it.

    But read that url if you intend to use it, its a tuning issue and you need to make sure it gets re-issued on boot.

    Also, this can only be viewed as a temporary solution, as the loading on the guest and on the host cluster is going to change and that is bound to affect the less-than-adequate hardware clock passing again.

    Basically, waiting for m$ to release an updated LIC with the hardware clock Time Sync issues fixed, I imagine that isn't too far off, and this will do in the meantime.

     

     

     

    Thursday, March 24, 2011 12:03 AM
  • What is the load in the VM that is causing the drift to be as extreme as it is?
    This posting is provided "AS IS" with no warranties, and confers no rights.


    It appears that 'over' loading is a red herring.  Its 'any' loading - in other words any use of the guest or host! :) 

    The most annoying/irritating part of the whole thing is that its a (typical, I would say, nuff said, grumble....) "two universes" problem - it affects some guests and not others, despite identical kernels, patching, software suites, guest specs, running on the same host in the same cluster and having near-identical loading.

    This is one of those problems that I doubt will ever be solved.  Rather the hardware clock syncing will get fixed in LIC and that will be that.

     

    Thursday, March 24, 2011 12:07 AM
  • Hyper-V, like most virtualization platforms, depends on a reliable interrupt source for accurate timekeeping. In a virtualized environment, the interrupt source is subject to the same fluctuation as everything else. The Linux community (with the help of VMware) implemented the concept of the pluggable time source, which helps normalize interrupt delivery to the virtual machine via a plugin model. We've delivered a pluggable time source module (hv_timesource) for those operating sytems that support it - RHEL 5 x86 and SLES 11 and higher. For RHEL 5 x64, Red Hat chose to not backport the infrastructure that is required to support the pluggable time source model, so we're stuck using ntp and adjtimex to attempt to keep the drift under control. There's nothing else we can do.

    Can you run adjtimex -p and copy the output so we can look at how much drift it's trying to correct?

    -Mike


    This posting is provided "AS IS" with no warranties, and confers no rights.
    Thursday, March 24, 2011 5:28 PM
  • hi this issue
    But we are using NTP however time goes ahead NTP Server like an hour the next day
    red hat 5.7 and 5.6
    any idea?
    Thursday, October 13, 2011 6:27 PM
  • Are you using a 32-bit or a 64-bit version of RHEL?
    This posting is provided "AS IS" with no warranties, and confers no rights.
    Thursday, October 13, 2011 6:29 PM
  • I am using x64 RH 5.5 so far I have NTP Configured but I have not tried the kernel parameters

    I already disabled Time Sync in VM Settings for Hyper-V

    notsc divider=10

    are these enough?

    these is my action plan:

    1. Kernel parameters

    notsc divider=10

    2. Install adjtimex rpm

    not sure if adjtimex must be installed when using NTP

    let's see but any other input would be appreciated 



    • Edited by karlo- Friday, October 14, 2011 1:36 AM
    Friday, October 14, 2011 1:31 AM
  • Hi,

    We use the following set of parameters:

    • RHEL5/x86_64: "divider=10 clock=pmtmr"
    • RHEL5/i386: "divider=10 clocksource=acpi_pm"
    • RHEL4/x86_64: "divider=10 notsc"
    • RHEL4/i386: "divider=10 notsc"

    and run ntpd on the VMs.

    On our ~1000 RHEL5 VMs we also install the Integration Components. We found a few cases where the clock still drifted, but replacing the emulated NIC with a synthetic NIC cured that.

    Note that this requires kernels >= 2.6.18-194.26.1.el5.

    On RHEL6 we haven't observed any clockdrift.

    A propos adjtimex: we install that RPM on RHEL5/x86_64, as it is required by the Integration Components. I don't think we benefit from it, though. Having said that: before we installed it, we had a VM kernel panicking after its host lost network connectivity, and the guest couldn't contact the NTP servers anymore. The ICs tried to use adjtimex, which wasn't installed, and the kernel panicked...

                          hth, cheers, Jan



    Friday, October 14, 2011 11:25 AM
  • thanks a lot

    looks like I have some parameters to try

    I saw these at: notsc divider=10

    http://social.technet.microsoft.com/Forums/en-US/winserverhyperv/thread/28c93dc4-3313-4121-8448-fb30c78d0359?prof=required

     

    and now yours thanks a lot guys

    I will try and I will let you know my journey

    now that you mention no drift on RH 6 I also have 1 RH 5.7 and it's working fine no time issues
    • Edited by karlo- Saturday, October 15, 2011 4:23 PM adding info about RH 5.7
    Saturday, October 15, 2011 4:20 PM
  • If I can revive this old thread, I am also having time sync headaches running RH on our 2008 R2 Hyper-V host. It is 64-bit RH 5.3. I believe IC is installed, as well as adjtimex.

    I added kernel parameters. Also tried turning off time sync on VH and only setting reliable servers as time sources in ntp.conf. But still get too much time drift (often the Linux guest is behind in time, clock is running slow).

    I've also tried turning off ntpd, and using ntpdate in crontab, but I don't like this because if the time gets fast (as it sometimes does without ntpd on), the ntpdate means it hits the same time twice, and crons can fire twice at that instant.

    If I manually run ntpdate it correctly syncs at that time. Also when I start the ntpd service it also correctly syncs at that time, but doesn't seem to keep up to date. (Does that have to do with drift values or something..?)

    Anyway, I'm no linux guru and wondering how I can troubleshoot this? How can I tell if the following are properly installed and working:

    * ntpd

    * adjtimex

    Also, if I turn of time sync in VH IC, should I remove adjtimex?

    Thanks!

    Thursday, May 31, 2012 10:40 PM
  • > Does that have to do with drift values (of ntpd) or something..?)
    Yes, if the drift gets too big, ntpd will give up synchronizing.

    > Also, if I turn of time sync in VH IC, should I remove adjtimex?
    It's been a whil since I took the chance to move to RHEL6 when on Hyper-V.
    But as prevous posters told: RHEL 5.x x86-64 doesn't support the pluggable time source, which means you can enable Time Sync for this VM but the VM won't get time updates via Hyper-V clocksource.

    You can validate that you're using the hyperv_clocksource on modern kernels with:

    cat /sys/devices/system/clocksource/clocksource0/current_clocksource
    > Should give: hyperv_clocksource

    In general for RHEL5 you can somehow try to get adjtimex working by tuning the boot parameter manually - and hopefully get a value that fits your env. - I always had too fast clock. BTW: RHEL 5.3 is EOL, you should really consider updating at least to a supported point release of RHEL5.

    A not so good interim fix can be to have a cron job stopping ntpd, launch ntpdate and restarting ntpd back on regular basis.
    But if you get the chance to migrate to RHEL6, go for it - the drivers are more mature and hv time sync work as expected there also on x86-64.

    Friday, June 1, 2012 4:39 AM