none
Heavy hypervisor workload by XP SP3 guest

    Question

  • Hello,

    I started to work with Hyper-V and found strange thing.

    Using Windows server 2008
    as a guest works fine, but when using XP SP3 as a guest , it runs very slow.

    I checked the performance monitor and found that Hypervisor consumes more then 60% of phisical CPU time,
    and Guest can use only 20 -30 % of the CPU time.

    I'm using RTM bits for Hyper-V(kb950050), give two virtual CPUs to the XP.
    Phisical CPU is Core2DUO 6300.


    There are people around me who are successfully using XP SP2/SP3 under Hyper-V.
    I tryed to use thier virtual image and found that the same thing happens again with my environment.

    Does this mean that Hyper-V 'supports' XP SP3 as a guest but not tuned very well for it (at least under some configuration)?
    Or am I doing something wrong? ( of cource, I've done integration service setup).


    Thank you for any help in advance,

    --
    tamagawa ryuji
    Friday, September 19, 2008 6:12 AM

Answers

  • Microsoft has informed me that the difference in performance documented in the previous post "can be explained by processor support for vTPR (Task Priority Register), a virtualization hardware assist". Additionally, 32-bit XP lacks "Lazy IRQL", which the faster operating systems have. There are no plans to add support for Lazy IRQL to XP.

    Microsoft PSS provided me with this information:

    The B2 stepping does not have vTPR; it was introduced in the E stepping (which was never released), and is present in the G stepping that does not exhibit the performance anomaly. 

     

    To clarify since this is marked as the answer, certain Intel processors with older steppings cannot run 32-bit XP virtual machines with good performance because of the vTPR issue. If you need good performance from 32-bit XP vms and you have, for example, B2 or B3 stepping, you must replace your processor with a later model that has F or newer stepping.

    The 32-bit XP performance problem is a hardware problem. As the previous post indicates, I/O performance is slowed by about an order of magnitude.
    Wednesday, January 21, 2009 6:04 AM

All replies

  • I saw this behavior pre-RTM and even demonstrated it to the product team. My problem went away with the RTM bits.

    How long does the high CPU usage last? Do you know? With the pre-RTM bits, I discovered that the slowness only lasted for about 10 minutes. After that, everything was fine. Do you have antivirus running inside the virtual machine?
    Friday, September 19, 2008 3:44 PM
  • Hello,

     

    This seems like a known issue that we received recently.

     

    The root cause seems related to special CPUs, when some virtual machines are running Windows XP SP3 or Windows 2000.

     

    So far, we have seen that the B2 stepping CPUs may cause this problem that high CPU is consumed even when no application is running in the virtual machine. Some other CPUs (e.g. G0 stepping) may work well with the same configuration of the virtual machine.

     

    We are trying our best to contact vendor companies to work together on this issue. However, I am sorry to say that we are still investigating the root cause and, this may not be fixed in a short time.

     

    I would suggest that, if possible, you choose another processor or, do not run Windows XP SP3 or Windows 2000 operating systems until a KB is published for this issue.

     

    Thank you for your understanding.

     

    Best regards,

    Chang Yin
    Tuesday, September 23, 2008 11:13 AM
  • Go to www.cpuid.com and download CPU-Z. It is an exe that you don't have to install. Run it and find out what your processor's stepping is. Please report back with your findings. Also, in reference to my previous post, have you let your virtual machine run a long enough time to find out if it settles down after running awhile?
    Tuesday, September 23, 2008 11:48 AM
  • Thank you Chang and John, and sorry for my late reply.

    To John:

    I use RTM bits, and the high CPU usage lasts over 10 minuites. Actually, I confirmed that it lasts at least several hours.

    I'm running antivirus software inside the VM, but while I am not doing any operation, the usage of the CPUs are quite low.

    I have checked my CPU with CPU-Z :
     Intel(R) Core(TM)2 CPU 6300@1.86GHz(ES)
     Family 6, Model F, Stepping 2, Ext.Model F, Revision L2.


    To Chan, FYI:

    My CPU info is above, and I have the problem not only with XP SP3, but also with XP SP2.
    I don't see the high CPU usage when I am not doing any operation. When CPU is idle, it's just idle: when CPU is doing some work, hypervisor layor consumes high CPU usage. I have also confirmed that the CPU load is really 'local' to VM(i.e. like while(True){a++;}), it does not affect the hypervisor workload.

    Hope this helps,


    Wednesday, September 24, 2008 12:22 AM
  • Are you familiar with the Hyper-V counters? See http://blogs.msdn.com/tvoellm/archive/2008/05/04/hyper-v-performance-counters-part-one-of-many.aspx for more information. If you aren't using those counters, the results might be skewed a little.

    Is using Vista in your virtual machine an option?
    Wednesday, September 24, 2008 3:24 AM
  • >Are you familiar with the Hyper-V counters?

    Yes, Of course.

    I am observing Hyper-V Hypervisor Logical Processor and
    Hyper-V Hypervisor Virtual Processor counters.

    >
    Is using Vista in your virtual machine an option?

    Practically, I am not in trouble anymore. I am using server 2008 as my working envirionment which works fine as a guest.

    I'm just curious, and hope this topic could help others.
    Wednesday, September 24, 2008 3:39 AM
  • I have had this problem with XP also.  An interesting input to this is that the XP virtual machine that I converted from Virtual Server shows much less processor use than a new, fresh, XP virtual Machine created with Hyper-V.   The converted VM has more running on it but still uses less CPU.  I'm going to check and see if Symantec Endpoing Protection is part of the problem.  The new VM is running SEP, the old one is running Avast, it's just a hunch but I've seen SEP do strange things with CPU usage before.
    Monday, October 27, 2008 4:00 PM
  • We have exactly the same problem with Windows XP SP2

    We are running Hyper-V on a IBM X3950 (8 processor cores, 32 GB Mem)

    We have a lot of Windows 2003 VM´s that are running fine...

    But the Windows XP SP2 (fresh install) is running very slow - just viewing the task manager takes 15% CPU

    There is nothing installed on the XP box, only integration services..
    Palle-DK
    Monday, November 10, 2008 8:51 AM
  • Does the processor usage settle down after a period of time. Pre-RTM I had a reproducible case of CPU usage settling down about 10 minutes after vm bootup. How about trying this? Take a snapshot and then apply XP SP3. See if the same behavior occurs.
    Monday, November 10, 2008 12:49 PM
  •  

    After switching from Virtual Server 2005 R2 SP1 x64 to Hyper-V, I have exactly the same slowdown problem with XP SP3 as Tamagawa described back in Sept 19, 2008.

    Thanks to Chang post the possible reason is clear: the Dell Precision 390 workstation has Intel Core 2 Duo E6420 Conroe (2.13Ghz), Family 6, Model F, Stepping 6, Revision B2.

    RTM Hyper-V is installed, patch http://support.microsoft.com/?kbid=956710 is applied.

    Simple operations (even responsiveness of Start menu or Windows Explorer) are very slow on Windows XP SP3, while Windows Server 2003, Vista or Server 2008 work just fine.

    To demonstrate some numbers, the time it takes to run NewSid (http://technet.microsoft.com/en-us/sysinternals/bb897418.aspx) on blank/fresh installations (with Hyper-V Integration Services already installed) of respective OS (only one VM was running on Hyper-V server during each experiment):

    -          Windows Server 2003 x86: 44 sec                                                                                              

    -          Windows Vista SP1 x86: 140 sec

    -          Windows XP SP3 x86: 360 sec

    Windows XP SP3 is about 8 times slower than Windows Server 2003 in this case.

    To John:

    After starting the XP SP3 VM and waiting for about 10 minutes before launching NewSid, the execution time is 320 seconds. Still more than 7 times slower than Win2003, so it seems that 10 minutes waiting does not help to solve compatibility issue with B2 CPUs. The difference 320 sec vs. 360 sec might be caused by host OS caching (Hyper-V was not rebooted between runs).

    To Chang:

    Unfortunately we cannot stop using Windows 2000 and Windows XP in virtual environment, because our customers are using these OS and we have to support them. Do you know, was the KB article that you had been writing about on Sept 23, 2008 already published? Or is there any kind of workaround to run Win2000/XP on Hyper-V Server with B2 CPUs?   

    Thank you in advance.

    Tuesday, November 18, 2008 10:57 PM
  • Seeing the same thing XP SP3 guest; Xeon E5320 CPU - Stepping 7...

    I thought it was just me; but the behavior is exactly the same as described here..  cpu use spikes for _ANYTHING_ and performance has been extremely poor (6x slower) running application processes on this VM.

    Any ETA on a resolution; or should I be re-doing things at this point?

    Ethan
    Wednesday, November 19, 2008 5:30 PM
  • Quick question: how many Virtual Processors did you configure on your XP VM?
    This posting is provided "AS IS" with no warranties, and confers no rights
    Thursday, November 20, 2008 7:17 AM
  • Just one...
    Ethan
    Thursday, November 20, 2008 3:05 PM
  • Each VM from my previous post (WinXP, W2K3, Vista) has 2 virtual processors. I have tested Windows XP SP3 VM with single virtual processor: everything is still very slow; it takes 404 seconds to run NewSid, slightly slower than with 2 virtual processors.
    Friday, November 21, 2008 12:52 AM
  • two and one. Both configuration shows the same behaivior.
    Monday, December 01, 2008 5:09 AM
  • Tested the same Windows XP SP3 x86 VM image on another Hyper-V Server: Dell Optiflex 960 with Intel Core 2 Duo E84000 Wolfdale (3.00Ghz), Family 6, Model 7, Ext. Model 17, Stepping A, Revision E0. VM runs fine; NewSid execution time is 35-48 seconds, almost 10 times faster than on Dell Precision 390 mentioned in the previous post.
    Wednesday, December 03, 2008 12:11 AM
  • Just curious if there has been any developments...?  Looking back at the machines reporting the problem; I'm pretty sure we are seeing the same issue... 

    Quad Core Xeon - Family 6, Model F, Stepping 7, Revision B3 - absolutely related to the Precision 390 mentioned in the first post.

    MS?  Any updates?

    Ethan
    Thursday, December 11, 2008 4:31 PM
  • I thought the problem went away with RTM bits, but the problem is showing up today. Here is the sequence of events:

    1. Using VPC 2007 SP1, installed XP SP3.
    2. Uninstalled Virtual Machine Additions.
    3. Copied the vhd to a Hyper-V server and configured a vm.
    4. Changed the number of processors to 2.
    5. Increased the ram from 512 to 2048 MB.
    6. Started the vm.
    7. Installed Integration Services.

    The vm was very slow. CPU usage was at 100% for a prolonged period of time. It finally settled to a lower but not normal level.

    Did another test:

    1. Uninstalled Virtual Machine Additions.
    2. Copied the vhd to a Hyper-V server and configured a vm.
    3. Started the vm.
    4. Installed Integration Services.

    The XP SP2 vm ran just fine. No changes were made to the service pack, processor count, or ram.

    The server has 4 Xeon MP 7140M dual core processors:

    3.4 GHz
    Family      F
    Model       6
    Stepping    8
    Ext. Family F
    Ext. Model  6
    Revision    B0


    UPDATE June 2009: If you read the entire thread, you know that the B0 revision indicates it is the vTPR problem explained in the marked answer.
    Friday, December 19, 2008 7:40 PM
  • After conducting additional tests, I have ruled out a Xeon processor as a root cause, although the evidence doesn't rule it out as a contributing factor.

    1. Virtual PC Service Pack 1 build 6.0.192.0

    2. Used volume licensed media to build an XP SP2 virtual machine in VPC. Did not apply any Windows Updates. Did not install any programs. Only changes were defragmentation, precompactor, vhd compaction. Resulting OS is:

       Build 2600.xpsp2_sp2_rtm.040803-2158 : Service Pack 2

    3. Made a copy of the vhd in the previous step. Created a new virtual machine and installed Virtual Machine Additions 13.820.0.0.

    4. Made a copy of the vhd in the previous step. Created a new virtual machine and uninstalled the Virtual Machine Additions. This is to control for the variable of where the additions are removed, VPC or Hyper-V.

    4. Made a copy of the vhd in step 3. Created a new virtual machine and installed XP SP3.

    5. Made a copy of the vhd in step 4. Created a new virtual machine and installed XP SP3.

    6. Copied all five vhds to a Hyper-V server. The server has 4 Xeon MP 7140M dual core processors. Keep in mind that some XP SP3 virtual machines on this server are showing the cpu spikes others have reported.

        3.4 GHz
        Family      F
        Model       6
        Stepping    8
        Ext. Family F
        Ext. Model  6
        Revision    B0


    7. Made five new virtual machines in Hyper-V using the five vhds that came from VPC. Their names are:

        XP SP2
        XP SP2 13.820
        XP SP2 13.820 removed
        XP SP3 13.820
        XP SP3 13.820 removed

    8. Started all five Hyper-V virtual machines. The Hyper-V server was fully patched as of December 12, 2008.

    9. Installed Integration Services on all five virtual machines. On the two machines (XP SP2 13.820, XP SP3 13.820) with the Virtual Machine Additions still installed, it was necessary to uninstall them before the Integration Services could install.

    10. Upon completion of the installation of Integration Services on all five Hyper-V vms, they were all rebooted. All five vms performed as expected. There was no unusual cpu usage in any of them. Device Manager shows an ACPI Uniprocessor PC HAL.

    Every XP SP3 virtual machine showing abnormally high cpu usage on this Hyper-V server is far from being a standard XP installation. Office 2003, custom .NET applications, and McAfee antivirus are installed in these problem virtual machines. The McAfee version is:

    VirusScan Enterprise ver 8.5i
    Scan Engine version 5300.2777
    DAT version 5472.0000

    Additionally, an HP Radia software agent is installed. More testing to follow...






    Tuesday, December 23, 2008 6:11 PM
  • I have a XP SP2 Hyper-V virtual machine that is running at 100% cpu for a prolonged period of time. It was made using Dell Ximage. It is a supposedly universal image for a large suite of Dell hardware and also Virtual PC.

    Device Manager shows an ACPI Uniprocessor PC HAL.

    Here is partial output from Process Explorer (http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx):

    Process                              PID    CPU   Description

    System Idle Process               0
       Interrupts                          n/a
       DPCs                               n/a
       System                                4    11.59
          smss.exe                      564
             crss.exe                     628
             winlogon.exe              652
                services.exe            696      5.80
                   svchost.exe          860     1.45
                      naPrdMgr.exe    248
                      wmiprvse.exe   1132    34.78   WMI
                   svchost.exe          928                                                                                
                   svchost.exe        1028     15.94  Generic Host Process for Win32 Processes
      

    Here are the particulars for the last scvhost.exe process:

    C:\WINNT\system32\svchost.exe                                                    
    Services:                                                                                       
      COM + Event System [EventSystem]                                            
      Cryptographic Services [CryptSvc]                                                 
      DHCP Client [Dhcp]                                                                      
      Distributed Link Tracking Client [TrkWks]                                       
      Error Reporting Service [ERSvc]                                                     
      Fast User Switching Compatibility [FastUserSwitchingCompatibility] 
      Help and Support [helpsvc]                                                            
      HID Input Service [HidServ]                                                            
      Logical Disk Manager [dmserver]                                                   
      Network Connections [Netman]                                                     
      Network Location Awareness (NLA] [NIa]                                       
      Remote Access Connection Manager [RasMan]                             
      Secondary Logon [seclogon]                                                         
      Server [lanmanserver]                                                                   
      Shell Hardware Detection [ShellHWDetection]                                
      System Event Notification [SENS]                                                 
      Task Scheduler [Schedule]                                                           
      Telephony [TapiSrv]                                                                      
      Themes [Themes]                                                                        
      Windows Audio [AudioSrv]                                                            
      Windows Management Instrumentation [winmgmt]                          
      Windows Time [W32Time]                                                             
      Workstation [lanmanworkstation]                                                   


    WMI takes up to 45% of the cpu. Applying SP3 to this vm doesn't change anything for better or worse as far as I can tell.

    Using Process Explorer, the entire wmiprvse.exe process tree was killed. It came back from the dead like an angry zombie seeking cpu for sustenance and it spawned Radia processes.

    When troubleshooting performance problems, it's imperative to identify the variables and account for them systematically. Use the right tools to provide insight into what is really going on. I have Process Explorer, CPU-Z (www.cpuid.com) and other tools on an iso image. By attaching the iso to a running vm, it's easy to use the tools on a vm lacking in diagnostic utilities. Finally, the Hyper-V performance counters are also useful at gaining insight into performance problems.

    I'm going to continue investigating this problem, but the evidence points to Radia. All of my problem vms to date have Radia installed.

    UPDATE June 7, 2009: Although having Radia installed and running made matters worse, as later posts in this long thread indicate, the underlying root cause of the performance problem was the particular Xeon processor being used.

    Tuesday, December 23, 2008 7:23 PM
  • I think you've successfully chased a similar symptom of a different problem.  Many of us with  Family 6, Model F processors can produce the problem with a clean install of the guest OS (no customizations; no 3rd party tools) under Hyper-V...  If you can remove a software component and resolve the issue it would appear that your  Family F, Model 6 processor is not inflicted with the same incompatibility.
    Ethan
    Wednesday, December 31, 2008 3:30 PM
  • My evidence doesn't conclusively rule out a problem with the processor. I'm still seeing an increased cpu workload relative to Virtual PC.

    Wednesday, December 31, 2008 3:43 PM
  • The evidence presented below does support the hypothesis that Family 6, Model F processors have a problem. In an attempt to quantify the problem, the same XP SP3 vm was run on three different Hyper-V machines. Additionally, Virtual PC results are included for comparison. The Family 6. Model F results are generally not as good as the Virtual PC results on lesser hardware.

    Here are the configurations in the order they appear in the graphs:

    1. Hyper-V 4 Xeon 7140M dual cores, vhds on a 6 disk RAID-0 15,000 rpm SCSI array as D. It is a Dell PowerEdge 6800.

    2. Hyper-V T7700 dual core Dell D830 laptop with two internal drives, vhds on a single PATA 7200 rpm hard drive as D.

    3. Hyper-V Q6600 quad overclocked to 3.0 GHz, vhds on a single internal SATA 7200 rpm 1 TB hard drive as D.

    4. Virtual PC 2007 SP1 Q6600 quad overclocked to 3.0 GHz, vhds on a single internal SATA 7200 rpm 1 TB hard drive as D.

    5. Virtual PC 2007 SP1 E6300 dual core Dell, vhds are on a single internal SATA 7200 rpm 0.5 TB hard drive as D.

    Based on processor specifications, the Xeon equipped server is expected to produce the best cpu results. It doesn't, except on the encryption test. On all other cpu measures, it underperforms the laptop and the desktops. That's pitiful.

    Virtual machine details:

    1. One virtual processor allocated per virtual machine.
    2. Two fresh builds of XP SP2 were done, one for Hyper-V, one for VPC. SP3 was applied afterwards. Windows Update was never used.
    3. The poor performance was confirmed with Task Manager.
    4. PassMark's PerformanceTest was installed as a final step. No other changes were made to the installation.

    PassMark's PerformanceTest product was used to produce the benchmarking results.

    CPU - Integer Math
                       scale 0 - 200      ------------------------------------------------------------------
    Xeon Dual Core  7140M 3.4  GHz   64.5            
    Core 2 Duo      T7700 2.4  GHz   97.4                                 
    Core 2 Quad     Q6600 3.0  GHz  127.3                                           
    VPC Core 2 Quad Q6600 3.0  GHz  119.4                                       
    VPC Core 2 Duo  E6300 1.86 GHz   76.5                           

    CPU - Floating Point Math
                       scale 0 - 400      ------------------------------------------------------------------
    Xeon Dual Core  7140M 3.4  GHz  201.9                                  
    Core 2 Duo      T7700 2.4  GHz  254.4                                           
    Core 2 Quad     Q6600 3.0  GHz  332.8                                                        
    VPC Core 2 Quad Q6600 3.0  GHz  301.9                                                   
    VPC Core 2 Duo  E6300 1.86 GHz  192.7                                 

    CPU - Find Prime Numbers
                       scale 0 - 500      ------------------------------------------------------------------
    Xeon Dual Core  7140M 3.4  GHz  182.2                         
    Core 2 Duo      T7700 2.4  GHz  360.5                                                 
    Core 2 Quad     Q6600 3.0  GHz  493.4                                                                   
    VPC Core 2 Quad Q6600 3.0  GHz  494.6                                                                  
    VPC Core 2 Duo  E6300 1.86 GHz  310.6                                          

    CPU - SSE
                       scale 0 - 3000     ------------------------------------------------------------------
    Xeon Dual Core  7140M 3.4  GHz 1237.2                            
    Core 2 Duo      T7700 2.4  GHz 2143.5                                                 
    Core 2 Quad     Q6600 3.0  GHz 2814.8                                                                
    VPC Core 2 Quad Q6600 3.0  GHz 2374.4                                                     
    VPC Core 2 Duo  E6300 1.86 GHz 1548.4                                   

    CPU - Compression
                       scale 0 - 4000     ------------------------------------------------------------------
    Xeon Dual Core  7140M 3.4  GHz 1877.7                                
    Core 2 Duo      T7700 2.4  GHz 2533.5                                            
    Core 2 Quad     Q6600 3.0  GHz 3258.1                                                       
    VPC Core 2 Quad Q6600 3.0  GHz 3318.8                                                       
    VPC Core 2 Duo  E6300 1.86 GHz 2051.1                                   

    CPU - Encryption
                       scale 0 - 20       ------------------------------------------------------------------
    Xeon Dual Core  7140M 3.4  GHz   18.6                                                              
    Core 2 Duo      T7700 2.4  GHz   15.2                                                    
    Core 2 Quad     Q6600 3.0  GHz   19.6                                                                   
    VPC Core 2 Quad Q6600 3.0  GHz   19.7                                                                  
    VPC Core 2 Duo  E6300 1.86 GHz   12.4                                 

    CPU - Image Rotation
                       scale 0 - 700      ------------------------------------------------------------------
    Xeon Dual Core  7140M 3.4  GHz  211.5                  
    Core 2 Duo      T7700 2.4  GHz  494.0                                                
    Core 2 Quad     Q6600 3.0  GHz  665.9                                                                 
    VPC Core 2 Quad Q6600 3.0  GHz  662.5                                                               
    VPC Core 2 Duo  E6300 1.86 GHz  411.3                               

    CPU - String Sorting
                       scale 0 - 3000     ------------------------------------------------------------------
    Xeon Dual Core  7140M 3.4  GHz 1244.5                               
    Core 2 Duo      T7700 2.4  GHz 1785.7                                        
    Core 2 Quad     Q6600 3.0  GHz 2381.7                                                      
    VPC Core 2 Quad Q6600 3.0  GHz 2238.9                                                  
    VPC Core 2 Duo  E6300 1.86 GHz 1218.9                            

    Memory - Allocate Small Block
                       scale 0 - 3000     ------------------------------------------------------------------
    Xeon Dual Core  7140M 3.4  GHz 1374.0                               
    Core 2 Duo      T7700 2.4  GHz 2074.3                                                
    Core 2 Quad     Q6600 3.0  GHz 2766.7                                                              
    VPC Core 2 Quad Q6600 3.0  GHz 2813.4                                                              
    VPC Core 2 Duo  E6300 1.86 GHz 1545.2                                   
     
    Memory - Read Cached
                       scale 0 - 3000     ------------------------------------------------------------------
    Xeon Dual Core  7140M 3.4  GHz 1714.5                              
    Core 2 Duo      T7700 2.4  GHz 1586.3                              
    Core 2 Quad     Q6600 3.0  GHz 2239.4                                                  
    VPC Core 2 Quad Q6600 3.0  GHz 2159.9                                                
    VPC Core 2 Duo  E6300 1.86 GHz 1360.9                         
     
    Memory - Read Uncached
                       scale 0 - 3000     ------------------------------------------------------------------
    Xeon Dual Core  7140M 3.4  GHz 1446.1                                 
    Core 2 Duo      T7700 2.4  GHz 1185.2                           
    Core 2 Quad     Q6600 3.0  GHz 2173.5                                                   
    VPC Core 2 Quad Q6600 3.0  GHz 2078.8                                               
    VPC Core 2 Duo  E6300 1.86 GHz 1192.9                           

    Memory - Write
                       scale 0 - 3000     ------------------------------------------------------------------
    Xeon Dual Core  7140M 3.4  GHz  907.8                     
    Core 2 Duo      T7700 2.4  GHz 1464.9                                 
    Core 2 Quad     Q6600 3.0  GHz 2057.3                                              
    VPC Core 2 Quad Q6600 3.0  GHz 1780.3                                        
    VPC Core 2 Duo  E6300 1.86 GHz 1103.3                         

    Memory - Large RAM
                       scale 0 - 300      ------------------------------------------------------------------
    Xeon Dual Core  7140M 3.4  GHz   34.3        
    Core 2 Duo      T7700 2.4  GHz  144.9                                 
    Core 2 Quad     Q6600 3.0  GHz  176.0                                        
    VPC Core 2 Quad Q6600 3.0  GHz  201.9                                             
    VPC Core 2 Duo  E6300 1.86 GHz  134.1                              

    Disk - Sequential Read
                       scale 0 - 200      ------------------------------------------------------------------
    Xeon Dual Core  7140M 3.4  GHz   24.6         
    Core 2 Duo      T7700 2.4  GHz   32.3            
    Core 2 Quad     Q6600 3.0  GHz   35.6             
    VPC Core 2 Quad Q6600 3.0  GHz  111.9                                     
    VPC Core 2 Duo  E6300 1.86 GHz   81.7                            

    Disk - Sequential Write
                       scale 0 - 30       ------------------------------------------------------------------
    Xeon Dual Core  7140M 3.4  GHz   23.0                                                   
    Core 2 Duo      T7700 2.4  GHz   15.8                                           
    Core 2 Quad     Q6600 3.0  GHz   21.0                                                
    VPC Core 2 Quad Q6600 3.0  GHz   28.4                                                               
    VPC Core 2 Duo  E6300 1.86 GHz   11.7                          

    Disk - Random Seek + RW
                       scale 0 - 70      ------------------------------------------------------------------
    Xeon Dual Core  7140M 3.4  GHz   24.4                        
    Core 2 Duo      T7700 2.4  GHz   0.90  
    Core 2 Quad     Q6600 3.0  GHz   1.31    
    VPC Core 2 Quad Q6600 3.0  GHz   41.9                                        
    VPC Core 2 Duo  E6300 1.86 GHz   68.1                                                                 





    Friday, January 02, 2009 8:17 AM
  • wow, what a thread.

    first of all - I have a comment.  Why give an XP vm two processors?
    There are actually a few reasons why I ask.

    1) Does the VM really _need_ it?
    2) Are you giving adequate RAM so that each virtual processor has a good share?
        Remember the old days of a dual proc box with 512 of RAM, the two processors split it and ended up crippled becuase they only got 256 each...
    3) under virtualization multiple processors are not always better.
        This was very common under VMware.  A PC might run just fine using dual processor hardware, but virtualized with two processors, something was just not right.  But then cutting back to a single virtual processor and all was perfect and good.  This ended up being an important point more often than not.

    Another commonality that I noticed.  NewSID.
    Did no one use SysPrep?  Everyone used NewSID?

    The reason that I ask is that SysPrep forces a hardware re-detection to happen that NewSID does not.  Thus there is potential for new processors, drivers, etc. not being detected properly and therefore new drivers not loading when they should.

    But that does not explain a new, fresh build behaving badly.

    Now, John's strangeness.  I am guessing that we have an agent issue.  Especially since most agents use WMI.  Something with the hardware is not being detected properly and therefore causing down stream issues.
    This harkens back to the fact that not all software survives well through a sysprep or a NewSID.

    John, get that darn Radia thing out of there.  Or, it may require an uninstall and re-install becuase it binds itself to something at the time of installation.  After the restore, it can't find it, therefore it keeps trying and overwhelms the subsystem.
    Personally, I don't like systems agents.  I know they have a place, but frequently they do not image and sysprep well.  Therefore they require a push after deplyment.

    In regards to CPU incompatibility.  This gets really hairy.  If MSFT claims that the found something, I am confident that they did because they should be testing the heck out of it.

    On a related note.  I have been running WinXP VMs on Hyper-V in tons of configurations and on multiple hosts.  I have not seen any of the results that people are reporting. 
    I have had VMs installed fresh and clean, I have converted installs to VHD, I have streamed the OS.

    In all cases I have had a nice, simple single proc and 512 or 256 Mb of RAM.  And with a little tuning of the Host NIC and management network all ran great.  Some hosts were really "this can barely run Hyper-V" workstations and some were decent server class hardware, even blade servers.

    Oh, and in regards to Hyper-V vs. VirtualPC.  This is interesting.  VirtualPC is all emulated while Hyper-V is a flavor of paravirtualization.  Yes, VMs should perform differently.  And in theory, Hyper-V should run better.  But, now we have a maturity of pltform issue to deal with.  VMAdditions were written for XP and Integration Services were made backward compatible for XP (my personall assumption, but I bet that I am not far off).

    I should probably stop now and see what comes back at me.


    Brian Ehlert (hopefully you have found this useful)
    Monday, January 05, 2009 12:56 AM
  • Brian, my benchmarks are from a vm that is a fresh build of XP SP2. There were two fresh builds done - one for VPC, another for Hyper-V. After installing XP, an XP SP3 iso was used to install SP3. Windows Update was never used. The only extra software ever installed was the PerformanceTest benchmarking software - and the poor performance was confirmed in Task Manager before installing PerformanceTest. All of these benchmarks are from single processor configurations.

    In this forum, there are six people including myself reporting Xeon processor problems.
    Monday, January 05, 2009 1:03 AM
  • Brian - I've run sysprep and the issue still happens...  For the purposes of this thread; someone was using newSID as a benchmarking tool...  Effectively the same VM image on two different host PC's has drastically different performance that is not to scale with the differences in the actual host hardware specs.

    I first noticed this issue on a very basic desktop PC that I did a P-V on...  RTM bits of Hyper-V...  XP SP3 running on a P4 1.8ghz desktop computer that we left running in the corner to run a batch download script...  It's a simple install to begin with, OS and anti-virus (which I tried disabled, no impact on performace in the VM) and one app that the download script runs on.  I didn't try to get fancy; I did the sysprep and reload to VM way of transfering to the VM, installed the guest tools and saw horrible performance.  Single processor on the physical machine (no HT) and single processor in the VM...  It should be a fairly compatible migration, and it went smooth...  it just runs 10x slower.

    Since then I've done clean installs of XP and Vista on that box trying to understand what is going on...  The short version is that XP x86 SP3 is 8x or 10x slower than Vista x86 SP1 on the same VM specs (1 processor w/512MB RAM)...  Vista performance is almost identical (actually it's better, but not by much) to the physical machine we set out to replace with a VM.  But it's only a 1.86 ghz quad core Xeon, so with one processor to the VM, I'd expect comporable real world processing speed.

    When testing; I only run one of the instances at a time (the P-V XP SP3, the fresh XP SP3, or the fresh Vista SP1)...  I also have been sure not to over allocate physical RAM.  The host has ~2.5GB untouched when I have the VM's running.

    ...

    For whatever it's worth the issue seems to be with the Conroe / Allendale / Woodcrest / Merom / Kentsfield / Clovertown [family 6, model F] processors...

    ...

    John - I know it's a lot to ask, but you seem to have some extra hardware to play with.  If there was any way you can throw a spare disk in that E6300 and run the same VM's on that physical machine - under hyper-V this time - and compare the results, I think that will truely show the "bug" in action.

    I know you are seeing strange performance, but I'm still not convinced the issue with the Cedar Mill / Presler / Dempsey / Tulsa [family F, model 6]  processors is the same issue.

    Ethan
    Monday, January 05, 2009 3:18 PM
  •  
    It is actually very interesting.  In theory, it should not be an issue.
     
    However, at the same time I know there are changes to the chip architecture and it is continuing to evolve.
     
    I wonder if it is a case where the processor is virtualization aware, but some other component such a memory management is not 100% - or the chipset requires a special driver that does not being loaded by the hypervisor engine itself.  Remember, there is a hypervisor under that parent partition.  hmm....

    One thing that I would like to see done, is these same tests run using XenServer as the hypervisor.
    Why?  One reason is that the architecture of the two hypervisors is similar (don't read into that) in the way they work.

    Another reason is to see if it is a hypervisor or chipset based issue.  In theory if the problem is rooted in the chipset or its implementation of virtualization, then this would affect other, similar, hypervisors as well.

    It is definately something I will be watching unfold.

    Brian Ehlert (hopefully you have found this useful)
    Monday, January 05, 2009 3:55 PM
  • Processor(s)  
    Number of processors 4
    Number of cores 2 per processor
    Number of threads 2 per processor
    Name Intel Xeon MP 7140M
    Code Name Tulsa
    Specification Intel(R) Xeon(TM) CPU 3.40GHz
    Package Socket 604 mPGA
    Family/Model/Stepping F.6.8
    Extended Family/Model F.6
    Core Stepping B0
    Technology 65 nm
    Core Speed 3392.4 MHz
    Multiplier x Bus speed 17.0 x 199.6 MHz
    Rated Bus speed 798.2 MHz
    Stock frequency 3400 MHz
    Instruction sets MMX, SSE, SSE2, SSE3, EM64T
    L1 Data cache (per processor) 2 x 16 KBytes, 8-way set associative, 64-byte line size
    Trace cache (per processor) 2 x 12 Kuops, 8-way set associative
    L2 cache (per processor) 2 x 1024 KBytes, 8-way set associative, 64-byte line size
    L3 cache (per processor) 2 x 16 MBytes, 16-way set associative, 64-byte line size
    Chipset & Memory  
    Northbridge Intel E8500 rev. 10
    Southbridge Intel 82801EB (ICH5) rev. 02
    Memory Type DDR2
    Memory Size 33024 MBytes
    System  
    System Manufacturer Dell Computer Corporation
    System Name PowerEdge 6800
    Mainboard Vendor Dell Computer Corporation
    Mainboard Model 0RD317
    BIOS Vendor Dell Computer Corporation
    BIOS Version A06
    BIOS Date 09/02/2008

    Monday, January 05, 2009 4:16 PM
  • Changing Hypervisors - I haven't done the test; but if I get my hands on some spare hardware I'll give it a shot. 

    I will say that I haven't seen anyone complain about this on similar hardware running VMWare ESX Server.  _zing_...  ;)

    What it honestly "feels" like on this end is some type of complete failure of the processor prefetch, branch prediction or etc. with certain guests...  Such that nearly every single CPU instruction for that guest requires stalling the processor while flushing the contents of the cpu pipeline and reloading the instruction...  If the processor stalls often, but only somehow for the instructions allocated by the hypervisor to that specific guest, it would explain the performance or lack there of.  That's also would explain why it doesn't crash.  The instructions are being executed - just very inefficiently.  Turning all the advanced widgetry of a "modern cpu" off effectively...

    It would also explain why vista x86 or x64 and server 2003 x64 or server 2008 x64 all run really well on that same host server.  Question is - why only a couple (older by the way) guests? 

    My money is on something the HAL is asking of the Hypervisor that isn't handled gracefully...  Maybe something that newer HAL's know enough not to ask?

    Ethan
    Monday, January 05, 2009 4:16 PM
  • Yep, you are right with my thinking there.

    And, I would not expect this under VMware (for the exact reason that I would not expect this under VirtualPC or Virtual Server) due to emulation being the model.  the feature set of the underlying hardware is totally hidden from the VM - all get a standard set of virtual hardware.

    The theory is that Hyper-V was modeled on the Xen architecture.  Which is why the chipset must have virtualization.  This exposes certain functions of the chipset to the VM (in emulation all guests sit on top of the emulation layer and never get to talk to any device features directly).

    The theory is a long term theory that virtualization will eventually become a hardware based commodity.  Thus all a hypervisor will do is prevent resource contention, presentation, and security, but the actual work happens at the processor, or the virtualization aware NIC, or the virtualization aware storage controller, etc.

    I have a new task of looking at PV aware Linux operating systems as VMs on various hypervisors.  I liken the concept to being a similar situation.  Here is my thinking - depending on how a processor and its capabilities are presented to a VM results in a different kernel being installed or loaded.  (just the Linux way of thinking about it).

    In this case, if you are right - it could be as simple as a new Integration Services driver to mask or 'differently present' the capabilities of the processor to the VM.

    Out of curiosity (I just thought of this while typing) - Has anyone tried enabling the "Limit Processor Functionality" bit on these VMs to see if anything behaves differently?
    Crazy idea, but if you have the VMs and hosts sitting around already..  It might be interesting (the operative word being 'might').


    Brian Ehlert (hopefully you have found this useful)
    Monday, January 05, 2009 5:12 PM
  • I can confirm the issue on Xeon 5160's, Stepping 6.   If you contact MS support, reference ID WinSE(RFC) 245712 and it should save you some time, although the root cause still doesn't seem to be well defined.  There is no patch expected anytime soon.  I'm currently trying to work with Dell to get a number of servers/CPU's replaced.
    Monday, January 05, 2009 6:29 PM
  • I spent a few hours on the phone with Microsoft product support. If you are like me and want to get this resolved, you need open a support incident with Microsoft. If you do, please contact me and I'll provide you my SRX number. If any of you have an SRX number, please let me know what it is.

    hughescj - I have contacted Microsoft support and I did "reference ID WinSE(RFC) 245712" as you suggested only to be told that they didn't know what that is. Can you provide more information?

    To be fair to Microsoft, those of us who are affected need to make a more compelling case (like, for example, actually opening a ticket). I have a simple, base vm that performs tolerably when running on Xeon processors. It provides much better on slower hardware, but that's a comparative case, not something that product support is for. Does anybody have a comparison case of Virtual PC or Virtual Server compared on Hyper-V on the same hardware that has Conroe / Allendale / Woodcrest / Merom / Kentsfield / Clovertown processors?

    Monday, January 05, 2009 6:36 PM
  • The SRX for my incident is SRX080910601140. I've been working with MS on this incident since September...  WinSE(RFC) 245712 was the issue report ID provided by Microsoft to make sure the case was referenced properly when working with Dell.
    Tuesday, January 06, 2009 1:52 PM
  • We can confirm that switching from a CPU with a B2 stepping to a CPU with a G0 stepping resolved the problem for us.  Our servers were using Xeon 5160 Stepping 6 (B2, S-SPEC SL9RT) processors.  Dell allowed us to swap out the processors for Xeon 5160 Stepping 11 (G0, S-SPEC SLAG9).  The performance of the VM's is now as expected, ~7 times as fast as it was previously.

    Wednesday, January 07, 2009 1:37 PM
  • My concern is that you had B2 stepping and as an earlier post of mine shows, we have B0 stepping. Is there anybody else with B0 stepping experiencing XP SP3 sluggishness?
    Wednesday, January 07, 2009 6:10 PM
  • I wonder if is in part due to improper handling / detection of Intel I/OAT?

    The difference between the B2 and the G0 Xeon 5160's is Intel I/OAT...  Same with the B3 and the G0 Xeon E5320 that I'm having problems with...  

    Doesn't explain the desktop chips, that all lack I/OAT support.  The E6300 also has the same problem (both the B2 and L2 revisions), yet ones like the E8400 do not have the problem.  Wonder if this is hyper-V detecting the wrong hardware capabilities (makes more sense mistaking one Xeon for another than a Desktop processor for a Xeon, but hey I guess I could see it happening) and asking the chip to do something it can't do?  Or; there is more than one ghost in the machine on this issue?


    Ethan
    Wednesday, January 07, 2009 6:30 PM
  • So working on that pet theory I've been comparing the spec sheet for the various EM64T capable Xeon processors...  I'd be very interested in feedback from anyone who has access to a couple specific pieces of hardware...

    First - Does this G0 Processor with out I/OAT perform well; or slow with Hyper-V and 2000 or XP Guest?

    SLAEQ    1.60 GHz    L5310    4    1066 MHz    65 nm    G0

    Second - Do any of these Bx or Lx Processors with I/OAT perform well; or slow with Hyper-V and 2000 or XP  Guest?

    SLAC2    1.86 GHz    3040    2    1066 MHz    65 nm    L2
    SLABZ    2.13 GHz    3050    2    1066 MHz    65 nm    L2
    SLACD    2.40 GHz    3060    2    1066 MHz    65 nm    B2
    SLACC    2.66 GHz    3070    2    1066 MHz    65 nm    B2
    SL9XA    1.86 GHz    5128    2    1066 MHz    65 nm    B2
    SL9RN    2.13 GHz    5138    2    1066 MHz    65 nm    B2

    If you have these deployed with Hyper-V and stumble on this thread - I'd love to hear your experience.

    Ethan
    Wednesday, January 07, 2009 10:36 PM
  • Below are the results of running seven different child operating systems on the Xeon based server for which the full specifications were published 10 posts ago.

    Notice that the 32 bit versions of XP Service Pack 2 and Service Pack 3 both have significantly impaired disk i/o metrics when compared to Windows Server 2003, Vista, and 64 bit XP.

    Note: There isn't a Service Pack 3 for 64 bit XP. It shares the same code base as 64 bit Windows Server 2003.

    Virtual machine details:

    1. One virtual processor allocated per virtual machine.
    2. Each virtual machine has 1 GB of ram allocated.
    3. Fresh builds of each operating systems were done. Windows Update was never used.
    4. PassMark's PerformanceTest was installed as a final step. No other changes were made to the installation.

    PassMark's PerformanceTest product was used to produce the benchmarking results.

    Disk - Sequential Read
                           scale 0 - 400 ------------------------------------------------------------------
    Windows Server 2003 R2 SP2 x32 347.7                                                           
    Windows Server 2003 R2 SP2 x64 257.3                                            
    Windows Vista SP1          x32 283.1                                                
    Windows Vista SP1          x64 281.1                                                
    Windows XP SP2             x32  24.8     
    Windows XP SP3             x32  24.5     
    Windows XP SP2             x64 282.3                                                

    Disk - Sequential Write
                           scale 0 - 200 ------------------------------------------------------------------
    Windows Server 2003 R2 SP2 x32 128.8                                            
    Windows Server 2003 R2 SP2 x64 139.5                                               
    Windows Vista SP1          x32 123.0                                          
    Windows Vista SP1          x64 126.2                                           
    Windows XP SP2             x32  22.1        
    Windows XP SP3             x32  22.3        
    Windows XP SP2             x64 129.4                                            

    Disk - Random Seek + RW
                           scale 0 - 400 ------------------------------------------------------------------
    Windows Server 2003 R2 SP2 x32 339.6                                                         
    Windows Server 2003 R2 SP2 x64 298.7                                                   
    Windows Vista SP1          x32 212.8                                     
    Windows Vista SP1          x64 244.4                                          
    Windows XP SP2             x32  22.2     
    Windows XP SP3             x32  22.2     
    Windows XP SP2             x64 300.4                                                   

    Monday, January 12, 2009 9:05 PM
  • Microsoft has informed me that the difference in performance documented in the previous post "can be explained by processor support for vTPR (Task Priority Register), a virtualization hardware assist". Additionally, 32-bit XP lacks "Lazy IRQL", which the faster operating systems have. There are no plans to add support for Lazy IRQL to XP.

    Microsoft PSS provided me with this information:

    The B2 stepping does not have vTPR; it was introduced in the E stepping (which was never released), and is present in the G stepping that does not exhibit the performance anomaly. 

     

    To clarify since this is marked as the answer, certain Intel processors with older steppings cannot run 32-bit XP virtual machines with good performance because of the vTPR issue. If you need good performance from 32-bit XP vms and you have, for example, B2 or B3 stepping, you must replace your processor with a later model that has F or newer stepping.

    The 32-bit XP performance problem is a hardware problem. As the previous post indicates, I/O performance is slowed by about an order of magnitude.
    Wednesday, January 21, 2009 6:04 AM
  • John Paul Cook said:

    Microsoft has informed me that the difference in performance documented in the previous post "can be explained by processor support for vTPR (Task Priority Register), a virtualization hardware assist". Additionally, 32-bit XP lacks "Lazy IRQL", which the faster operating systems have. There are no plans to add support for Lazy IRQL to XP.




    Did you get any type of reference id for this (i.e. a Knowledge Base article or a white paper)?

    The CPU problem can be handled with the proper hardware.  However, I am concerned about the XP I/O performance.  It sounds like Server 2008 with Hyper-V will not be a suitable host for XP VMs that have medium to heavy I/O loading.

    Saturday, February 07, 2009 2:15 AM
  • saberman said:

    Did you get any type of reference id for this (i.e. a Knowledge Base article or a white paper)?

    I was told a KB article will be forthcoming.

    The CPU problem can be handled with the proper hardware.  However, I am concerned about the XP I/O performance.  It sounds like Server 2008 with Hyper-V will not be a suitable host for XP VMs that have medium to heavy I/O loading.

    I think you've misinterpreted the results. 32-bit XP virtual machines run just fine on Hyper-V using newer processors. My first set of graphs on the previous page clearly show that. Also on the previous page, the person who described his under warranty processor replacement said I/O performance was as it should be after getting the new processors. (I know who he is and where he is and we've exchanged several emails - he is happy now.)

    I have absolutely no reservations whatsoever about running 32-bit XP on Hyper-V with Intel G stepping or later. My testing is far more extensive than I've published here.

    Saturday, February 07, 2009 3:01 AM
  • I think Microsoft has a measure of liability on this one... 

    The information available about the officially supported guest operating systems for Hyper-V does not mention any specific hardware requirements to use certain guest OS's.  If Microsoft was forthcoming about the subset of EM64T capable hardware that was _actually_ required to use some of the "legacy" guests in the real world under Hyper-V I think a fair number of us may have opted for other technologies - namely VMWare ESX 3.5i... 

    Since this information is still not readily available Microsoft is deliberately leading people to make the wrong decision when evaluating the potential for a P2V project of existing production infrastructure.  It is clear that they want market share in the important virtual machine arena, and don't care if it comes at the customers expense. 

    There are many places on the MS websites; but here are a couple...

    http://technet.microsoft.com/en-us/library/cc731898.aspx
    http://technet.microsoft.com/en-us/library/cc816844.aspx

    [quote]Hardware requirements

    Hyper-V requires specific hardware. To install and use the Hyper-V role, you will need the following:

    • An x64-based processor. Hyper-V is available in 64-bit editions of Windows Server 2008—specifically, the 64-bit editions of Windows Server 2008 Standard, Windows Server 2008 Enterprise, and Windows Server 2008 Datacenter. Hyper-V is not available for 32-bit (x86) editions or Windows Server 2008 for Itanium-Based Systems.

    • Hardware-assisted virtualization. This is available in processors that include a virtualization option—specifically processors with Intel Virtualization Technology (Intel VT) or AMD Virtualization (AMD-V) technology.

    • Hardware-enforced Data Execution Prevention (DEP) must be available and enabled. Specifically, you must enable Intel XD bit (execute disable bit) or AMD NX bit (no execute bit).[/quote]

    The Hardware Considerations for Hyper-V was last updated Feb 4th, 2009 and still has no mention of this issue.  In the insurance world errors and omissions of this nature bring full responsibility to the "seller"...  I don't think it's any different in this case, the damages are time lost to undo an impossible implementation or new hardware purchases to gain the full level of compatibility that was expected by the representation of the capabilities published for Hyper-V.

    Ethan
    Monday, February 09, 2009 8:55 PM
  • John Paul Cook said:
    I think you've misinterpreted the results. 32-bit XP virtual machines run just fine on Hyper-V using newer processors. My first set of graphs on the previous page clearly show that. Also on the previous page, the person who described his under warranty processor replacement said I/O performance was as it should be after getting the new processors. (I know who he is and where he is and we've exchanged several emails - he is happy now.)

    I have absolutely no reservations whatsoever about running 32-bit XP on Hyper-V with Intel G stepping or later. My testing is far more extensive than I've published here.


    I am confused.  Both the graphs and Microsoft's response indicates that disk I/O in an XP VM is significantly slower than in a native XP machine.  The Microsoft comment was "Additionally, 32-bit XP lacks "Lazy IRQL", which the faster operating systems have. There are no plans to add support for Lazy IRQL to XP."

    The graphs were in: 

    John Paul CookMVP - Posted on Monday, January 12, 2009 4:05:20

    (I do not know how to create a hyper link to the post.)

    It appeared that the I/O performance of 32 bit XP VMs (SP2 and SP3) was significantly worse (an order of magnitude) than the I/O performance of VM machines with operating systems that included "Lazy IRQL".  Since the native performance of XP I/O is not an order of magnitude worse than Vista there is a strong implication that XP I/O in a virtual machine would be significantly slower than XP running native on the same hardware.

    Did I misinterpret the information you provided?  Have you rerun the I/O benchmarks using a CPU with a vTPR?  Do you have the same benchmarks for a native 32 bit XP system on the same hardware?

     


     

    Tuesday, February 10, 2009 9:09 AM
  • I reviewed my data and the graphs. I now see how you could be confused. Thanks for persisting and giving me a chance to clarify.

    hughescj said:

    We can confirm that switching from a CPU with a B2 stepping to a CPU with a G0 stepping resolved the problem for us.  Our servers were using Xeon 5160 Stepping 6 (B2, S-SPEC SL9RT) processors.  Dell allowed us to swap out the processors for Xeon 5160 Stepping 11 (G0, S-SPEC SLAG9).  The performance of the VM's is now as expected, ~7 times as fast as it was previously.




    The experience of hughescj is the best evidence. He has about 10 servers. His problem went away when he started running with the new processors.

    My G stepping processors run 32-bit XP just fine. I have additional data that I haven't published. I suppose that I'm opening the door for conspiracy theorists by not publishing my data. I'm confident that the cause and the remedy have been identified. I don't have the time to continue pursuing something that is resolved. Making those graphs takes far more time than I'd care to admit. It's a time management issue for me. I have the answer to my problem and I've moved on - or at least I'm trying to.

    Have you rerun the I/O benchmarks using a CPU with a vTPR?

    Yes. Performance was fine. As the previous quote from hughscj shows, so did he, and performance was fine.

    Operating systems prior to 2003 Server, which includes XP and 2000, do not have lazy IRQL. When an operating system without IRQL runs in Hyper-V on a processor without vTPR, disk performance is slow. Switching to a vTPR processor makes the problem go away. hughescj and I have seen this. The point is that the solution is not to rewrite an old operating system, but to use vTPR processors.




     

    Tuesday, February 10, 2009 1:01 PM
  • John Paul Cook said:

    The point is that the solution is not to rewrite an old operating system, but to use vTPR processors.




     


    And yet to this day there is no mention of that requirement.  Only hardware requirements for Hyper-V listed are still Intel VT or AMD-V and Hardware DEP...

    Ethan
    Tuesday, February 10, 2009 3:01 PM
  • nanunanu said:

    John Paul Cook said:

    The point is that the solution is not to rewrite an old operating system, but to use vTPR processors.


    And yet to this day there is no mention of that requirement.  Only hardware requirements for Hyper-V listed are still Intel VT or AMD-V and Hardware DEP...

    Ethan


    I put together a machine to use Server 2008 with Hyper-V at home.  It has an ASUS M3A76-EM motherboard and an AMD64 Athlon X2 6000 CPU.  According to AMD's Microsoft Hyper-V compatibility check utility "This system is compatible with Hyper-V.  This AMD64 system combines AMD Virtualization (AMD-V) Technology, high performance, and power efficiency."

    I am having an interesting conversation with AMD about this system.  AMD technically support now says AMD does not offically support running Hyper-V on the Athlon -- only on the Opteron.  I have searched the AMD web site and their technical specifications and can find no mention of vTPR for any of their processors. 

    Anyone have any information on AMD processors?

    Wednesday, February 11, 2009 4:02 AM
  • saberman said:

    II have searched the AMD web site and their technical specifications and can find no mention of vTPR for any of their processors. 

    Anyone have any information on AMD processors?



    This discussion from the original post has been about poor performance on Intel processors. vTPR is an Intel feature also known as FlexPriority, not an AMD feature. Discussing AMD processors is a perfectly valid topic, but it is a change of subject, so please start a completely new thread. I think AMD processors need to be at least revision E for good results with Hyper-V.
    Wednesday, February 11, 2009 4:34 AM
  • I have similar problem, Windows Server 2008 std x64 w/ Hyper-V, fresh install, all patch from windows update were installed.
    fresh installed XP SP3 VM runs *VERY* slow, only apps running are guest component and SAV.  It took seconds to open control panel or IE and CPU usage was 100%. After control panel opened, CPU usage drop to 0%.
    My other VMs upgraded from VS2005R2SP1 has same problem. (NOTE: these VMs worked great on VS2005R2SP1)

    My CPU is Core Quad Q6600, stepping 7, revision B3, model F.
    Monday, April 06, 2009 6:15 AM
  • Where did you obtain those processor specs from? B3 is the stepping revision of the original Q6600s, not 7. The B3 Q6600s have the problem. The newer G stepping Q6600s like mine don't have the problem with 32-bit XP virtual machines.
    Monday, April 06, 2009 11:31 AM
  • I got the CPU model name from CPU-Z
    Sunday, June 07, 2009 3:34 AM
  • You have the older Q6600. The B3 stepping revision is the problem.

    Here is the Intel specification sheet http://processorfinder.intel.com/details.aspx?sSpec=SL9UM showing the B3 stepping on the SL9UM Q6600. You will never have good performance with 32-bit XP virtual machines running on that version of the processor.

    Here is the Intel specification sheet http://processorfinder.intel.com/details.aspx?sSpec=SLACR showing the G0 stepping on the SLACR Q6600, which was released later.

    Use Perfmon on your Hyper-V machine when the XP vm is running:

    1. Click the + button to bring up the Add Counters dialog box.
    2. Select Hyper-V Hypervisor Virtual Processor.
    3. Underneath your selection, now select APIC TPR Accesses/Sec.
    4. Under Instances of selected object, you will see the vm's machine name. Select all of the virtual processors associated with it.

    You will probably see elevated APIC TPR Accesses/Sec.


    Sunday, June 07, 2009 5:01 AM
  • Hello!

    I'm exposed to exactly the same behavior with Hyper-V Server R2 RC and a WinXP SP3 guest. This is with the following processor:
    Name: Intel Xeon 5160
    Codename: Woodcrest
    Specification: Intel(R) Xeon(R) CPU 5160  @ 3.00GHz
    Package: Socket 771 LGA (platform ID = 2h)
    CPUID: 6.F.6
    Extended CPUID: 6.F
    Core Stepping: B2


    I have another machine left which can be virtualized. It has this processor:
    Name: Intel Xeon X7350
    Codename: Tigerton
    Specification: Intel(R) Xeon(R) CPU X7350  @ 2.93GHz
    Package: Socket 604 mPGA (platform ID = 3h)
    CPUID: 6.F.B
    Extended CPUID: 6.F
    Core Stepping: G0


    Will I be ok?

    Thanks!

    Sunday, June 14, 2009 6:44 PM
  • Hello!

    I'm exposed to exactly the same behavior with Hyper-V Server R2 RC and a WinXP SP3 guest. This is with the following processor:
    Name: Intel Xeon 5160
    Codename: Woodcrest
    Specification: Intel(R) Xeon(R) CPU 5160  @ 3.00GHz
    Package: Socket 771 LGA (platform ID = 2h)
    CPUID: 6.F.6
    Extended CPUID: 6.F
    Core Stepping: B2


    B2 means you'll never get good performance with 32-bit XP.

    I have another machine left which can be virtualized. It has this processor:
    Name: Intel Xeon X7350
    Codename: Tigerton
    Specification: Intel(R) Xeon(R) CPU X7350  @ 2.93GHz
    Package: Socket 604 mPGA (platform ID = 3h)
    CPUID: 6.F.B
    Extended CPUID: 6.F
    Core Stepping: G0


    Will I be ok?


    You won't know with certainty until you try, but I expect it to work.
    Sunday, June 14, 2009 7:50 PM
  • Thanks for your quick answer!

    It's working very well. Even when the virtual XP has much CPU time the host just has 1-5% on G0.

    Will this problem also occure for Windows Server 2003 32bit on B2 or is it XP specific?
    Tuesday, June 16, 2009 3:19 PM
  • It would be OS's that don't have Lazy IRQL implemented.  So those that would show slow performance are XP, 2000, NT4 etc.  I believe that Server 2003 has this optimization implemented in the 32 bit version as of SP2...
    Ethan
    Tuesday, June 16, 2009 3:27 PM
  • Any insight as to why this is an issue with Hyper-V and not Virtual Server 2005? Would it be, then, safe to say that Windows XP 32bit VMs should continue to housed on Virtual Server 2005 running on Windows 2003 x64? I'm trying to create a testing/development environment where one of the target platforms is Windows XP 32bit, however the Xeon processors avalable to me are 5110 B2s. Any useful workarounds to get the VMs to run better under Hyper-V?

    Thursday, September 10, 2009 2:31 PM
  • On Intel's processor finder the feature you need is I/OAT... 

    http://processorfinder.intel.com/List.aspx?ParentRadio=All&ProcFam=0&SearchKey=5110

    If you look,  you'll see SLABR and SL9RZ are both B2 stepping and don't have the I/OAT...  The SLAGE version is the G0 stepping and has I/OAT... 

    Replace your processors with the SLAGE 5110 G0 flavor or run Vista 32bit as your guest OS since it has Lazy IRQL implemented...  I've been down this road, you don't have any other options.
    Ethan
    Thursday, September 10, 2009 4:13 PM
  • Ivan, Hyper-V and Virtual Server are completely different architecturally. As the previous poster pointed out, there is no workaround. If you want good XP performance on Hyper-V, you must replace the processor. If you don't replace the processor, then you probably will get better performance with Virtual Server 2005 R2 SP1 than Hyper-V.
    Thursday, September 10, 2009 10:33 PM
  • Actually the key difference is that Virtual Server do not virtualize the APIC at all while Hyper-V does. Try changing the HAL to one that do not use an APIC, like ACPI PC or Standard PC.
    Friday, November 13, 2009 1:44 AM
  • Actually the key difference is that Virtual Server do not virtualize the APIC at all while Hyper-V does. Try changing the HAL to one that do not use an APIC, like ACPI PC or Standard PC.

    Doing this means you won't be able to install Integration Services because VMBus relies on APIC.
    Friday, November 13, 2009 12:23 PM
  • Is there a definitive list of the processors exhibiting problems?  I am seeing this on the following processors:

        Number of cores        2 (max 2)
        Number of threads    2 (max 2)
        Name            AMD Opteron 8220
        Codename        Santa Rosa
        Specification        Dual-Core AMD Opteron(tm) Processor 8220
        Package         Socket Fr4 (1207)
        CPUID            F.1.3
        Extended CPUID        F.41
        Brand ID        4
        Core Stepping       
        Technology        90 nm
        Core Speed        1004.8 MHz
        Multiplier x FSB    5.0 x 201.0 MHz
        HT Link speed        1004.8 MHz
        Stock frequency        2800 MHz
        Instructions sets    MMX (+), 3DNow! (+), SSE, SSE2, SSE3, x86-64
        L1 Data cache        2 x 64 KBytes, 2-way set associative, 64-byte line size
        L1 Instruction cache    2 x 64 KBytes, 2-way set associative, 64-byte line size
        L2 cache        2 x 1024 KBytes, 16-way set associative, 64-byte line size
        FID/VID Control        yes
        Max FID            14.0x
        Max VID            1.400 V

        K8 Thermal sensor    yes
        K8 Revision ID        5.3
        Attached device        PCI device at bus 0, device 24, function 0
        Attached device        PCI device at bus 0, device 24, function 1
        Attached device        PCI device at bus 0, device 24, function 2
        Attached device        PCI device at bus 0, device 24, function 3
    Thursday, January 07, 2010 6:11 PM
  • Please post to an open AMD thread discussing this issue or open a new post. I'm trying to keep this thread restricted to Intel. One thread, one problem, please.

    Thursday, January 07, 2010 10:03 PM
  • We can confirm that switching from a CPU with a B2 stepping to a CPU with a G0 stepping resolved the problem for us.  Our servers were using Xeon 5160 Stepping 6 (B2, S-SPEC SL9RT) processors.  Dell allowed us to swap out the processors for Xeon 5160 Stepping 11 (G0, S-SPEC SLAG9).  The performance of the VM's is now as expected, ~7 times as fast as it was previously.


    Hi hughescj,

    Do you have a reference # from Dell for this incident?

    Was it Dell in the USA that you dealt with?
    Wednesday, January 13, 2010 3:52 AM
  • The Dell reference number is 6250062.  This case number was from the US Microsoft High Complexity Group at Dell.  If you need more information feel free to email me at my username@ufl.edu.
    Wednesday, January 13, 2010 11:26 AM
  • The Dell reference number is 6250062.  This case number was from the US Microsoft High Complexity Group at Dell.  If you need more information feel free to email me at my username@ufl.edu.

    Dell was more helpful on this than IBM was...  Might be due to your order size and total business volume with them.  In the end IBM decided there was no problem with the hardware, so I had to buy new SLAEL E5320 processors out of pocket to resolve the issue here.
    Ethan
    Wednesday, January 13, 2010 3:25 PM
  • Hello,

    I have the same issues with Windows XP SP3 on HyperV R2 / 2 X AMD SixCore Opteron Processors 2435, 2600mHz ? Does anybody know if there is a fixed solution for this issue running virtual XP?

    Thanks!

    Sigitas

    Thursday, August 12, 2010 4:13 PM
  • Hallo Sigitas

    I think the Problem is the missing hardware virtualisation support for ACPI TPR by AMD CPU's or older Intel CPU's.

    Former OS`s like Windows XP or older uses this feature and causes a slow down of the os, particularly if you use a more than one CPU in your VM.

    Only CPU`s with hardware virtualisation support for ACPI TPR  (Like actual Intel CPU's, or perhabs upcoming AMD BULLDOZER) will work with expected speed.

    ps: i hope that microsoft will work on a fixed hal for windows xp to speed up xp under hyper -V.

    over and out

     

     

    • Proposed as answer by Sigitas Wednesday, August 18, 2010 7:38 PM
    Tuesday, August 17, 2010 7:07 PM
  • After I moved XPs to HyperV with Intel CPU, workstations performs just fine.
    Wednesday, August 18, 2010 7:45 PM
  • Has there been any new developments from MS for a fix for XP?

    We purchased MANY Dell M905 blades to virtulize just about everything using Hyper-V and they all have Quad Quad-core AMD processors.  I'm pushing as many people forward to Windows 7 x64 as I can but we still have many legacy apps that require WinXP x86.

    Thursday, February 10, 2011 1:45 PM
  • This has been a really touchy issue as most indications have pointed to it being a hardware related issue.

    Possibly kernel code, or in processor issue that for most folks have been resolved with changing the CPU.

    But quite honestly, if your VMs are XP and they only have 1 vCPU and you have this issue - then I would point straight at the hardware.  Specifically a particular moment in time of CPU production.

    If your VMs are XP and you have more than 1 vCPU in the VM - then most folks get relief by dropping to 1 vCPU.

     


    Brian Ehlert (hopefully you have found this useful) http://ITProctology.blogspot.com
    Thursday, February 10, 2011 5:03 PM
  • Is there any change of a fix by the Hyper-V SP1 release. Or is this something that can never be fixed by the Hypervisor
    DirkK.
    Thursday, February 24, 2011 11:09 AM
  • This has been a really touchy issue as most indications have pointed to it being a hardware related issue.


    Brian Ehlert (hopefully you have found this useful) http://ITProctology.blogspot.com

    Touchy indeed.  It's hardware and software working together to cause the problem, and I still say that the lack of disclosure about this in marketing material (see:  http://www.microsoft.com/hyper-v-server/en/us/system-requirements.aspx no specific hardware clarifications mentioned) is on MS not the chip maker or system builder.
    Ethan
    Saturday, March 05, 2011 6:49 AM
  • Is there any change of a fix by the Hyper-V SP1 release. Or is this something that can never be fixed by the Hypervisor
    DirkK.


    I replaced my hardware so I can't test it any more, but I doubt it.  If you have hardware that experiences the performance issue, you must run a guest OS with Lazy IRQL implemented.  Vista and Windows 7 (both 32 and 64 bit), or XP 64bit only.  Server 2003 SP2 and higher and all server 2008 + flavors will also be just fine.

    If you must run XP 32 bit as a VM guest, you'll need to replace your hardware with a stepping model that has vTPM implemented. 

    The software fix would be either for Lazy IRQL to be implemented in XP SP4 which will never happen obviously, or for the guest integration tools to implement it somehow for guests that don't have it natively implemented which (with out getting into how difficult that may be to actually implement) is IMHO what MS should have done...  I understand why they have not done such a thing, but it just gets me the wrong way that either a software fix or clearer hardware requirements have not been offered to customers, now nearly two and a half years removed from the problem.

    Edit:  Oh, I never did review the R2 supported software configs.  Looks like the official and simple answer is to remove XP from the list of supported guests with 2008 R2 Hyper-V.  So with that, LOL, the answer is No.  This will never get fixed.  Someday the hardware will cycle out of production and the problem will only be a bad memory.  :)


    Ethan
    Saturday, March 05, 2011 6:58 AM
  • While I would love to drop XP from my supported OS list we still have legacy apps that the highest they will run is on XP so we ended up having to buy a few additional servers specifically to virtualize XP.  Dell sells the M610 blade that has an Intel processor that works with XP virtuals.  Stupid solution but guess we are stuck with it until we phase out XP.

    Thursday, April 21, 2011 12:22 PM