none
Volume Missing RRS feed

  • Question

  • DPM 2010 installed on a Server 2008 r2 box.  The server is connected to three MSAs (HP MSA 2012fc DC).  Each MSA has two 10 TB virtual disks carved out making a total of six 10 TB virtual disks.  In disk management on the DPM server I can see all six disks.  I initialized the disk GPT and the disks are online.  I added one disk to the DPM storage pool and created a protection group consisting of a virtual host that I am protecting 7 VMs, 3 volumes, and BMR. I have co-located the VMs.  I was able to create the replicas and DPM will create recovery points and even sync up.  The problem is that I am constantly loosing the disk in the DPM storage pool.  It comes up volume missing in DPM and when I go to Disk Management it is gone or unhealthy.  All the other virtual disks that I have not added to the storage pool are still connected.  When this happens it takes a long to time to open Dsik Management and sometimes the Virtual disk will still be there but with errors.  I can reactivate the disk and the problem is fixed temporarily.  This happens a couple times per day. 

    Troubleshooting steps:

    Event logs were showing MPIO errors and Server was blue screening - Logged a call with MS and installed KB 978500.  I believe this updated the DSM driver. May have fixed BSOD but volume is still coming up missing.

    Logged call with HP and they confirmed that the MSA was configured correctly.

    The server was taking a really long time to reboot and would hang during shutdown.  I disconnected the one MSA with the virtual disk that DMP has in it's storage pool and the server is blazing fast.  Shutsdown and restarts with no problems.

    Any ideas?

    Friday, August 27, 2010 12:29 AM

Answers

  • We tried both HP and Microsoft's MPIO DSM and it did not make a difference.  HP has actually recommended the Microsoft MPIO for our configuration.  We are on a HP BL460 G6 with Server 2008 R2SAN connected via QLogic card to a HP MSA 2012 fc.  Hp has recently replaced the controllers, enclosures and multiple power supplies and we have not experienced the problem since.
    Wednesday, December 22, 2010 10:34 PM

All replies

  •  

    This is not a DPM issue, you need to troubleshoot your storage / connectivity issue, are you still seeing events in the event log ?  You can install this qfe to help monitor i/o to the san storage.

    978000 Add a fix to improve the logging capabilities of the Storport.sys driver to troubleshoot poor disk I/O performance in Windows Server 2008 R2
    http://support.microsoft.com/default.aspx?scid=kb;EN-US;978000


    Regards, Mike J [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Friday, August 27, 2010 3:01 PM
    Moderator
  • Yes, l am still seeing events in the logs.

    Source = MPIO Event ID = 23  All paths have failed. \Device\MPIODisk3 will be removed.

    Source = MPIO Event ID = 16  A fail-over on \Device\MPIODisk3 occurred.

    Those errors are followed by a bunch of Disk errors:

    Source = Disk Event ID = 15 The device, \Device\Harddisk5\DR5, is not ready for access yet.

    Then I get a volsnap error:

    Source = volsnap Event ID = 85

    Volume D:\...es\Microsoft DPM\DPM\Volumes\Replica\File System\vol_3820dd57-fe06-4d89-a3f7-1e5be21f1957 is offline for shadow copy protection. A read failure occurred during a shadow copy on write operation. Please try clearing the protection fault or restart the computer. If all else fails, revert out of shadow copy protection mode to reclaim the use of the volume while losing the shadow copies.

    Followed by more disk errors and the following Ntfs warning:

    Source = Ntfs Event ID = 141

    The description for Event ID 141 from source Ntfs cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

    If the event originated on another computer, the display information had to be saved with the event.

    The following information was included with the event:

    D:\Program files\Microsoft DPM\DPM\Volumes\Re .. \vol_3820dd57-fe06-4d89-a3f7-1e5be21f1957\$Mft

    the message resource is present but the message is not found in the string/message table

     

    I will install 97800 to trouble shoot the disk i/o.  Any other suggestions?

     

     

    Friday, August 27, 2010 4:06 PM
  • These two events say it all:

    Source = MPIO Event ID = 23  All paths have failed. \Device\MPIODisk3 will be removed.
    Source = MPIO Event ID = 16  A fail-over on \Device\MPIODisk3 occurred

    Please work with SAN vendor to fix the connectivity flakyness.


    Regards, Mike J [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    • Marked as answer by Data Hoarder Friday, August 27, 2010 7:19 PM
    • Unmarked as answer by Data Hoarder Friday, August 27, 2010 7:20 PM
    • Marked as answer by Data Hoarder Friday, August 27, 2010 7:20 PM
    • Unmarked as answer by Data Hoarder Tuesday, November 23, 2010 11:07 PM
    Friday, August 27, 2010 5:06 PM
    Moderator
  • Logged a case with HP and received the following:

    Based on the symptoms you mention (Losing access to the Vdisk, having to reboot to gain access, etc).  You are having problems with the type of disk is configure. Dynamic disk, witch a Microsoft feature is supported, however, on the MSA it could cause problems with the connection to the logical units.

     

    Unfortunately there is no documentation regarding the issues. But us as a support level at HP… We don’t recommend to have the disk as dynamic since it could cause problems as the ones you are having with at this time. We always recommend to have it as basic disk.

     

    This is bad news for me.  Converting the disk to basic would cause me to loose the auto-grow feature and I would have to manually manage all of the different volumes and disks.  Loosing the connection throughout the day is not an option.  Hmmmmm........what to do??

    Friday, August 27, 2010 7:22 PM
  • This is bunk - the MSA knows nothing about how that disk is used or what is stored on that disk, and dynamic disks have been a feature since Windows 2000.  Find your self a different (seasoned) HP engineer who actually knows something about physical disk connectivity. 
    Regards, Mike J [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Friday, August 27, 2010 7:57 PM
    Moderator
  • Ended up being a problem with the SAN zoning configuration.  Typically we use the World Wide Port Name to configure storage devices on the SAN.  For this MSA setup we needed to configure using the World Wide Node Name. We also changed the topology of the MSA from Loop to point-to-point.

     

    Below is from the HP documentation:

     

    The floating WWNN is part of the failover design on the HP StorageWorks MSA2012fc and MSA2212fc fibre channel arrays, therefore, the correct use of the WWNN is recommended, however, it is not strictly required in loop configuration.

     

    Use the WWNN in all Soft Zones, because the physical location of the WWNN can change according to the state of the controller failover, however, the WWNN and WWPN remain available during failover events.

    Friday, October 1, 2010 4:42 PM
  • Marked post as unanswered becuase the problem returned and we continue to have problems with the missing disk.  We currently are still working with HP to resolve this issue.  I believe it is a hardware problem.  We now have another DPM server that is using a different MSA and it never has a missing disk.  Will post outcome when issue is resolved.

    Just curious, has anyone else had any simmilar issues with missing disks?

    Thursday, December 2, 2010 7:30 PM
  •  

    We're seeing a similar issue.  One of our SAN volumes suddenly disappeared.  Another SAN volume remained connected on the same server.

    We have a new HP blade server connected via QLogic card to HP EVA SAN.  We saw event ID's 23 and 16 (same as above).  The server was rebooted and the mapping came back. 

    We are trying to perform root cause analysis and are wondering if possibly MPIO version is correct?  We are also following up with HP on the issue.

     

    Tuesday, December 14, 2010 5:32 PM
  • Are you using the MSFT native msdsm or the HP DSM for multipathing. This appears to be a dsm or mpio issue.

    Did you open a case with MSFT on this?


    ranga
    Monday, December 20, 2010 11:54 PM
  • We tried both HP and Microsoft's MPIO DSM and it did not make a difference.  HP has actually recommended the Microsoft MPIO for our configuration.  We are on a HP BL460 G6 with Server 2008 R2SAN connected via QLogic card to a HP MSA 2012 fc.  Hp has recently replaced the controllers, enclosures and multiple power supplies and we have not experienced the problem since.
    Wednesday, December 22, 2010 10:34 PM