none
VMGS files missing in backup - unable to recover VMs RRS feed

  • Question

  • Hello,

    I have a trouble with VM backup in our DPM 2016: it looks like the DPM is not backing VMGS files for VMs.
    I am using Hyper-V backup feature of DPM, DPMs are hosted on Windows Server 2019, with resilient change tracking 
    (RCT), all VMs are stored on ReFS volumes.

    The backup contains vhdx, VMCX and VMRS file, but ***VMGS*** file is missing, this behavior is identical for all backed up VMs.

    The restore or VM import fails with following Event Error in Hyper-V-VMMS, ID 18440

    Import failed. Import task failed to copy file from 'F:\SomeVM-2_8-1-2019_18.23.2\d-Vol\vm\SomeVM\Virtual Machines\E51373A3-3030-452F-996E-0B09E8ECCF1C.vmgs' to 'C:\ProgramData\Microsoft\Windows\Hyper-V\Planned Virtual Machines\E51373A3-3030-452F-996E-0B09E8ECCF1C.vmgs': The system cannot find the file specified. (0x80070002).

    (because VMGS file is really not present in the backup set restored on disk).

    DPM Version 5.0.409.0, Update Rollup 7 for SCDPM 2016
    DPM OS Version :  Microsoft Windows Server 2019 Standard  10.0.17763.557

    Hypervisor OS Version : Microsoft Windows Server 2019 Standard 10.0.17763

    Any ideas?
    Thank you,
    Jan


    Friday, August 2, 2019 5:03 PM

Answers

  • Hey Jan,

    Unfortunately SCDPM 2016 doesn't support backing up Hyper-V 2019 as you've found out, and you can see this here on the support matrix: https://docs.microsoft.com/en-us/system-center/dpm/dpm-protection-matrix?view=sc-dpm-2016

    Going forward, you should upgrade SCDPM to 2019 to get support

    If you need to restore a VM for the time being, you'll need to restore the files, and then re-create the VM rather than import it. What that means is create a new VM, and re-attach the VHD(x)s files to the new VM in the correct order, set the vNIC MAC address to match and then you'll be able to boot the restore fine.


    Cheers, Ben Thomas

    Microsoft Certified Professional
    Microsoft MVP Cloud and Datacenter Management

    Disclaimer: This posting is provided "AS IS" with no warranties or guarantees, and confers no rights.

    • Marked as answer by janrovner Saturday, August 3, 2019 9:47 PM
    Saturday, August 3, 2019 12:45 AM

All replies

  • Hello Jan,

    Are your Hyper-V host(s) standalone or clustered?

    If clustered, are you storing the Hyper-V virtual machines on Scale-Out File Server (SoFS) share(s) formatted on ReFS disks?

    If you're not using SoFS share(s), or if you are using a standalone Hyper-V with simple ReFS formatted disks, then I'm afraid it is not a supported scenario, and problems may occur because of this.

    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:

    Friday, August 2, 2019 10:34 PM
  • Hey Jan,

    Unfortunately SCDPM 2016 doesn't support backing up Hyper-V 2019 as you've found out, and you can see this here on the support matrix: https://docs.microsoft.com/en-us/system-center/dpm/dpm-protection-matrix?view=sc-dpm-2016

    Going forward, you should upgrade SCDPM to 2019 to get support

    If you need to restore a VM for the time being, you'll need to restore the files, and then re-create the VM rather than import it. What that means is create a new VM, and re-attach the VHD(x)s files to the new VM in the correct order, set the vNIC MAC address to match and then you'll be able to boot the restore fine.


    Cheers, Ben Thomas

    Microsoft Certified Professional
    Microsoft MVP Cloud and Datacenter Management

    Disclaimer: This posting is provided "AS IS" with no warranties or guarantees, and confers no rights.

    • Marked as answer by janrovner Saturday, August 3, 2019 9:47 PM
    Saturday, August 3, 2019 12:45 AM
  • Thank you very much for an explanation. As a temporary workaround, I think about backing up vmgs file separately (daily, manual copy), however I don't know what kind of - and how important - information it contains and whether the file changes during time.

    All my vmgs files are fixed size 4,194,816 bytes and contain some JSON information like

    "Version": {
    "Major": 0,
    "Minor": 0
    },
    "Devices": {
    "ac6b8dc1-3257-4a70-b1b2-a9c9215659ad": {
    "Version": {
    "Major": 6,
    "Minor": 0
    },
    "Type": "ac6b8dc1-3257-4a70-b1b2-a9c9215659ad",
    "States": {
    "Nvram": {
    "Vendors": {
    "4c19049f-4137-4dd3-9c10-8b97a83ffdfa": {
    "Variables": {
    "MemoryTypeInformation": {
    "Attributes": 3,
    "Data": [10, 0, 0, 0, 4, 0, 0, 0, 9, 0, 0, 0, 21, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 6, 0, 0, 0, 110, 0, 0, 0, 5, 0, 0, 0, 48, 0, 0, 0, 3, 0, 0, 0, 251, 2, 0, 0, 4, 0, 0, 0, 0, 15, 0, 0, 15, 0, 0, 0, 0, 0, 0, 0]
    }
    }
    },
    "5b446ed1-e30b-4faa-871a-3654eca36080": {
    "Variables": {
    "00155DBFFD0A": {
    "Attributes": 3,
    "Data": [232, 255, 1, 0, 16, 0, 1, 0, 4, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]
    },
    "00155DBFFD0B": {
    "Attributes": 3,
    "Data": [232, 255, 1, 0, 16, 0, 1, 0, 4, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]
    }
    }
    },
    "77fa9abd-0359-4d32-bd60-28f4e78f784b": {
    "Variables": {
    "BootDebugPolicyApplied": {
    "Attributes": 3,
    "Data": [42]
    },
    "CurrentActivePolicy": {
    "Attributes": 3,
    "Data": [0]
    },
    "CurrentPolicy": {
    "Attributes": 3,
    "Data": [4]
    },
    "Kernel_CopyOfUSN": {
    "Attributes": 3,
    "Data": [255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 127]
    },
    "RevocationList": {
    "Attributes": 3,
    "Data": ....
    },
    "WindowsBootChainSvn": {
    "Attributes": 3,
    "Data": [1, 0, 0, 0]
    }
    }
    },
    "8be4df61-93ca-11d2-aa0d-00e098032b8c": {
    "Variables": {
    "Boot0000": {
    "Attributes": 7,
    "Data": [1, 0, 0, 0, 117, 0, 69, 0, 70, 0, 73, 0, 32, 0, 78, 0, 101, 0, 116, 0, 119, 0, 111, 0, 114, 0, 107, 0, 0, 0, 2, 2, 24, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 86, 77, 66, 117, 115, 0, 0, 0, 1, 4, 52, 0, 162, 229, 23, 155, 145, 8, 221, 66, 182, 83, 128, 181, 194, 40, 9, 186, 99, 81, 97, 248, 62, 223, 197, 70, 145, 63, 242, 210, 249, 101, 237, 14, 160, 221, 15, 227, 169, 146, 230, 65, 161, 101, 65, 254, 162, 53, 36, 233, 3, 11, 37, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, 255, 4, 0]
    },
    "Boot0001": {
    "Attributes": 7,
    "Data": [1, 0, 0, 0, 88, 0, 69, 0, 70, 0, 73, 0, 32, 0, 83, 0, 67, 0, 83, 0, 73, 0, 32, 0, 68, 0, 101, 0, 118, 0, 105, 0, 99, 0, 101, 0, 0, 0, 2, 2, 24, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 86, 77, 66, 117, 115, 0, 0, 0, 1, 4, 52, 0, 162, 229, 23, 155, 145, 8, 221, 66, 182, 83, 128, 181, 194, 40, 9, 186, 217, 99, 97, 186, 161, 4, 41, 77, 182, 5, 114, 226, 255, 177, 220, 127, 121, 175, 187, 202, 19, 179, 49, 73, 171, 1, 72, 216, 242, 240, 117, 175, 3, 2, 8, 0, 0, 0, 0, 0, 127, 255, 4, 0]
    },
    "Boot0002": {
    "Attributes": 7,
    "Data": [1, 0, 0, 0, 88, 0, 69, 0, 70, 0, 73, 0, 32, 0, 83, 0, 67, 0, 83, 0, 73, 0, 32, 0, 68, 0, 101, 0, 118, 0, 105, 0, 99, 0, 101, 0, 0, 0, 2, 2, 24, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 86, 77, 66, 117, 115, 0, 0, 0, 1, 4, 52, 0, 162, 229, 23, 155, 145, 8, 221, 66, 182, 83, 128, 181, 194, 40, 9, 186, 217, 99, 97, 186, 161, 4, 41, 77, 182, 5, 114, 226, 255, 177, 220, 127, 121, 175, 187, 202, 19, 179, 49, 73, 171, 1, 72, 216, 242, 240, 117, 175, 3, 2, 8, 0, 0, 0, 1, 0, 127, 255, 4, 0]
    },
    "Boot0003": {
    "Attributes": 7,
    "Data": ....
    },
    "Boot0004": {
    "Attributes": 7,
    "Data": ...
    },
    "BootOrder": {
    "Attributes": 7,
    "Data": [3, 0, 2, 0, 0, 0, 4, 0, 1, 0]
    },
    "ConIn": {
    "Attributes": 7,
    "Data":...
    },
    "ConOut": {
    "Attributes": 7,
    "Data": ...
    },
    "KEK": {
    "Attributes": 39,
    "Data": ....
    },
    "Lang": {
    "Attributes": 7,
    "Data": [101, 110, 103, 0]
    },
    "PK": {
    "Attributes": 39,
    "Data": ....
    },
    "PlatformLang": {
    "Attributes": 7,
    "Data": [101, 110, 45, 85, 83, 0]
    },
    "Timeout": {
    "Attributes": 7,
    "Data": [0, 0]
    }
    }
    },
    "937fe521-95ae-4d1a-8929-48bcd90ad31a": {
    "Variables": {
    "00155D6F3418": {
    "Attributes": 3,
    "Data": [116, 223, 93, 111, 132, 19, 3, 0, 88, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 80, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 2, 21, 93, 255, 254, 111, 52, 24]
    },
    "00155D6F3419": {
    "Attributes": 3,
    "Data": [191, 29, 125, 255, 25, 68, 3, 0, 88, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 80, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 2, 21, 93, 255, 254, 111, 52, 25]
    },
    "00155DBFFD0A": {
    "Attributes": 3,
    "Data": [253, 191, 221, 255, 153, 95, 3, 0, 52, 0, 68, 0, 8, 0, 0, 0, 1, 0, 0, 0, 48, 0, 65, 0, 4, 0, 0, 0, 2, 0, 0, 0, 44, 0, 0, 0, 4, 0, 0, 0, 3, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 21, 93, 255, 254, 191, 253, 10]
    },
    "00155DBFFD0B": {
    "Attributes": 3,
    "Data": [255, 127, 221, 191, 150, 222, 3, 0, 52, 0, 68, 0, 8, 0, 0, 0, 1, 0, 0, 0, 48, 0, 66, 0, 4, 0, 0, 0, 2, 0, 0, 0, 44, 0, 0, 0, 4, 0, 0, 0, 3, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 21, 93, 255, 254, 191, 253, 11]
    }
    }
    },
    "9fb9a8a1-2f4a-43a6-889c-d0f7b6c47ad5": {
    "Variables": {
    "ClientId": {
    "Attributes": 3,
    "Data": [18, 0, 0, 4, 216, 212, 154, 156, 0, 224, 193, 66, 180, 24, 68, 131, 190, 83, 27, 37]
    }
    }
    },
    "c095791a-3001-47b2-80c9-eac7319f2fa4": {
    "Variables": {
    "FirmwarePerformance": {
    "Attributes": 3,
    "Data": [0, 32, 255, 127, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    }
    }
    },
    "d719b2cb-3d3a-4596-a3bc-dad00e67656f": {
    "Variables": {
    "db": {
    "Attributes": 39,
    "Data": ....
    },
    "dbx": {
    "Attributes": 39,
    "Timestamp": "2010-03-06T19:17:21Z",
    "Data": ....
    }
    }
    },
    "eaec226f-c9a3-477a-a826-ddc716cdc0e3": {
    "Variables": {
    "OfflineUniqueIDRandomSeed": {
    "Attributes": 7,
    "Data": ...
    },
    "OfflineUniqueIDRandomSeedCRC": {
    "Attributes": 7,
    "Data": [8, 51, 242, 87]
    },
    "UnlockID": {
    "Attributes": 3,
    "Data": ....
    }
    }
    },
    "eb704011-1402-11d3-8e77-00a0c969723b": {
    "Variables": {
    "MTC": {
    "Attributes": 7,
    "Data": [45, 0, 0, 0]
    }
    }
    },
    "fab7e9e1-39dd-4f2b-8408-e20e906cb6de": {
    "Variables": {
    "HDDP": {
    "Attributes": 7,
    "Data": ...
    }
    }
    }
    },
    "LastUpdateTime": "2019-06-18T15:10:28.4192693Z"
    }
    }
    },
    "db8b9818-b4bb-4725-b99d-b4612716b6b4": {
    "Version": {
    "Major": 1,
    "Minor": 0
    },
    "Type": "db8b9818-b4bb-4725-b99d-b4612716b6b4",
    "States": {
    "Power": {
    "ShutdownType": "Unknown"
    }
    }
    }
    }
    }

    Saturday, August 3, 2019 7:12 AM
  • The .vmgs file was introduced for virtual machines with a configuration version of 8.2 and higher, the "VMGS" stands for VM guest state and is a new internal file which includes device state that was previously part of the VM runtime state file.

    You should be able to restore by creating a new VM and simply attach the VHDX files to it, did you try that?

    As for the support, I do believe that there's no rush in updating to DPM 2019 yet (but it's a good thing to start planning), I'm pretty confident that DPM 2016 will get support for Windows Server 2019/Hyper-V 2019 in an upcoming Update Rollup.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Saturday, August 3, 2019 8:55 AM
  • Hello, thank you, I know about this approach, however I faced serious problems with that:

    a) on Linux VMs - Ubuntu 16 Gen2 VM did not boot because of missing some information/boot file stored in UEFI firmware (GRUB had to be fixed)

    b) on Windows VMs - after the boot, a network has missing NICs and related config (after entering the IP address, the user sees "duplicate default gateway" error etc).

    Those are ones I can remember... I need a simple and bullet-proof way to recover a VM for non-experts :-)


    Saturday, August 3, 2019 9:11 AM
  • Windows will see and complain about a duplicate IP address because if you forcefully remove a Hyper-V network adapter it will still have remains in the operating system.

    Go to the Device Manager and uninstall the old network adapter from there, then try to apply the same network configuration on the new network adapter.

    If this doesn't help, you will need to locate the network card from the Windows registry and delete the entries there.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Saturday, August 3, 2019 9:48 AM