none
Replica inconsistent - SQL database RRS feed

  • Question

  • Hi,

    I have a problem with backup SQL databases with DMP 2012. I don’t know exactly why this stopped to work, but I know, that there is responsible SQL VSS writer.
    My environment is:
    - DPM 2012
    - SQL 2008R2 Express installed on non-domain Server 2008R2, virtualized on Server 2012

    DPM agent and communication between servers is OK, because I am able to backup file shares on the same server. I can also see all SQL databases in DPM console, but when I am trying to backup them, I receive “Replica is inconsistent”. I am unable to repair this and if I manually start “Perform Consistency Check”, I receive the following error:
    An unexpected error occurred on DPM server machine during a VSS operation. (ID 207 Details: VssError:The shadow copy provider had an error. Check the System and Application event logs for more information.
     (0x80042306))

    On SQL server, I looked with vssadmin list writers and SQL writer is listed with status Stable, No Errors.

    Can you please help me to solve my issue?


    Regards,


    Elvis Gustin

    Monday, December 3, 2012 8:48 AM

Answers

  • Hi,

    Yes, I suppose that will resolve the issue, but scratching my head on what all looks OK but still getting that error.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, December 5, 2012 5:35 PM
    Moderator

All replies

  • Hi,

    The ErrorCode: 0x80042306 (-2147212538): VSS_E_PROVIDER_VETO: The shadow copy provider had an error. Please see the system and application event logs for more information.

    So this isn't a VSS writer issue, it's a VSS provider issue.  What do you show for providers when you run "vssadmin list providers" ? 

    Are there any event messages in the application or system event log inside the guest around the time of the failure ?

    If you try to protect a folder on a volume in the guest, can DPM successfully protect that ?


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Monday, December 3, 2012 6:53 PM
    Moderator
  • Hi Mike,

    You right. I make a mistake with writers.
    However in »vssadmin list providers« I see only »Microsoft Software Shadow Copy provider 1.0«.
    There are no errors in application or system event log in time when I am trying to repair inconsistent replicas. The last error is in application log three days ago:
    Source: VSS
    EventID: 1
    Volume Shadow Copy Service initialization error: the control dispatcher cannot be started [0x80070427, The service process could not connect to the service controller.].

    This is the only error in event logs.
    As I described in first post, I am able to protect shares and volumes. I have all replicas without any error.

    Regards, 


    Elvis Gustin -- (Please mark as answer if it is usefull.)

    Tuesday, December 4, 2012 6:25 AM
  • Hi,

    Try using DiskShadow.exe to make a snapshot of the volume(s) that the SQL databases reside on.  It will list the writers used to make the snapshot and may show you where we're failing.

    example:

    C:\Windows\system32>diskshadow.exe
    Microsoft DiskShadow version 1.0
    Copyright (C) 2007 Microsoft Corporation
    On computer:  Machine_name,  date / time


    DISKSHADOW> begin backup

    DISKSHADOW> add volume c:   <----- always include c:

    DISKSHADOW> add volume d:   <---- volume where SQ DB / Logs reside.

    DISKSHADOW> set verbose on

    DISKSHADOW> create

    From here, it will detail the writers used to make the snapshot, the components added and then try to make a snapshot. See what that does.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, December 4, 2012 3:10 PM
    Moderator
  • Hi Mike,

    Thank you for your suggestion. I did what you are asking for and there are details:

    Excluding writer "BITS Writer", because all of its components have been excluded
    .
    Excluding writer "Shadow Copy Optimization Writer", because all of its component
    s have been excluded.
    Component "\BCD\BCD" from writer "ASR Writer" is excluded from backup,
    because it requires volume  which is not in the shadow copy set.
    The writer "ASR Writer" is now entirely excluded from the backup because the top
    -level
    non selectable component "\BCD\BCD" is excluded.

    * Including writer "Task Scheduler Writer":
            + Adding component: \TasksStore

    * Including writer "VSS Metadata Store Writer":
            + Adding component: \WriterMetadataStore

    * Including writer "Performance Counters Writer":
            + Adding component: \PerformanceCounters

    * Including writer "SqlServerWriter":
            + Adding component: \WEBPLAT\SQLEXPRESS\master
            + Adding component: \WEBPLAT\SQLEXPRESS\model
            + Adding component: \WEBPLAT\SQLEXPRESS\msdb
            + Adding component: \WEBPLAT\SQLEXPRESS\WebsitePanel
            + Adding component: \WEBPLAT\SQLEXPRESS\1_db
            + Adding component: \WEBPLAT\SQLEXPRESS\2_Db
            + Adding component: \WEBPLAT\SQLEXPRESS\3_db
            + Adding component: \WEBPLAT\SQLEXPRESS\4_DB
            + Adding component: \WEBPLAT\SQLEXPRESS\5_db
            + Adding component: \WEBPLAT\SQLEXPRESS\6_DB
            + Adding component: \WEBPLAT\SQLEXPRESS\7_DB
            + Adding component: \WEBPLAT\SQLEXPRESS\8_DB
            + Adding component: \WEBPLAT\SQLEXPRESS\9_DB
            + Adding component: \WEBPLAT\SQLEXPRESS\10_DB

    * Including writer "System Writer":
            + Adding component: \System Files

    * Including writer "IIS Metabase Writer":
            + Adding component: \IISMETABASE

    * Including writer "IIS Config Writer":
            + Adding component: \IISCONFIG

    * Including writer "WMI Writer":
            + Adding component: \WMI
    * Including writer "COM+ REGDB Writer":
            + Adding component: \COM+ REGDB

    * Including writer "Registry Writer":
            + Adding component: \Registry

    Alias VSS_SHADOW_1 for shadow ID {f372690a-5e88-4611-a9a0-d592b0cb1eb3} set as e
    nvironment variable.
    Alias VSS_SHADOW_2 for shadow ID {9ab7a7cf-8193-4c2e-a31e-74be32761988} set as e
    nvironment variable.
    Alias VSS_SHADOW_SET for shadow set ID {8d5dd279-d80d-43be-94a7-81957b802db3} se
    t as environment variable.
    Inserted file Manifest.xml into .cab file 30-4.12.2012-17_--_WEBPLAT.cab
    Inserted file BCDocument.xml into .cab file 30-4.12.2012-17_--_WEBPLAT.cab
    Inserted file WM0.xml into .cab file 30-4.12.2012-17_--_WEBPLAT.cab
    Inserted file WM1.xml into .cab file 30-4.12.2012-17_--_WEBPLAT.cab
    Inserted file WM2.xml into .cab file 30-4.12.2012-17_--_WEBPLAT.cab
    Inserted file WM3.xml into .cab file 30-4.12.2012-17_--_WEBPLAT.cab
    Inserted file WM4.xml into .cab file 30-4.12.2012-17_--_WEBPLAT.cab
    Inserted file WM5.xml into .cab file 30-4.12.2012-17_--_WEBPLAT.cab
    Inserted file WM6.xml into .cab file 30-4.12.2012-17_--_WEBPLAT.cab
    Inserted file WM7.xml into .cab file 30-4.12.2012-17_--_WEBPLAT.cab
    Inserted file WM8.xml into .cab file 30-4.12.2012-17_--_WEBPLAT.cab
    Inserted file WM9.xml into .cab file 30-4.12.2012-17_--_WEBPLAT.cab
    Inserted file WM10.xml into .cab file 30-4.12.2012-17_--_WEBPLAT.cab
    Inserted file WM11.xml into .cab file 30-4.12.2012-17_--_WEBPLAT.cab
    Inserted file WM12.xml into .cab file 30-4.12.2012-17_--_WEBPLAT.cab
    Inserted file DisC915.tmp into .cab file 30-4.12.2012-17_--_WEBPLAT.cab

    Querying all shadow copies with the shadow copy set ID {8d5dd279-d80d-43be-94a7-
    81957b802db3}
            * Shadow copy ID = {9ab7a7cf-8193-4c2e-a31e-74be32761988}
    %VSS_SHADOW_2%
                    - Shadow copy set: {8d5dd279-d80d-43be-94a7-81957b802db3}
    %VSS_SHADOW_SET%
                    - Original count of shadow copies = 2
                    - Original volume name: \\?\Volume{d5854622-d2de-11e0-a212-00155
    d01010e}\ [E:\]
                    - Creation time: 12/4/2012 5:29:51 PM
                    - Shadow copy device name: \\?\GLOBALROOT\Device\HarddiskVolumeS
    hadowCopy10
                    - Originating machine: WebPlat
                    - Service machine: WebPlat
                    - Not exposed
                    - Provider ID: {b5946137-7b9f-4925-af80-51abd60b20d5}
                    - Attributes:  Auto_Release Differential

            * Shadow copy ID = {f372690a-5e88-4611-a9a0-d592b0cb1eb3}
    %VSS_SHADOW_1%
                    - Shadow copy set: {8d5dd279-d80d-43be-94a7-81957b802db3}
    %VSS_SHADOW_SET%
                    - Original count of shadow copies = 2
                    - Original volume name: \\?\Volume{d1d0c888-d314-11e0-a5e1-806e6
    f6e6963}\ [C:\]
                    - Creation time: 12/4/2012 5:29:51 PM
                    - Shadow copy device name: \\?\GLOBALROOT\Device\HarddiskVolumeS
    hadowCopy9
                    - Originating machine: WebPlat
                    - Service machine: WebPlat
                    - Not exposed
                    - Provider ID: {b5946137-7b9f-4925-af80-51abd60b20d5}
                    - Attributes:  Auto_Release Differential

    Number of shadow copies listed: 2

    Seems that SQL writer is working. Am I wrong?

    Regards,


    Elvis Gustin -- (Please mark as answer if it is usefull.)

    Tuesday, December 4, 2012 5:37 PM
  • Hi,

    Yes, that appears to work, so which database are you trying to protect with DPM that causes the error.  Can you copy / paste the DPM job details for the failed job.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, December 4, 2012 5:45 PM
    Moderator
  • Hi Mike,

    I am trying to backup all databases and all they have the same error. In same protection group, I am protecting two folders. This two are protected without errors.
    This is an error of one job:

    Affected area: WEBPLAT\SQLEXPRESS\1_DB
    Occurred since: 4.12.2012 18:15:56
    Description: The replica of SQL Server 2008 database WEBPLAT\SQLEXPRESS\1_DB on WebPlat is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent.

    For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106). An unexpected error occurred on DPM server machine during a VSS operation. (ID 207 Details: VssError:The shadow copy provider had an error. Check the System and Application event logs for more information. (0x80042306))
     
    More information
    Recommended action: Retry the operation.
     Synchronize with consistency check.
     Run a synchronization job with consistency check...
    Resolution: To dismiss the alert, click below
     Inactivate

    There is an error in event viewer – Application log on DMP server:

    Log Name:      Application
    Source:        VSS
    Date:          4.12.2012 18:15:55
    Event ID:      12289
    Task Category: None
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      computer
    Description:
    Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\\?\Volume{d63cb72d-e760-11e1-9327-00155d100604} - 00000000000001E8,0x0053c06c,000000659C789FF0,0,000000659C78B000,4096,[0]).  hr = 0x80070005, Access is denied.

    Operation:
       Query diff area for this volume

    Context:
       Volume Name: \\?\Volume{d63cb72d-e760-11e1-9327-00155d100604}\

    Regards,


    Elvis Gustin -- (Please mark as answer if it is usefull.)

    Tuesday, December 4, 2012 7:24 PM
  • Ah, this shed new light on the problem.  The problem is on the DPM Server itself and seems like we're getting access is denied on volume \\?\Volume{d63cb72d-e760-11e1-9327-00155d100604}

    From an administrative command prompt, Please run:

        chkdsk /f \\?\Volume{d63cb72d-e760-11e1-9327-00155d100604}

    Then run the following command and see what permissions are on the root, note the addition of the backslash \ at the end.

       Icacls \\?\Volume{895a20e5-1f7f-11e2-8f7c-0019b9f81999}\

     It should return the following but with your volume guid:

    C:\Windows\system32>icacls \\?\Volume{e07bd411-67b3-11e0-9913-0019b9f81999}\
    \\?\Volume{e07bd411-67b3-11e0-9913-0019b9f81999}\ BUILTIN\Administrators:(F)
                                                      NT AUTHORITY\SYSTEM:(F)

    Successfully processed 1 files; Failed processing 0 files


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, December 4, 2012 7:48 PM
    Moderator
  • Hi Mike,

    Also here seems to be all OK. I executed all commands and results are here.

    C:\Windows\system32>chkdsk /f \\?\Volume{d63cb72d-e760-11e1-9327-00155d100604}
    The type of the file system is NTFS.
    Volume label is DPM-vol_b2e845de-b285-4e56-.

    CHKDSK is verifying files (stage 1 of 3)...
      256 file records processed.
    File verification completed.
      0 large file records processed.
      0 bad file records processed.

    CHKDSK is verifying indexes (stage 2 of 3)...
      504 index entries processed.
    Index verification completed.
      0 unindexed files scanned.
      0 unindexed files recovered.

    CHKDSK is verifying security descriptors (stage 3 of 3)...
    Security descriptor verification completed.
      124 data files processed.

    Windows has scanned the file system and found no problems.
    No further action is required.

      10485759 KB total disk space.
        140768 KB in 91 files.
            80 KB in 126 indexes.
             0 KB in bad sectors.
         55471 KB in use by the system.
         54480 KB occupied by the log file.
      10289440 KB available on disk.

          4096 bytes in each allocation unit.
       2621439 total allocation units on disk.
       2572360 allocation units available on disk.

    C:\Windows\system32>Icacls \\?\Volume{d63cb72d-e760-11e1-9327-00155d100604}\
    \\?\Volume{d63cb72d-e760-11e1-9327-00155d100604}\ BUILTIN\Administrators:(F)
                                                      NT AUTHORITY\SYSTEM:(F)

    Successfully processed 1 files; Failed processing 0 files

    However, seems that permissions are OK and nothing is wrong with volume.

    Any other Idea? Is in this case maybe better completely remove protection group and allocated space and then later create new one with same settings?

    Regards,


    Elvis Gustin -- (Please mark as answer if it is usefull.)

    Wednesday, December 5, 2012 6:09 AM
  • Hi,

    Yes, I suppose that will resolve the issue, but scratching my head on what all looks OK but still getting that error.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, December 5, 2012 5:35 PM
    Moderator
  • Hi Mike,

    As we expect. Deleting the old protection group and creating a new one solves the problem. Now I know that the problem was really on the DPM volume. I don’t know what kind, I have no idea…

    Thank you a lot for your support

    Regards,


    Elvis Gustin -- (Please mark as answer if it is usefull.)

    • Proposed as answer by K Phung Thursday, March 28, 2013 8:48 PM
    Thursday, December 6, 2012 6:31 AM
  • Thanks for the update. Not sure what the real problem was, but hopefully it was a one time occurrence.

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Thursday, December 6, 2012 1:09 PM
    Moderator
  • Hi Mike,

    I am experiencing the same symptoms with several of my datasources (not only SQL) following a ungraceful reset of my DPM 2012 SP1 RU1 server. I would like to avoid creating new protection groups, since the datasources having the problem are from several groups. Is there any more information I can provide for troubleshooting this?  

    When running ProcessMonitor and filtering for DeviceIOControl operations when trying to run a consistency check on one of the failed datasources, I see IOCTL_VOLSNAP_QUERY_DIFF_AREA events returning successes. 

    I noticed that the dynamic disks holding my volumes only have 10MB of free space--both that I looked into.  

    Thanks!
    Matthew


    Matthew

    Wednesday, March 6, 2013 12:07 AM