locked
DPM agent communication RRS feed

  • Question

  • Hi

    I was trying to figure out why there is a difference between data transfer rate during consistency checks and synchronization. Sometimes the data transfer rate of one will be higher than the other and vice versa. I just wanted to know out of the two processes (Consistency check and Synchronization) which one actually communicates with the agent? Does 'consistency check' check the consistency of the replica by comparing it with the agent? Does 'synchronization' copy all the changes from last recovery point to the Replica Volume? And at the time of tape backups, I believe there is no agent communication required as it just backs up data from tapes to disks? Or DPM communicates with the agent during tape backup also?

    Regards

    Pavit

    Tuesday, July 31, 2012 12:50 PM

Answers

  • Hi,

    Below is an overview of how DPM protection works.

    The below pertains to volume and file share protection, application (Exchange / Sql / Sharepoint) data protection works lightly different)

    NTFS USN journal
    =================
    To replicate changes from a file system volume, DPM uses the USN journal for tracking changed files and a volume filter driver for tracking changed blocks within changed files on the protected volume. Typically, the file system tracker is interested in any operations which modify data and metadata.  This includes the following:
    • All metadata operations: create, delete, rename, property changes, security changes, etc.
    • All non-cached writes to disk.  Cached writes to disk are ignored since they will eventually be followed by a non-cached write and we care about what is actually on the disk. 
    Turning on USN journaling on a file system volume gives DPM visibility to all interesting changes and allows the incremental replication of these changes in an efficient manner. By default, we reserve 300MB for the change journal on each volume. 


    Volume Filter
    ==============
    DPM deploys a component called the volume filter on the production server.  The volume filter (Dpmfltr.sys for DPM 2007 or DPMFilter.sys for DPM 2010 and later) will be installed by the replication agent setup as upper volume filter driver to Volsnap. The DPM volume filter tracks changes to a volume from one replication snapshot to another replication snapshot. The DPM volume filter is responsible for tracking modified blocks of a monitored volume in a change bitmap. The replication agent on the Production server uses the bitmap to optimize delta replication of non-resident data streams in a file or directory.

     

    Tracking Changes
    ================
    After the replica is created, the protection agent on the server begins tracking all changes to protected data on that server. Changes to files are passed through a filter before being written to the volume. This process is similar to the filtering of files through antivirus software, but the performance load of DPM tracking changes is less than the performance load of antivirus software.  A bitmap is created for the data source being protected (volume, File share) and tracks changes on the volume 1 bit per 16K (1MB for 128GB) and replicates the whole 16K block if any change has been made. It’s possible more than one file can be utilizing the same 16K block and may not be part of a protection group. If this occurs DPM will still replicate the whole 16K block and only apply the protected object changes to the replica then discard the remaining data.  The application of changes is done at the file level.

     
    Synchronization
    ===============
    Synchronization is the process by which DPM transfers data changes from the protected server to the DPM server, and then applies the changes to the replica of the protected data source.
    For a file volume or share, the protection agent on the protected server tracks changes to blocks, using the volume filter and the change journal that is part of the operating system to determine whether any protected files were modified. DPM also uses the volume filter and change journal to track the creation of new files, deletion or renaming of protected files.

    Consistency Check
    ===============
    Consistency Checks are used to compare protected data stored on the DPM Replica volume with the data on the protected server. The data comparison is done in two stages, a light weight fix up comparison of NTFS metadata and a heavy weight comparison of file data for files that are detected as being changed by the light weight fix up. For possible long periods of time that no file differences are found, the amount transferred will be only the CRC blocks send between DPMRA agents.  This means you cannot use the data transferred as a gauge of throughput since most of the time we're just sending small packets and no real changed data.

    Recovery Point Creation
    ========================
    Recovery points for volumes and shares are maintained in the replica volume. Windows built-in feature called Volume Shadow Copy (VSS) is utilized to track changes to the replica volume and move changed blocks to the recovery point volume.  This is done by utilizing a copy on write algorithm - whenever a block of data is changed on the replica during a synchronization, VSS first copies the original block to the recovery volume before overwriting the block on the replica volume. This occurs only once for each changed block between shadow copies (recovery points) - meaning a block may be changed multiple times during multiple synchronization, but only the first write to any given block will be copied to the recovery point volume until after a new recovery point is created. At that time, the whole process starts over for the current snapshot.

    In summary - DPM is extremely efficient at tracking changes to file and folders, and will only replicate changed file data blocks between the production server and DPM server during each synchronization job.  Between recovery point jobs, VSS keeps track of changes on the replica volume and moves changed blocks that get applied by synchronization jobs to the recovery point volume.

    For Long Term Disk to Disk to Tape (D2D2T), the DPMRA agent on the DPM server mounts the last disk based snapshot (recovery point) and writes it to tape.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, July 31, 2012 4:39 PM

All replies

  • Hi,

    Below is an overview of how DPM protection works.

    The below pertains to volume and file share protection, application (Exchange / Sql / Sharepoint) data protection works lightly different)

    NTFS USN journal
    =================
    To replicate changes from a file system volume, DPM uses the USN journal for tracking changed files and a volume filter driver for tracking changed blocks within changed files on the protected volume. Typically, the file system tracker is interested in any operations which modify data and metadata.  This includes the following:
    • All metadata operations: create, delete, rename, property changes, security changes, etc.
    • All non-cached writes to disk.  Cached writes to disk are ignored since they will eventually be followed by a non-cached write and we care about what is actually on the disk. 
    Turning on USN journaling on a file system volume gives DPM visibility to all interesting changes and allows the incremental replication of these changes in an efficient manner. By default, we reserve 300MB for the change journal on each volume. 


    Volume Filter
    ==============
    DPM deploys a component called the volume filter on the production server.  The volume filter (Dpmfltr.sys for DPM 2007 or DPMFilter.sys for DPM 2010 and later) will be installed by the replication agent setup as upper volume filter driver to Volsnap. The DPM volume filter tracks changes to a volume from one replication snapshot to another replication snapshot. The DPM volume filter is responsible for tracking modified blocks of a monitored volume in a change bitmap. The replication agent on the Production server uses the bitmap to optimize delta replication of non-resident data streams in a file or directory.

     

    Tracking Changes
    ================
    After the replica is created, the protection agent on the server begins tracking all changes to protected data on that server. Changes to files are passed through a filter before being written to the volume. This process is similar to the filtering of files through antivirus software, but the performance load of DPM tracking changes is less than the performance load of antivirus software.  A bitmap is created for the data source being protected (volume, File share) and tracks changes on the volume 1 bit per 16K (1MB for 128GB) and replicates the whole 16K block if any change has been made. It’s possible more than one file can be utilizing the same 16K block and may not be part of a protection group. If this occurs DPM will still replicate the whole 16K block and only apply the protected object changes to the replica then discard the remaining data.  The application of changes is done at the file level.

     
    Synchronization
    ===============
    Synchronization is the process by which DPM transfers data changes from the protected server to the DPM server, and then applies the changes to the replica of the protected data source.
    For a file volume or share, the protection agent on the protected server tracks changes to blocks, using the volume filter and the change journal that is part of the operating system to determine whether any protected files were modified. DPM also uses the volume filter and change journal to track the creation of new files, deletion or renaming of protected files.

    Consistency Check
    ===============
    Consistency Checks are used to compare protected data stored on the DPM Replica volume with the data on the protected server. The data comparison is done in two stages, a light weight fix up comparison of NTFS metadata and a heavy weight comparison of file data for files that are detected as being changed by the light weight fix up. For possible long periods of time that no file differences are found, the amount transferred will be only the CRC blocks send between DPMRA agents.  This means you cannot use the data transferred as a gauge of throughput since most of the time we're just sending small packets and no real changed data.

    Recovery Point Creation
    ========================
    Recovery points for volumes and shares are maintained in the replica volume. Windows built-in feature called Volume Shadow Copy (VSS) is utilized to track changes to the replica volume and move changed blocks to the recovery point volume.  This is done by utilizing a copy on write algorithm - whenever a block of data is changed on the replica during a synchronization, VSS first copies the original block to the recovery volume before overwriting the block on the replica volume. This occurs only once for each changed block between shadow copies (recovery points) - meaning a block may be changed multiple times during multiple synchronization, but only the first write to any given block will be copied to the recovery point volume until after a new recovery point is created. At that time, the whole process starts over for the current snapshot.

    In summary - DPM is extremely efficient at tracking changes to file and folders, and will only replicate changed file data blocks between the production server and DPM server during each synchronization job.  Between recovery point jobs, VSS keeps track of changes on the replica volume and moves changed blocks that get applied by synchronization jobs to the recovery point volume.

    For Long Term Disk to Disk to Tape (D2D2T), the DPMRA agent on the DPM server mounts the last disk based snapshot (recovery point) and writes it to tape.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, July 31, 2012 4:39 PM
  • Thanks Mike

    This really throws some light on the process. Although Im still a bit confused on Consistency Checks. This is used to compare the replica volume with the protected server, but does that mean there will be a communication only from DPM server towards the agent? Or will the metadata be transferred from the agent to the DPM server as well?

    Regards

    Pavit

    Friday, August 3, 2012 1:27 PM
  • Hi,

    If you open process monitor from task manager, you can monitor disk read/write and network activity for the DPMRA services.  You will mostly reads being done on both servers.  If no changes occur on the PS between CC jobs, you wil notice the amount xfered will be the same for each job, those are CRC blocks being compared.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Friday, August 3, 2012 5:18 PM
  • Thanks Mike, that really helps.

    Regards

    Pavit

    Wednesday, August 22, 2012 7:05 AM