DFS Replication has experienced an update sequence number (USN) journal loss on D:

Thu, 25 Jun 2009 12:21:19 Z

DFS Replication has experienced an update sequence number (USN) journal loss on D:

What can be the cause? I know there was a fibre break between two of our offices that day. can this implement an error such as suddenly files disappearing from the DFS?

DFS Replication has experienced an update sequence number (USN) journal loss on D:

Fri, 26 Jun 2009 04:07:38 Z

Hello HAL07,

Thank you for posting here.

Generally speaking, a USN journal *loss* means that the USN journal was deleted. A USN journal *wrap* indicates that files are being changed faster than DFSR can manage.

On Windows Server 2008, there are two separate events that will be logged in DFS Replication event log if the above error could occur.

Event ID 2202: indicates a journal wrap

Event ID 2204: indicates a journal loss

To identify which process deleted the NTFS change journal on the D: volume and causes the error, you can open Event Viewer and try to find the above 2 events, if you can see the journal loss event 2204, and then you will know that someone or some process deleted the NTFS change journal on the D: volume.

According to your description, it seems that the fibre broke at that time. It is possible that the USN journal was corrupt or deleted due to the hardware issue.

Solution:

DFSR should recover from this error by re-scanning all of the volume's replicated files, and then resuming normal replication activity. However, if there are lots of files on that volume, this recovery process can take a while.

Hope the information can helpful.

This posting is provided "AS IS" with no warranties, and confers no rights.

DFS Replication has experienced an update sequence number (USN) journal loss on D:

Fri, 26 Jun 2009 10:48:10 Z

yes I have a 2204. and it was 22.06.09 at 18:44
Now it's still not fixed. I wonder how long it takes before it's synced?

it's about 1,6tb of data

Update 28.june: The DFS finally started working after I scheduled a boot-time disk repair during boot-up of the volume D: on the central file server.

DFS Replication has experienced an update sequence number (USN) journal loss on D:

Mon, 29 Jun 2009 05:44:32 Z

I ran a chkdsk/F (had to schedule it at boot time) and the problem finally went away.

The real cause was some kind of bug in volsnap. I got the following event in System log

Log Name:      System
Source:        volsnap
Date:          19.06.2009 12:45:13
Event ID:      8
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      servername.domain.com
Description:
The flush and hold writes operation on volume D: timed out while waiting for a release writes command.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
    <Provider Name="volsnap" />
    <EventID Qualifiers="49158">8</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2009-06-19T10:45:13.555Z" />
    <EventRecordID>49824</EventRecordID>
    <Channel>System</Channel>
    <Computer>servername.domain.com</Computer>
    <Security />
</System>
<EventData>
    <Data>
    </Data>
    <Data>D:</Data>
    <Binary>000000000200300000000000080006C000000000000000001B000000000000000000000000000000</Binary>
</EventData>
</Event>

Then later:
Log Name:      System
Source:        Service Control Manager
Date:          19.06.2009 18:10:22
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      servername.domain.com
Description:
The DFS Replication service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 1800000 milliseconds: Restart the service.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
    <Provider Name="Service Control Manager" Guid="{555908D1-A6D7-4695-8E1E-26931D2012F4}" EventSourceName="Service Control Manager" />
    <EventID Qualifiers="49152">7031</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2009-06-19T16:10:22.000Z" />
    <EventRecordID>49852</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>System</Channel>
    <Computer>servername.domain.com</Computer>
    <Security />
</System>
<EventData>
    <Data Name="param1">DFS Replication</Data>
    <Data Name="param2">1</Data>
    <Data Name="param3">1800000</Data>
    <Data Name="param4">1</Data>
    <Data Name="param5">Restart the service</Data>
</EventData>
</Event>

Then various volsnap errors the days after. Not sure how this really affected the journal, but there was some kind of inconsistency on the drive.

1. I wonder where the journal is saved
2. where is the log of what got fixed on chkdsk/f at boot time. The server is Windows 2008.

DFS Replication has experienced an update sequence number (USN) journal loss on D:

Mon, 29 Jun 2009 07:15:49 Z

Hello HAL07,

Sorry for the delay. Glad to hear that the issue has went away.

Based on my research, here is some detailed information which might be helpful for you.

Q1. I wonder where the journal is saved?

A1: Generally speaking, USN change journal is saved as the metadata of NTFS on the file system.

In Windows Server 2008, an NTFS volume can use an update sequence number (USN) change journal. A change journal provides a complete log of all changes made to the volume. It records additions, deletions, and modifications regardless of who made them or how the additions, deletions, and modifications occurred. As with system logs, the change log is persistent, so it isn’t reset if you shut down and restart the operating system.

The operating system writes records to the NTFS change log when an NTFS checkpoint occurs. The checkpoint tells the operating system to write changes that would allow NTFS to recover from failure to a particular point in time.

The change journal is enabled when you install certain services, including File Replication Service (FRS), Indexing Service, and Distributed File System Service. Domain controllers and any other computer in the domain that uses these services rely heavily on the change journal. The change journal allows these services to be very efficient at determining when files, folders, and other NTFS objects have been modified. Rather than checking time stamps and registering for file notifications, these services perform direct lookups in the change journal to determine all the modifications made to a set of files. Not only is this faster, it uses system resources more efficiently as well.

We can query the USN data for a volume by running the following command

fsutil usn queryjournal <Drive letter>

We can also get the most recent change journal entry for a file by typing following command

fsutil usn readdata <FilePath>

where FilePath is the name of the file for which you want to retrieve change information.

For example, if you want to obtain the most recent change journal information on a file with the path C:\Domain-Computers.txt, you’d type

fsutil usn readdata c:\domaincomputers.txt

If you want to enumerate all the USN journal data for all the files on a volume, you can use the following command

fsutil usn enumdata <Drive letter>

Q2. Where is the log of what got fixed on chkdsk/f at boot time. The server is Windows 2008.

A2: Generally, the log information about Chkdsk is saved in the Event logging system, you can double-click on "Adminstrative Tools", double-click on the "Event Viewer", then expand "Windows Logs", and then click on "Application". In the "Source" column, look for the "Winlogon" item. Double-click it and you will see the results of the Chkdsk during startup progress.

We can also run the command to output the log to other location,

Chkdsk /f > C:\output.txt

Hope the information can be helpful.

This posting is provided "AS IS" with no warranties, and confers no rights.