none
Errors 9840, 9814, 2007 9006 and 8229 daily on Exchange 2010 SP1 Server RRS feed

  • Question

  • Every evening between 9:00 and 9:30 PM there is chain of these 5 errors repeated over and over again on my Exchange server. i'm using DPM 2010. I'm not sure what is going on with the exchange database between these times that is preventing it from being backed up. any idea how to resolve this?

    Saturday, October 23, 2010 3:29 PM

Answers

  • Hello,

    I'm glad that you found a work around that helps you accomplish your goals.

    DPM triggers the exchange VSS writer to both take a backup and truncate the logs.  Once the backup is taken the blocks then are given to the DPMRA to move over to the DPM server.  If this piece is broken then DPM never comes into play.

    If you had to stop the information store service, I'd engage the Exchange forum as to how best figure out what would cause you to have to do this.

     

    Thanks
    Shane

    Saturday, December 3, 2011 5:28 PM
    Moderator

All replies

  • Hello,

    Can you please paste the exact error details in this forum? Please also be mindful that DPM uses the Exchange VSS writer to take the storage group snapshot.  If that fails then so will DPM on the backup of Exchange storage groups.  If this is what is happening on your backups then DPM is just a victim of the Exchange VSS writer failing and that is where the troubleshooting needs to to be at.

    When time  permits, please paste the details.

    Thanks,
    Shane

    Monday, October 25, 2010 2:06 PM
    Moderator
  • Log Name:      Application
    Source:        MSExchangeIS
    Date:          10/24/2010 9:11:13 PM
    Event ID:      9840
    Task Category: General
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      SERVERNAME.domain.local
    Description:
    An attempt to prepare the database 'Mailbox Database'for backup failed because the database is already in the process of being backed up. The error code is -2403. (Note that if a backup was recently aborted, then it may take several minutes for the system to detect the aborted backup and initiate backup cleanup procedures, so this message may be generated if an attempt was made to backup a database before a previous backup attempt had fully terminated.)
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="MSExchangeIS" />
        <EventID Qualifiers="49158">9840</EventID>
        <Level>2</Level>
        <Task>6</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2010-10-25T01:11:13.000000000Z" />
        <EventRecordID>318245</EventRecordID>
        <Channel>Application</Channel>
        <Computer>SERVERNAME.domain.local</Computer>
        <Security />
      </System>
      <EventData>
        <Data>Mailbox Database</Data>
        <Data>-2403</Data>
      </EventData>
    </Event>

     

    Log Name:      Application
    Source:        MSExchangeIS
    Date:          10/24/2010 9:11:13 PM
    Event ID:      9814
    Task Category: Exchange VSS Writer
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      SERVERNAME.domain.local
    Description:
    Exchange VSS Writer (instance 2b02e7de-2661-4823-a1f3-a70ed12a0aba:1775) failed with error code -2403 when preparing the database engine for backup of database 'Mailbox Database'.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="MSExchangeIS" />
        <EventID Qualifiers="49193">9814</EventID>
        <Level>2</Level>
        <Task>41</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2010-10-25T01:11:13.000000000Z" />
        <EventRecordID>318246</EventRecordID>
        <Channel>Application</Channel>
        <Computer>SERVERNAME.domain.local</Computer>
        <Security />
      </System>
      <EventData>
        <Data>2b02e7de-2661-4823-a1f3-a70ed12a0aba:1775</Data>
        <Data>-2403</Data>
        <Data>Mailbox Database</Data>
        <Binary>4356737349457863685772697465723A3A4563507265706172654A696E73744261636B75703A204661696C656420707265706172696E6720736E617073686F7420696E7374616E636520626563617573652061206261636B757020697320616C726561647920696E2070726F67726573732E00</Binary>
      </EventData>
    </Event>

     

    Log Name:      Application
    Source:        ESE
    Date:          10/24/2010 9:11:13 PM
    Event ID:      2007
    Task Category: ShadowCopy
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      SERVERNAME.domain.local
    Description:
    Information Store (2456) Shadow copy instance 1775 aborted.

    For more information, click http://www.microsoft.com/contentredirect.asp.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="ESE" />
        <EventID Qualifiers="0">2007</EventID>
        <Level>2</Level>
        <Task>16</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2010-10-25T01:11:13.000000000Z" />
        <EventRecordID>318247</EventRecordID>
        <Channel>Application</Channel>
        <Computer>SERVERNAME.domain.local</Computer>
        <Security />
      </System>
      <EventData>
        <Data>Information Store</Data>
        <Data>2456</Data>
        <Data>
        </Data>
        <Data>1775</Data>
      </EventData>
    </Event>

     

    Log Name:      Application
    Source:        MSExchangeIS
    Date:          10/24/2010 9:11:13 PM
    Event ID:      9609
    Task Category: Exchange VSS Writer
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      SERVERNAME.domain.local
    Description:
    Exchange VSS Writer (instance 2b02e7de-2661-4823-a1f3-a70ed12a0aba:1775) failed with error code -2403 when preparing for Snapshot.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="MSExchangeIS" />
        <EventID Qualifiers="49193">9609</EventID>
        <Level>2</Level>
        <Task>41</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2010-10-25T01:11:13.000000000Z" />
        <EventRecordID>318248</EventRecordID>
        <Channel>Application</Channel>
        <Computer>SERVERNAME.domain.local</Computer>
        <Security />
      </System>
      <EventData>
        <Data>2b02e7de-2661-4823-a1f3-a70ed12a0aba:1775</Data>
        <Data>-2403</Data>
        <Binary>4356737349457863685772697465723A3A4F6E50726570617265536E617073686F743A204661696C696E672063616C6C696E67204A657420746F207072657061726520696E7374616E6365206261636B75702E00</Binary>
      </EventData>
    </Event>

     

    Log Name:      Application
    Source:        VSS
    Date:          10/24/2010 9:11:13 PM
    Event ID:      8229
    Task Category: None
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      SERVERNAME.domain.local
    Description:
    A VSS writer has rejected an event with error 0x800423f3, The writer experienced a transient error.  If the backup process is retried,
    the error may not reoccur.
    . Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer.

    Operation:
       PrepareForSnapshot Event

    Context:
       Execution Context: Writer
       Writer Class Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
       Writer Name: Microsoft Exchange Writer
       Writer Instance Name: Exchange Information Store
       Writer Instance ID: {8e5ea956-7280-44bd-8deb-06cda51cd361}
       Command Line: "E:\Program Files\Microsoft\Exchange Server\V14\bin\store.exe"
       Process ID: 2456
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="VSS" />
        <EventID Qualifiers="0">8229</EventID>
        <Level>3</Level>
        <Task>0</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2010-10-25T01:11:13.000000000Z" />
        <EventRecordID>318249</EventRecordID>
        <Channel>Application</Channel>
        <Computer>SERVERNAME.domain.local</Computer>
        <Security />
      </System>
      <EventData>
        <Data>0x800423f3, The writer experienced a transient error.  If the backup process is retried,
    the error may not reoccur.
    </Data>
        <Data>

    Operation:
       PrepareForSnapshot Event

    Context:
       Execution Context: Writer
       Writer Class Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
       Writer Name: Microsoft Exchange Writer
       Writer Instance Name: Exchange Information Store
       Writer Instance ID: {8e5ea956-7280-44bd-8deb-06cda51cd361}
       Command Line: "E:\Program Files\Microsoft\Exchange Server\V14\bin\store.exe"
       Process ID: 2456</Data>
        <Binary>2D20436F64653A20575254575254494330303030353139372D2043616C6C3A20575254575254494330303030323838302D205049443A202030303030323435362D205449443A202030303031303832342D20434D443A202022453A5C50726F6772616D2046696C65735C4D6963726F736F66745C45786368616E6765205365727665725C5631345C62696E5C73746F72652E6578652220202D20557365723A204E616D653A204E5420415554484F524954595C53595354454D2C205349443A532D312D352D313820</Binary>
      </EventData>
    </Event>

    Monday, October 25, 2010 3:35 PM
  • Hello,

    Do you have any other backup program attempting to back up the exchange database? or even a backup agent from another application not in use but that is installed on this exchange server?

    What operating system, service pack level and architecture are you running for this server?

    Do you have any scheduled jobs, DPM or otherwise, that is running during these times? Any scripts running in the afternoon?

    Even though these errors happen in the evening, afterwards if you perform a manual consistency check does that succeed or does that also fail?

    Shane

     

    Monday, October 25, 2010 7:26 PM
    Moderator
  • There are no other backup programs or backup agents installed on this server. the OS is Windows Server 2008 R2 x64. the only jobs in the DPM that are scheduled for this time are the regular 15 minute syncronization that take place all day. long The Express full backup is set to the default of 8:00pm and the protection group has a 5 minute offset.

    I do not have to run a manual consistancy check as this problem resolves itself around 9:30 pm each day.

    Does Exchange 2010 SP1 have any internal built-in backup that could be taking place during this time frame?

    Tuesday, October 26, 2010 8:49 PM
  • Hello,
    Another question if I may, are there any other currently running DPM jobs for this same SG in question?
    Exchange is not my particular area of concentration so I can not speak to the depth of its native backup capabilities and all the caveats therein, however : http://technet.microsoft.com/en-us/library/dd876874.aspx 

    I personally have not seen any issue like this with Exchange 2010 that occurs everyday at a specific time unless some other job was interferring. I am partial to using process monitor and process explorer to see what is running in the background on a server.  Many times you can find things that most people did not know was running.

    Process monitor: http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
    Process explorer: http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

    If you are sure however that there is no othe conflicting process, you may want to pose your question "Does Exchange 2010 SP1 have any internal built-in backup that could be taking place during this time frame?" to the Exchange 2010 forum http://social.technet.microsoft.com/Forums/en-US/category/exchange2010 as to the details and\or settings within Exchange 2010 itself.

    There is an artilce that applies to Exchange 2007 and 2003 that has the same error as 9840 with error code 2403: http://support.microsoft.com/default.aspx?scid=kb;EN-US;930800 

    Shane

    Wednesday, October 27, 2010 1:36 PM
    Moderator
  • After doing some more research as to the timeline on these errors, I found that they began appearing the same evening that I upgraded exchange from 2010 to 2010 SP1. However, the errors began at 9pm that evening and the actual upgrade took place late that night. the only things that had been installed just prior to the upgrade were (4) hotfixes.

    KB977020, KB979744, KB982867 and KB983440.

    These hotfixes were required for the upgrade of Exchange to SP1.

    The other thing I had considered was that the DPM may have been trying to do a backup when the upgrade was in progress. I could understand this causing the backup to fail that night, but that does not explain the errors re-occuring at the same time each evening.

    Monday, November 1, 2010 2:06 PM
  • Hi,

                    This can happen if you are protecting multiple copies of an Exchange 2010 DAG DB in parallel from different nodes.

                    Exchange does not support parallel backups of a DB from multiple DAG nodes. DPM 2010 has inbuilt retry logic for this. DPM retries for 15mins for incremental backups and 1hr for Express Full/Copy backup before failing the job.  Looks like your backup windows for a DB are overlapping on two or more nodes, please reconfigure your backup schedules so that there are no overlaps.

     

    Thanks & Regards,

    Nagesh


    Nagesh[MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Thursday, February 10, 2011 9:09 AM
  • I do not have any Database Availability Groups in my scenario. I have (2) mailbox databases and (1) public Folder store. My Exchange configuration is (1) Server holding the Edge Transport role and (1) Server holding the Mailbox, Client Access and Hub Transport roles. I have a single Exchange Protection Group which protects all 3 of these databases. The errors appear everyday when the Express/Full backup runs. I wonder if DPM is also trying to run an incremental backup at the same time and that's what's failing.
    Tuesday, February 22, 2011 3:57 PM
  • Did you ever find a resolution to this issue?  We are running into the same problem.  So far, MS support has been unable to correct the issue.  What runs at 9PM and can this schedule be changed?
    Wednesday, March 2, 2011 2:34 AM
  • I, unfortunately, am still having this issue. I think it is evident that the express/full backup is what is running and failing. However, I have not been able to figure out what is conflicting. The incremental backups run fine all day long. It's just the full backup that fails continuously. In our scenario, DPM continues to try and eventually completes the backup. However, it shouldn't be failing everyday and continuously trying until it succeeds.
    Wednesday, March 2, 2011 5:17 AM
  • Hello,

    I think that detailed logging needs to be gathered and analyzed on both the DPM server and exchange server.  Unfortunately this is not the sort of thing that can be easily done via forum post as the logs can get quite indepth. 

    Are you able to open a support incident with Microsoft on this issue so that not only can a DPM engineer look into this but also an Exchange engineer if needed to look at the Exchange VSS writer.  

    Thanks
    Shane
    Wednesday, March 2, 2011 9:47 PM
    Moderator
  • mk473764,

    I still am having this issue. Have you been able to find a resolution?

    Monday, April 4, 2011 9:18 PM
  • Hello,

    Are you still getting errors on the exchange backup with "An attempt to prepare the database 'Mailbox Database'for backup failed because the database is already in the process of being backed up." ?


    Thanks,
    Shane
    Friday, July 29, 2011 12:27 PM
    Moderator
  • I am also getting this exact same thing. 1 database using symantec backup exec that starts at 9pm every night and my errors are all at 12:01.

    IS it a Database maintenance issue. Is there a way to see when the last time database maintenance has run.

    • Edited by Timgaik Monday, September 12, 2011 2:52 PM
    Monday, September 12, 2011 2:21 PM
  • I am still having this issue as well.
    Monday, September 12, 2011 2:34 PM
  • Hello,

    You can determine if the database have the "BackupInProgress" flag set to True by executing the command below on the Exchange DAG member via the Exchange Management Shell (PowerShell) and looking in the output for the BackupInProgress property.

    Get-MailboxDatabase 'DBName' -Status |fl

    On the Exchange server the store itself must be restarted, or the MSExchangeIS service restarted. If you do this does this work


    Thanks,
    Shane

    Monday, September 12, 2011 11:14 PM
    Moderator
  • I think I have gotten to the bottom of this issue.

    I had the Express-Fuill Backups for both my Exchange Protection Group and my BMR/System State protection groups set for 8:00 PM daily. Therefore, they were conflicting with each other. I have changed the times and moved BMR/System State to 10:30 PM daily and the problem improves somewhat. At this point I only get one notification per Exchange Database around 10:30 PM or 11:00 PM indicating that the Exchange Protection group is failing on each of my Mailbox and Public Folder Databases. Around 1:30 AM when the BMR/System State Express-Full Backup completes, the Exchange errors resolve themselves.

    Is there any way to prevent these errors from happening entirely? Obviously I'll take 1 set of errors per evening over dozens of them, but if there is a way to resolve this entirely, I'd like to do so.

     

     

    Friday, October 28, 2011 2:18 PM
  • Hello,

    If you were to give the BMR backup more of a time difference this way it does not overlap any into the Exchange database snapshot, does this make a difference. 


    Thanks
    Shane

    Friday, October 28, 2011 8:34 PM
    Moderator
  • I have tried spreading the window out which is how I came up with the 8:00 PM time for the Exchange group at the 10:30PM time for the BMR. I think I tried 8:00 and 9:00 my first attempt to resolve this. If I had to guess I suspect that it may be my short-term syncroniztions on the 15 minute interval that are interfering with the BMR. If the Exchange Express-Full backup runs at 8 and is finished before 10:30 and I'm not starting to see errors till 11, it shouldn't be the exchange express-full backup as that should be complete for the evening.

    Friday, October 28, 2011 8:41 PM
  • Hello,

    Are you still having the issue?


    Thanks,
    Shane
    Friday, November 4, 2011 11:29 PM
    Moderator
  • Yes I am unfortunately.
    Monday, November 7, 2011 2:15 PM
  • Hello,

    I suggest to open up a case with Microsoft in which there can be a collaborative effort between Exchange and DPM to figure out the cause and solution.


    Thanks
    Shane
    Tuesday, November 8, 2011 9:49 PM
    Moderator
  • We have same issue in our environment.

    Exchange 2010SP1 RU6, DPM 2010.

    One of databases have BackupInProgress=True flag.

    Thursday, November 10, 2011 7:25 AM
  • Hello,

    Then there is some other process backing up that database. 


    Thanks
    Shane
    Monday, November 14, 2011 4:08 PM
    Moderator
  • hi all,

     

    i found a solution, LOL solution.

    i put a script at 22:30PM that stop information store service and a script that start the service at 22:50 and the backup start at 23:00.

    this is work fine for me now but i cant say im happy with it:D

     

     

     


    lipaz@yazamco.co.il, נא לשנן כתובת זו למקרים קשים :)
    Friday, December 2, 2011 7:26 AM
  • Hello,

    I'm glad that you found a work around that helps you accomplish your goals.

    DPM triggers the exchange VSS writer to both take a backup and truncate the logs.  Once the backup is taken the blocks then are given to the DPMRA to move over to the DPM server.  If this piece is broken then DPM never comes into play.

    If you had to stop the information store service, I'd engage the Exchange forum as to how best figure out what would cause you to have to do this.

     

    Thanks
    Shane

    Saturday, December 3, 2011 5:28 PM
    Moderator
  • i am having the exact same scenario and couldnt find a solution.

    there are no any scripts / tasks run in background at the same interval time

    please support

    Tuesday, February 7, 2012 9:41 AM