none
Deleting a RAID 5 array without a backup

    Question

  • Hi,

    I'm in a bad situation as you can probably tell from the thread title.

    SBS2008 on a Dell Poweredge T710 with exchange server 2007, trend micro anti virus.

    A few weeks ago my backup started to fail. At the same time I also noticed that one of the disks in my 4-disk hardware RAID5 array had failed. Another disk had a predicted failure according to Dell OpenManage Server. I called Dell Tech support (They were very good) and they sent me 2 new disks. As the disks are hot swappable I took the failed disk out and replaced it. The replacement disk would keep going Offline before the full rebuild had finished. Eventually I got the replacement disk to stay online and all my data seemed OK.

    Next, the disk with a predicted failure appeared to fail so I put the 2nd replacement disk in. However, once the data rebuild gets to 92% if fails. OMSA reports the 2nd replacement disk as Failed. If I physically remove the disk and reinsert, the rebuild starts again but it always fails at 92%.

    To make matters worse, my windows server backup always fails at 62%. I tried downloading a 3rd party backup solution but this also fails at exactly 62%. Both backup software complete the backup of C: (RAID1) but fail at 62% on D:\ (RAID5).

    The server event log shows the following errors:

    Event ID: 2273: A block on the physical disk has been punctured by the controller:  Physical Disk 0:0:2 Controller 0, Connector 0

    Event ID: 2350: There was an unrecoverable disk media error during the rebuild or recovery operation:  Physical Disk 0:0:2 Controller 0, Connector 0

    Event ID: 51: An error was detected on device \Device\Harddisk2\DR10 during a paging operation.

    I reported this to Dell tech support and they have said that as the RAID is punctured a rebuild will never work and I need to delete the RAID and recreate it. Then, the disks will be fixed when the new RAID is initialised? As I don't have a backup of the data I'm really hesitant to do this.

    I could manually copy the data files (word, excel, images etc) to an external drive, but I have other things on this drive such as my Exchange databases (first storage group, 2nd storage group), my backup software is installed on this drive.

    I'm pretty sure I couldn't just copy these folders to an external drive, recreate the RAID then copy them back and expect everything to be the same?

    Any suggestions or help would be most appreciated. At least it's Friday and I can spend all weekend here if I need to..


    • Edited by Chris30-UK Friday, September 14, 2012 10:01 AM Spelling / grammar
    Friday, September 14, 2012 8:56 AM

Answers

  • Hello,

    Checkdisk will not fix the puncture, the part of the drive that the RAID controller will not grant access to, but it may fix up the OS enough to allow you to move your data. The puncture will still be there but the os may be able to work around it after a chkdsk. There is also a utility called spinrite by GRC.com that may help you recover. You will still need to re-initialize the RAID afterwards.

    Also, I would try this only as a last resort because after chkdsk things may get better or worse.


    Miguel Fra | Falcon IT Services, Miami, FL
    www.falconitservices.com | www.falconits.com | Blog

    • Marked as answer by Chris30-UK Thursday, September 20, 2012 7:25 AM
    Saturday, September 15, 2012 3:33 PM

All replies

  • Hi Chris,

    Sorry to hear that, you are having trouble with your Server.

    AFAIK, the only way to fix disk related errors (in this context) is to fully initialize the faulty disk and then restore the data from back up. However, as you already mentioned backups are failing, it's out out of question here.That's really unfortunate.

    Looking at the problem description, what I can suggest is, it's better to involve support personnel's from both Microsoft and Dell from creating a support case's with them as you are dealing with the multiple issues both on hardware as well as failing backup on operating system. 



    I do not represent the organisation I work for, all the opinions expressed here are my own.

    This posting is provided "AS IS" with no warranties or guarantees and confers no rights.

    - .... .- -. -.- ... --..-- ... .- -. - --- ... ....

    Friday, September 14, 2012 1:28 PM
    Moderator
  • Thanks for the reply Santosh.

    OK, I've manually copied most of the files. If I can just get a backup of the Exchange Database (company.edb) I'm willing to wipe the RAID and start again and resinstall the programs I need to.

    I tried to un-mount the exchange database (which succeeded) then manually copy the .edb file to a different drive but the paste section fails after a while saying Invalid MS-DOS Function. Are .edb files copy and paste-able?

    I then tried to move it through the SBS console. I did this when I first got the server to move the database onto the large RAID5 array rather than the OS C: drive. However, this also fails.

    I even tried moving the database through EMC and it also fails.

    Basically, I want some way of backing this database up, then I can get going rebuilding my array :-(

    please help.

    Friday, September 14, 2012 4:36 PM
  • Chris,

    I am unfamiliar with SBS, won't be able to help much on SBS queries. However, following might help

    Move Data (SBS) :http://technet.microsoft.com/en-us/library/cc527609(v=ws.10).aspx

    Also, you might get some better help in SBS forum on Exchange part

    http://social.technet.microsoft.com/Forums/en/smallbusinessserver/threads


    I do not represent the organisation I work for, all the opinions expressed here are my own.

    This posting is provided "AS IS" with no warranties or guarantees and confers no rights.

    - .... .- -. -.- ... --..-- ... .- -. - --- ... ....

    Friday, September 14, 2012 6:02 PM
    Moderator
  • The biggest problem in your case is the SBS. (Your Exchange is tied to your AD...)

    Really do a backup of your system state at the minimum...  after you can move back your file.

    -System-state backup, restore it to the new raid, and move back all file after.

    If you loose your AD, your exchange db will be in trouble, and you will have to re-migrate all computer.

    So, I just hope you can do a system-state backup at the minimum..

    Added: If you CAN'T do a system-state backup : Make sure all your user export their email to a local PST before any step !  (so if you re-do your domain, you can easilly import it)


    MCP | MCTS 70-236: Exchange Server 2007, Configuring

    Want to follow me ?  |  Blog: http://www.jabea.net | http://blogs.technet.com/b/wikininjas/



    Friday, September 14, 2012 7:34 PM
    Moderator
  • Hi Yagmoth,

    I do have a system state backup (my C drive, raid 1)

    If I try and do a data backup (my d drive, raid5) it fails at 64%. As my exchange .edb files are on the D: drive they haven't been backed up.

    I need to recreate this RAID5 ASAP as a failure is imminent, however, I want a copy of this exchange .edb first.

    You've got me thinking, if I export all the .pst files and recreate the RAID5. Exchange wouldn't be able to find the .edb file. Could I create a new blank database then import all the .pst files? sorry if that doesn't make sense..

    Friday, September 14, 2012 9:57 PM
  • Chris,

    Please make sure you have stopped the Exchange IS store service before copying the Exchanged DB files, otherwise they will not copy.

    Use a utility such as MS RichCopy to copy the Exchange DB files.

    After you have a comfortable backup, try doing a chkdsk/r and see if that saves you a rebuild. The server found bad media on one of the drives and tagged in in the rest of the drives. In a non-raid it's the equivalent of bad/tracks and sectors. Depending on what part of the disk/data got affected, you may be able to get away with a checkdisk.

    It's possible that your file allocation table or the Exchange DB has data on part of the disk that is 'punctured' and that may cause the DB to be unmovable. If that's the case, you might try an eseutil repair. If the file allocation table is bad, a track by track copy utility may do the trick.


    Miguel Fra | Falcon IT Services, Miami, FL
    www.falconitservices.com | www.falconits.com | Blog


    Saturday, September 15, 2012 12:41 AM
  • Hi Yagmoth,

    I do have a system state backup (my C drive, raid 1)

    If I try and do a data backup (my d drive, raid5) it fails at 64%. As my exchange .edb files are on the D: drive they haven't been backed up.

    I need to recreate this RAID5 ASAP as a failure is imminent, however, I want a copy of this exchange .edb first.

    You've got me thinking, if I export all the .pst files and recreate the RAID5. Exchange wouldn't be able to find the .edb file. Could I create a new blank database then import all the .pst files? sorry if that doesn't make sense..

    Perfect if you got a system-state backup! and that your raid5 does not host your c:.

    Yes for the pst, it's a path that would work (I actually do that in reverse for switching a sendmail site to a exchange'site), an export, and copy the database too. If something went wrong mounting the DB, importing the data will remake your user content back online. I would maybe disable the receive connector while you export, to be sure all mailbox content are uptodate, and remote user sending you mail will receive a not delivered answer too.

    I would follow Falcon IT Services advice too.


    MCP | MCTS 70-236: Exchange Server 2007, Configuring

    Want to follow me ?  |  Blog: http://www.jabea.net | http://blogs.technet.com/b/wikininjas/



    Saturday, September 15, 2012 1:08 AM
    Moderator
  • Hi,

    Dell have informed me that a "punctured RAID" is basically the end, nothing will fix it including chkdsk.

    I have downloaded ms richcopy and during the copying of the .edb file it failed at around 60% with "incorrect function" as per my backups! This looks like it is something to do with my punctured RAID.

    My new plan is to export all the .pst files, delete the RAID array, recreate it then import the .pst files. Can I just export all mailbox .pst files to my file server?

    Help with the syntax would be appreciate. All the guides I can find are for doing much more complex tasks that this...

    EDIT: On my exchange server logged in as administrator I've tried to run the following cmd in the EMS:

    export-mailbox -identity username -PSTFolderPath C:\PST -Confirm:$false

    Obviously, I swapped username for a real person. I have already created the "PST" folder under C:. The erro I get back is:

    The log file directory 'C:\program files\microsoft\exchange server\logging\migrationlogs' does not exist.

    I've checked manually, and I can see that the folder mentioned does not exist. Should this folder exist? Should this folder be created when I execute this cmd and it is failing? Or something else?

    I appreciate all comments.


    • Edited by Chris30-UK Saturday, September 15, 2012 11:00 AM additions
    • Proposed as answer by James XiongModerator Thursday, September 20, 2012 1:13 AM
    Saturday, September 15, 2012 9:33 AM
  • Update:

    I found on another page where people were saying if they made this folder on the client PC which I'm running the command and create a registry key it should work. details here: 

    http://social.technet.microsoft.com/Forums/en-GB/exchangesvrmigration/thread/a143664a-71b1-4c64-9928-71ec3ea809c9

    This did get me passed my initial error but I came up against another one "MAPI exceptionnetworkerror"

    It turns out this was because my database was unmounted and the MX Information Store was stopped (from when I was trying to copy the .edb file!)

    Finally, I am able to export the .pst files. 

    ______

    EDIT: I've just noticed I'm getting a message at the bottom of the export output which says "Error occurred in the step: Moving messages. This mailbox exceeded the maximum number if corrupted items specified for this move mailbox operation."

    In the target destination folder I can see all the .pst files I've exported and they all seem like they are the correct file size etc. Should I be worried?


    Saturday, September 15, 2012 2:52 PM
  • Hello,

    Checkdisk will not fix the puncture, the part of the drive that the RAID controller will not grant access to, but it may fix up the OS enough to allow you to move your data. The puncture will still be there but the os may be able to work around it after a chkdsk. There is also a utility called spinrite by GRC.com that may help you recover. You will still need to re-initialize the RAID afterwards.

    Also, I would try this only as a last resort because after chkdsk things may get better or worse.


    Miguel Fra | Falcon IT Services, Miami, FL
    www.falconitservices.com | www.falconits.com | Blog

    • Marked as answer by Chris30-UK Thursday, September 20, 2012 7:25 AM
    Saturday, September 15, 2012 3:33 PM
  • Hi Miguel,

    Thanks for the advice.

    I see what you mean about the chkdsk now. I will try this on Friday night just before I delete the RAID and reinitialise it.

    I'll let you guys know how it goes!

    Monday, September 17, 2012 8:01 AM
  • Ok, good luck, hope all goes well.

    Miguel Fra | Falcon IT Services, Miami, FL
    www.falconitservices.com | www.falconits.com | Blog

    Monday, September 17, 2012 8:18 PM
  • Hi Miguel,

    I managed to get all the mailboxes exported without errrors using the -BadItemLimit 9999 key last night.

    I thought I'd give the chkdsk a try. I ran chkdsk /r and the server restarted and began at 17:00....it finished at 22.25! :-).

    I then tried to move the exchange database file using the SBS Console Wizard (same as the wizard in the EMC) and this time it worked! Previously, it failed no matter what I tried so it looks like the chkdsk fixed that.

    I then deleted the RAID from the PERC config and rebuilt it.

    Things were going well and I thought my backup would work again, but it failed :-(.

    I'm going to start a new thread just for this problem as it doesn't seem to be related to the RAID issue.

    thanks again for everyone's help. 

    Saturday, September 22, 2012 10:00 AM
  • I love it when a plan comes together! Make sure you initialize the drives and look for any errors during the init process in case one of the drive has bad media.

    Congrats on recovering your DB!


    Miguel Fra | Falcon IT Services, Miami, FL
    www.falconitservices.com | www.falconits.com | Blog


    • Edited by Miguel Fra Saturday, September 22, 2012 8:31 PM
    Saturday, September 22, 2012 8:29 PM
  • hey, i'm experiencing almost the same punture issues.  everything is identical except my db is on the C partition.

    how does exporting psts allow for chkdsk to complete?

    Tuesday, September 25, 2012 11:17 PM
  • Hello,

    PST export had nothing to do with checkdisk. Checkdisk allowed the DB to be moved in its entirety by fixing things up a bit.

    chkdks inspects the physical and logical structures of a disk drive to make sure everything is the way it's supposed to be. If it finds currupt data, inconstent data, bad tracks and sectors, it can map around those so that the OS and its programs do not hang or freeze while the OS is trying to read from bad data.


    Miguel Fra | Falcon IT Services, Miami, FL
    www.falconitservices.com | www.falconits.com | Blog

    Wednesday, September 26, 2012 12:00 AM