none
What's the fastest way to copy 151GB, 375000 files from win 2003 server to win 2008 server

    Question

  • Non techie here.

    I have a project where I need to get 151GB of data spread over 375000+ files moved from a win 2003 FAP to a 2008 server. Copy, xCopy, Robocopy all take in excess of 50hours to move it to an external HDD. Has to be external move for security reasons.

    I have 40 hours max to get it off and onto the new server.

     

    Cheers 

    Ian

    Thursday, February 10, 2011 9:14 PM

Answers

  • Ok, you are possibly thinking about this all wrong.  I will assume you already have the 2003 and 2008 server, and are running tests, and these files are used daily..thus making changes?

    So your 40 hour limitation, it just a window to complete the final copy, not a situation where you get the 2008 server at hour 1 and it leaves at hour 40?  You have the server now, you most likely need to complete the copy over a weekend.

    Simple....use robocopy, XCopy, etc to pre-stage the server in advance and prior to your 40 hour window.  Then use robocopy, DFS-Replication, or Beyond Compare to copy over the changes that happen after your initial stage.

    So your plan in looks like this

    1 week in advance copy data to 2008 server, that will take 2-3 days

    In your 40 hours window copy over changes....this should take a fraction of the time.

    We use Beyond Compare for this type of migration all the time.

    Friday, February 11, 2011 2:26 PM

All replies

  • Couldn't you set up to do it over LAN?
    The best way I could think of doing it without that is to transfer the hard drive to the other server, however I am unsure if you have to have zero downtime.

    I honesly think that setting up a private network over LAN that that is on a /31 subnet is your best choice, inless they are in different geographic areas.

    Friday, February 11, 2011 1:16 AM
  • Hello Ian,

    Based on my experience, the most fast way to migration so much data is to use File Server Migration Toolkit.

    We can use FSMT to finish the migration project, which is very simple to use.

    Overview of the Microsoft File Server Migration Toolkit
    http://www.microsoft.com/downloads/en/details.aspx?FamilyID=604BE797-D19E-4BC1-A148-B48FC731135E&displaylang=en

    For the detail about File Server Migration Toolkit, please refer to the FSMT white paper.

    Overview of the Microsoft File Server Migration Toolkit
    http://www.microsoft.com/downloads/details.aspx?FamilyID=604BE797-D19E-4BC1-A148-B48FC731135E&displaylang=en&displaylang=en

    Hope this can be helpful.

    Scorprio


    TechNet Software Assurance Managed Newsgroup MCTS: Windows Vista | Exchange Server 2007 MCITP: Enterprise Support Technician | Server & Enterprise Admin | System Architect
    Friday, February 11, 2011 2:19 AM
  • Ok, you are possibly thinking about this all wrong.  I will assume you already have the 2003 and 2008 server, and are running tests, and these files are used daily..thus making changes?

    So your 40 hour limitation, it just a window to complete the final copy, not a situation where you get the 2008 server at hour 1 and it leaves at hour 40?  You have the server now, you most likely need to complete the copy over a weekend.

    Simple....use robocopy, XCopy, etc to pre-stage the server in advance and prior to your 40 hour window.  Then use robocopy, DFS-Replication, or Beyond Compare to copy over the changes that happen after your initial stage.

    So your plan in looks like this

    1 week in advance copy data to 2008 server, that will take 2-3 days

    In your 40 hours window copy over changes....this should take a fraction of the time.

    We use Beyond Compare for this type of migration all the time.

    Friday, February 11, 2011 2:26 PM
  • I copied over 12TB in 24 hours using the method below. A lot of this depends on your infrastructure. The scripts I used are unmodified for your case. I suggest you give them a look and understand the process and change it to fit your needs.

    There are 2 parts. The first is a Main script that schedules PowerShell jobs that actually do the work. The main script will read a file called JobCount every loop to see how many jobs it can run at one time, the reason I did this was to change the number of jobs depending on Day (production) times and Night times. Also, the Main loop reads a nomig file that tells the script, don't move these folders because they were our test cases, you can even do test cases during the migration since you can modify the file while the script is running. The example was use to move thousands of home folders. Using Robocopy if you tell a single command to do everything, it will take hours to start, just looking around. If you do one root folder at a time, it will run much faster which is the reason I created this. If you have a small number root folders, you may want to point it at folders where you do have a lot of subs, remember you can have more than one main process running in different runspaces.

    Main Script 

    VVVVVVVVVV

    $homeOld = (Get-ChildItem -Path \\server\share | select name)

    $JobRun = 10

    $i = 0

    $Count = $homeOld.Count

    foreach ($homeDir in $homeOld) { 

        $i = $i + 1

        $Sdir = $homeDir.Name

        Write-Progress -Activity "Migrating Homes" -Status "Processing $i of $Count"  -PercentComplete ($i/$homeOld.Count*100) -CurrentOperation "Next to Process $Sdir"

        $not = gc \\serverl\share\script\nomig.txt -ErrorAction "Stop"

        $JobRun = gc \\server\share\script\jobcount.txt -ErrorAction "Stop"

        if ($not -notcontains ($homeDir.Name).ToLower()) {

                While ((Get-Job -State "Running").Count -gt ($JobRun-1)) {

                    Start-Sleep -Seconds 3

                } 

     

                if ((Get-Job -State "Completed").Count -gt 0) {

                $Comp = Get-Job -State "Completed"

                foreach ($Job in $Comp) { 

                    $outfile = $Job.name + ".txt"

                    Receive-Job -Job $Job | Out-File -FilePath "\\server\share\verify\$outfile"

                    Remove-Job -Job $Job}

                }

                Start-Job -Name $Sdir -ArgumentList "\\server\share\$Sdir", "\\newserver\share\$Sdir", "/COPYALL", "/MIR", "/W:1", "/R:1", "/MT:5" -FilePath \\server\share\script\robothread.ps1 > $null

     

        }

        else {

            Write-Host $HomeDir.Name " Excluded" -ForegroundColor Green

        }

    }

     

    =====

    Thread Script - where Robocopy does the work.

    VVVVVVVVV

    & robocopy $args[0] $args[1] $args[2] $args[3] $args[4] $args[5] $args[6]

    ============

     

    This comes with no warranty, it is just an idea I used to do a very fast copy with permissions and all attributes, where no other method was useable.

    Thanks,

    Allan


    Allan
    Friday, February 11, 2011 3:38 PM
  • I cannot confirm if this can help. Try to use robocopy /mt to see if it can help speed up the whole process.
    Shaon Shan |TechNet Subscriber Support in forum |If you have any feedback on our support, please contact tngfb@microsoft.com
    Wednesday, February 16, 2011 3:26 PM
  • Hey allan,,

       can u explain the above script with an example, such as source and destination folder, or C drive D Drive.....

     

     Can we copy files through the network using your script??

     

    regards,

    Jugal

    Thursday, November 24, 2011 5:44 AM
  • I found this PowerShell very useful. The only problem I have is the script seem to end and not all the robocopy logs get written.

    How can I fix this?

    Monday, July 07, 2014 9:46 PM