none
Powershell Script takes to long to finish

    질문

  • I have a script that will go through will go through 42 directories (that have security camera pictures dumped into) move the files contained in each direct into another directory for each date:

    $flist = dir
    $PicCounter = 0
    #echo "Inside $curcam directory"
    #Loop that looks at every file
    for($j = 0; $j -lt ($flist.Length); $j++)
    {
    	$ftemp = $flist[$j]
    	#echo "Working on $ftemp"
    	#Check to see that the file is not a directory (should be just pictures)
    	if($flist[$j].Attributes -ne "Directory")
    	{
    		$PicCounter = $PicCounter + 1
    		$tstamp = $flist[$j].LastWriteTime
    		$mdir = $tstamp.GetDateTimeFormats()[6]
    		#create directory for that date if it doesn't yet exist
    		if(! (test-path $mdir -pathtype container))
    		{
    			#echo "Creating directory $mdir"
    			mkdir $mdir | out-null
    		}
    		#move the file into that directory
    		mv $flist[$j] $mdir
    	}
    }

    There is the code that does this.  However when it gets going.....  it takes WAY to long to accomplish and it seems to want to take up ALL memory and create a 10 gig cache file.  How do I make it work better?

    2012년 2월 17일 금요일 오후 6:57

답변

  • Ok, I've simplified your script substantially, and used foreach-object instead of a for loop.  This should reduce memory usage, and perhaps increase speed.  What it does it sort the files into folders based on date, so more than one file may end up in any given folder, depending on how many had that date (time is not considered).

    $files = Get-ChildItem | Where-Object {(!($_.PSIsContainer))}
    $files | ForEach-Object {
        $date = $_.lastwritetime
        $date = Get-Date $date -Format 'dd_MMM_yyyy'
        New-Item -ItemType directory -Path $date -ErrorAction SilentlyContinue
        Move-Item -Path $_ -Destination $date
        } # end foreach object


    Grant Ward, a.k.a. Bigteddy

    What's new in Powershell 3.0 (Technet Wiki)


    2012년 2월 17일 금요일 오후 7:51
    1. Get all lines from 'dir' that contain a forward-slash.
    2. Skip the first 2 lines in array (. and ..)
    3. For each of the lines, split on the forward-slash and assign the first 7 leftovers to variables starting with %%G.
    4. In each iteration, call the print function with arguments G, H, I and M (day, month, year, filename)
    5. If %4 (fourth argument passed to function) is empty, do nothing.
    6. Else set the date to the format OP requested
    7. Check for existence of the folder with date from file; if it doesn't exist, create it
    8. Copy filename to foldername

    Rich Prescott | Infrastructure Architect, Windows Engineer and PowerShell blogger | MCITP, MCTS, MCP

    Engineering Efficiency
    @Rich_Prescott
    Windows System Administration tool
    AD User Creation tool

    2012년 2월 18일 토요일 오후 1:34
  • Here's another example of it in batch format:

       @echo off
    
       FOR /f "tokens=1-4 delims=/-. " %%G IN ('date /t') DO (call :s_fixdate %%G %%H %%I %%J)
       goto :s_print_the_date
       
       :s_fixdate
       if "%1:~0,1%" GTR "9" shift
       FOR /f "skip=1 tokens=2-4 delims=(-)" %%G IN ('echo.^|date') DO (
           Set %%G=%1&set %%H=%2&Set %%I=%3)
       goto :eof
    
       :s_print_the_date
       SET yy=%yy%&SET mm=%mm%&SET dd=%dd%
       SET MyDate=%dd%_%mm%_%yy%
       echo %MyDate%
    
       if not exist %MyDate% mkdir %MyDate%
       copy *.* %MyDate%
    This would work if you set it as a scheduled task every night at midnight.


    Rich Prescott | Infrastructure Architect, Windows Engineer and PowerShell blogger | MCITP, MCTS, MCP

    Engineering Efficiency
    @Rich_Prescott
    Windows System Administration tool
    AD User Creation tool


    2012년 2월 18일 토요일 오전 1:01
  • You can also use this version, which does exactly what your original script does, except with batch.

    @echo off
    for /f "skip=2 tokens=1-7 delims=/ " %%G IN ('dir ^| find "/"') DO (call :print %%G %%H %%I %%M)
    
    :print
    if "%4"=="" (echo.) else (
    SET MyDate=%1_%2_%3
    if not exist %MyDate% mkdir %MyDate%
    copy %4 %MyDate%
    )


    Rich Prescott | Infrastructure Architect, Windows Engineer and PowerShell blogger | MCITP, MCTS, MCP

    Engineering Efficiency
    @Rich_Prescott
    Windows System Administration tool
    AD User Creation tool

    2012년 2월 18일 토요일 오전 1:55

모든 응답

  • Ok, I've simplified your script substantially, and used foreach-object instead of a for loop.  This should reduce memory usage, and perhaps increase speed.  What it does it sort the files into folders based on date, so more than one file may end up in any given folder, depending on how many had that date (time is not considered).

    $files = Get-ChildItem | Where-Object {(!($_.PSIsContainer))}
    $files | ForEach-Object {
        $date = $_.lastwritetime
        $date = Get-Date $date -Format 'dd_MMM_yyyy'
        New-Item -ItemType directory -Path $date -ErrorAction SilentlyContinue
        Move-Item -Path $_ -Destination $date
        } # end foreach object


    Grant Ward, a.k.a. Bigteddy

    What's new in Powershell 3.0 (Technet Wiki)


    2012년 2월 17일 금요일 오후 7:51
  • Here's another example of it in batch format:

       @echo off
    
       FOR /f "tokens=1-4 delims=/-. " %%G IN ('date /t') DO (call :s_fixdate %%G %%H %%I %%J)
       goto :s_print_the_date
       
       :s_fixdate
       if "%1:~0,1%" GTR "9" shift
       FOR /f "skip=1 tokens=2-4 delims=(-)" %%G IN ('echo.^|date') DO (
           Set %%G=%1&set %%H=%2&Set %%I=%3)
       goto :eof
    
       :s_print_the_date
       SET yy=%yy%&SET mm=%mm%&SET dd=%dd%
       SET MyDate=%dd%_%mm%_%yy%
       echo %MyDate%
    
       if not exist %MyDate% mkdir %MyDate%
       copy *.* %MyDate%
    This would work if you set it as a scheduled task every night at midnight.


    Rich Prescott | Infrastructure Architect, Windows Engineer and PowerShell blogger | MCITP, MCTS, MCP

    Engineering Efficiency
    @Rich_Prescott
    Windows System Administration tool
    AD User Creation tool


    2012년 2월 18일 토요일 오전 1:01
  • You can also use this version, which does exactly what your original script does, except with batch.

    @echo off
    for /f "skip=2 tokens=1-7 delims=/ " %%G IN ('dir ^| find "/"') DO (call :print %%G %%H %%I %%M)
    
    :print
    if "%4"=="" (echo.) else (
    SET MyDate=%1_%2_%3
    if not exist %MyDate% mkdir %MyDate%
    copy %4 %MyDate%
    )


    Rich Prescott | Infrastructure Architect, Windows Engineer and PowerShell blogger | MCITP, MCTS, MCP

    Engineering Efficiency
    @Rich_Prescott
    Windows System Administration tool
    AD User Creation tool

    2012년 2월 18일 토요일 오전 1:55
  • You can also use this version, which does exactly what your original script does, except with batch.

    @echo off
    for /f "skip=2 tokens=1-7 delims=/ " %%G IN ('dir ^| find "/"') DO (call :print %%G %%H %%I %%M)
    
    :print
    if "%4"=="" (echo.) else (
    SET MyDate=%1_%2_%3
    if not exist %MyDate% mkdir %MyDate%
    copy %4 %MyDate%
    )


    Rich Prescott | Infrastructure Architect, Windows Engineer and PowerShell blogger | MCITP, MCTS, MCP

    Engineering Efficiency
    @Rich_Prescott
    Windows System Administration tool
    AD User Creation tool

    Rich, would it be possible to explain, in plain language, how that for loop works?  I'm not too good with batch programming, and I'd like to understand.

    PS:  I've run for /?, but it doesn't explain it very well, IMO.


    Grant Ward, a.k.a. Bigteddy

    What's new in Powershell 3.0 (Technet Wiki)


    • 편집됨 Bigteddy 2012년 2월 18일 토요일 오전 7:13
    2012년 2월 18일 토요일 오전 7:11
    1. Get all lines from 'dir' that contain a forward-slash.
    2. Skip the first 2 lines in array (. and ..)
    3. For each of the lines, split on the forward-slash and assign the first 7 leftovers to variables starting with %%G.
    4. In each iteration, call the print function with arguments G, H, I and M (day, month, year, filename)
    5. If %4 (fourth argument passed to function) is empty, do nothing.
    6. Else set the date to the format OP requested
    7. Check for existence of the folder with date from file; if it doesn't exist, create it
    8. Copy filename to foldername

    Rich Prescott | Infrastructure Architect, Windows Engineer and PowerShell blogger | MCITP, MCTS, MCP

    Engineering Efficiency
    @Rich_Prescott
    Windows System Administration tool
    AD User Creation tool

    2012년 2월 18일 토요일 오후 1:34
  • Thank you!  I didn't realise it would involve so much typing.

    Grant Ward, a.k.a. Bigteddy

    What's new in Powershell 3.0 (Technet Wiki)

    2012년 2월 18일 토요일 오후 2:13
  • I used this code, however it still ends up taking 78% down from 95% of 12 gig of memory.  Sense there is 42 cameras dumping images through filezilla and once a night moving them.....  do I have to clear the memory after the program is done running?  If so How do I accomplish this  or how do I free up memory?
    2012년 2월 23일 목요일 오후 8:01
  • I used this code, however it still ends up taking 78% down from 95% of 12 gig of memory.  Sense there is 42 cameras dumping images through filezilla and once a night moving them.....  do I have to clear the memory after the program is done running?  If so How do I accomplish this  or how do I free up memory?

    Memory will clear automatically when the process is complete.  If a system is idle it will use nearly all memory if it can.

    I assume you are talking about either page file or physical memory.

    If page file is expanding that much it usually means you do not have enough physical for the process you are using.

    PowerShell may not clean up memory if there is no demand to do so.  The memory will be released as soon as another program demands it or when PowerShell finishes.

    The Windows memory manager is very efficient.  It doesn't waste time releasing page file space until no processes might ask for it.

    For efficiency and disk performance you should be sure the files you are receiving and copying are not on the same disk as the page file. 

    You have never explained where you see this 10 gig cache file so I am assuming you are talking about the system page file.  PowerShell does not create cache files.  Some operations may create a temporary file in the temp folder but just copying files will not do this.

    It there is a cache file then it may well be filezilla doing this.  Filezilla is known to eat up resources when it runs with large image/video files.

     


    ¯\_(ツ)_/¯

    2012년 2월 23일 목요일 오후 8:35
  • After doing more investigating and using RamMap from www.sysinternals.com the 10 gig file is a Metafile

    From http://blogs.technet.com/b/askperf/archive/2010/08/13/introduction-to-the-new-sysinternals-tool-rammap.aspx

    Metafile: Metafile is part of the system cache and consists of NTFS metadata. NTFS metadata includes the MFT as

    well as the other various NTFS metadata files (see How NTFS Works for more details, and of course Windows

    Internals is a great reference). In the MFT each file attribute record takes 1k and each file has at least one attribute record.

    Add to this the other NTFS metadata files and you can see why the Metafile category can grow quite large on servers with lots of files.

    How do I clear this file or mv files with using this file?

    2012년 2월 24일 금요일 오후 7:07
  • After doing more investigating and using RamMap from www.sysinternals.com the 10 gig file is a Metafile

    From http://blogs.technet.com/b/askperf/archive/2010/08/13/introduction-to-the-new-sysinternals-tool-rammap.aspx

    Metafile: Metafile is part of the system cache and consists of NTFS metadata. NTFS metadata includes the MFT as

    well as the other various NTFS metadata files (see How NTFS Works for more details, and of course Windows

    Internals is a great reference). In the MFT each file attribute record takes 1k and each file has at least one attribute record.

    Add to this the other NTFS metadata files and you can see why the Metafile category can grow quite large on servers with lots of files.

    How do I clear this file or mv files with using this file?

    Yu cannot.

    It is a system file that you are looking at.  This is not normally visible or important.

    YO are copy a large numeber of very larg eimage files so this will happen.  There is nothing you can do about it.

    If your copies are very slow perhaps your disk is very heavily fragmented due to many copty and paste operations.

    If this is your boot drive then you are probably creating issues with the system by copying many large files to the boot drive.

    It has nothing to do with copying files except that you have a lot of very large files on your system.

    YOu cannot chaneg the system metafile which is called $metafile.  It contains all of teh metadata that dewscritbes the files on  your system.  A good defrag utility can  improve performance and comp[act the file and may need to be run frequently if you are creating and deleting many large files.


    ¯\_(ツ)_/¯

    2012년 2월 24일 금요일 오후 7:17
  • I would defrag the server but:

    - 42 security cameras upload images  constantly when motion is detected (most of these are in class rooms so there WILL be motion)

    - Every night we want to move the previous days pictures to its own folder for ease

    This will cause the issue more than once unless I can figure out how to prevent it.  Is there a way to move the files (using move-item/mv now) without causing this?

    2012년 2월 24일 금요일 오후 7:27
  • You should note that the size of the MetaFile is completely dependent on the number and size of the files in your system and is completely controlled by the operating system.  As the article says each file takes a minimum of 1kb and actually takes 1kb per extent I believe.  Very large files take a lot of kbs in the MetaFile. 

    The MetaFile is virtualized to disk and does not take up that much memory in physical Ram.  It is basically a fast way to finding and update files in the system.


    ¯\_(ツ)_/¯


    • 편집됨 jrv 2012년 2월 24일 금요일 오후 7:34
    2012년 2월 24일 금요일 오후 7:33
  • Well When we reboot the machine and with filezilla running and the images being uloaded constantly (61kb each file)....  the server runs at about 17% of 12gb of ram.

    Once the script runs and copies the files over (averaging about 20000 files in each of 42 directories)....  THIS is when it bogs down to about 80 to 90% used and stays that way until reboot the system.  The only way around it is to reboot the system?  There is no other way to copy these files to prevent it?

    2012년 2월 24일 금요일 오후 7:51
  • I don't think the metafile has anything to do wqith this.  I think you are running out of memory fo to an OS bug.

    I would contact MS support or search teh KB for memory issues for your OS version.

    Since we don't have access to your system there is little anyone her can do .  It is not a scripting error.

    If yu stop the FIlezilla processes and stop POwerSHell and the memory is not released then you have a system bug.  Thisis most likely caused by Filezilla but may be a bug in some other part of the system.


    ¯\_(ツ)_/¯

    2012년 2월 24일 금요일 오후 8:39