none
Read in specific line numbers from a text file (very large 1.5GB) using PowerShell RRS feed

  • Question

  • Hi all,

    I have a large file (a text file) which is over 1.5GB in size. 

    I need to read in specific line numbers from the file, I have a list of exact line numbers I wish to read.

    If I use "get-content" to read in all the contents, PowerShell consumes 1.3GB in memory then crashes out.  I don't have enough memory to cope with using get-content to read in all the contents.

    Is there any way of doing the following:

           $entry = get %linenumber% from %filename%

    Any help is greatly appreciated!

    Kathy


    Kathleen Hayhurst Senior IT Support Analyst

    Friday, December 6, 2013 1:53 PM

Answers

  • Hi Kathy,

    a StreamReader should be able to do this faster:

    function Get-Lines($path, $lineNumbers){
    	$arrList = New-Object System.Collections.ArrayList($null)
    	$arrList.AddRange(($lineNumbers | sort))
        $reader = New-Object 'System.IO.StreamReader'($path, $true)
    	$arrIndex=0
        try{
            while (($line = $reader.ReadLine()) -ne $null -and $arrList.Count){
                if ($currentIndex++ -eq $arrList[0]-1){
                    $line
    				$arrList.Remove($arrList[0])
    				$arrIndex++
                }
            }
        }
        finally{
            $reader.Close();
        }
        return $null;
    }
    
    Get-Lines -path test.txt -lineNumbers 3,7,9
    
    


    Friday, December 6, 2013 2:29 PM

All replies

  • Hi Kathy,

    a StreamReader should be able to do this faster:

    function Get-Lines($path, $lineNumbers){
    	$arrList = New-Object System.Collections.ArrayList($null)
    	$arrList.AddRange(($lineNumbers | sort))
        $reader = New-Object 'System.IO.StreamReader'($path, $true)
    	$arrIndex=0
        try{
            while (($line = $reader.ReadLine()) -ne $null -and $arrList.Count){
                if ($currentIndex++ -eq $arrList[0]-1){
                    $line
    				$arrList.Remove($arrList[0])
    				$arrIndex++
                }
            }
        }
        finally{
            $reader.Close();
        }
        return $null;
    }
    
    Get-Lines -path test.txt -lineNumbers 3,7,9
    
    


    Friday, December 6, 2013 2:29 PM
  • Hi,

    Have you tried above suggestion? Any update about the issue? If there is anything we could help, please feel free let us know.

    TechNet Subscriber Support

    If you are TechNet Subscription user and have any feedback on our support quality, please send your feedback here

    Regards, Yan Li

    Monday, December 9, 2013 5:35 AM
    Moderator
  • Just to update the thread, I ended up using "select-string" directly against the file like this:
     
          select-string-path  c:\filname.txt  -pattern"Started : (.*)"-AllMatches

    I was then able to process the file(s) ok.
     
    Thanks for all your replies by the way!
     
    :)


    Kathleen Hayhurst Senior IT Support Analyst


    • Edited by KMH.1 Monday, March 3, 2014 1:07 PM
    Monday, March 3, 2014 1:05 PM