none
Array becomes empty when storing huge amount of data

    Question

  • Hello All,

    In the below code, I'm recursively searching for folders/sub-folders files which are having LastWriteTime current day plus previous seven days. This code works fine in DEV server(less no of files). 

    In PROD server, i've huge volume of folders/files and this code is not working. The GCI output stored in array($Global:SourceFiles) always comes with empty.  

    $Global:SourceFiles = $null

    $Global:SourceFiles = GCI -Recurse $Source | ? { $_.PSIsContainer -eq $false}| Where-Object {$_.LastWriteTime.ToShortDateString() -ge (Get-Date).AddDays(-7).ToShortDateString() -and $_.LastWriteTime.ToShortDateString() -le (Get-Date).ToShortDateString()} | Select Fullname,LastWriteTime

    Please help me to find why my array value is showing empty for huge volume of data?

    Is there any alternate solution for this scenario?

    Thanks in Advance

    Saturday, July 14, 2018 7:45 AM

All replies

  • Don't use strings.  Strings cannot be compared as dates.

    There is no need to set a variable to null before assigning it.
    You need to go back 8 days to get the "previous" seven days

    $sourceFiles = Get-ChildItem $source -Recurse -File |
        Where-Object{
            $_.LastWriteTime -ge [datetime]::Today.AddDays(-8) -and
            $_.LastWriteTime -lt [datetime]::Today
        } |
        Select-Object Fullname,LastWriteTime

    Use "help" to learn how to use new CmdLets.  Take time to learn PowerShell instead of copying things and guessing.


    \_(ツ)_/





    Saturday, July 14, 2018 8:14 AM
    Moderator
  • What version of .NET is installed on that server? Prior to version 4.5 the maximum .NET object size was, I believe, 2GB.

    With .NET 4.5 you can get up to 4GB if you follow this:

    https://docs.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/runtime/gcallowverylargeobjects-element

    Either way, your code may run very slowly as the number of elements in the array grows. Instead of keeping the data in an array, try writing it to a file.

    See JRV's reply for a better way to deal with dates.


    --- Rich Matheisen MCSE&I, Exchange Ex-MVP (16 years)

    Saturday, July 14, 2018 6:53 PM
  • I agree that the results may exceed limits but there would be an exception.  No exception seems to indicate other  issues  such as the date match filter is not working as it is unlikely that any sizes would be exceeded if we only return the files for one week.  Matching of strings will not work for dates when we use GT or LT or any other boundary management because date strings do not collate correctly.

    DateTime objects match correctly.


    \_(ツ)_/

    Saturday, July 14, 2018 8:52 PM
    Moderator
  • Consider the following example of comparing dates.

    PS D:\scripts> $d1 = '12-21-2000'
    PS D:\scripts> $d2 = '11-21-2001'
    PS D:\scripts> $d2 -gt $d1
    False
    PS D:\scripts> $d1 = [datetime]$d1
    PS D:\scripts> $d2 = [datetime]$d2
    PS D:\scripts> $d2 -gt $d1
    True

    Strings are compared left-to-right.  Dates are compared as magnitudes because they have a "Compare" override that knows how to compare dates.

    Dates are compared by comparing tick count which is a pure number:

     $d2.Ticks

    Which is an INT64.


    \_(ツ)_/


    Saturday, July 14, 2018 8:57 PM
    Moderator
  • Well, dates in the format returned by ToShortDate() certainly aren't usually useful for comparisons. Using GetDate -Format "yyyyMMdd" works, as would (Get-Date).ToString("yyyyMMdd") . . . provided you need only the date and not the clock time.

    But, as you say, why bother with the conversion if it isn't necessary.


    --- Rich Matheisen MCSE&I, Exchange Ex-MVP (16 years)

    Saturday, July 14, 2018 9:05 PM
  • Well, dates in the format returned by ToShortDate() certainly aren't usually useful for comparisons. Using GetDate -Format "yyyyMMdd" works, as would (Get-Date).ToString("yyyyMMdd") . . . provided you need only the date and not the clock time.

    But, as you say, why bother with the conversion if it isn't necessary.


    --- Rich Matheisen MCSE&I, Exchange Ex-MVP (16 years)

    We need a date to doo the comparison correctly.  We want a day boundary to include a correct and complete range of days.

    THe boundary for "before" today  is:

    $d -lt [datetime].Today

    This includes all matches before midnight today.

    To get the starting boundary we then subtract exactly 7 days + 1 from that date and we have a date object at the beginning boundary.

    PS D:\scripts> [datetime]::Today
    
    Saturday, July 14, 2018 12:00:00 AM
    
    
    PS D:\scripts> [datetime]::Today.AddDays(-8)
    
    Friday, July 6, 2018 12:00:00 AM
    

    Note the boundaries are both set at midnight.

    The end of the range we compare with "LT" to exclude the endpoint.  The start use "GE" to include the startpoint.  We will now have all 7 days of matches in the interval.  This cannot be done if we convert the dates to strings.


    \_(ツ)_/

    Saturday, July 14, 2018 9:15 PM
    Moderator
  • Here is an alternate method to get the week exactly previous to today.

    $start = [datetime]::Today.AddDays(-8)
    $end = $start.AddDays(7)

    This makes the interval more explicit since we are picking the start and using it to define the interval's end.

    If we are looking for the range of dates in the  current 7 day interval we would just take the everything after the start.

    $start = [datetime]::Today.AddDays(-7)

    Now we retrieve everything that is "GE" $start.


    \_(ツ)_/

    Saturday, July 14, 2018 9:51 PM
    Moderator
  • You can do it this way.  The '[timespan]' is optional, since it will get cast that way anyway.

    gci -recurse $source -file | where { (get-date) - $_.lastwritetime -lt [timespan]7. }



    • Edited by JS2010 Sunday, July 15, 2018 7:53 PM
    Sunday, July 15, 2018 7:52 PM
  • You can do it this way.  The '[timespan]' is optional, since it will get cast that way anyway.

    gci -recurse $source -file | where { (get-date) - $_.lastwritetime -lt [timespan]7. }



    Yes that is another method of setting it up but not what was asked for.  We need end boundaries at the start of the day and not at the current time.  Also "[timespan]8" is 8 ticks and not 8 days.

    8 days:

    [timespan]'8:0:0:0'

    [timespan]'dd:hh:mm:ss'

    You have to exclude anything including midnight of "[datetime]::Today".   This is much harder to do using your method.


    \_(ツ)_/

    Sunday, July 15, 2018 8:09 PM
    Moderator
  • 8 with a period after it would be 8 days:  [timespan]8.

    Sunday, July 15, 2018 8:14 PM
  • 8 with a period after it would be 8 days:  [timespan]8.

    Sorry.  I missed that tiny little dot.  You still need to account for the boundaries of the interval and that is harder with a timespan.  Wit a TS you have to set an interval between 8 and 1.   Which would look something like this:

    gci -recurse $source -file | 
        where {
            $ts = [datetime]::Today -$_.lastwritetime
            if($ts -gt [timespan]1. -and $ts -le [timespan]8.){$true}else{$false}
        }


    \_(ツ)_/




    Sunday, July 15, 2018 8:25 PM
    Moderator
  • Well had to repost that three times before finally getting all of it right.  I think  that is how to make it work with a timespan.


    \_(ツ)_/


    Sunday, July 15, 2018 8:32 PM
    Moderator
  • I don't know why this is, but [timespan]8. (with the period) and [timespan]'8' (with the quotes) is eight days, but [timespan]8 (without the period) is 8 ticks.



    • Edited by JS2010 Monday, July 16, 2018 12:30 PM
    Monday, July 16, 2018 12:29 PM
  • Don't forget:  [timespan]8.0

    Look at:

     [timespan]::New


    \_(ツ)_/

    Monday, July 16, 2018 4:51 PM
    Moderator