locked
Sanity check prior to starting script RRS feed

  • Question

  • Hello there Scripting Guys,

    I am trying to build a wee sanity check at the beginning of a script to make sure that the files I intend to be working with later on have been created.

    It looks like this right now during testing:

    $path = "F:\Backup\*"

    $limit = (Get-Date).AddMinutes(-5)

    $filelist = (Get-ChildItem -Path $path -include *.trn,*.bak -Recurse -Force | where-object { !$_.PSIsContainer -and $_.LastWriteTime.Date -lt $limit } | select-object -property Name)

    # Sanity check for missing backup files
    $backupsexist = Test-Path $filelist
    if ($backupsexist -eq "True")
    { write-Host "At least 1 uncompressed file found - ok to proceed" }
    Else
    { write-host "No backup files found - has SQL backup been run?"}

    ***

    $filelist Shows a bunch of file names, eg

    hostname_DBATools_FULL_20150929_114756.bak                                                                                                                                                                                                         
    hostname$SQLEXPRESS_ReportServer$SQLEXPRESS_FULL_20150929_114825.bak                                                                                                                                                                                          
    hostname$SQLEXPRESS_ReportServer$SQLEXPRESSTempDB_FULL_20150929_114826.bak

    but "Test-Path $filelist" returns a string of "False" results.

    I'll admit it, I'm bamboozled as to why a variable that has been populated  with tasty filenames is not returning a "true" result in this case. Perhaps I'm missing some basic syntax rule here ( most likely as I am new to this ) ?

    Thanks in advance for a nudge in the right direction!


    • Edited by adriannzde Tuesday, September 29, 2015 10:36 AM
    Tuesday, September 29, 2015 10:33 AM

Answers

  • I understand that and it as good info but not really a good solution.  Using Test-Path after a Get-ChildItem is silly.

    Why not try this:

    Get-ChildItem *.bak | Test-Path

    I guarantee it will always return true so the test serves no purpose.


    \_(ツ)_/

    • Marked as answer by adriannzde Wednesday, September 30, 2015 7:41 AM
    Tuesday, September 29, 2015 4:11 PM

All replies

  • I recommend that you actually try to learn PowerShell. What you are trying to do and what you are asking make no sense.

    Start here: https://technet.microsoft.com/en-us/scriptcenter/dd793612.aspx?f=255&MSPPError=-2147217396

    We do not write scripts and we cannot read your mind or decode what you are trying to do.

    Hint - if "Get-ChildItem" returns a file name then it exists.  You do not need to test it.

    You also need to learn how to use a program to automate or create a result. We cannot teach you how to do that.

    I suspect that you need to have a list in a file or an array that you will test for. Use the learning materials to learn how to do this. Post back with specific questions about a script.


    \_(ツ)_/

    Tuesday, September 29, 2015 10:55 AM
  • well I have got my test-path working, but I still don't understand why it was returning false on my initial test.

    I dropped the "select-object" clause on the $filelist variable, so now I get the complete path returned and my test path returns "true"

    Tuesday, September 29, 2015 11:51 AM
  • What you are doing still does not make any sense. Why would you test a file that you already know exists?

    To test if you have found a file at all we would do this:

    if($filelist=(Get-ChildItem -Path $path -include *.trn,*.bak -File -Recurse -Force | where-object {$_.LastWriteTime.Date -lt $limit)){
         # files found
    }else{
         # no files found
    }


    \_(ツ)_/

    • Proposed as answer by jrv Tuesday, September 29, 2015 1:04 PM
    Tuesday, September 29, 2015 11:58 AM
  • I certainly don't expect you to write my script for me, and I certainly did come here to learn, so thanks on both counts. I read the Forum FAQ before posting and belive my question fits the requirements described.

    The script that creates the daily SQL Server backups runs as a separate scheduled task, and creates the files my script needs to work with.

    If it doesn't run for some reason then there will be no files for the rest of the script to work with, and so I want it to exit early on and report back on that condition.

    I am not testing for a file I already know exists, I'm building a safety mechanism into the beginning of the script to make sure I exit if the overnight backups have failed or not run for any reason.

    Thanks again for your code example.


    • Edited by adriannzde Tuesday, September 29, 2015 12:14 PM
    Tuesday, September 29, 2015 12:11 PM
  • $filelist shows a bunch of file names because that's what you asked for with select-object -property Name

    Your Test-Path statement returns false because the argument you're passing is a PSCustomObject, not a String type. Your argument should have been $filelist.name which contains the actual file name. Note, this test could still potentially return false.

    As it's currently written, the script must be located in the directory you're searching, because $filelist does not contain a full directory path and file name. To fix that, remove | select-object -property Name. Use $filelist.FullName as the argument to Test-Path.

    You can also remove the wildcard from "F:\Backup\*" since you're using the -Recurse switch with Get-ChildItem

    Read the documentation. Keep at it. It'll become clearer over time with persistent use. You may find a simpler, more efficient solution.


    Please remember to: "Vote" if my response was helpful. "Mark As Answered" if I answered your question correctly. V/R, Darrick West - Senior Systems Engineer, ConfigMgr: OSD







    • Proposed as answer by Darrick West Tuesday, September 29, 2015 12:33 PM
    • Edited by Darrick West Tuesday, September 29, 2015 12:38 PM
    Tuesday, September 29, 2015 12:14 PM
  • It would make more sense to test the backup logs in SqlServer as you may have files even if the backup failed.  The logs will tell you if it succeeded.

    Post in SqlServer forum for more information on how to manage backups.


    \_(ツ)_/

    Tuesday, September 29, 2015 12:23 PM
  • Try these changes in your code:

    $path = "F:\Backup"
    
    $filelist = (Get-ChildItem -Path $path -include *.trn,*.bak -Recurse -Force | where-object { !$_.PSIsContainer -and $_.LastWriteTime.Date -lt $limit }).FullName

    I'm using the property dereference operator to return the FullName property, full directory path and file name of the Get-ChildItem cmdlet, note the syntax. This will return a String type in $filelist that can be parsed by Test-Path

    Please remember to: "Vote" if my response was helpful. "Mark As Answered" if I answered your question correctly. V/R, Darrick West - Senior Systems Engineer, ConfigMgr: OSD


    • Edited by Darrick West Tuesday, September 29, 2015 12:33 PM
    • Proposed as answer by Darrick West Tuesday, September 29, 2015 12:33 PM
    Tuesday, September 29, 2015 12:31 PM
  • I'm implementing the backup scripts from Ola Hallengren for the SQL backups, and these produce their own logs that need to be checked for success or fail.

    The script I am working here on has the task of zipping all the database backup files from each database daily to conserve space on the database server. It also deletes the zip archives older than "x" days , after the file System backups have picked them up.

    The step I am asking for help with here is at the beginning of the script, where I check if new files have landed on the file System overnight - or in my test version of the script in the past 5 minutes.

    I would be happy to post the whole script if People would like more reasons to attack my lack of mad powershell coder skillz ;-) , but I wanted to stick to the principal of one problem, one question.

    Tuesday, September 29, 2015 12:54 PM
  • Thanks Derrick, but your command variation Returns a NULL string for me. I'll continue to research the property dereference operator you mention in your post.  
    Tuesday, September 29, 2015 1:27 PM
  • Hmmm...

    Are their files located in the target path?

    Try (Get-ChildItem).FullName as a test. 


    Please remember to: "Vote" if my response was helpful. "Mark As Answered" if I answered your question correctly. V/R, Darrick West - Senior Systems Engineer, ConfigMgr: OSD

    Tuesday, September 29, 2015 2:47 PM
  • Once again guys thisis how we would do this in PowerShell.  There is no need to invent anything more difficult.

    if($filelist=(Get-ChildItem -Path $path -include *.trn,*.bak -File -Recurse -Force | where-object {$_.LastWriteTime.Date -lt $limit)){
         # files found
    }else{
         # no files found
    }


    \_(ツ)_/

    Tuesday, September 29, 2015 3:12 PM
  • Yes, great example of one way to do it ;-)

    I voted on it BTW.

    My intent was not to "invent anything more difficult" as you put it.

    I was just replying to adriannzde's comment regarding the NULL value returned and suggested a simple test using the property dereference operator


    Please remember to: "Vote" if my response was helpful. "Mark As Answered" if I answered your question correctly. V/R, Darrick West - Senior Systems Engineer, ConfigMgr: OSD

    Tuesday, September 29, 2015 3:52 PM
  • I understand that and it as good info but not really a good solution.  Using Test-Path after a Get-ChildItem is silly.

    Why not try this:

    Get-ChildItem *.bak | Test-Path

    I guarantee it will always return true so the test serves no purpose.


    \_(ツ)_/

    • Marked as answer by adriannzde Wednesday, September 30, 2015 7:41 AM
    Tuesday, September 29, 2015 4:11 PM
  • Thanks jrv, this is the method I am now using.

    For the record here is the code block I will be using to check for the existence of a couple of items before the real work of the script Begins:

    $bkpath = "F:\Backup\"
    $chkfl = (Get-ChildItem -Path $bkpath -include *.trn,*.bak -Recurse -Force)
    
    $workdir = "F:\Backup\temp\" 
    $Zipfiledir = "F:\Backup\zipfiles"
    
    # Sanity test on variables
    $Zipfiledirexists = $Zipfiledir | Test-Path
    $workdirexists = $workdir | Test-Path
    
    if ($Zipfiledirexists -eq "True" -and $workdirexists -eq "True")
    {
    write-Host "Path variables are set to existing locations"
    }
    Else
    {
    write-host "Check your path variables are set correctly"
    exit
    }
    
    # Sanity test on existence of uncompressed backup files
    $backupsexist = $chkfl | Test-Path
    if ($backupsexist -eq "True")
    {
    write-Host "At least 1 uncompressed file found - ok to proceed"
    }
    Else
    {
    write-host "No backup files found - has SQL backup been run?"
    exit
    }

     


    • Edited by adriannzde Wednesday, September 30, 2015 8:11 AM
    Wednesday, September 30, 2015 8:09 AM
  • Yeah, you're right. But I wasn't implying the use of Test-Path for his test.

    Please remember to: "Vote" if my response was helpful. "Mark As Answered" if I answered your question correctly. V/R, Darrick West - Senior Systems Engineer, ConfigMgr: OSD


    Monday, October 5, 2015 10:29 AM
  • Your code involving   $chkfl  and   $backupsexist   is probably not doing what you intended.
    What do you think that the code involving   $chkfl  and $backupsexist  is doing?
    Monday, October 5, 2015 4:47 PM
  • I understand that and it as good info but not really a good solution.  Using Test-Path after a Get-ChildItem is silly.

    Why not try this:

    Get-ChildItem *.bak | Test-Path

    I guarantee it will always return true so the test serves no purpose.


    \_(ツ)_/

    andrianzde, why is that post by jrv marked as the "answer" ?  

    I don't think that jrv intended that as "working" code. 
    Monday, October 5, 2015 6:17 PM
  • I understand that and it as good info but not really a good solution.  Using Test-Path after a Get-ChildItem is silly.

    Why not try this:

    Get-ChildItem *.bak | Test-Path

    I guarantee it will always return true so the test serves no purpose.


    \_(ツ)_/

    andrianzde, why is that post by jrv marked as the "answer" ?  

    I don't think that jrv intended that as "working" code. 

    I intended to point pout that it serves no purpose.  It is just wasted code.

    \_(ツ)_/

    Monday, October 5, 2015 6:21 PM
  • Hi Larry,

    Thanks for your comment!

    what I _think_ I am doing with these 2 variables is checking for the existance of any backup files, and then with the conditional clause that comes directly after performing 1 set of actions if files are found - actually what I do is create a zip archive in the "real" script , or if no files are found do something else - in the real script a mail goes out telling me the backup job Needs to be checked.

    If at least 1  file is found , the $chkfl | test-path Returns "true"

    If not the test returns "false" .

    What am I missing here? - so far as I can see this test is doing exactly as I require.

    I can attach screen shots of the behavior of this test if you like.

    Wednesday, October 7, 2015 2:47 PM
  • But what you are not understanding is that the following is the same as TEst-Path.

    $chkfl = (Get-ChildItem -Path $bkpath -include *.trn,*.bak -Recurse -Force)

    We would just do this:

    if(Get-ChildItem -Path $bkpath -include *.trn,*.bak -Recurse -Force){
        # at least one file found
    }else{
        # no files fouond
    }

    Why do everything twice.


    \_(ツ)_/

    Wednesday, October 7, 2015 2:53 PM
  • Nope not working code, but the method seems correct for my purposes. Which is why I cut to the chase and marked it as the answer.

    ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
    PS F:\> $chkfl = (Get-ChildItem -Path $bkpath -include *.trn,*.bak -recurse) 

    ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
    PS F:\> $chkfl


        Directory: F:\Backup\FEL-NT73TEST$SQLEXPRESS\VIM_VCDB\FULL


    Mode                LastWriteTime     Length Name                                                                                                                                                                                                                 
    ----                -------------     ------ ----                                                                                                                                                                                                                 
    -a---        05.10.2015     13:27 3602673152 FEL-NT73TEST$SQLEXPRESS_VIM_VCDB_FULL_20151005_132550.bak                                                                                                                                                            

    ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
    PS F:\> $chkfl | test-path
    True

    ***

    at this point I rename the backup file , so that $chkf is now "null" ...

    ***

    PS F:\> $chkfl | test-path
    False

     this is exactly what I need: a simple true if I have at least 1 file or a false if I dont.

    Thanks very much

    Wednesday, October 7, 2015 2:54 PM
  • So you are just using it as a way of saving the name.  What happens if there is more than one file?


    \_(ツ)_/

    Wednesday, October 7, 2015 2:59 PM
  • Hi Larry,

    Thanks for your comment!

    what I _think_ I am doing with these 2 variables is checking for the existance of any backup files, and then with the conditional clause that comes directly after performing 1 set of actions if files are found - actually what I do is create a zip archive in the "real" script , or if no files are found do something else - in the real script a mail goes out telling me the backup job Needs to be checked.

    If at least 1  file is found , the $chkfl | test-path Returns "true"

    If not the test returns "false" .

    What am I missing here? - so far as I can see this test is doing exactly as I require.

    I can attach screen shots of the behavior of this test if you like.

    Thanks.  A better way to code that is this

    $bkpath = "F:\Backup\"
    $chkfl = @(Get-ChildItem -Path $bkpath -include *.trn,*.bak -Recurse -Force)
    if ($chkfl.count -gt 0)
    {
    write-Host "At least 1 uncompressed file found - ok to proceed"
    }
    Else
    {
    write-host "No backup files found - has SQL backup been run?"
    exit
    }
    

    Wednesday, October 7, 2015 4:03 PM
  • Note that the return of trest-path with multiole files is an array.  An array of falses wil still return true.

    $x=$false,$false
    if($x){'true'}else{'false'}

    Try that to see what I mean.

    The only way test path woks correctly is with a single file. Get-ChildItem will work correctly to detect that no files are left to process.

    Simple and basic logic and simple and basic PowerShell.  This is a major crux of learning how to program.  Much of what seems to work doesn't until you understand the basics and the internals.


    \_(ツ)_/

    Wednesday, October 7, 2015 4:09 PM
  • Hi Larry,

    Thanks for your comment!

    what I _think_ I am doing with these 2 variables is checking for the existance of any backup files, and then with the conditional clause that comes directly after performing 1 set of actions if files are found - actually what I do is create a zip archive in the "real" script , or if no files are found do something else - in the real script a mail goes out telling me the backup job Needs to be checked.

    If at least 1  file is found , the $chkfl | test-path Returns "true"

    If not the test returns "false" .

    What am I missing here? - so far as I can see this test is doing exactly as I require.

    I can attach screen shots of the behavior of this test if you like.

    Thanks.  A better way to code that is this

    $bkpath = "F:\Backup\"
    $chkfl = @(Get-ChildItem -Path $bkpath -include *.trn,*.bak -Recurse -Force)
    if ($chkfl.count -gt 0)
    {
    write-Host "At least 1 uncompressed file found - ok to proceed"
    }
    Else
    {
    write-host "No backup files found - has SQL backup been run?"
    exit
    }

    Larry -

    This is the best way to doo this in  PoowerShell


    if(Get-ChildItem -Path f:\backup -include *.trn,*.bak -Recurse -Force){
        # at least one file found
    }else{
        # no files fund
    }

    We can also do this when this construct has an advantage:

    while($files=Get-ChildItem -Path f:\backup -include *.trn,*.bak -Recurse -Force){
        # process files
    }



    \_(ツ)_/


    • Edited by jrv Wednesday, October 7, 2015 4:14 PM
    Wednesday, October 7, 2015 4:12 PM
  •  

    This is the best way to do this in  PowerShell


    if(Get-ChildItem -Path f:\backup -include *.trn,*.bak -Recurse -Force){
        # at least one file found
    }else{
        # no files fund
    }

     



    \_(ツ)_/


    I like that.  
    Wednesday, October 7, 2015 4:51 PM

  • We can also do this when this construct has an advantage:

    while($files=Get-ChildItem -Path f:\backup -include *.trn,*.bak -Recurse -Force){
        # process files
    }



    \_(ツ)_/


    You would need to make sure either that that loop eventually terminates, or add code to make it run "nicely" (not 100% CPU aggressive)
    Wednesday, October 7, 2015 4:59 PM

  • We can also do this when this construct has an advantage:

    while($files=Get-ChildItem -Path f:\backup -include *.trn,*.bak -Recurse -Force){
        # process files
    }



    \_(ツ)_/


    You would need to make sure either that that loop eventually terminates, or add code to make it run "nicely" (not 100% CPU aggressive)

    No.  Why would you think that.  It is just a loop.  When all files have been renamed or deleted or moved the loop will terminate.  As soon as there are no more files to process the loop will terminate.


    \_(ツ)_/

    Wednesday, October 7, 2015 5:06 PM

  • We can also do this when this construct has an advantage:

    while($files=Get-ChildItem -Path f:\backup -include *.trn,*.bak -Recurse -Force){
        # process files
    }



    \_(ツ)_/


    You would need to make sure either that that loop eventually terminates, or add code to make it run "nicely" (not 100% CPU aggressive)

    No.  Why would you think that.  It is just a loop.  When all files have been renamed or deleted or moved the loop will terminate.  As soon as there are no more files to process the loop will terminate.


    \_(ツ)_/

    There is  a chance that a file collected in the loop will be locked by another process and therefore not moveable.   I just recreated that scenario with a version of the UNIX-like  less  utility that was browsing the file when I ran your version of the script.   Not pretty!
    Wednesday, October 7, 2015 7:20 PM
  • I said "when this construct has an advantage:".  It will have to be altered for every scenario..  However, you are missing the point.  We can  use a file list to maintain a queue and process things out of the queue.  It is a fairly common design pattern.

    See: https://en.wikipedia.org/wiki/Software_design_pattern


    \_(ツ)_/

    Wednesday, October 7, 2015 7:49 PM
  • I said "when this construct has an advantage:".  It will have to be altered for every scenario..  However, you are missing the point.  We can  use a file list to maintain a queue and process things out of the queue.  It is a fairly common design pattern.

    See: https://en.wikipedia.org/wiki/Software_design_pattern


    \_(ツ)_/

    I might be missing your point, but you most certainly are not acknowledging my point.

    And it is hardly a queue if you need to regenerate it on every iteration.

    Wednesday, October 7, 2015 8:35 PM
  • I didn't say it is a queue.  I said it was like a queue.

    Who made the rule that a queue cannot be refreshed?

    I think you ae making more of this than exist.  The point s you can use the construct to continue/end a loop.  The rest is up too the coder and the context.

    I am not nor was I not acknowledging your point.  I was just replying that it has nothing to do with the purpose of the suggestion.  It goes without saying, at lest for programmers, that loops have to be deterministic in some way.  Basic loops 101.


    \_(ツ)_/

    Wednesday, October 7, 2015 8:42 PM
  • I didn't say it is a queue.  I said it was like a queue.

    Who made the rule that a queue cannot be refreshed?

    I think you ae making more of this than exist.  The point s you can use the construct to continue/end a loop.  The rest is up too the coder and the context.

    I am not nor was I not acknowledging your point.  I was just replying that it has nothing to do with the purpose of the suggestion.  It goes without saying, at lest for programmers, that loops have to be deterministic in some way.  Basic loops 101.


    \_(ツ)_/

    OK - your "Basic loops 101" should cover it.   Some coders don't consider the effects of the "outside world" on their code.  In this case it takes an expanded viewpoint to realize that other processes might interfere with the mission.
    Wednesday, October 7, 2015 11:17 PM
  • I think I'll take a hiatus from commenting on your posts.
    There is a "Principal of Charity" that I expect that is missing.
    See https://en.wikipedia.org/wiki/Principle_of_charity
    And you probably think that I'm not honoring that principle either, so I'm taking a break.
    Wednesday, October 7, 2015 11:21 PM
  • I think I'll take a hiatus from commenting on your posts.
    There is a "Principal of Charity" that I expect that is missing.
    See https://en.wikipedia.org/wiki/Principle_of_charity
    And you probably think that I'm not honoring that principle either, so I'm taking a break.

    No Larry. I generally appreciate your comments however, this time, you seemed to be changing the subject from the use of the pattern to a lesson in loops.  That was not my intent for the post.

    Don't worry about it.  It is not a big deal.  I apologize if it seemed that I was being negative or too critical.


    \_(ツ)_/


    • Edited by jrv Wednesday, October 7, 2015 11:33 PM
    Wednesday, October 7, 2015 11:33 PM
  • Hello Larry and jrv,

    I for one have learned something every step through this discussion. It would be a shame if you stopped commenting Larry, for me its like watching you and jrv play tennis :-)

    Coming from a non-programming background as I do I its the basics like running loops and pattern matching I need to learn to take the step up from simply running commands one after another and calling it a script and to actually having a small amount of decision making along the way.

    So I appreciate every one of your comments on this thread, thanks for helping me get some basic concepts seated. 

    I'll need a day or two before I can try out the coding suggestions made in this thread, but I certainly will, and I'll report back on progress.

    regards Adrian

    Thursday, October 8, 2015 6:40 AM
  • Nice feedback Adrian.

    A little debate can be illuminating.

    Larry - be glad I am not Donald The Trumpster.  At least our debate had substance.

    Ciao all.


    \_(ツ)_/

    Thursday, October 8, 2015 11:01 AM