locked
Using Powershell to download from FTP site, file name has wildcard RRS feed

  • Question

  • In Powershell I am able to upload multiple files to an FTP site using a wildcard but unable to download files using a wildcard. 

    Below is the code to upload multiple files and it works fine:

    #Directory where files are located 
    $FromDir="<path here>"   

    #ftp server info 
    $ftp = "<ftp address>"
    $user = "<username>"
    $pass = "<password>" 
     
    $webclient = New-Object System.Net.WebClient
    $webclient.Credentials = New-Object System.Net.NetworkCredential($user,$pass) 
    $YestDate = (get-date).AddDays(-1)
    $FileNameDate = date $YestDate -f yyyyMMdd

    #Pick up files
    foreach($item in (dir $FromDir "FTP_*$FileNameDate.txt")){
        "Uploading $item..."
        $uri = New-Object System.Uri($ftp+$item.Name)
        $webclient.UploadFile($uri, $item.FullName)
     }

    #Download files - Below code works but I need to use a wildcard in the file name as it changes and I can't figure out how to do that.

    $target = "pc name\C$\FTP\Download\Notes.txt"
    $source =  "<ftp address>/Sample_file.txt"

    $WebClient.DownloadFile($source, $target)

    Thanks in advance for any assistance!

    Friday, October 25, 2013 7:11 PM

Answers

  • I agree with Mike. FTP is not able to handle the wildcards itself. In the upload scenario PowerShell resolves the pathes before they are used by the WebClient. If you pull a directory listing and filter the files to download from PowerShell you should be able to achieve what you want:

    function Get-FtpDir ($url,$credentials) {
        $request = [Net.WebRequest]::Create($url)
        $request.Method = [System.Net.WebRequestMethods+FTP]::ListDirectory
        if ($credentials) { $request.Credentials = $credentials }
        $response = $request.GetResponse()
        $reader = New-Object IO.StreamReader $response.GetResponseStream() 
    	$reader.ReadToEnd()
    	$reader.Close()
    	$response.Close()
    }
    $folderPath="ftp://folder/"
    $files=Get-FTPDir $folderPath
    
    $target = "c:\folder"
    Foreach ($file in ($files | where {$_ -like "FTP_*"})){
    	$source=$folderPath+$file 
    	$WebClient.DownloadFile($source, $target)
    }
    
    

    • Proposed as answer by jrv Friday, October 25, 2013 10:22 PM
    • Marked as answer by IamMred Wednesday, October 30, 2013 2:21 AM
    Friday, October 25, 2013 9:03 PM

All replies

  • Hi,

    Are you able to query the list of files available on the FTP site before starting the download? If so, get the file list and check for which file(s) you need to download. Once you have that, you can use DownloadFile to fetch the file(s) you need.


    Don't retire TechNet! - (Maybe there's still a chance for hope, over 12,225+ strong and growing)

    Friday, October 25, 2013 7:57 PM
  • I agree with Mike. FTP is not able to handle the wildcards itself. In the upload scenario PowerShell resolves the pathes before they are used by the WebClient. If you pull a directory listing and filter the files to download from PowerShell you should be able to achieve what you want:

    function Get-FtpDir ($url,$credentials) {
        $request = [Net.WebRequest]::Create($url)
        $request.Method = [System.Net.WebRequestMethods+FTP]::ListDirectory
        if ($credentials) { $request.Credentials = $credentials }
        $response = $request.GetResponse()
        $reader = New-Object IO.StreamReader $response.GetResponseStream() 
    	$reader.ReadToEnd()
    	$reader.Close()
    	$response.Close()
    }
    $folderPath="ftp://folder/"
    $files=Get-FTPDir $folderPath
    
    $target = "c:\folder"
    Foreach ($file in ($files | where {$_ -like "FTP_*"})){
    	$source=$folderPath+$file 
    	$WebClient.DownloadFile($source, $target)
    }
    
    

    • Proposed as answer by jrv Friday, October 25, 2013 10:22 PM
    • Marked as answer by IamMred Wednesday, October 30, 2013 2:21 AM
    Friday, October 25, 2013 9:03 PM
  • I understand what you are saying but for some reason I'm not able to get my code working based on the below, can you see what I might be doing wrong?  My script isn't erroring  but I'm second-guessing if I'm actually connecting.  I had to use the webclient in the Get-FtpDir otherwise the script would fail.  Thanks!

    $url = "ftp://##.#.#.#/"
    $user = "username"
    $pass = "123456" 
    $credentials = new-object System.Net.NetworkCredential($user, $pass)

    $webclient = New-Object System.Net.WebClient
    $webclient.Credentials = New-Object System.Net.NetworkCredential($user,$pass) 


    $target = "\\computername\C$\FTP\Download\"


    function Get-FtpDir ($url,$webclient) {
        $request = [Net.WebRequest]::Create($url)
        $request.Method = [System.Net.WebRequestMethods+FTP]::ListDirectoryDetails
        if ($credentials) { $request.Credentials = $credentials }
        $response = $request.GetResponse()
        $reader = New-Object IO.StreamReader $response.GetResponseStream()
     $reader.ReadToEnd()
     $reader.Close()
     $response.Close()
    }
    $folderPath="ftp://##.#.#.#/folder/folder/"
    $files=Get-FTPDir $folderPath

    Foreach ($file in ($files | where {$_ -like "Test_*.txt"})){
     $source=$folderPath+$file
     $WebClient.DownloadFile($source, $target)
    }


    CINDI

    Monday, October 28, 2013 3:38 PM
  • I added [System.Console]::Writeline($reader.ReadToEnd()) to the function and was able to confirm that I am connecting and I can see the list of files available.  But, that's as far as I can get, I am not able to download any files.

    Thanks


    CINDI

    Monday, October 28, 2013 5:42 PM
  • Hi,

    if $files contains the list of files from the directory the next thing you should check if whether the where clause meets your requirements. Do the files you want to download all start with Test_ and end in .txt? Does the code below return anything. Otherwise you'll need to adjust according to your needs.

    Foreach ($file in ($files | where {$_ -like "Test_*.txt"})){
         $folderPath+$file
    }

    Monday, October 28, 2013 7:15 PM
  • Yes, the files start with Test_ and end in .txt.  The code doesn't return anything.

    CINDI

    Monday, October 28, 2013 7:34 PM
  • The issue seems to be that it's not truly reading the files.  I changed the code to:

    $folderPath="ftp://##.#.#.#/folder/"
    $files=Get-FTPDir $folderPath

    Foreach ($file in ($files )){
        $source=$folderPath+$file
     [System.Console]::Writeline($source)

    }

    The output of the $source ends up just being the folderPath not the filename.


    CINDI

    Monday, October 28, 2013 7:58 PM
  • Ok, after more troubleshooting I realized that the $file is reading the filenames on the server with all the properties.  For example when I output the $file I get:

    10-28-13  01:58PM                   31 Test_Notes_20131028.txt

    So, in my query if I put

    Foreach ($file in ($files | Where {$_ -like "*Test*"})){

    It now see's the files because the wildcard before Test, it thinks the file starts with the10-28-13.  It still doesn't do the Download though and I'm guessing it's because of the way it is reading the file name.


    CINDI

    Monday, October 28, 2013 8:21 PM
  • if you change the following within Get-FTPDir it will only return the file names. That was actually my initial intention I just forgot about to change it back after playing with it:

    $request.Method = [System.Net.WebRequestMethods+FTP]::ListDirectoryDetails
    
    #to
    $request.Method = [System.Net.WebRequestMethods+FTP]::ListDirectory

    then you should get some output here:

    Foreach ($file in ($files | where {$_ -like "Test_*.txt"})){
     $source=$folderPath+$file
     #$WebClient.DownloadFile($source, $target)
    }
    if you get the right files back you can un-comment the line that starts with $webclient

    Monday, October 28, 2013 8:33 PM
  • Thank you everyone for your assistance, some really great input!

    Unfortunately, I was still not able to get it working, even if it saw the file list it didn't loop correctly so the download would crash.  I ended up taking a different approach.  The multiple files I need to download have the same base name each day, the only thing that changes in the file name is the date. So, I created unique source variables for each filename and ran the download command multiple times.  Example:

    $WebClient.DownloadFile($sourcebd, $targetbd)
    $WebClient.DownloadFile($sourcepre, $targetpre)
    $WebClient.DownloadFile($sourcenotes, $targetnotes)

    This seems to be working fine.


    Thanks again!


    CINDI

    Tuesday, October 29, 2013 6:04 PM
  • Quick note on this. Overall this is a great script. I had to tweak one thing in order to get the foreach loop to work. `$files` didn't parse out into an array nicely for the foreach loop to work.

    $files = Get-FTPDir $folderPath | Out-String

    $files = $files.replace("zip","zip,")

    $array = $files -split ","


    Thursday, March 6, 2014 11:07 PM
  • Just in case anybody comes across this, if you're trying to specify a directory in the `$folderPath`, for instance:

    $folderPath="ftp://10.10.10.10/home/unixuser/something"

    That will not work, you must escape the first slash like so:

    $folderPath="ftp://10.10.10.10/%2fhome/unixuser/something"

    Took me a good 30 minutes or more just to figure this out (yikes!), hope this helps someone else out!

    Source (scroll down to remarks):
    http://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest(v=vs.110).aspx

    Monday, November 10, 2014 11:31 PM
  • See my reply above, you can't use the actual folder path in the path without escaping, so this

    $folderPath="ftp://##.#.#.#/folder/folder/"

    Should be this:

    $folderPath="ftp://##.#.#.#/%2ffolder/folder/"
    That is, for absolute path, otherwise if you dont add %2f it will be relative to the FTP directory

    Monday, November 10, 2014 11:34 PM
  • Taking what snoop89 did, I had to do some additional finagling of the Get-FtpDir output to make the foreach loop work.
        $files=Get-FTPDir $folderPath | out-string
        #need to format the output since this all comes back as a blob of data
        #remove LFCR
        $files = $files.replace("`r",",")
        $files = $files.replace("`n","")
        #remove trailing commas
        $files = $files.trimend(",")
        #split values into an array
        $array = $files -split ","
        #write-output $files
        #write-output $array

    Foreach ($file in ($array | where {$_ -like "Test*.txt"})){

    ....

    Wednesday, December 9, 2015 10:01 PM