none
Get-ChildItem files, includes, literalpath and square brackets

    General discussion

  • It seems like I'm dealing with a worst case scenario for GCI here. I'm trying to create a foreach loop to rename files with specific extensions after the folder in which they reside. However, I don't seem to be able to successfully return the list of files I want to change, correctly. The extra challenge here is that there are square brackets in the folder names.

    The closest I've managed is:

    gci -File -include('*.jpg','*.jpeg','*.png','*.gif') -Recurse -LiteralPath ".\Test `[blah`]\"

    This almost works, but I end up getting all of the files; the Include doesn't seem to work. I've read this might be a side effect of  the way the filter works, and appending a * to the Path might return the filtered collection I want. Except that the square brackets in the folder name appear to break gci if I don't use -LiteralPath, and -LiteralPath means I can't append the * and have it expand as wildcard properly.

    So....I'm out of ideas at the moment. It's starting to feel like maybe I'll have to apply another filter on a return set of all the files, and just live with it, but I'm hoping someone can show me something I missed to return just the file set I want.

    Hopefully I explained that in an way that is understandable.

    Thank you!

    • Changed type jrvModerator Thursday, July 12, 2018 7:23 PM Not a question.
    Thursday, June 14, 2018 5:48 PM

All replies

  • Get-ChildItem  .\Test\blah -include *.jpg,*.jpeg,*.png,*.gif -File -Recurse

    \_(ツ)_/


    Thursday, June 14, 2018 6:36 PM
    Moderator
  • That's not my path though. There are square brackets in the folder name, hence the need for -LiteralPath.

    The rest of that would imply the order of the switches matters? As does specifying the -Include items in parens vs just as a comma separated list?

    Friday, June 15, 2018 3:57 PM
  • You should never use those characters in a folder name.  They cannot be escaped.


    \_(ツ)_/

    Friday, June 15, 2018 6:00 PM
    Moderator
  • How about instead of -include:

     | where name -match 'jpg$|jpeg$|png$|gif$'



    • Edited by JS2010 Friday, June 15, 2018 11:37 PM
    Friday, June 15, 2018 11:37 PM
  • How about instead of -include:

     | where name -match 'jpg$|jpeg$|png$|gif$'



    The "Include" works fine.  It is the incorrect path characters that are the issue.


    \_(ツ)_/

    Friday, June 15, 2018 11:53 PM
    Moderator
  • I disagree.  -include in PS 5 only works if there's a wildcard in the path, or with the -recurse option.  It's resolved in PS 6.

    Saturday, June 16, 2018 9:06 PM
  • I disagree.  -include in PS 5 only works if there's a wildcard in the path, or with the -recurse option.  It's resolved in PS 6.

    Yes which is why we add the options.  If we use a post filter (Where) the performance will be very bad on large file systems.

    The real issue here is the illegal characters in the path name.


    \_(ツ)_/

    Saturday, June 16, 2018 9:26 PM
    Moderator
  • Using the "Where" may be the only solution.


    \_(ツ)_/

    Saturday, June 16, 2018 9:34 PM
    Moderator
  • Yes - This does seem to work in all cases:

    Get-Item d:\test?junk?1\* | where{$_.Extension -match 'ps1|txt'}


    \_(ツ)_/

    Saturday, June 16, 2018 9:36 PM
    Moderator
  • You should never use those characters in a folder name.  They cannot be escaped.


    \_(ツ)_/

    if they should never be used, the should be disallowed in the file system, just like ?, *, and a number of others. Powershell being unable to cope with them well is a terrible reason to not use a character allowed by the file sytem. What if it couldn't handle the letter S?

    But at least that tells me there's not a simple workaround. I ended up adding a switch statement and just processing the extensions myself, it works well enough and is fast enough it's not an issue.

    Thanks

    Tuesday, June 19, 2018 8:01 PM

  • The real issue here is the illegal characters in the path name.


    \_(ツ)_/

    There's nothing illegal about the characters. They are perfectly fine in NTFS. They are perfectly fine in File Explorer. The problem is Powershell can't manage them. Don't blame a weakness of Powershell on the file system.

    Tuesday, June 19, 2018 8:04 PM
  • THe characters are not allowed by the Nwt Framework or the current version of NTFS.  Much of the older documentation fails to address this:

    Here is what the Net Framework reports.

    PS D:\scripts> [System.IO.Directory]::GetFiles('\test[me]now\*')
    Exception calling "GetFiles" with "1" argument(s): "Illegal characters in path."
    At line:1 char:1
    + [System.IO.Directory]::GetFiles('\test[me]now\*')
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
        + FullyQualifiedErrorId : ArgumentException

    Note that the message is "Illegal characters in path."

    The issue has always been a problem and it is an issue with the documentation and not PowerShell.

    Using the file/folder creation API calls does allow all characters to be used.  There is no protection.  Explorer and other utilities can access these paths but use a different method of identifying objects in the file system.

    Avoid using all special characters in  file/folder names to avoid these issues.

    Again. It is not a PowerShell issue but is an artifact of the design choices for NTFS and the high level APIs that access it.

    Things like long file paths have also always been possible and are accessible via Explorer but will not work in most file system APIs witout special handling.

    There is a good book on the NTFS.  Look it up if you want more background on these odd and, at time, silly behaviors of NTFS.


    \_(ツ)_/

    Tuesday, June 19, 2018 8:23 PM
    Moderator
  • I should also have noted that using "LiteralPath" will allow the characters but it will not allow the correct use of "Exclude" and "Include".  That is because the literal version of the path cannot be used as a filter in the API if the path includes illegal characters. 


    \_(ツ)_/

    Tuesday, June 19, 2018 8:48 PM
    Moderator
  • I suspect because square brackets are wildcard characters:

    get-childitem [a-c]*




    • Edited by JS2010 Tuesday, June 19, 2018 8:53 PM
    Tuesday, June 19, 2018 8:53 PM
  • I suspect because square brackets are wildcard characters:

    get-childitem [a-c]*




    Yes.  In a "Path" statement and in the underlying file API. "LiteralPath" causes the path to suppress wild card characters and see all characters as presented or typed.   Passing a pathname with those characters to the internal filter sees then as wild card specifications; not literal characters.


    \_(ツ)_/

    Tuesday, June 19, 2018 9:17 PM
    Moderator
  • THe characters are not allowed by the Nwt Framework or the current version of NTFS.  Much of the older documentation fails to address this:


    Ok, that's fair. However, Powershell seems like it's as culpable as anyone else then, because this works:

    # Results in a directory named [blah]
    New-Item -ItemType Directory [blah]
    
    It will let me create an item that you say has disallowed characters. Maybe it shouldn't do that? Or acknowledge the fundamental reality here, that in practice square brackets have to be valid because they used to be valid. Or Microsoft needs to bite the bullet and break systems that have characters that are now considered illegal. You don't get to have it both ways. The characters are either allowed or they aren't. If you keep letting me make new ones, they are allowed, and you are perpetuating the problem. I had no reason to think that they were disallowed and I should stop using them because nothing stops me from making new ones.

    Monday, June 25, 2018 3:02 PM
  • Incidentally, do you have linkage to documentation that indicates this? I'm having trouble locating any, which is unsurprising based on your supposition.

    Monday, June 25, 2018 3:05 PM
  • These work for me:

    mkdir "[hi]"

    [System.IO.Directory]::CreateDirectory('[hi]')






    • Edited by JS2010 Monday, June 25, 2018 3:34 PM
    Monday, June 25, 2018 3:33 PM
  • These work for me:

    mkdir "[hi]"

    [System.IO.Directory]::CreateDirectory('[hi]')







    The APIs all allow using characters that can cause issues. 


    \_(ツ)_/

    Monday, June 25, 2018 3:56 PM
    Moderator
  • Incidentally, do you have linkage to documentation that indicates this? I'm having trouble locating any, which is unsurprising based on your supposition.

    Years of experience.  I have never saved links to discussions about this.   The issue has been repeatedly debated.  If you search long enough you will find more information. 


    \_(ツ)_/

    Monday, June 25, 2018 3:57 PM
    Moderator
  • Note that many characters that are allowed (not restricted or won't cause an error) are unusable.  It is possible to create a name with a null but the name will be unusable.

    There is a difference between "not allowed" and "not usable".  The MS explanation is usually that the API has to support multiple file systems. It is the file system that determines usability and not the API.  For example, when accessing a MAC file system on a share you can easily create file names from the MAC that are unusable from Windows; the reverse is also possible.


    \_(ツ)_/

    Monday, June 25, 2018 4:07 PM
    Moderator
  • Note that many characters that are allowed (not restricted or won't cause an error) are unusable.  It is possible to create a name with a null but the name will be unusable.

    There is a difference between "not allowed" and "not usable".  The MS explanation is usually that the API has to support multiple file systems. It is the file system that determines usability and not the API.  For example, when accessing a MAC file system on a share you can easily create file names from the MAC that are unusable from Windows; the reverse is also possible.



    Your example is some what irrelevant because of what you said, a null character actually creates an unusable filename. Square brackets in file names are largely usable everywhere. They don't produce an un-usuable state for most things. Powershell is the first thing I've had a problem with them, which is why I was confused. And even then, they only create a somewhat unusuable state in Powershell, since I was able to work around it within Powershell.

    Square brackets have been, in my experience, until now, both allowed and usable, and until now, I've never had any reason to think otherwise. Frankly, in the absence of documentation(which I haven't found so far), I really don't have any reason to think that they aren't allowed. I think I could still make a solid argument that it's a Powershell problem, until I can see Microsoft documentation that says otherwise. I get that contextually, they have some special use, but so do backslashes. But unlike a backslash, I can't create a filename with a slash. Nothing in Windows lets me, and I consider that an invalid character for that use, because it's enforced that it's one of the invalid characters for that use.

    This is more like declaring a bug as working as intended so there's nothing to fix, and just blaming the user for doing something that someone said isn't allowed but you can still do. I get that this isn't your fault, but it does seem like something Microsoft should either fix by really disallowing things, or acknowledging the reality they created, and dealing with the fact that they are actually allowed.


    Monday, July 9, 2018 7:04 PM
  • The APIs all allow using characters that can cause issues. 


    \_(ツ)_/

    Not to belabor the point, but usually, if an API allows something it's classified as one of two things:

    1) Feature/Working as intended

    2) Bug

    It should either be considered supported, or a bug. From what you are saying, it's a massive, system wide, system pervasive bug.

    Monday, July 9, 2018 7:08 PM
  • The Windows API clearly documents all of this.  Why are you continuing to complain.  If you insist it is a bug then post it in "UserVoice" and not here.  (it has been posted there a number of times).

    I suggest starting here to gain an understanding of the subtleties of NTFS: https://www.amazon.com/Windows-File-System-Internals-Developers/dp/9351101045/ref=sr_1_4?s=books&ie=UTF8&qid=1531163602&sr=1-4&keywords=windows+nt+file+system+internals

    And here: https://www.amazon.com/Developer-Reference-Windows-Internals-Book/dp/B072PC3MCX/ref=sr_1_8?s=books&ie=UTF8&qid=1531163602&sr=1-8&keywords=windows+nt+file+system+internals


    \_(ツ)_/

    Monday, July 9, 2018 7:15 PM
    Moderator
  • There is some kind of a bug here in PS 5.  I've always had bad luck with -include and -exclude.  With -path, it works ok.  And -recurse is required for -include.  In PS 6, this works as expected.  '[' and ']' are .net wildcard characters.  It is what it is.


    PS C:\Users\admin> Get-ChildItem -include *.jpg -literalPath foo -Recurse

        Directory: C:\Users\admin\foo

    Mode                LastWriteTime         Length Name
    ----                -------------         ------ ----
    d-----        6/28/2018   5:25 PM                foo2
    -a----        6/19/2018  12:34 PM             10 file
    -a----         7/9/2018   4:25 PM             10 file.jpg
    -a----        6/28/2018   5:22 PM             10 file2


    PS C:\Users\admin> [wildcardpattern]::ContainsWildcardCharacters('[')
    True
    PS C:\Users\admin> [wildcardpattern]::ContainsWildcardCharacters(']')
    True
    • Edited by JS2010 Thursday, July 12, 2018 10:26 PM
    Monday, July 9, 2018 8:33 PM
  • The Windows API clearly documents all of this.  Why are you continuing to complain.  If you insist it is a bug then post it in "UserVoice" and not here.  (it has been posted there a number of times).


    I'm not complaining, I'm responding to your replies.  I asked for documentation. You shrugged and said it was word of mouth. Now you are saying it's in the Windows API. If you would have said that when I asked for documentation that calls it out, I would have had a better place to narrow my search to. When I tried to find it before, based on your claims, I came up empty handed. You should have led with that information instead of waiting until now, after making a bunch of statements of fact but unable to provide any kind of place to back it up. If I'm complaining about anything, it's about how you've been presenting the information.

    Having said all that, it does sound like a bug or documentation issue. I appreciate that this is the wrong place for that, and since you are saying I'm far from the only person that's had issue with it, I'll head over to UserVoice and see what's there, so thank you for that part. Please don't take it personally, I'm just trying to understand it, and it wasn't always clear to me that what I was trying to do really should be considered a problem on my end.

    • Edited by MasterChiefmas Thursday, July 12, 2018 6:16 PM clarification
    Thursday, July 12, 2018 6:12 PM