none
Powershell ISE vs [Console]::OutputEncoding

    Question

  • Hello guys,

    I have already went through multiple articles and still cannot find answer. Example:
    http://stackoverflow.com/questions/22349139/utf-8-output-from-powershell
    http://blogs.msdn.com/b/powershell/archive/2006/12/11/outputencoding-to-the-rescue.aspx
    https://bazaarbytes.com/blog/beware-powershell-ise-powershellexe

    What I am trying to do:

    • create in.txt in notepad++ with UTF8 encoding (no BOM). The content of file is (special character is alt + 753951):
      • hello ▼ word
    • run following command in powershell ISE (Win8.1 x64, powershell 4.0, [Console]::OutputEncoding and $OutputEncoding returns BodyName: iso-8859-2):
      • findstr hello in.txt | Out-File -FilePath out.txt -Encoding utf8
    • special character in out.txt is broken and that is something I want to avoid

    How I was able to resolve it:

    • close and open powershell ISE again and run following command:
      • ping | Out-Null
      • [Console]::OutputEncoding = [System.Text.Encoding]::UTF8
      • findstr hello in.txt | Out-File -FilePath out.txt -Encoding utf8
    • this works, special character in out.txt is not broken

    What I do not understand:

    • I close and open powershell ISE again and run following command: 
      • [Console]::OutputEncoding = [System.Text.Encoding]::UTF8
      • findstr hello in.txt | Out-File -FilePath out.txt -Encoding utf8
    • again out.txt gets broken character. Moreover I get following error:
      • Exception setting "OutputEncoding": "The handle is invalid.
      • At C:\script.ps1:1 char:1
      • + [Console]::OutputEncoding = [System.Text.Encoding]::UTF8
      • ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      • + CategoryInfo          : NotSpecified: (:) [], SetValueInvocationException
      • + FullyQualifiedErrorId : ExceptionWhenSetting

    If I run the same command from powershell console, I get no error. If I run the same command from Win7 x64 machine through powershell ISE (powershell 2.0) I get no error. Do you have any idea how avoid this error or why I am getting it? I do not want to be calling fake ping or other command at the beginning of my script.

    Thank you.


    • Edited by garwel Sunday, March 01, 2015 10:13 AM
    Sunday, March 01, 2015 12:32 AM

Answers

  • The ping was just an example. I tried to demonstrate that my command to set a console was working only if I launched some application before that.

    I am working at CSS, I am analysing log files. Imagine situation where you have 100 log files with size 10MB. You need to go through all of these log files and do following:

    1. Find all lines that contain some string (array of strings) in any of these files
    2. Write all lines found in point 1 to a single file and sort them out per time. Content of files must not be broken as you may need to send the output of this file for further processing (manual or automatic) or just copy some of lines and search them in input files.
    3. The fast speed of processing is also one of requirements.

    Now although I know that select-string is pretty capable of that, per my tests it is about 2-3 times slower than findstr or findstr in combination with get-content... With multiple big files we are talking about minutes of difference (idea also supported by one of articles linked in my second comment). That is why I turned to findstr - to gain the speed. The quality is still there, I do not need objects at this point, encoding work just fine when I can set it for console. And as I am writing my powershell code in powershell ISE, I started to wonder why I am getting that error about console configuration.

    Thanks to your comment I did some more tests with procmon utility and I now understand that when I launch Powershell ISE, I do not have console running (well, some commands such as [Console]::BackgroundColor does return an output but it might be just some predefined value) and thus running my command [Console]::OutputEncoding = [System.Text.Encoding]::UTF8 ends with failure as there is no console for which I could apply this. When I launch an external application (ping, findstr...), the first thing powershell_ise.exe does is it launches its own console (conhost.exe) and then it starts those external applications such as ping. So from this moment on Powershell ISE has its console running and I can configure it the way I want -> any application I call afterwards will use the settings of the console I just set.
    I also found that powershell.exe launches conhost.exe right after it starts and that is the reason why I can call [Console]::OutputEncoding = [System.Text.Encoding]::UTF8 withing powershell.exe without any error.

    Just btw, majority of properties for [Console]:: returns null value before I launch external application such as ping, another indication conosle is really not initialized at the start:
    $objTemp = New-Object System.Object        
    $objTemp | Add-Member -type NoteProperty -name BackgroundColor -value ([Console]::BackgroundColor
    $objTemp | Add-Member -type NoteProperty -name BufferHeight -value ([Console]::BufferHeight
    $objTemp | Add-Member -type NoteProperty -name BufferWidth -value ([Console]::BufferWidth
    $objTemp | Add-Member -type NoteProperty -name CapsLock -value ([Console]::CapsLock
    $objTemp | Add-Member -type NoteProperty -name CursorLeft -value ([Console]::CursorLeft
    $objTemp | Add-Member -type NoteProperty -name CursorSize -value ([Console]::CursorSize
    $objTemp | Add-Member -type NoteProperty -name CursorTop -value ([Console]::CursorTop
    $objTemp | Add-Member -type NoteProperty -name CursorVisible -value ([Console]::CursorVisible
    $objTemp | Add-Member -type NoteProperty -name Error -value ([Console]::Error
    $objTemp | Add-Member -type NoteProperty -name ForegroundColor -value ([Console]::ForegroundColor
    $objTemp | Add-Member -type NoteProperty -name In -value ([Console]::In
    $objTemp | Add-Member -type NoteProperty -name InputEncoding -value ([Console]::InputEncoding
    $objTemp | Add-Member -type NoteProperty -name IsErrorRedirected -value ([Console]::IsErrorRedirected
    $objTemp | Add-Member -type NoteProperty -name IsInputRedirected -value ([Console]::IsInputRedirected
    $objTemp | Add-Member -type NoteProperty -name IsOutputRedirected -value ([Console]::IsOutputRedirected
    $objTemp | Add-Member -type NoteProperty -name KeyAvailable -value ([Console]::KeyAvailable
    $objTemp | Add-Member -type NoteProperty -name LargestWindowHeight -value ([Console]::LargestWindowHeight
    $objTemp | Add-Member -type NoteProperty -name LargestWindowWidth -value ([Console]::LargestWindowWidth
    $objTemp | Add-Member -type NoteProperty -name NumberLock -value ([Console]::NumberLock
    $objTemp | Add-Member -type NoteProperty -name Out -value ([Console]::Out
    $objTemp | Add-Member -type NoteProperty -name OutputEncoding -value ([Console]::OutputEncoding
    $objTemp | Add-Member -type NoteProperty -name Title -value ([Console]::Title
    $objTemp | Add-Member -type NoteProperty -name TreatControlCAsInput -value ([Console]::TreatControlCAsInput
    $objTemp | Add-Member -type NoteProperty -name WindowHeight -value ([Console]::WindowHeight
    $objTemp | Add-Member -type NoteProperty -name WindowLeft -value ([Console]::WindowLeft
    $objTemp | Add-Member -type NoteProperty -name WindowTop -value ([Console]::WindowTop
    $objTemp | Add-Member -type NoteProperty -name WindowWidth  -value ([Console]::WindowWidth

     
    So I think that clears out the issue I had and the topic can be closed.


    • Marked as answer by Bill_StewartModerator Tuesday, March 03, 2015 3:11 PM
    • Edited by garwel Wednesday, February 17, 2016 9:32 AM a few typos (select-string instead of findstr)
    Monday, March 02, 2015 5:15 PM

All replies

  • Why are you using findstr instead of select-string cmdlet?

    -- Bill Stewart [Bill_Stewart]

    Sunday, March 01, 2015 5:53 PM
    Moderator
  • I find findstr much faster than select-string. 

    ## txt file with 20MB size

    Measure-Command { Select-String -Path something.txt -Pattern 'Invalid/Unknown' -SimpleMatch | Select-String -Pattern 'TerminalListener' } ## Seconds : 1 ## Milliseconds : 760 Measure-Command { Select-String -Path something.txt -Pattern 'Invalid/Unknown' | Select-String -Pattern 'TerminalListener' } ## Seconds : 0 ## Milliseconds : 214 Measure-Command {findstr 'Invalid/Unknown' something.txt | findstr 'TerminalListener'} ## Seconds : 0 ## Milliseconds : 58


    I could probably get speed with Powershell as well (http://blogs.msdn.com/b/fsharpteam/archive/2012/10/03/rethinking-findstr-with-f-and-powershell.aspx) but it seems to complex for me so far.


    • Edited by garwel Sunday, March 01, 2015 6:30 PM
    Sunday, March 01, 2015 6:30 PM
  • So the question then, is: Does findstr support the encoding you want to use?


    -- Bill Stewart [Bill_Stewart]

    Sunday, March 01, 2015 9:14 PM
    Moderator
  • I think so as running "findstr hello in.txt > out.txt" in command line creates valid output. I am not sure whether the point of my enquiry is clear. What I complain about is that Powershell ISE reports error when I call "[Console]::OutputEncoding = [System.Text.Encoding]::UTF8" before calling any external application such as ping or others. If I call ping and then call "[Console]::OutputEncoding = [System.Text.Encoding]::UTF8", no error is reported and then everything else works as expected.

    Why does Powershell ISE report this error?

    Monday, March 02, 2015 12:26 PM
  • PowerShell ISE is not a console program. It is a GUI program.


    -- Bill Stewart [Bill_Stewart]

    Monday, March 02, 2015 1:39 PM
    Moderator
  • Thanks for responses Bill. I a partially understand that what I am trying to do is a bit weird as e.g. mentioned here: http://blogs.msdn.com/b/powershell/archive/2009/04/17/differences-between-the-ise-and-powershell-console.aspx

    But why:
    1/ Calling [Console] command before launching ping fails while it does not fail if calling it after launching ping? What does change in Powershell ISE that forces it to work? Is there possibly some powershell command I can call to "initialize" that console part?
    2/ Why is this error shown only on PS ISE within Win 8.1 and not on PS ISE within Win 7 where it works just fine?
    3/ Would you know any other way how to call findstr from powershell ISE and get valid output without broken encoding? So far I know that e.g. changing $OutputEncoding has no effect on my test case, encoding is broken if I set this one to UTF8.

    To me it seems as some glitch in PS ISE.

    Monday, March 02, 2015 3:52 PM
  • I don't recommend calling findstr because it's an external executable (and thus returns only text), because PowerShell already has Select-String which returns objects. You also don't need ping (again, external executable) when PowerShell has Test-Connection.

    What is the purpose/goal? What are you trying to do that this problem is stopping you from doing?


    -- Bill Stewart [Bill_Stewart]

    Monday, March 02, 2015 4:04 PM
    Moderator
  • The ping was just an example. I tried to demonstrate that my command to set a console was working only if I launched some application before that.

    I am working at CSS, I am analysing log files. Imagine situation where you have 100 log files with size 10MB. You need to go through all of these log files and do following:

    1. Find all lines that contain some string (array of strings) in any of these files
    2. Write all lines found in point 1 to a single file and sort them out per time. Content of files must not be broken as you may need to send the output of this file for further processing (manual or automatic) or just copy some of lines and search them in input files.
    3. The fast speed of processing is also one of requirements.

    Now although I know that select-string is pretty capable of that, per my tests it is about 2-3 times slower than findstr or findstr in combination with get-content... With multiple big files we are talking about minutes of difference (idea also supported by one of articles linked in my second comment). That is why I turned to findstr - to gain the speed. The quality is still there, I do not need objects at this point, encoding work just fine when I can set it for console. And as I am writing my powershell code in powershell ISE, I started to wonder why I am getting that error about console configuration.

    Thanks to your comment I did some more tests with procmon utility and I now understand that when I launch Powershell ISE, I do not have console running (well, some commands such as [Console]::BackgroundColor does return an output but it might be just some predefined value) and thus running my command [Console]::OutputEncoding = [System.Text.Encoding]::UTF8 ends with failure as there is no console for which I could apply this. When I launch an external application (ping, findstr...), the first thing powershell_ise.exe does is it launches its own console (conhost.exe) and then it starts those external applications such as ping. So from this moment on Powershell ISE has its console running and I can configure it the way I want -> any application I call afterwards will use the settings of the console I just set.
    I also found that powershell.exe launches conhost.exe right after it starts and that is the reason why I can call [Console]::OutputEncoding = [System.Text.Encoding]::UTF8 withing powershell.exe without any error.

    Just btw, majority of properties for [Console]:: returns null value before I launch external application such as ping, another indication conosle is really not initialized at the start:
    $objTemp = New-Object System.Object        
    $objTemp | Add-Member -type NoteProperty -name BackgroundColor -value ([Console]::BackgroundColor
    $objTemp | Add-Member -type NoteProperty -name BufferHeight -value ([Console]::BufferHeight
    $objTemp | Add-Member -type NoteProperty -name BufferWidth -value ([Console]::BufferWidth
    $objTemp | Add-Member -type NoteProperty -name CapsLock -value ([Console]::CapsLock
    $objTemp | Add-Member -type NoteProperty -name CursorLeft -value ([Console]::CursorLeft
    $objTemp | Add-Member -type NoteProperty -name CursorSize -value ([Console]::CursorSize
    $objTemp | Add-Member -type NoteProperty -name CursorTop -value ([Console]::CursorTop
    $objTemp | Add-Member -type NoteProperty -name CursorVisible -value ([Console]::CursorVisible
    $objTemp | Add-Member -type NoteProperty -name Error -value ([Console]::Error
    $objTemp | Add-Member -type NoteProperty -name ForegroundColor -value ([Console]::ForegroundColor
    $objTemp | Add-Member -type NoteProperty -name In -value ([Console]::In
    $objTemp | Add-Member -type NoteProperty -name InputEncoding -value ([Console]::InputEncoding
    $objTemp | Add-Member -type NoteProperty -name IsErrorRedirected -value ([Console]::IsErrorRedirected
    $objTemp | Add-Member -type NoteProperty -name IsInputRedirected -value ([Console]::IsInputRedirected
    $objTemp | Add-Member -type NoteProperty -name IsOutputRedirected -value ([Console]::IsOutputRedirected
    $objTemp | Add-Member -type NoteProperty -name KeyAvailable -value ([Console]::KeyAvailable
    $objTemp | Add-Member -type NoteProperty -name LargestWindowHeight -value ([Console]::LargestWindowHeight
    $objTemp | Add-Member -type NoteProperty -name LargestWindowWidth -value ([Console]::LargestWindowWidth
    $objTemp | Add-Member -type NoteProperty -name NumberLock -value ([Console]::NumberLock
    $objTemp | Add-Member -type NoteProperty -name Out -value ([Console]::Out
    $objTemp | Add-Member -type NoteProperty -name OutputEncoding -value ([Console]::OutputEncoding
    $objTemp | Add-Member -type NoteProperty -name Title -value ([Console]::Title
    $objTemp | Add-Member -type NoteProperty -name TreatControlCAsInput -value ([Console]::TreatControlCAsInput
    $objTemp | Add-Member -type NoteProperty -name WindowHeight -value ([Console]::WindowHeight
    $objTemp | Add-Member -type NoteProperty -name WindowLeft -value ([Console]::WindowLeft
    $objTemp | Add-Member -type NoteProperty -name WindowTop -value ([Console]::WindowTop
    $objTemp | Add-Member -type NoteProperty -name WindowWidth  -value ([Console]::WindowWidth

     
    So I think that clears out the issue I had and the topic can be closed.


    • Marked as answer by Bill_StewartModerator Tuesday, March 03, 2015 3:11 PM
    • Edited by garwel Wednesday, February 17, 2016 9:32 AM a few typos (select-string instead of findstr)
    Monday, March 02, 2015 5:15 PM
  • It is a bug and should be posted to "Connect".  It is an oversight in the design where the console does not exists and so is not settable.  Since you will never be running scripts solely in the ISE (it is just a fance editor) but from PowerShell.exe this should be a big problem.

    Using Select-String can be sped up by using Regex direct.

    Get-Content $file -Readcount 2000 |%{$_ -match 'some criteeria'}

    In fact, with proper tuning, it can be faster than findstr.


    ¯\_(ツ)_/¯

    Monday, March 02, 2015 5:23 PM
  • There is also the Log Parser tool for examining log files, which may be useful in your circumstances and prevent some amount of writing code. Questions about Log Parser should be asked in the Log Parser forums, though.

    -- Bill Stewart [Bill_Stewart]

    Monday, March 02, 2015 5:26 PM
    Moderator
  • Also note that output to the console is throttled in PowerShell and not in CMD.exe.  This makes measuring with output faulty.

    Use a null output to test:

    Measure-Command {$matches=Get-Content $file -readcount 2000 |%{$_ -match 'stuff'}}

    By catching the output you avoid throttling.


    ¯\_(ツ)_/¯

    Monday, March 02, 2015 5:28 PM