locked
Why Doesn't -Wait Work, And Why Doesn't Anyone Seem To Care? RRS feed

  • Question

  • I've seen quite a few suggestions bouncing around the internet recommending PowerShell's Get-Content command to provide *nix "tail-like" functionality. The recommended syntax is:

    gc <file_name> -Wait -Tail 20

    But sadly -Wait doesn't actually seem to do anything--it just sits there blocking and never is updated by the file stream supplied as an argument. Meanwhile, I'm using Cygwin tail and everything works exactly as expected--no sad errors from the OS about how "so and so already has that file open" no unlimited blocking while staring at 20 minute old log messages.

    Does get-content -Wait actually *work* for anyone? If so, what's the magic formula that the PS adepts are using?  If not, when is it going to get unbroken? 

    Tuesday, October 21, 2014 6:11 PM

Answers

  • Interesting, looks like this has been broken for a while now.

    http://stackoverflow.com/questions/19919180/get-content-wait-not-working-as-described-in-the-documentation

    Maybe Powershell 5.0 will have a solution.

    • Marked as answer by AnnaWY Monday, November 3, 2014 9:13 AM
    Tuesday, October 21, 2014 7:38 PM
  • The problem appears to be a limitation with the FileSystemWatcher class.  This is what the FileSystem provider uses (when you use the -Wait switch) to determine when to open up the file and read more lines.  I've been able to reproduce the behavior described for Get-Content here by just creating a FileSystemWatcher under the same conditions, and seeing that it's not firing events (and the WaitForChanged() method is timing out.)

    (These tests were done on PowerShell v4; not sure if PowerShell has always used the FSW, or if there was an explicit 1-second long polling loop in earlier versions.)


    • Edited by David Wyatt Friday, November 14, 2014 1:10 PM
    • Marked as answer by ianbc1 Friday, November 14, 2014 7:04 PM
    Friday, November 14, 2014 1:08 PM

All replies

  • The -Tail parameter and the -Wait parameter cannot be used together as they are not both from the same parameter set (Get-Help Get-Content). That's probably about as helpful I can be to this thread. If you'd like the cmdlet to work differently then I would recommend you submit a suggestion to the PowerShell team via Connect: https://connect.microsoft.com/PowerShell.
    Tuesday, October 21, 2014 6:35 PM
  • The -Wait parameter works exactly as it says it does in the Help for me. It checks the content of the file every second and puts that result into the console. It will only return the prompt to you when you issue a Ctrl+C at the console.
    Tuesday, October 21, 2014 6:36 PM
  • Works as advertised for me.  I've never seen Get-Content complain about a file being open.

    Can you give us steps to repro?


    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "


    • Edited by mjolinor Tuesday, October 21, 2014 6:38 PM
    Tuesday, October 21, 2014 6:38 PM
  • Hi mjolinor,

    Sure, I'm just trying to tail the output logs of a running tomcat instance. (Based on your response, I was able to write a trivial script that echoes to a file. I can -Wait -Tail that file, and things work as expected.)

    But When I point get-content at the output of a .NET MVC .log running under IIS I get no updates at all. When I point it at a log from a running Tomcat instance I get no real-time updates of the output at all.

    You can test this by running a cygwin terminal, firing up an instance of Tomcat, and pointing both `Get-Content foo.log -Wait -Tail 200` and Cygwin's `tail -f -n 200 foo.log` at the resulting log file.

    For me, tail shows log messages updating in real-time. gc just sits there doing nothing. Does it work differently on your end?

    Thanks...


    Tuesday, October 21, 2014 7:24 PM
  • tommy,

    I think you're using an old version of PowerShell, at least -Wait and -Tail work together on the Windows 8 server I'm looking at, but not my Windows 7 development box. I'm glad they added support for using both together, since I can't think of anything more frustrating than having to wait for several minutes as a 5 GB log file scrolls by before you can actually "wait" on the end of that file. :)

    Tuesday, October 21, 2014 7:26 PM
  • Hi mjolinor,

    Sure, I'm just trying to tail the output logs of a running tomcat instance. (Based on your response, I was able to write a trivial script that echoes to a file. I can -Wait -Tail that file, and things work as expected.)

    But When I point get-content at the output of a .NET MVC .log running under IIS I get no updates at all. When I point it at a log from a running Tomcat instance I get no real-time updates of the output at all.

    You can test this by running a cygwin terminal, firing up an instance of Tomcat, and pointing both `Get-Content foo.log -Wait -Tail 200` and Cygwin's `tail -f -n 200 foo.log` at the resulting log file.

    For me, tail shows log messages updating in real-time. gc just sits there doing nothing. Does it work differently on your end?

    Thanks...


    Maybe someone who has those already installed can test that.  

    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

    Tuesday, October 21, 2014 7:28 PM
  • Interesting, looks like this has been broken for a while now.

    http://stackoverflow.com/questions/19919180/get-content-wait-not-working-as-described-in-the-documentation

    Maybe Powershell 5.0 will have a solution.

    • Marked as answer by AnnaWY Monday, November 3, 2014 9:13 AM
    Tuesday, October 21, 2014 7:38 PM
  • I have lost confidence in being able to see the actual parametersets from the help text.
    I use the method shown below to see the actual parametersets:

    (gcm get-content).Definition

    Tuesday, October 21, 2014 9:29 PM
  • I think this is a matter of when the producer actually flushes the buffers to make the most recent content available to the consumers.

    Tuesday, October 21, 2014 9:31 PM
  • Or also with less complexity:

    Get-Command get-content -Syntax

    Wednesday, October 22, 2014 9:30 PM
  • Hi Larry,

    I'm not sure that's the case, though. If you run a process like Tomcat, Jetty or a .NET ASP/MVC/ IIS webapp that logs to a file, then open a cygwin window and tail the file next to a PowerShell window that's gc-ing the file with `gc <file> -Wait -Tail 200` the cygwin process will be spitting out logging messages in real-time while nothing comes out of the gc... 

    Wednesday, October 22, 2014 9:37 PM
  • That's interesting.  Maybe it will take an inspection of the PowerShell code that implements Get-Content to find the exact I/O call that is used to retrieve the freshest part of a file.

    I suspect that Get-Content is using a less aggressive method to find new content.

    • Edited by Larry Weiss Wednesday, October 22, 2014 10:01 PM
    Wednesday, October 22, 2014 9:51 PM
  • Especially the second part of the question is pretty annoying "why doesn't anyone seem to care?"

    you can simply reproduce this issue with the following steps:

    1. start a console and redirect the output of ping -t to a textfile
      ping localhost -t >> c:\temp\ping.txt
    2. open a second console and type "gc c:\temp\ping.txt -tail 5 -wait"
      You will see the last 5 lines of the TXT file and then it stops.
    3. open windows explorer and browse to "c:\temp"
      in the moment you open the folder in explorer, whatch your console window and you will see the output is updating. This behavior will repeat every time you refresh the explorer window (F5).
    4. you can see the same behavior by opening a third console and typing "test-path c:\temp\ping.txt"

    So everytime you "touch" the textfile, the get-content cmdlet is updating.

    And here comes the funny part:

    1. keep pinging your localhost
    2. type "gc c:\temp\ping.txt -tail 5 -wait" in your second console
    3. type "gc c:\temp\ping.txt -tail 5 -wait" in your third console
    4. let the magic happen
      The get-content cmdlet will check the file one time per second. This check seems to trigger the event (or what ever the background mechanics may be) for the other get-content instance to recognize that the file has changed. So if you have 2 instances of "get-content -wait" reading on the same file, it works like expected.

    Has anyone tested the -wait parameter in powershell 5.0?

    Wednesday, November 12, 2014 2:43 PM
  • The problem is not In PowerShell,  I use this all of the time to trap output from the firewall logs, IIS logs anfd other logs like WU.  The issue is that some utilities do not release the file or open it shared.

    Try this test as it works better:

    while(1){ sleep -s 1; [datetime]::Now.ToString()|Out-File c:\temp\temp.txt -append }

    Now tail that file and you will get the updates.


    ¯\_(ツ)_/¯

    • Proposed as answer by jrv Wednesday, November 12, 2014 8:17 PM
    Wednesday, November 12, 2014 8:16 PM
  • "The problem is not In PowerShell,  I use this all of the time to trap output from the firewall logs, IIS logs anfd other logs like WU.  The issue is that some utilities do not release the file or open it shared."

    And yet.... install cygwin, and run `tail -f foo.log` and it just works.  

    Maybe it's because I'm relatively new to Windows systems administration, but it seems like the above sentiment is a bit overly tolerant of poor tooling.

    gc's -Wait does not work in many cases, in the experience of many people. Maybe we're using it wrong, but it would be nice if the documentation set expectations appropriately. This is absolutely basic stuff, and sometimes it's not possible to install cygwin on every Windows box you come into contact with.

    Wednesday, November 12, 2014 8:36 PM
  • Not the same. The "tail" option is not the Unix "tail" and is not intended to be.

    I have a couple of tail utilities for Windows that are designed to hook a process and sneak around limits.  If cygwin works so well then why aren't you using it on Windows? 

    My point was that is it not a bug. It does what it was intended to do.  It is a simple way to monitor system log files.  Some old utilities that use the older APIs do not behave well with Net components.  PING will go away.   I don't expect anyone will fix it.  It may be replaced by a simple net equivalent also called ping.

    I believe that Microsoft is trying to get rid of all old bits like CMD.EXE and other older non-CLR components for security and support reasons.  Maybe in V6 the team will take the time to crate an extension although baretail still works well.

    Even on Unix you cannot tail locked files.


    ¯\_(ツ)_/¯

    Wednesday, November 12, 2014 10:12 PM
  • "If cygwin works so well then why aren't you using it on Windows?"

    Well, of course I do use it on Windows. There are times when one has to log into a production box and perform debugging tasks such as inspecting log files on a running system. In that scenario, I'm hardly going to log in and start immediately installing third-party POSIX environments.  :)

    While apo32 outlined a trivial test using ping above, the behavior is not limited to ping. You can get the exact same behavior on java application servers logging via log4j. Or .NET applications using log4net. That's great that ping may be replaced by a .NET equivalent some day. But unfortunately my clients are still going to be making my life miserable by running Tomcat on Windows boxes.

    "Even on Unix you cannot tail locked files."

    While I've never come across this problem, the case we're talking about here is one where tailing a log file is trivially easy using cygwin and impossible using get-content--in identical circumstances.  If that's not what get-content is for, that's one thing, but if you google "tail in windows" you're inundated with results claiming that get-content with -Tail and -Wait provides the exact same functionality.

    Wednesday, November 12, 2014 10:36 PM
  • Yes I know that.  Almost all old Unix ports like Tomcat, Java, Apache and others use blocking IO for compatibility with cross-platform C libraries.  They also did it for performance. I remember when Unix always outperformed Windows back at NT 4.  In W2K and later MS modified the file system to allow handles to be shared or, at least, mot really closed.  They called it logically closed and this allows other programs access and it allows a program to open, write, close a file very quickly.  All modern programs do not open files for exclusive writing but almost all Unix compatibility libraries do as do almost all older command line utilities.  I know of no Microsoft systems that do this now.  IIS/SQLSERVER/Exchange all close after write. For the few that don' there is a utility that can be kept on a share and used as needed.  I now keep it on a thumb drive.  No need for Posix.

    So don't pick on Get-Content.  It works most of the time for me but then I do not allow ported Unix utilities anywhere near my systems due to many bad experiences and the one customer that has them is kept locked down.


    ¯\_(ツ)_/¯


    • Edited by jrv Wednesday, November 12, 2014 11:01 PM
    Wednesday, November 12, 2014 11:00 PM
  • "Almost all old Unix ports like Tomcat, Java, Apache and others use blocking IO for compatibility with cross-platform C libraries."

    Java has had NIO since JDK 1.4, and Tomcat runs on top of that. I'm not sure one could really characterize Java on Windows as an "old Unix port" :)  Anyway, Tomcat has had native nio support since Tomcat 6...

    In any case, what do you tell someone who wants to tail a log file in Windows? It can't be done? If there's anything about Windows development & systems administration that irks me, it's a common reflex that if something's not working, the user's to blame, and you should wipe your machine, install Windows 10 Ultra, and use whatever toolchain Microsoft has pushed out this past month. Tailing a logfile shouldn't be beyond the grasp of 21st century technology. 


    • Edited by ianbc1 Thursday, November 13, 2014 10:04 PM
    Thursday, November 13, 2014 10:03 PM
  • I have no trouble tailing Windows log files with Get-Content.  It always works. Like I posted. Old Unix ports don't behave well because the compatibility libraries were built in a way that blocks them if they are the log creators.  On the Posix subsystem things may work but full interoperability with Windows is poor.

    Microsoft dropped Posix quite a while ago infavor of virtualization. That way the two APIs can coexist peacefully.

    I prefer Unix on a Unix core and not on an NT HAL.  NT was based on Dec's  port of VMS before they completed the Unix bases.  I worked on all of them.   DEC Alpha Unix was awesome.  It could hos anything.  VMS 5.1 and later could run with Alpha Unix co-resident. Too bad all of that broke DEC's back. I really like those Massachusetts guys.

    Anyway I don't have problems with the Windows log files.


    ¯\_(ツ)_/¯

    Friday, November 14, 2014 12:10 AM
  • I think the basic problem here is getting confused by the solution (i.e. cygwin's tail). Tomcat's runtime environment is not an "old unix port". It runs under the Windows JVM--which is equivalent to the CLR.

    While you may have no trouble tailing Windows lo files with Get-Content, it's clear many others have a different experience.

    Friday, November 14, 2014 1:45 AM
  • Right.  I can't get it to work either using PowerShell V3 under Windows 7.
    Friday, November 14, 2014 3:05 AM
  • Right.  I can't get it to work either using PowerShell V3 under Windows 7.

    It works for me on 7 and 8 with 3 and 4.  Always has worked and always will work.  Won't work with legacy apps that use the older API open calls.  This is documented and has been since W2K.


    ¯\_(ツ)_/¯

    Friday, November 14, 2014 4:50 AM
  • I think the basic problem here is getting confused by the solution (i.e. cygwin's tail). Tomcat's runtime environment is not an "old unix port". It runs under the Windows JVM--which is equivalent to the CLR.

    While you may have no trouble tailing Windows lo files with Get-Content, it's clear many others have a different experience.


    What makes you think that JVM is the same as CLR.  JVM use older Win32API direct.  CLR uses NET Framework and access the newest file system API only.  JVM on Posix uses the C compatibility libraries.  At least it always did in the past.  If Posix was redesigned for the Net Framework then you may have something.  Get-Content works for me except against files that are opened using the older API calls and fail to ask for "read-shared-deny-none".

    ¯\_(ツ)_/¯

    Friday, November 14, 2014 12:34 PM
  • I think the basic problem here is getting confused by the solution (i.e. cygwin's tail). Tomcat's runtime environment is not an "old unix port". It runs under the Windows JVM--which is equivalent to the CLR.

    While you may have no trouble tailing Windows lo files with Get-Content, it's clear many others have a different experience.

    ian - I think I know of a simple way to demonstrate what I am trying to show you but I have to do some testing later to be sure it works the way I think it will. Give me a bit and maybe we can agree or maybe I will agree with you. we'll see.

    ¯\_(ツ)_/¯

    Friday, November 14, 2014 12:39 PM
  • Right.  I can't get it to work either using PowerShell V3 under Windows 7.

    Did you try this:

    First copy of PowerShell CLI

    while(1){ sleep -s 1; [datetime]::Now.ToString()|Out-File c:\temp\temp.txt -append }

    New copy of PowerShell CLI

    Now tail that file and you will get the updates.

    Get-Content c:\temp\temp.txt -wait -tail 1

    That works as does tailing almost any Windows log file like IIS.



    ¯\_(ツ)_/¯


    • Edited by jrv Friday, November 14, 2014 3:33 PM
    Friday, November 14, 2014 1:05 PM
  • The problem appears to be a limitation with the FileSystemWatcher class.  This is what the FileSystem provider uses (when you use the -Wait switch) to determine when to open up the file and read more lines.  I've been able to reproduce the behavior described for Get-Content here by just creating a FileSystemWatcher under the same conditions, and seeing that it's not firing events (and the WaitForChanged() method is timing out.)

    (These tests were done on PowerShell v4; not sure if PowerShell has always used the FSW, or if there was an explicit 1-second long polling loop in earlier versions.)


    • Edited by David Wyatt Friday, November 14, 2014 1:10 PM
    • Marked as answer by ianbc1 Friday, November 14, 2014 7:04 PM
    Friday, November 14, 2014 1:08 PM
  • I think the basic problem here is getting confused by the solution (i.e. cygwin's tail). Tomcat's runtime environment is not an "old unix port". It runs under the Windows JVM--which is equivalent to the CLR.

    While you may have no trouble tailing Windows lo files with Get-Content, it's clear many others have a different experience.

    CYGWIN is built on the old Win32 API. It is not built on Net. Tomcat runs on the Posix JVM which runs on Win32. tail under Cygwin cannot tail a windows api that uses exclusive opens unless it has been rebuilt to use the Debug API. If that it true then Cygwin is a potential security nightmare.  Of course that has been claimed repeatedly before so maybe it does.

    I have worked on many versions of Unix.  tail on Uniix cannot open files that are opened exclusively.


    ¯\_(ツ)_/¯

    Friday, November 14, 2014 1:27 PM
  • Yes, that works. (after fixing the typo in the second Get-Content call)
    Friday, November 14, 2014 3:04 PM
  • Thanks.  I wonder now if the PowerShell team is aware of this.

    It really limits the use of Get-Content -Wait 

    Friday, November 14, 2014 3:06 PM
  • The problem appears to be a limitation with the FileSystemWatcher class.  This is what the FileSystem provider uses (when you use the -Wait switch) to determine when to open up the file and read more lines.  I've been able to reproduce the behavior described for Get-Content here by just creating a FileSystemWatcher under the same conditions, and seeing that it's not firing events (and the WaitForChanged() method is timing out.)

    (These tests were done on PowerShell v4; not sure if PowerShell has always used the FSW, or if there was an explicit 1-second long polling loop in earlier versions.)


    The FSW is probably not what is used.  The code, if in C++ or C# would call the SENS subsystem and park a SENS trigger on the file.  This trigger is more sensitive than the one used by FSW which is designed to be multipurpose.  I cannot prove this but I do not see the exact same behavior.  I can receive cascades of lines as long as I stay away from using the > redirector in a CMD shell. All IO from PS does not block.  IO from system to IIS, SQL and other logs all can be tailed.


    ¯\_(ツ)_/¯

    Friday, November 14, 2014 3:37 PM
  • Ok, because it's friday, and I've got too much time on my hands :) Running "native" node.js 64-bit v.0.10.33:

    > npm install winston [the node.js logging package]

    var http = require('http');
    var winston = require('winston');

    winston.add(winston.transports.File, { filename: 'winston-nodejs.log'});
    winston.remove(winston.transports.Console);

    http.createServer(function (request, response) {
        response.writeHead(200, {'Content-Type': 'text/plain'});
        response.end('Hello World\n');
        winston.info('You can\'t see me if you\'re using PowerShell!!! Hello everyone else!!!');
    }).listen(8124);

    console.log('Server running at http://127.0.0.1:8124/');

    > node hello.js

    <<cygwin terminal>>

    > tail -f winston-nodejs.log

    <<powershell terminal>>

    > gc .\winston-nodejs.log -Wait

    I can hit localhost:8124 as much as I like and I get no response in Windows 7 using PowerShell 3 or 4. My cygwin terminal updates in real-time. It's only when I ctrl-C the node.js process that get-content updates my window. 

    Friday, November 14, 2014 5:03 PM
  • The FSW is probably not what is used.  The code, if in C++ or C# would call the SENS subsystem and park a SENS trigger on the file.  This trigger is more sensitive than the one used by FSW which is designed to be multipurpose.  I cannot prove this but I do not see the exact same behavior.  I can receive cascades of lines as long as I stay away from using the > redirector in a CMD shell. All IO from PS does not block.  IO from system to IIS, SQL and other logs all can be tailed.


    ¯\_(ツ)_/¯

    Trust me, it is.  I looked at the code in ILSpy.  :)  You can find it in the System.Management.Automation.dll assembly, class Microsoft.PowerShell.Commands.FileSystemContentReaderWriter , method WaitForChanges().
    Friday, November 14, 2014 6:05 PM
  • The FSW is probably not what is used.  The code, if in C++ or C# would call the SENS subsystem and park a SENS trigger on the file.  This trigger is more sensitive than the one used by FSW which is designed to be multipurpose.  I cannot prove this but I do not see the exact same behavior.  I can receive cascades of lines as long as I stay away from using the > redirector in a CMD shell. All IO from PS does not block.  IO from system to IIS, SQL and other logs all can be tailed.


    ¯\_(ツ)_/¯

    Trust me, it is.  I looked at the code in ILSpy.  :)  You can find it in the System.Management.Automation.dll assembly, class Microsoft.PowerShell.Commands.FileSystemContentReaderWriter , method WaitForChanges().

    That performs almost as badly as WMI events.  They are all based on SENS but the direct API setup seems to be more reliable or it was when I was using it.

    In any case I have no issues reading Windows log files or any files I have created with PowerShelll of Net calls.  The only regular failure is from Win32 API calls like the CMD shell redirectors.  CYGWIN uses those old calls.  CYGWIN was built 20 yeas ago.  Long before the CLR was designed and before the NTFS files system and API were updated.  I guarantee if you run it under the Software Compatibility Checker it will alert on all of the file open calls.


    ¯\_(ツ)_/¯

    Friday, November 14, 2014 8:05 PM