none
KB2707511 causes NTVDM crash when a DOS program attempts to open a pipe

    Question

  • We have a large number of customers using some legacy 16-bit DOS code. This code communicates with our Windows application using named pipes.

    Since KB2707511 started rolling out, the legacy DOS code crashes when it attempts to open a pipe. The crash is in NTVDMD.DLL (0xC00000005: Access violation).

    I have reduced the code to a bare minimum to reproduce the crash:

    #include <dos.h>
    #include <fcntl.h>
    #include <share.h>
    #include <stdio.h>

    unsigned fdout;
    int err;

    int main() {
    printf("Opening pipe %s\n");
    err = _dos_open("\\\\.\\PIPE\\SGO0001.TMP", O_WRONLY, &fdout);
    if(err) {
    printf("Open failed:%d\n", err);
    _exit(1);
    }
    printf("Done\n");
    return 0;
    }

    This code is compiled with DOS C (V5), and crashes every time.

    What do I do now?

    Thursday, June 21, 2012 6:29 PM

Answers

  • I might add that there is a hotfix - installing that, if possible would be the better solution: http://support.microsoft.com/kb/2732488
    Friday, November 23, 2012 11:08 PM

All replies

  • Same problem here.  I suggest to uninstall KB2707511.
    Monday, June 25, 2012 11:06 AM
  • That's going to really impress all our customers.

    Many of them are not technically proficient (so would struggle to uninstall the update without help).

    The situation is clearly a bug in the update - I would expect Microsoft to investigate the bug report, fix the update, and reissue it.

    Monday, June 25, 2012 11:26 AM
  • Any luck finding a solution that doesn't require uninstalling the update?  This is really a pain for our customers who's IT demand updates be applied.

    In my case the code looks like this creating something like a named pipe as well:

    DOSHANDLE xOpen( LPTSTR name, UCHAR flags )
    {
       WORD savPSP;
       WORD handle;
       WORD savDS;
       handle = INVALID_DOS_HANDLE;

       savPSP = xSwitchPSP(); // Program Segment Prefix

       savDS = _DS;
       _DS = FP_SEG(name);
       _DX = FP_OFF(name);
       _AL = flags;
       _AH = 0x3d;
    asm {
       int   21h
       jc    x1
    }
       handle = _AX;
    x1:
       _DS = savDS;
       xSetPSP(savPSP);

       return handle;
    }

    static WORD xSwitchPSP(VOID)
    {
       WORD savPSP;

       savPSP = xGetPSP();

       xSetPSP(_psp);
       return savPSP;
    }

    static VOID xSetPSP( WORD psp )
    {
       _BX = psp;
       _AH = 0x50;

    asm {
       int   21h
    }

    }

    Thursday, June 28, 2012 1:05 PM
  • As an MSDN subscriber, I solicited an accident vs. Microsoft technical support. The reply was to uninstall the KB. They don't seem to realize that they suddendly broke Dos sessions. Contrary to the specifications, Windows XP don't support DOS sessions any more. Microsoft should really fix it !!!

    Sunday, July 08, 2012 3:30 PM
  • I have opened a support incident too. I have spoken on the phone to a Senior Support Engineer in the Windows Server Setup & Core Cluster Team, and he passed the incident on to the Development team on 3 July 2012. I have heard nothing since, but live in hope!
    Monday, July 09, 2012 9:13 AM
  • We have the same issue here. The NTVDM crashes. I have since removed the update. Any new information?

    Thanks,

    Mike

    Wednesday, July 11, 2012 2:36 PM
  • I will report back if we ever hear anything. Holding your breath is NOT recommended!

    I am investigating a workround. I have had some success with the following:

    Windows program runs a 32-bit console app instead of running the DOS program directly.

    This app opens the pipes, and assigns the pipe handles to stdin and stderr when it starts the DOS process. DOS process uses stdin and stderr instead of the pipe handles. Of course, this only works if the DOS process is not using stdin or stderr for something else!

    This works on my machine, but it doesn't seem to work on some of my colleagues machines. I am awaiting a copy of a VM on which it is alleged not to work, to see if it is finger trouble, something I have done, or something in Windows.

    Wednesday, July 11, 2012 3:30 PM
  • I am interested in this thread because I also have to support DOS sessions in XP (no named pipes thankfully). Just a quick question and it probably wont make any difference but is the behaviour the same if you use command.com instead of cmd.exe? 
    Wednesday, July 11, 2012 4:54 PM
  • I don't understand the question.

    We are running a Windows app that starts a 16-bit DOS app. What have command.com and cmd.exe got to do with it?

    Wednesday, July 11, 2012 5:14 PM
  • If your running DOS under XP one of those commands is setting the enviroment for the application.
    So your windows application that calls your DOS application must use either command.com or cmd.exe to start the DOS window.
    • Edited by Ed888 Wednesday, July 11, 2012 6:00 PM
    Wednesday, July 11, 2012 5:57 PM
  • How do I tell?

    The code which starts the process is:

    STARTUPINFO sti;
    PROCESS_INFORMATION pi;
    sti.cb = sizeof(sti);
    GetStartupInfo(&sti);
    sti.lpReserved = 0;
    sti.lpDesktop = 0;
    sti.lpTitle = "Window title";
    sti.dwFlags = STARTF_USESHOWWINDOW;
    sti.wShowWindow = SW_HIDE;
    sti.cbReserved2 = 0;
    sti.lpReserved2 = 0;
    CreateProcess(0,
    "dosprogram dos parameters", 
    0,
    0,
    0,
    CREATE_NEW_CONSOLE|CREATE_SEPARATE_WOW_VDM,
    0,
    0,
    &sti,
    pi);

    Thursday, July 12, 2012 8:47 AM
  • One easy way to tell is to run process explorer and see what starts when you run your code. If it is cmd.exe it will show that, if it is command.com it will show ntvdm.exe. I am not a developer but I think your code would use ntvdm.exe with the create_new_console call. Not sure though.

    Honestly I think I may be sidetracking you on this, I expect either way the behaviour might be the same regarding your problem but I was interested in seeing if there was some workaround or fix that could be applied while leaving the patch in place.

    Thursday, July 12, 2012 3:46 PM
  • I can tell you are not a developer! And you are indeed side tracking.

    The following is to the best of my knowledge...

    Neither command.com nor cmd.exe are involved. All 16-bit programs are run within NTVDM (including command.com, and my 16-bit program). Command.com is the 16-bit command interpreter (what you see as a "Dos Box"). Cmd.exe is the 32-bit command interpreter (which looks much the same at first glance). If you run a 16-bit program from command.com, it will run in the same 16-bit NTVDM session as the command.com is currently running in.

    If you run a 16-bit program from cmd.exe, it will have to start a new NTVDM 16-bit session in which to run the program.

    The only workround I have found so far is the one I detailed above (involving opening the pipes in a 32-bit console program and redirecting the 16-bit programs stdin/stderr to those pipes). This is still in the process of being tested. I will report back if the workround actually works.

    Thursday, July 12, 2012 4:52 PM
  • I have the same problem with a dos program that initializes a proprietary serial card that communicates with  a silicon wafer measuring device. With KB2707511 dos program crashes and exits. Without KB2707511 all is fine. Please release updates that are backward compatible – this is a mission critical device, and there are no upgrades or replacements for the device. Most DOS programs call named pipes, why on earth would Microsoft release an update that break these critical pipes?

    Thursday, July 12, 2012 8:31 PM
  • Did anyone manage to get a workaround working with this security patch in place?

    Monday, October 22, 2012 9:05 AM
  • I found that disabling DEP on a machine that is affected makes the crash disappear, even with the update in question installed.
    Friday, November 23, 2012 10:42 PM
  • I might add that there is a hotfix - installing that, if possible would be the better solution: http://support.microsoft.com/kb/2732488
    Friday, November 23, 2012 11:08 PM
  • The KB2732488 was the answer I needed to get the systems working again with the KB2707511 security update.  Thanks for the pointer!

    Saturday, November 24, 2012 6:59 AM