none
Weird extended ACSII characters in NTDVM Windows 10 RRS feed

  • Question

  • In anticipation of the withdrawal of support for Windows 7 in January I am starting to transition my company’s computers to Windows 10.  We have used Win 7 32 bit in order to use a long standing custom DOS application.  It has run in NTVDM faultlessly with the advantage that it can make calls to the external Windows system such as displaying and printing PDF files.

    I have downloaded the Windows 10 update tool (32 bit of course) from Microsoft and attempted the transition on the first computer.  I ran it with the option to keep all programs and data.  Surprisingly, it ran without trouble with the result that all the Windows programs work as expected under the new Win 10.  That is, with the exception of NTVDM.

    In itself, NTVDM appears to work just fine.  But the extended ASCII character set is displaying some weird characters.  Since we use some of the extended characters (eg, £, ½, º, ±, ©, ®, µ) this will be a problem for us.

    Under Win 7 in order to display the correct ASCII extended codes we added the following line to the batch file that calls the DOS application:

    C:\Windows\System32\KB16 UK,850,C:\Windows\System32\KEYBOARD.SYS

    That gave the ‘correct' ASCII extended character set as displayed here:

           ┌───────┬───────┬───────┬───────┬─────┬─┬───────┬───────┬───────┐
        │ 128 Ç │ 144 É │ 160 á │ 176 ░ │ 192 └ │ 208 ð │ 224 Ó │ 240 │
        │ 129 ü │ 145 æ │ 161 í │ 177 ▒ │ 193 ┴ │ 209 Ð │ 225 ß │ 241 ± │
        │ 130 é │ 146 Æ │ 162 ó │ 178 ▓ │ 194 ┬ │ 210 Ê │ 226 Ô │ 242 ‗ │
        │ 131 â │ 147 ô │ 163 ú │ 179 │ │ 195 ├ │ 211 Ë │ 227 Ò │ 243 ¾ │
        │ 132 ä │ 148 ö │ 164 ñ │ 180 ┤ │ 196 ─ │ 212 È │ 228 õ │ 244 ¶ │
        │ 133 à │ 149 ò │ 165 Ñ │ 181 Á │ 197 ┼ │ 213 ı │ 229 Õ │ 245 § │
        │ 134 å │ 150 û │ 166 ª │ 182 Â │ 198 ã │ 214 Í │ 230 µ │ 246 ÷ │
        │ 135 ç │ 151 ù │ 167 º │ 183 À │ 199 Ã │ 215 Î │ 231 þ │ 247 ¸ │
        │ 136 ê │ 152 ÿ │ 168 ¿ │ 184 © │ 200 ╚ │ 216 Ï │ 232 Þ │ 248 ° │
        │ 137 ë │ 153 Ö │ 169 ® │ 185 ╣ │ 201 ╔ │ 217 ┘ │ 233 Ú │ 249 ¨ │
        │ 138 è │ 154 Ü │ 170 ¬ │ 186 ║ │ 202 ╩ │ 218 ┌ │ 234 Û │ 250 ─ │
        │ 139 ï │ 155 ø │ 171 ½ │ 187 ╗ │ 203 ╦ │ 219 █ │ 235 Ù │ 251 ¹ │
        │ 140 î │ 156 £ │ 172 ¼ │ 188 ╝ │ 204 ╠ │ 220 ▄ │ 236 ý │ 252 ³ │
        │ 141 ì │ 157 Ø │ 173 ¡ │ 189 ¢ │ 205 ═ │ 221 ¦ │ 237 Ý │ 253 ² │
        │ 142 Ä │ 158 × │ 174 « │ 190 ¥ │ 206 ╬ │ 222 Ì │ 238 ¯ │ 254 ■ │
        │ 143 Å │ 159 ƒ │ 175 » │ 191 ┐ │ 207 ¤ │ 223 ▀ │ 239 ´ │       │
        └───────┴───────┴───────┴─────┴─┴───────┴───────┴───────┴───────┘

    However, under Win 10 this displays mostly wrong, basically a different extended character set - sorry the forum wouldn't let me post an image because it tells me I'm not verified...

    I notice that both KB.COM and KEYBOARD.SYS are present in the same C:\Windows\System32 folder in Win 10.  So why is a different extended character set displaying?

    I have configured NTVDM under Win 10 to ‘Use legacy console’.   I notice a new parameter ‘Current code page’ configured for 'OEM – Multilingual Latin I’ which can't be altered whether in legacy console or not.

    In Control Panel, Region Settings, the system locale is set to ‘English (United Kingdom)'. I don't know if this affects (or should affect) NTDVM in Win 10.

    Interestingly, this behaviour seems to only affect the raster font.  When selecting either of the TrueType fonts, Consolas or Lucinda Console, the ‘correct’ extended character set is displayed.

    However, the TrueType fonts introduce screen artefacts (mainly when toggling through menus) which don’t appear with the raster font.  So I’d prefer to use the raster font in Win 10 as I did with Win 7.

    I’ve searched the internet for possible solutions but can find nothing.  Maybe people don’t use the extended ASCII set so they don't see a problem.

    I have recently looked at alternatives to NTVDM.  Although vDOS would appear to be the answer for my application (both 32 and 64 bit) I can’t get it to work.  NTVDM is simple, comes with Windows (32 bit) and I know it works (until now).

    Would much appreciate some assistance with this.



    • Edited by Robert Lenk Sunday, October 20, 2019 10:12 PM
    Saturday, October 19, 2019 10:48 PM

Answers

  • Mario

    Thanks, but that thread is not related to my issue.

    Turns out that the muted colours are correct. All is explained here:

    https://devblogs.microsoft.com/commandline/updating-the-windows-console-colors/

    Since I don't like the muted palette that MS have now chosen for us I can at least restore the old palette - click on Properties / Colours and set each Screen Background colour according to these values:

    Color Name Console Legacy RGB Values New Default RGB Values
    BLACK         0,0,0                 12,12,12
    DARK_BLUE 0,0,128         0,55,218
    DARK_GREEN 0,128,0         19,161,14
    DARK_CYAN 0,128,128                 58,150,221
    DARK_RED 128,0,0         197,15,31
    DARK_MAGENTA 128,0,128         136,23,152
    DARK_YELLOW 128,128,0                 193,156,0
    DARK_WHITE 192,192,192         204,204,204
    BRIGHT_BLACK 128,128,128         118,118,118
    BRIGHT_BLUE 0,0,255         59,120,255
    BRIGHT_GREEN 0,255,0         22,198,12
    BRIGHT_CYAN 0,255,255                 97,214,214
    BRIGHT_RED 255,0,0         231,72,86
    BRIGHT_MAGENTA 255,0,255         180,0,158
    BRIGHT_YELLOW 255,255,0         249,241,165
    WHITE 255,255,255         242,242,242

    I now have a fully functioning NTVDM within Win 10 that replicates the behaviour in older versions of Windows and can now go ahead and transition my company's computers to Win 10.

    Mario, thank you for your time and patience with this.

    Regards

    Robert Lenk

    • Marked as answer by Robert Lenk Saturday, October 26, 2019 11:41 PM
    Saturday, October 26, 2019 11:41 PM

All replies

  • Have yo already tried using CHCP??

    Change CodePage is a little tool that iwll set the system codepage while you are running in a console .

    By example if you open a command prompt and want to copy files with ? and % in the name you must call
    chcp 1252

    in your case you may want to try with

    chcp 850

    Try and let us know

    HTH
    -mario

    Monday, October 21, 2019 7:15 AM
  • Hi Mario

    Thanks, I didn't know about CHCP.

    When I type just chcp it returns the line 'Active code page: 850' so I'm in the right area.

    However, I still get the weird ASCII extended set using the raster font, which I presume must be from a different codepage although I can't work out which one.

    I'm still not 'verified' so I can't include an image highlighting the 'weird' characters. 😢

    By the way, I have been able to change the default 413 codepage to 850 from information on another forum:

    "...here is the config.nt, and autoexec.nt modifications (for turkish keyboard:
    config.nt;
    dos=high, umb
    device=%SystemRoot%\system32\himem.sys
    COUNTRY=090,857, %SystemRoot%\system32\COUNTRY.SYS
    files=99
    autoexec.nt;
    %SystemRoot%\system32\KB16 TR,857, %SystemRoot%\system32\KEYBOARD.SYS
    For Danish;
    DK,850
    For Belgium;
    BE,850..."

    Your assistance is much appreciated.

    Robert
    Monday, October 21, 2019 11:28 PM
  • did you try with Code page 437 or 1252??

    437 is the OEM font

    Unfortunately I no longer have any 16 bit app to try with..

    HTH
    -mario

    Tuesday, October 22, 2019 12:36 PM
  • Neither 437 or 1252 gives the 'correct' extended ascii set in the CMD window.

    (What is codepage 1252?)

    When I try to run the command <chcp codepage> from within my 16-bit program I get the error message 'NLSFUNC not installed'.  However, just <chcp> returns the codepage, 850.

    To give you some background I test for the ascii extended characters by running a batch file, ascii.bat:

    @echo off
    cls
    type ascii.txt
    pause

    Contents of ascii.txt:

    ASCII Character Table - Hold down ALT key whilst entering number then release

        ┌───────┬───────┬───────┬───────┬─────┬─┬───────┬───────┬───────┐
        │ 128 Ç │ 144 É │ 160 á │ 176 ░ │ 192 └ │ 208 ð │ 224 Ó │ 240 ­ │
        │ 129 ü │ 145 æ │ 161 í │ 177 ▒ │ 193 ┴ │ 209 Ð │ 225 ß │ 241 ± │
        │ 130 é │ 146 Æ │ 162 ó │ 178 ▓ │ 194 ┬ │ 210 Ê │ 226 Ô │ 242 ‗ │
        │ 131 â │ 147 ô │ 163 ú │ 179 │ │ 195 ├ │ 211 Ë │ 227 Ò │ 243 ¾ │
        │ 132 ä │ 148 ö │ 164 ñ │ 180 ┤ │ 196 ─ │ 212 È │ 228 õ │ 244 ¶ │
        │ 133 à │ 149 ò │ 165 Ñ │ 181 Á │ 197 ┼ │ 213 ı │ 229 Õ │ 245 § │
        │ 134 å │ 150 û │ 166 ª │ 182 Â │ 198 ã │ 214 Í │ 230 µ │ 246 ÷ │
        │ 135 ç │ 151 ù │ 167 º │ 183 À │ 199 Ã │ 215 Î │ 231 þ │ 247 ¸ │
        │ 136 ê │ 152 ÿ │ 168 ¿ │ 184 © │ 200 ╚ │ 216 Ï │ 232 Þ │ 248 ° │
        │ 137 ë │ 153 Ö │ 169 ® │ 185 ╣ │ 201 ╔ │ 217 ┘ │ 233 Ú │ 249 ¨ │
        │ 138 è │ 154 Ü │ 170 ¬ │ 186 ║ │ 202 ╩ │ 218 ┌ │ 234 Û │ 250 ─ │
        │ 139 ï │ 155 ø │ 171 ½ │ 187 ╗ │ 203 ╦ │ 219 █ │ 235 Ù │ 251 ¹ │
        │ 140 î │ 156 £ │ 172 ¼ │ 188 ╝ │ 204 ╠ │ 220 ▄ │ 236 ý │ 252 ³ │
        │ 141 ì │ 157 Ø │ 173 ¡ │ 189 ¢ │ 205 ═ │ 221 ¦ │ 237 Ý │ 253 ² │
        │ 142 Ä │ 158 × │ 174 « │ 190 ¥ │ 206 ╬ │ 222 Ì │ 238 ¯ │ 254 ■ │
        │ 143 Å │ 159 ƒ │ 175 » │ 191 ┐ │ 207 ¤ │ 223 ▀ │ 239 ´ │       │
        └───────┴───────┴───────┴─────┴─┴───────┴───────┴───────┴───────┘

    (I tried to set a non-proportional font for clarity but without success).

    I hope this helps to resolve my issue.

    Regards

    Robert


    • Edited by Robert Lenk Tuesday, October 22, 2019 11:33 PM
    Tuesday, October 22, 2019 11:30 PM
  • Hi Robert,

    honestly I've done nothing that creating the ascii file with notepad++ in the right codepage, 850, and when I show it in the NTVDM of windows 10, without even touching autoexec.nt or config.nt I can see it correctly..

    even if i open the file in edit it is correct

    HTH

    -mario

    Wednesday, October 23, 2019 8:47 AM
  • Hi Mario

    Sadly it doesn't work for me.  Loading the ascii.txt file in both Notepad and Edit I still get the wrong extended characters.

    I wish I could post pictures which would explain a lot, but the system won't let me until I'm verified.  I've been waiting for more than a month now with no action on Microsoft's part to engage in verification. Perhaps you could prod someone at your end?

    Anyway, back to this issue.  I did the upgrade from Windows 7 to 10 whilst keeping all my programs and data.  Perhaps a clean install of W10 may be the way forward.

    What would you suggest?

    Regards

    Robert

    Wednesday, October 23, 2019 2:46 PM
  • I've done a clean install of Windows 10 Italian but I think the trick has been done in Notepad++

    When you create a new file, it is set to Unicode utf-8 and Widnows.

    But you can go to Encoding, Character Set, Western Euroean, and choose OEM 850

    At this point you can insert extended character using ALT+Number

    I think that if you save this file correctly encoded then it will display correctly in your ntvdm..

    That is I think you have to verify the original file.. probably with all the passage it is no longer OEM 850 at the start.. try recreating it with Notepad++. I don't think the OS has influence on this.. you have set CP 850 and verified that with CHCP.. so it should be ok.. try with another file..

    HTH
    -mario

     

    Wednesday, October 23, 2019 7:07 PM
  • Mario

    The text file containing the extended character set is a pure text file, so holds no information on encoding.

    These are the extended ascii characters that I want displayed (copied and pasted from original post):

    £, ½, º, ±, ©, ®, µ, Å

    The ALT+numbers for these are, respectively:

    156, 171, 248, 241, 184, 169, 230, 143

    Personally, I use TextPad as my Notepad replacement but I've downloaded and installed Notepad++ to best duplicate your results.

    But before I do that I'll copy and paste the results from CMD (ie, NTVDM) using the raster font:

    First, check the codepage:

    C:\Users\Robert Lenk>chcp
    Active code page: 850

    Next type in the ALT+numbers direct into the CMD window:

    £, ½, °, ±, É, ®, µ, Å

    You will see that the 5th character (copyright) is translated.  It's actually <E grave> but I've only been able to show this here using <E acute - ALT-144>, that is what displays 'wrong' in the CMD window displays 'correct' in this post window.  Also, the other characters look very rough - most unlike their appearance in NTVDM under Windows 7.

    So, whatever I am seeing, it's certainly not codepage 850.  I do really wish that I could post some pictures here...   :-(

    Next, I created a text file in Notepad++ 7.8 32-bit , set encoding as Western European, OEM 850 ...

    All characters, display correctly in the Notepad++ window - save as testnp.txt.

    Next, displayed the file created by Notepad++ in the CMD (NTVDM - raster font) window then copy and paste the result here:

    C:\Users\Robert Lenk\Desktop>type testnp.txt
    ┬ú, ┬¢, ┬░, ┬▒, ┬®, ┬«, ┬Á, ├à

    Surprisingly, this has copied over correctly (that is, as displayed in the CMD window) but, as you will see, it's a right mess.

    Final point, verified the codepage again to make sure that it hadn't changed:

    C:\Users\Robert Lenk\Desktop>chcp
    Active code page: 850

    Hopefully this may give an insight into what's going wrong here.  Something's not right but I just can't work out what it is.

    Regards

    Robert


    Thursday, October 24, 2019 11:06 PM
  • Hopefully when you refer to cmd you are really referring to Command.com and not cmd.exe.

    Command.com is the 16 bit command interpreter to which gets attached NTVDM.

    So, in my example below I started the batch file via a .PIF file and command.com directly.

    As you can see I have 2 NTVDM attached to the command.com window.

    If I type chcp it is 850

    And is I add the characters you pointed out, they are correct.

    and they are all correct and as expected..

    Honestly I can't remember anything else that could go wrong the way is going to you.

    If you are using command.com then you should be fine.. 

    I tried also in the cmd at that point just to be sure it is the same thing..

    and it is correct also in the cmd..

    At this point I would install a clean machine to see if there is any difference..

    Thanks
    -mario

     
    Friday, October 25, 2019 8:29 AM
  • Mario

    I have been using CMD and not COMMAND.COM.

    So, I tried COMMAND.COM and got a different result to CMD...

    Microsoft(R) Windows DOS
    (C)Copyright Microsoft Corp 1990-2001.

    C:\WINDOWS\SYSTEM32>chcp
    Active code page: 850

    C:\WINDOWS\SYSTEM32>£, ½, °, ±, ©, ®, µ, Å

    Unfortunately, when I copy and paste here, the 'correct' characters appear!  In the COMMAND.COM window the characters have a Russian feel about them. In any case, it's clearly not codepage 850.

    I'll do a clean install of Windows 10 and keep my fingers crossed.

    Thank you for your time and patience.

    Regards

    Robert

    Friday, October 25, 2019 10:34 AM
  • So it is at all possible that the original file is now, sort of, corrupted..

    try rewrite that file using Edit inside command.com. It will be code page 850 for sure and will display the corret characters..

    Leave the complete install as a last chance..

    HTH
    -mario

    Friday, October 25, 2019 11:46 AM
  • Mario

    Sorry, I missed your last post.  I went ahead with a clean install.

    However, there is good news and bad news...

    The good news is that the 'weird characters' issue is now sorted!  All extended ASCII characters are now displaying correctly with the raster font. 

    The lesson here is to do a clean install.  It's a lesson we all learn the hard way...

    The bad news is that the colours of my 16-bit application are now muted, they are no longer the vibrant colours they should be. This is the same whether I load <cmd> or <command.com>.

    Installing the correct driver for the monitor doesn't help.  Windows programs and images all have correct colour.

    Connecting to a remote PC running my 16-bit application rendered the correct colours, so it's not a monitor calibration issue.

    I loaded the <edit> application in cmd and got the same muted colours. The worst are that cyan is rendered as a light blue and yellow as a very very light yellow. So it would seem that all 16-bit dos applications are affected. But why?  I've never seen this problem in Win 2000, XP, 7.

    Selecting the Colours tab in Properties, I noticed that the clean installed setting for Selected Colour Values was 12 for each colour (Red, Green, Blue).  On my Win 7 computer these are all set to zero.  But setting to zero in Win 10 had no effect.

    If I had hair I'd be pulling it out now...   :-(

    Any ideas?

    Regards

    Robert


    • Edited by Robert Lenk Saturday, October 26, 2019 11:19 AM
    Friday, October 25, 2019 10:51 PM
  • You should set the video driver to use only 256 colors.. probably..

    See this interesting thread:
    https://answers.microsoft.com/en-us/insider/forum/all/8-bit-256-colour-support/71a133fc-27cd-4c81-95f9-29d6ff8834f8

    HTH
    -mario

    Saturday, October 26, 2019 4:30 PM
  • Mario

    Thanks, but that thread is not related to my issue.

    Turns out that the muted colours are correct. All is explained here:

    https://devblogs.microsoft.com/commandline/updating-the-windows-console-colors/

    Since I don't like the muted palette that MS have now chosen for us I can at least restore the old palette - click on Properties / Colours and set each Screen Background colour according to these values:

    Color Name Console Legacy RGB Values New Default RGB Values
    BLACK         0,0,0                 12,12,12
    DARK_BLUE 0,0,128         0,55,218
    DARK_GREEN 0,128,0         19,161,14
    DARK_CYAN 0,128,128                 58,150,221
    DARK_RED 128,0,0         197,15,31
    DARK_MAGENTA 128,0,128         136,23,152
    DARK_YELLOW 128,128,0                 193,156,0
    DARK_WHITE 192,192,192         204,204,204
    BRIGHT_BLACK 128,128,128         118,118,118
    BRIGHT_BLUE 0,0,255         59,120,255
    BRIGHT_GREEN 0,255,0         22,198,12
    BRIGHT_CYAN 0,255,255                 97,214,214
    BRIGHT_RED 255,0,0         231,72,86
    BRIGHT_MAGENTA 255,0,255         180,0,158
    BRIGHT_YELLOW 255,255,0         249,241,165
    WHITE 255,255,255         242,242,242

    I now have a fully functioning NTVDM within Win 10 that replicates the behaviour in older versions of Windows and can now go ahead and transition my company's computers to Win 10.

    Mario, thank you for your time and patience with this.

    Regards

    Robert Lenk

    • Marked as answer by Robert Lenk Saturday, October 26, 2019 11:41 PM
    Saturday, October 26, 2019 11:41 PM
  • You are welcome!

    Nice find!!!

    Thanks!

    -mario

    Sunday, October 27, 2019 7:56 AM