none
problem with record terminator on ftp'd files RRS feed

  • Question

  • Using Win7 Pro 64bit

    This is not strictly a scripting issue, but it doesn't seem to fit neatly into any other forum, so I figured that there might be sufficient expertise here.

    and I am doing this in a script ... ;-)

    I ftp a plain text file from a linux server.  File gets proper record-delimiter translation.  Open the file with a hex editor and I can see the x'0D0A'.  But if I open the file in Notepad, it comes up as one long run-on line.  And if I try to process it with a vbscript, a Line Input command returns the entire file. 

    I've done a lot of ftp across multiple systems (dos, Windows, OS/2, OS/MVS, various *nix) over many years, and have never seen anything like this.

    Ideas?

    Thursday, June 12, 2014 2:58 PM

Answers

  • Step 1 is to identify the actual problem (sometimes, perhaps even often, this isn't what the problem seems to be). This can be tricky and time consuming.

    Step 2 is to create a reproducible use case (i.e., give someone else the ability to recreate the exact thing you're seeing). Usually this step helps bring the problem into sharp focus.

    Step 3 is to search for information regarding this specific problem. Very often the previous steps will help you to figure out what you can start searching for.


    -- Bill Stewart [Bill_Stewart]

    Thursday, June 12, 2014 6:00 PM
    Moderator

All replies

  • You are not setting your download to convert the line delimiters.  You need to set that or know that Unoix text files do not use the same delimiters as Windows and mat notbebyte ordered the same.

    If you are setting a binary download this will be the case.

    This is not a scripting issue.  It is a basic issue of system compatibility and how to use FTP.

    If you are on Windows Vista or later the file should open correctly as long as the character sets are correct and the Unicode header is correct.

    Contact you system support people to help you sort this out.

    Try also researching Unicode differences.

    I retrieve Unix files in text mode all of the time with no issues.  The problem is in how you are doing this.  We cannot see what you are doing so we cannot help.

    Again: not a scripting issue.


    ¯\_(ツ)_/¯

    Thursday, June 12, 2014 3:13 PM
  • I am fully aware that line delimiters are not the same in *nix as in dos/Windows.  That's why I specifically pointed out that viewing the file in a hex editor reveals the proper dow/Windows terminators of x'0D' x'0A'.

    I am specifically setting the download to ASCII to force the transalation, and everything I see in a hex editor says the translation is being done properly.

    I, too, retrieve unix files in txt mode all the time.  See my original comments on the subject. I am fully aware of the translation between *nix and dos/Windows - as well as ebcdic/ascii translations, when needed.

    I am fully aware that this is not a scripting issue, per se.  See my disclaimer on that subject.

    Thursday, June 12, 2014 3:54 PM
  • It is still not a scripting issue.


    ¯\_(ツ)_/¯

    Thursday, June 12, 2014 3:59 PM
  • What are the first 4 bytes of the file.


    ¯\_(ツ)_/¯

    Thursday, June 12, 2014 3:59 PM
  • The first 4 bytes are '2014', or hex '32 30 31 34'

    Plain text, exactly as expected.

    Thursday, June 12, 2014 4:37 PM
  • Then the issue is with your system or your analysis.

    This is not a scritping issue.  Yu need to take a different approach.

    I will try to help you but only if you can upload the file for me to look at.


    ¯\_(ツ)_/¯

    Thursday, June 12, 2014 4:40 PM
  • What is notepad seeing the files as.  What does it say if you do a "Save As" and look at the file type.


    ¯\_(ツ)_/¯

    Thursday, June 12, 2014 4:42 PM
  • It is still not a scripting issue.


    ¯\_(ツ)_/¯


    Fine.  So recommend a forum that is a better fit.
    Thursday, June 12, 2014 4:44 PM
  • I still think you are downloading the file in binary mode.  There is no "Ascii" setting in FTP downloads.  It is either binary mode or text mode.

    YOu are downloading a Unicode Unix file You binary editor is not showing you the real first four characters of the file.  This are in strict binary and not printable.  THe first character is usually 0xFF


    ¯\_(ツ)_/¯


    • Edited by jrv Thursday, June 12, 2014 4:54 PM
    Thursday, June 12, 2014 4:50 PM
  • So recommend a forum that is a better fit.

    You can use a search engine to look for a suitable forum (?).

    It is not considered good etiquette to ask off-topic questions just because you might find helpful respondents.


    -- Bill Stewart [Bill_Stewart]

    Thursday, June 12, 2014 4:51 PM
    Moderator
  • So recommend a forum that is a better fit.

    You can use a search engine to look for a suitable forum (?).

    It is not considered good etiquette to ask off-topic questions just because you might find helpful respondents.


    -- Bill Stewart [Bill_Stewart]

    I know.  That's why I gave disclaimers.  Sometimes there isn't a clear place to start, so one starts where one might at least find some knowledgeable people.  I'm willing to be pointed to a more appropriate forum.  Can someone recommend a more appropriate forum?  Or at least a search argument that's not so broad and vague as to be meaningless to my question?
    Thursday, June 12, 2014 5:45 PM
  • I still think you are downloading the file in binary mode.  There is no "Ascii" setting in FTP downloads.  It is either binary mode or text mode.



    ¯\_(ツ)_/¯


    My Windows ftp client would beg to differ

    331 Please specify the password.
    Password:
    230 Login successful.
    ftp> asci
    200 Switching to ASCII mode.
    ftp>
    ftp> text
    Invalid command.
    ftp>

    But this isn't a scripting question. I get it.  I knew that coming in.  My question doesn't fit neatly to search for a better place.  Surely someone has an idea of a forum that is a better fit.

    Thursday, June 12, 2014 5:58 PM
  • Your issue is more one of a need for technical support.  The issues you outline are technically not possible unless you are missing key information or you have system or software failures in your system.  This is best addressed by a competent technical support group.

    Example: 

    You say you are transferring in ascii.  ASCII is a character encoding it is not a file format like binary or text/Unicode.  Notepad does not display ASCII as an option. It displays ANSI/Unicode/UTF-8. The character encoding is set globally by Windows. This determins how hex values like 0xD and 0xA are displayed.

    If you are looking at binary and you see 00 0D 00 0A then your file is Unicode and not ANSI.  If it is missing the first four binary characters it will not display correctly.  This is usually because you have a mismatch in the download.

    I have used and modified FTP clients for decades.  The issue iss not uncommon when a remote file is downloaded incorrectly.

    You can try posting in the forum for the FTP server that you are downloading from.  That would be the site for the version of Linux you are using.

    If as yo say the first absolute 4 characters of the file are in fact 2014 and you are using a true binary editor and not just a binary enabled text editor then you likely have a character set issue or hardware issue.  For that contact MS Support for assistance.

    As I posted earlier.  I will help you if you zip up the file and upload it to a sharing service.  I have very good binary analysis tools on muy development system and can quickly tell you if the file is build correctly.  If you can't do that then I guess you are just on your own and will need to call a consultant or support.

    Sorry but that is as much as we can do.


    ¯\_(ツ)_/¯

    Thursday, June 12, 2014 5:59 PM
  • Step 1 is to identify the actual problem (sometimes, perhaps even often, this isn't what the problem seems to be). This can be tricky and time consuming.

    Step 2 is to create a reproducible use case (i.e., give someone else the ability to recreate the exact thing you're seeing). Usually this step helps bring the problem into sharp focus.

    Step 3 is to search for information regarding this specific problem. Very often the previous steps will help you to figure out what you can start searching for.


    -- Bill Stewart [Bill_Stewart]

    Thursday, June 12, 2014 6:00 PM
    Moderator
  • I still think you are downloading the file in binary mode.  There is no "Ascii" setting in FTP downloads.  It is either binary mode or text mode.



    ¯\_(ツ)_/¯


    My Windows ftp client would beg to differ

    331 Please specify the password.
    Password:
    230 Login successful.
    ftp> asci
    200 Switching to ASCII mode.
    ftp>
    ftp> text
    Invalid command.
    ftp>

    But this isn't a scripting question. I get it.  I knew that coming in.  My question doesn't fit neatly to search for a better place.  Surely someone has an idea of a forum that is a better fit.

    Sorry you are correct.  THe toggle is ascii.  ASCII is the default. I always use binary when I need to switch.

    Try downloading in binary mode then looking at the file.  ASCII may strip the first 4 bytes on some systems.  The mode is set on the remote system which is why I say to contact the Linux people.  BINARY will transfer the file with no changes.


    ¯\_(ツ)_/¯

    Thursday, June 12, 2014 6:04 PM
  • OK, part of the mystery is solved.  The hex editor I was using on the Windows desktop (UltraEdit) was doing its own translation of the record terminator. 

    Looking at the original file, on the linux system, with vi in hex mode, clearly shows it is a pure text file (no leading 4 characters of something other than my data), and clearly shows the unix record terminator of x'0A'. 

    I installed a different hex editor on my desktop (HxD).  Did a binary ftp and confirmed that the record terminator was still simpy x'0A'.  Did an ascii ftp (very explicit) and the resulting file still had just x'0A' -- not x'0A' x'0D'.

    Looks like something is awry with my ftp client.  But at least with that I have something I can get my hands around and can find a more appropriate forum.

    Thursday, June 12, 2014 6:43 PM
  • Just for the sake of closure ... looks like I'll have to let this one lie.

    File has sensitive info, so I can't upload it. 

    I can now draw a pretty tight fence around the problem.  Along with other tests I've run, I think I can find a more appropriate forum.

    Thursday, June 12, 2014 8:27 PM
  • Just for the sake of closure ... looks like I'll have to let this one lie.

    File has sensitive info, so I can't upload it. 

    I can now draw a pretty tight fence around the problem.  Along with other tests I've run, I think I can find a more appropriate forum.

    The problem is what I said it was in the beginning.  The editor you are using is not a true binary editor and the file you are getting is a Unix file not a Windows file.

    Setting ASCII does not6 translate line terminiators.  You need to use a file converter for that.

    The issue you claimed in the beginning was misleading to us and to yourself because this waould never have worked any differently any time in the past.  On any system.

    You need to 5take issues like this to your support people or to those trained in computer technology and not just in use of the desktop.  These issues are common across all platforms.  That is why you claim from the beginning was so suspicious.  You were not telling us or seeing the real story.  The file never had CrLf pairs.  Your tools fooled you. This also why computer engineers and certified techs get the big bucks.  Call you tech support first the next time after searching the net and refining the problem as Bill suggested above.  You will save yourself a lot of frustration.


    ¯\_(ツ)_/¯


    • Edited by jrv Thursday, June 12, 2014 8:51 PM
    Thursday, June 12, 2014 8:51 PM