Buffer size too small ?

[dingdongdingding]

Hi ! Hello there !

I have a script which would extract line by line from a file

for /F "tokens=* delims= " %%a in (%1) do (

Worked fine until I had a very long line. The script just fail to extract that line. Still ok for those before and after this long line.

Any solution for me ?

Thanks very much in advance !

 

[Twayne]

In

Hi ! Hello there !

I have a script which would extract line by line from a
file

for /F "tokens=* delims= " %%a in (%1) do (

Worked fine until I had a very long line. The script
just fail to extract that line. Still ok for those
before and after this long line.

Any solution for me ?

Thanks very much in advance !

Just how long is "very long"?
How long are the lines that work?

Give the folks something to work with! Like, a script in what? Batch or ...
?

 

[Paul]

In

Just how long is "very long"?
How long are the lines that work?

Give the folks something to work with! Like, a script in what? Batch or ...
?

If the source file is totally uncontrolled, the OP can
use a separate tool to pre-process it. For example, if
a Linux/Unix system puts the wrong line termination on
the file, the file could be converted to something
Windows could use. Sed, Perl, Awk could be used to
pre-process it. Once the input is "sane", it can
be passed to some other script.

Using tools like that, you can determine the longest
line in the input, and pass that information to the
batch script.

http://www.perlmonks.org/?node_id=847747

wc -L test

A Windows port of wc, as wc.exe, is available.

http://gnuwin32.sourceforge.net/packages/coreutils.htm

If you needed SED, there's even one of those.

http://gnuwin32.sourceforge.net/packages/sed.htm

Since SED is stream oriented, a long line of input
can't break it. You could even pipe the output of
SED, into some other tool you're using.

http://en.wikipedia.org/wiki/Sed

I can't vouch for whether gnuwin32 works on a
64 bit Windows OS. There might be 16 bit code in
there for all I know.

Paul

 

[dingdongdingding]

Thanks for the replies. Ok, sorry about it. I thought this was a WindowsXP group, so I just assume I'm limiting myself to talking about windows command batch files, those *.bat files, which is what I'm using here. Apologies...

Those line that works are a few hundred characters, eg 500 chars. The line which I have that does not work is 12000 characters and above.

I don't know if it's related to the buffer size of a command line. Just a thought.

Thanks a lot for your helps !

 

[Paul]

Thanks for the replies. Ok, sorry about it. I thought this was a WindowsXP group, so I just assume I'm limiting myself to talking about windows command batch files, those *.bat files, which is what I'm using here. Apologies...

Those line that works are a few hundred characters, eg 500 chars. The line which I have that does not work is 12000 characters and above.

I don't know if it's related to the buffer size of a command line. Just a thought.

Thanks a lot for your helps !

There are some limits listed here.

http://blogs.msdn.com/b/oldnewthing/archive/2003/12/10/56028.aspx

CreateProcess 32767 characters

CMD.EXE 8192 character command line length limit

ShellExecute/Ex 2048 (INTERNET_MAX_URL_LENGTH)

*******

You might have bumped into the 8192 limit.

If a file has a 12,000 character line in it, that's
no ordinary file. And perhaps a different approach to
the problem is needed. (Such as treating the file
as a stream, rather than delimited lines.) The 12,000
really implies the input file is unbounded on line length.

Paul