Process.BeginErrorReadLine and BeginOutputReadLine; buffer notflushed?

[robert.waters]

I have a thread that uses the Process object to execute an external
script, and redirects the stdout and stderr output from that script
asyncronously (using BeginErrorReadLine and BeginOutputReadLine and a
DataReceivedEventHandler for each output handle).
However, the output is not being sent to the event handlers when it is
flushed in the external script.

The (perl) script is simple, it just iterates and prints a string to
stdout when the iterator is even, and a string to stderr when it is
odd; output is flushed after each print. When I run this script in
the console, I get the expected output (alternating stdout/stderr
strings). However, when I run this script using the Process object,
the strings appear in bunches of one or the other. (If I add a
1/10th second delay to the perl script during each iteration, the
Process object works as expected, most of the time.)

So, the event handlers are not being triggered when output from the
script is flushed; they seem to have their own internal buffers that
flush at an indeterminate (to me) time. The
Process.StandardOutput.BaseStream object has a Flush() method, but
this cannot be called when the stream is being read asynchronously.

Please help!
Thanks,
Robert Waters

 

[robert.waters]

I have a thread that uses the Process object to execute an external
script, and redirects the stdout and stderr output from that script
asyncronously (using BeginErrorReadLine and BeginOutputReadLine and a
DataReceivedEventHandler for each output handle).
However, the output is not being sent to the event handlers when it is
flushed in the external script.

The (perl) script is simple, it just iterates and prints a string to
stdout when the iterator is even, and a string to stderr when it is
odd; output is flushed after each print.  When I run this script in
the console, I get the expected output (alternating stdout/stderr
strings).  However, when I run this script using the Process object,
the strings appear in bunches of one or the other.  [...]

I suspect that this is a side-effect of the complete end-to-end  
implementation details.  Unfortunately, none of the parts you're using  
provide any sort of guarantee that the stdout and stderr will in any way  
be synchronized, which means that there's a fundamental problem with the  
specific approach.

That said, you _might_ be able to improve the situation by handling the  
reads yourself rather than letting the Process "read line" behavior do  
it.  In particular, call BeginRead() directly on the stream returned by 
Process.StandardOutput.BaseStream (and likewise for StandardError).  That  
will at least allow you to bypass the buffering that goes on in the text  
reader (but you'll have to handle the character output encoding yourself)..

I'm not all that hopeful the above would work though.  I mean, it's worth  
a try.  But it's not just the StreamReader that does buffering.  The  
OS/console itself is also buffering.  There's a certain amount of output  
that the process can send out on the stdout and stderr streams, a few  
hundred bytes if I recall correctly, before the process itself will  
block.  That means that if the process gets ahead of your reading process  
at all, the reading process can still wind up in a thread that's happy to 
keep reading the stream it's working on, while the OS buffers output sent 
to the other string.

All of this assumes that there's not actually a set order to how output  
goes to each stream.  I'm assuming that your "odd/even" test case is just  
a simplification to narrow down the problem and that in reality you've got  
output going to each string arbitrarily.  Of course, if it's not, then one  
obvious solution is to simply set up a bottleneck queue that alternates  
between which stream it allows output from, on a line-per-line basis.

But somehow I suspect your actual real-world problem isn't that simple.  
The bottom line here is that you seem to be using stdout and stderr in a  
way that is contrary to their intended use, wanting for a guarantee that  
was never designed into the console output architecture in the first  
place.  Ultimately, that sort of approach is likely to lead to failure,or  
at the very least to some fairly awkward or ugly hacking to get things to 
work the way you want.

Pete

Thanks Pete. "Ugly Hacking" is looking like the only resolution to
this problem.
I found some related information:
"The Process class use a internal class AsyncStreamReader of which
constructor the buffer size is fixed as 1024."; so, without re-
implementing the Process class, I am out of luck.

The bottom line here is that you seem to be using stdout and stderr in a
way that is contrary to their intended use

Pet, you've read me perfectly. I have a tendency to use stderr as a
logging mechanism (for debugging), which keeps stdout clean-and-neat.
I suppose I'll have to start logging to file like everyone else

Thank you so much for your time and your help; I wish my question
could have been as succinct as your answer.
-Robert