Handling Process Input and Output Streams

[jpalmer]

Hi,

I would like to implement some c# .NET 2.0 code that pipes an input .NET
stream into an external process and then captures the process’s stdout and
stderr into.NET streams. I was wondering what the best way to do this is.

My current thinking is to use threads to implement the reading and writing
streams in parallel. However starting all of these threads seems quite a lot
of overhead for such a simple task.

The basic sort of thing I trying to do as run on the command line looks like
this (using a command line utility called crx2rnx):

cat infile | crx2rnx - > outfile.

Note that running this command works and no data is output on the stderr.

Here's my attempt at the code using threads. This current implementation
runs but the the thread that is running the WriteProcStdOut() method does not
complete. It seems to be locked on the "while (stdout.Read(buf, 0, bufSize) >
0)" method call.

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.Diagnostics;
using System.Threading;

public class Program
{
private static Process _process;
private static FileStream _inputStream;
private static FileStream _outputStream;

public static void Main(string[] args)
{
string compactFile = "infile";
string outputRinexFile = "outfile";
try
{
_inputStream = new FileStream(
compactFile, FileMode.Open, FileAccess.Read);
_outputStream = new FileStream(
outputRinexFile, FileMode.Create, FileAccess.Write);

_process = new Process();
_process.StartInfo.FileName = "crx2rnx";
_process.StartInfo.Arguments = "-";
_process.StartInfo.RedirectStandardOutput = true;
_process.StartInfo.RedirectStandardInput = true;
_process.StartInfo.UseShellExecute = false;
_process.StartInfo.CreateNoWindow = true;

_process.Start();
new Thread(new ThreadStart(ReadProcStdIn)).Start();
new Thread(new ThreadStart(WriteProcStdOut)).Start();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}

private static void ReadProcStdIn()
{
try
{
const int bufSize = 1024;
byte[] buf = new byte[bufSize];
Stream stdin = _process.StandardInput.BaseStream;
while (_inputStream.Read(buf, 0, bufSize) > 0)
{
stdin.Write(buf, 0, bufSize);
}
}
catch (Exception e)
{
Console.Error.WriteLine("Error Writing to process stdin:"
+ e.Message);
}
}

private static void WriteProcStdOut()
{
try
{
const int bufSize = 1024;
byte[] buf = new byte[bufSize];
Stream stdout = _process.StandardOutput.BaseStream;
while (stdout.Read(buf, 0, bufSize) > 0)
{
_outputStream.Write(buf, 0, bufSize);
}
}
catch (Exception e)
{
Console.Error.WriteLine("ShowProcStdOut:" + e.Message);
}
}
}


Initially I thought that the "crx2rnx" process was not flushing its stdout.
However I have access to the program's C source code as was able to ensure
that a fflush() call was made before the process ended. So I'm currently not
sure where to go from here...

Thanks
Jeremy

 

[Pavel Minaev]

Hi,

I would like to implement some c# .NET 2.0 code that pipes an input .NET
stream into an external process and then captures the process’s stdout and
stderr into.NET streams. I was wondering what the best way to do this is.

My current thinking is to use threads to implement the reading and writing
streams in parallel. However starting all of these threads seems quite a lot
of overhead for such a simple task.

It's not even clear why you would even need threads in this case. If
the process you're running is a simple filter/converter from its stdin
to stdout, then couldn't you just alternate Write & Read calls on its
stdin/stdout on a single thread for the same effect?

 

[jpalmer]

So how could this be done for Stdin, stderr, stdout within one thread without
first reading the whole stdin, and not causing a deadlock with alternate
read and write calls?

Example...?

 

[Pavel Minaev]

So how could this be done for Stdin, stderr, stdout within one thread without
first reading the whole stdin, and not  causing a deadlock with alternate
read and write calls?

Yes, I see the problem now. I must admit I'm not sure about this, but
have you tried using StreamReader.EndOfStream to check if there's
anything to read before reading it? So basically it would go like
this:

write stuff to stdin of the process
while there's anything to read from stdout or stderr of the process,
read it; repeat until there's nothing left in either one

 

[jpalmer]

The first part, "write stuff to stidin", can block if too much is written
at once. Then the second part will never be reached. MSDN is pretty
clear about making sure to not attempt to handle input and output from the
same thread. If both (or all three) are redirected, they need to be dealt
with in _some_ kind of asynchronous manner to ensure no deadlock happens.

Can I please have the link to the MSDN reference?

I'm still not clear about your comments here. My current example deals with
the stdin and stdout in separate threads. Are you talking about having a
separate thread that reads my file input stream into a buffer, then have
another thread that reads from that buffer (in a thread safe manner) and then
writes that data to the stdin? And repeat for stdout and output file steam.

As far as the original question goes, I don't see the problem with having
separate threads. Assuming you're only creating a handful of them, it's a
perfectly reasonable approach. That said, if you really want to be
(slightly) more efficient, you can get the streams directly and use the
async i/o methods (e.g. BeginRead(), BeginWrite()). That's a little
harder, because you wind up having to deal with character encoding
explicitly (there aren't any async methods for the
StreamReader/StreamWriter classes). But it's somewhat more efficient
because it uses (when supported) IOCP and the associated i/o thread pool.

Personally, I'd just run some threads.

Pete

Ok I'll take the thread approach.

 

[jpalmer]

I was responding to Pavel's post, not yours. His seemed to imply doing
all of the writing at once, and not reading from the redirected output
until that was all done. That won't work, at least not for any
significant amount of data.

OK thanks

I'm not talking about any specific implementation; just pointing out that
it's not good to mix synchronous access to more than one redirected
process stream in a single thread.

You could do as you suggest, though it seems to me it would make more
sense to simply have one thread write to the process StandardInput stream
using data read from your file in the same thread, and likewise have one
thread read from the StandardOutput stream and write that output to your
output file in that same thread. In other words, just two threads...one
to deal with the process's input and one to deal with its output.

Pete

OK so no synchronous access to the redirected streams. I've done that in the
above example by using threads. So why does a deadlock occur in the code?

 

[jpalmer]

there's no way for anyone to _run_ your code so as to reproduce the
problem (in the event that the bug is not self-evident upon inspection of
the code).

Ok thanks. I should have really posted the binaries and source for the
program upfront!

The source code for crx2rnx is located here
ftp://terras.gsi.go.jp/software/RNXCMP_4.0.3/RNXCMP_4.0.3_src.tar.gz. The
source file is located at "source/crx2rnx.c" within the tar.

The binary is located here:
ftp://terras.gsi.go.jp/software/RNXCMP_4.0.3/RNXCMP_4.0.3_Windows.tar.

A test input file is located here:
ftp://ftp.geonet.org.nz/gps/rinex/2008/001/auck0010.08d.Z. Just remember to
UNIX decompress the file first.