NNTP or Basic I/O?

[sdf]

Hey all, I have a problem here. I got some code from the web/a book (it
was actually the exactly the same), on how to do NNTP. The code works except
it's VERY cpu intensive and VERY slow. I looked into the code and noticed it
calls a Response method over and over, each time it returns a line of text.
The problem is the method itself. Response() reads data from the
NetworkStream 1 byte at a time! This of course is a terrible idea, and takes
forever. I know there's got to be a better way, b/c outlook express(which
I'm using to write this) pulls up newsgroups really fast. I've posted the
method on the end of this message. I can't seem to get the thing to go
faster - I've tried BufferedStream, StreamReader, etc. - They all read part
of the message and then block indefintly on the read method.
Hope someone can come up with something, I tried the web and no one will
answer this one.

Thanks,
Nick


Code:

private string Response()
{
System.Text.ASCIIEncoding enc = new System.Text.ASCIIEncoding();
byte [] serverbuff = new Byte[4096];
NetworkStream stream = GetStream();
int count = 0;
byte[] buff = new byte[1];

while (true)
{
int bytes = stream.Read(buff, 0, 1 );
if (bytes == 1)
{

serverbuff[count] = buff[0];
count++;

if(count > serverbuff.Length - 5)
break;

if (buff[0] == '\n')
{
break;
}
}
else
{
break;
};
};

 

[Jon Davis]

Try this, dunno if it'll work or help.

1. Wrap the stream with a StreamReader.
2. Loop ReadLine() instead of Read().

Jon

 

[Nicholas Paldino [.NET/C# MVP]]

Nick,

Unfortunately, there is little that can be done except to read it byte
by byte. The problem is in the protocol itself. There is nothing in the
protocol that lets you know how large the content is, so you can't read
chunks of data at a time. Rather, there are markers in the stream to
indicate when a request/response is complete.

Hope this helps.

 

[sdf]

Ok, but shouldn't there be a way to read until there is no more data? I've
tried StreamReader, and it says it should return -1 when the end of stream
has been reached, but it just blocks. Now there is a line in the header of
an NNTP message that goes like Lines: 514. I suppose I could parse that and
just read that many lines, but I thought there would be a way to use the
managed classes to just keep reading until I hit the end. I think I even
tried the CanRead and DataAvailable properties, and no luck.

Nick,

Unfortunately, there is little that can be done except to read it byte
by byte. The problem is in the protocol itself. There is nothing in the
protocol that lets you know how large the content is, so you can't read
chunks of data at a time. Rather, there are markers in the stream to
indicate when a request/response is complete.

Hope this helps.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

Hey all, I have a problem here. I got some code from the web/a book (it
was actually the exactly the same), on how to do NNTP. The code works except
it's VERY cpu intensive and VERY slow. I looked into the code and

noticed

it
calls a Response method over and over, each time it returns a line of text.
The problem is the method itself. Response() reads data from the
NetworkStream 1 byte at a time! This of course is a terrible idea, and takes
forever. I know there's got to be a better way, b/c outlook express(which
I'm using to write this) pulls up newsgroups really fast. I've posted the
method on the end of this message. I can't seem to get the thing to go
faster - I've tried BufferedStream, StreamReader, etc. - They all read part
of the message and then block indefintly on the read method.
Hope someone can come up with something, I tried the web and no one will
answer this one.

Thanks,
Nick


Code:

private string Response()
{
System.Text.ASCIIEncoding enc = new System.Text.ASCIIEncoding();
byte [] serverbuff = new Byte[4096];
NetworkStream stream = GetStream();
int count = 0;
byte[] buff = new byte[1];

while (true)
{
int bytes = stream.Read(buff, 0, 1 );
if (bytes == 1)
{

serverbuff[count] = buff[0];
count++;

if(count > serverbuff.Length - 5)
break;

if (buff[0] == '\n')
{
break;
}
}
else
{
break;
};
};

 

[Nicholas Paldino [.NET/C# MVP]]

sdf,

With network streams, you don't know when there is no more data, unless
the connection is closed. It is for this reason that you have delimited
messages, or fixed-length messages.

You could use the Lines header, but I don't believe that it is a
standard header, and I don't know if you can depend on it.

You could read each line using ReadLine (since the specification
indicates that a line is finished by a control feed/new line). What you
have to do is know when the message is complete, which is indicated by a
single period on a line (I believe).

This makes the processing from a programattic view not much different
from reading the lines header, since you would just change how you control
your loop (instead of looping to the number of lines, you loop while the
line is not a single period).


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

Ok, but shouldn't there be a way to read until there is no more data? I've
tried StreamReader, and it says it should return -1 when the end of stream
has been reached, but it just blocks. Now there is a line in the header of
an NNTP message that goes like Lines: 514. I suppose I could parse that and
just read that many lines, but I thought there would be a way to use the
managed classes to just keep reading until I hit the end. I think I even
tried the CanRead and DataAvailable properties, and no luck.


message news:[email protected]...

Nick,

Unfortunately, there is little that can be done except to read it byte
by byte. The problem is in the protocol itself. There is nothing in the
protocol that lets you know how large the content is, so you can't read
chunks of data at a time. Rather, there are markers in the stream to
indicate when a request/response is complete.

Hope this helps.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

Hey all, I have a problem here. I got some code from the web/a book (it
was actually the exactly the same), on how to do NNTP. The code works except
it's VERY cpu intensive and VERY slow. I looked into the code and

noticed

it
calls a Response method over and over, each time it returns a line of text.
The problem is the method itself. Response() reads data from the
NetworkStream 1 byte at a time! This of course is a terrible idea, and takes
forever. I know there's got to be a better way, b/c outlook express(which
I'm using to write this) pulls up newsgroups really fast. I've posted the
method on the end of this message. I can't seem to get the thing to go
faster - I've tried BufferedStream, StreamReader, etc. - They all

read
part

of the message and then block indefintly on the read method.
Hope someone can come up with something, I tried the web and no one will
answer this one.

Thanks,
Nick


Code:

private string Response()
{
System.Text.ASCIIEncoding enc = new System.Text.ASCIIEncoding();
byte [] serverbuff = new Byte[4096];
NetworkStream stream = GetStream();
int count = 0;
byte[] buff = new byte[1];

while (true)
{
int bytes = stream.Read(buff, 0, 1 );
if (bytes == 1)
{

serverbuff[count] = buff[0];
count++;

if(count > serverbuff.Length - 5)
break;

if (buff[0] == '\n')
{
break;
}
}
else
{
break;
};
};

 

[Jon Davis]

Ok, but shouldn't there be a way to read until there is no more data?

Yes, it's called ReadToEnd() on the StreamReader, but that's not what you
want.

The stream doesn't terminate until the TCP session terminates, so CanRead
will always be true even if it blocks. Don't get HTTP and NNTP confused.
HTTP is stateless, so it typically closes the connection when the response
is complete, but NNTP is stateful, so it remains connected and you will have
to read the output line by line.

Jon

Ok, but shouldn't there be a way to read until there is no more data? I've
tried StreamReader, and it says it should return -1 when the end of stream
has been reached, but it just blocks. Now there is a line in the header of
an NNTP message that goes like Lines: 514. I suppose I could parse that and
just read that many lines, but I thought there would be a way to use the
managed classes to just keep reading until I hit the end. I think I even
tried the CanRead and DataAvailable properties, and no luck.


message news:[email protected]...

Nick,

Unfortunately, there is little that can be done except to read it byte
by byte. The problem is in the protocol itself. There is nothing in the
protocol that lets you know how large the content is, so you can't read
chunks of data at a time. Rather, there are markers in the stream to
indicate when a request/response is complete.

Hope this helps.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

Hey all, I have a problem here. I got some code from the web/a book (it
was actually the exactly the same), on how to do NNTP. The code works except
it's VERY cpu intensive and VERY slow. I looked into the code and

noticed

it
calls a Response method over and over, each time it returns a line of text.
The problem is the method itself. Response() reads data from the
NetworkStream 1 byte at a time! This of course is a terrible idea, and takes
forever. I know there's got to be a better way, b/c outlook express(which
I'm using to write this) pulls up newsgroups really fast. I've posted the
method on the end of this message. I can't seem to get the thing to go
faster - I've tried BufferedStream, StreamReader, etc. - They all

read
part

of the message and then block indefintly on the read method.
Hope someone can come up with something, I tried the web and no one will
answer this one.

Thanks,
Nick


Code:

private string Response()
{
System.Text.ASCIIEncoding enc = new System.Text.ASCIIEncoding();
byte [] serverbuff = new Byte[4096];
NetworkStream stream = GetStream();
int count = 0;
byte[] buff = new byte[1];

while (true)
{
int bytes = stream.Read(buff, 0, 1 );
if (bytes == 1)
{

serverbuff[count] = buff[0];
count++;

if(count > serverbuff.Length - 5)
break;

if (buff[0] == '\n')
{
break;
}
}
else
{
break;
};
};

 

[Dave Downs]

Hey all, I have a problem here. I got some code from the web/a book (it
was actually the exactly the same), on how to do NNTP. The code works except
it's VERY cpu intensive and VERY slow. I looked into the code and noticed it
calls a Response method over and over, each time it returns a line of text.
The problem is the method itself. Response() reads data from the
NetworkStream 1 byte at a time! This of course is a terrible idea, and takes
forever. I know there's got to be a better way, b/c outlook express(which
I'm using to write this) pulls up newsgroups really fast. I've posted the
method on the end of this message. I can't seem to get the thing to go
faster - I've tried BufferedStream, StreamReader, etc. - They all read part
of the message and then block indefintly on the read method.
Hope someone can come up with something, I tried the web and no one will
answer this one.

Thanks,
Nick

Have a look at RFC 850: Standard for Interchange of USENET messages:
http://www.faqs.org/rfcs/rfc850.html

and also RFC 977: Network News Transfer Protocol:
http://www.faqs.org/rfcs/rfc977.html

no code, but it gives you a view as to how the protocol works. You may also
want to look into using asynchronous data transfer so that, despite your app
taking a long time to run the NNTP code, you can still do other things.

 

[Niki Estner]

Have you tried setting the ReceiveTimeout using Socket.SetSocketOption
(TcpClient has a property for it)?
I don't know if it works, but if the docs are right, setting the timeout to
a few ms should make it possible to read big chunks of data without blocking
forever if the end of the data is reached.

Niki

 

[Michael Culley]

Have you tried setting the ReceiveTimeout using Socket.SetSocketOption
(TcpClient has a property for it)?
I don't know if it works, but if the docs are right, setting the timeout to
a few ms should make it possible to read big chunks of data without blocking
forever if the end of the data is reached.

But what happens if it takes a few ms for your data to arrive?

 

[Niki Estner]

But what happens if it takes a few ms for your data to arrive?

Maybe I got sdf's question wrong: I had the impression he was looking for a
way to read couple of bytes in a single call (without blocking forever)
instead of doing one call per byte. Wouldn't a low timeout do exactly that?

To answer your question: He would of course have to keep reading from the
socket until some finish-marker is reached (if I understood his code, until
'\n' arrives). Without adjusting the timeout, he would have to do pretty
much the same (reading until '\n' arrived), wouldn't he?
The only difference I see is that the code should be way faster if the data
does arrive in time. And that's what sdf asked for...

Niki

 

[Michael Culley]

Maybe I got sdf's question wrong: I had the impression he was looking for a
way to read couple of bytes in a single call (without blocking forever)
instead of doing one call per byte. Wouldn't a low timeout do exactly that?

To answer your question: He would of course have to keep reading from the
socket until some finish-marker is reached (if I understood his code, until
'\n' arrives). Without adjusting the timeout, he would have to do pretty
much the same (reading until '\n' arrived), wouldn't he?
The only difference I see is that the code should be way faster if the data
does arrive in time. And that's what sdf asked for...

If you set a low timeout you will keep getting timeouts because data from the internet can easily be delayed. I think the preferred
option is to start a new thread and set the timeout at 30 or 60 seconds and read the data byte at a time. I did this for an nntp
program and it worked fine.

 

[sdf]

Ok, well I found the timeout method. It's called ReceiveTimeout, and it's a
member of TcpClient. I set it, and it does timeout, but it doesn't seem to
do it very fast. I wrapped a StreamReader around the stream, and I'm reading
line by line, but it's either blocking elsewhere, or just taking forever.
Here's a link to where I got the code:
http://www.developerfusion.com/show/4472/ He's got some small errors in his
code. Not so much errors really as things that don't make sense, like using
a two byte array when he only needs one, and only using the first 1k of the
message, or getting only 100 messages at a time.
But if anyone maybe improve on the (in)effeciency of that response method,
I'd be very interested in seeing your code.

Thanks for all the help,

Nick

Have you tried setting the ReceiveTimeout using Socket.SetSocketOption
(TcpClient has a property for it)?
I don't know if it works, but if the docs are right, setting the timeout to
a few ms should make it possible to read big chunks of data without blocking
forever if the end of the data is reached.

Niki

Hey all, I have a problem here. I got some code from the web/a book (it
was actually the exactly the same), on how to do NNTP. The code works except
it's VERY cpu intensive and VERY slow. I looked into the code and

noticed

it
calls a Response method over and over, each time it returns a line of text.
The problem is the method itself. Response() reads data from the
NetworkStream 1 byte at a time! This of course is a terrible idea, and takes
forever. I know there's got to be a better way, b/c outlook express(which
I'm using to write this) pulls up newsgroups really fast. I've posted the
method on the end of this message. I can't seem to get the thing to go
faster - I've tried BufferedStream, StreamReader, etc. - They all read part
of the message and then block indefintly on the read method.
Hope someone can come up with something, I tried the web and no one will
answer this one.

Thanks,
Nick


Code:

private string Response()
{
System.Text.ASCIIEncoding enc = new System.Text.ASCIIEncoding();
byte [] serverbuff = new Byte[4096];
NetworkStream stream = GetStream();
int count = 0;
byte[] buff = new byte[1];

while (true)
{
int bytes = stream.Read(buff, 0, 1 );
if (bytes == 1)
{

serverbuff[count] = buff[0];
count++;

if(count > serverbuff.Length - 5)
break;

if (buff[0] == '\n')
{
break;
}
}
else
{
break;
};
};

 

[Joris Dobbelsteen]

The best way it to read large blocks.

public int Socket.Read(byte[] buffer)

is what you need. This reads the largest possible block that is currently
available.

If you put the socket in non-blocking mode, it throws an exception when no
data is available, otherwise it will block until its available.
It reads the data until either: no more data received or buffer full.

- Joris

 

[sdf]

Ok I finally got a solution. I tried the multithreaded approach like
someone here suggested, but it didn't matter when i read it byte by byte
because it was such a CPU intensive task that my P4 was maxed out on only 1
thread to begin with. But when you use a StreamReader and do a
reader.ReadLine() it actually does work. You just have to keep checking for
the '.' single period line meaning EOF in NNTP talk, or a line starting with
423, indicating No Such Article.
The thing about this method is, it's not actually any faster than byte by
byte for a given article download, but it is MUCH less work for the
processor, meaning that multiple threads are now a good idea again, and you
should be able to get 5-10 times the speed if you're still willing to max
out the CPU. Of course my NNTP server(Alltel here) only allows 6
simultaneous connections from 1 IP, but still, 6 times the speed is a big
deal.

Thanks for all the ideas!

Nick