XmlDocument read from Network Stream

[Yechezkal Gutfreund]

I have been using the following code (successfully) to read Xml formated
text packets from a TCP stream. The output from the server stream consists
of a sequence of well formed Xml documents written to the output stream.


We are willing to pay $ to any expert (e.g. MVP) consultant who has to time
to help us track down this problem.
(we will discuss rates if you can prove your expertise, and problem solving
approach).

The code below is for the client that is reading the stream.

All this worked well, as long as long delays were between each "packet". But
when multiple packets are sent in quick succession, it is dropping packets.
I have been trying to do this more "correctly" with using a XmlTextReader,
but with no luck.

E.g.

XmlTextReader r = new XmlTextReader();
reader.MoveToContent();
string s = reader.ReadOuterXml();
dgram.Load(s);

It basically hangs on the ReadOuterXml();

Here is the original "working" code:
--------------------------------------------

private void OrchTalk()
{
int bytesRead = 0;
XmlDocument dgram = new XmlDocument();

while (true)
{
byte[] buffer = new byte[1024]; // buffer for socket read stream
bytesRead = 0;
try { bytesRead = oStream.Read(buffer, 0, buffer.Length); }
catch (Exception e)
{
if (stateCode == Common.stateCodes.shutdown) {break;}
Tools.Debug(10, "OrchTalk: read error {0} ", e.ToString());
break;
}
if (bytesRead == 0)
{
if (this.stateCode == Common.stateCodes.shutdown)
{
Tools.Debug(10, "OrchTalk: Orchestrator died");
this.Disconnect();
return;
}
break;
}
else
{
string s = Tools.ByteToStr(buffer, 0);
try { dgram.LoadXml(s); }
catch (Exception e)
{ Tools.Debug(0, "OnReadComplete: {0} Runt packet: {1}", e.ToString(),
s); }
int endTime = Environment.TickCount;
this.inPacket = dgram.DocumentElement;
Tools.Debug(10, "packet: {0}", inPacket.OuterXml);
this.Dispatch();
int startTime = (int)this.stats["start"];
int elapsed = endTime - startTime;
this.stats["stop"] = endTime;
this.stats["time"] = elapsed;
this.stats["bytes"] = bytesRead;
this.stats["packetText"] = inPacket.OuterXml;
if (StationMaster.isConnected)
{
string outPacket = inPacket.OuterXml;
FlashComm.itself.Send(outPacket);
}
}
}
this.Disconnect();
}

 

[Yan-Hong Huang[MSFT]]

Hello Yechezkal,

Thanks for posting in the group.

Based on my understand, now the problem is that: If there is enough delay
time, the problem won't happen. If the packets is received quickly, the
program will hang when calling ReadOuterXML(). Please correct me if I have
misunderstood the problem.

I have done many network programming before and met such problem also.
Sometimes if the transfer high is too high, we may lose some data from
network. I am willing to share my experience here.

Under this situation, firstly we have data buffers in the main thread which
will receive all the data from socket. The main work here is to copy the
data into buffers. How to design the buffer is up to you. I designed it as
a buffer list before and there are some bits in the buffer for sign.

For the main job of reading the content of the buffer, we spawn a worker
thread to do it specially. It continues reading contents from buffers if it
has new data. In this way, we could avoid the problem as much as possible.

Hope that helps. Also, let's keep an eye out in the community to see if
others have any suggestions on this issue.

Best regards,
Yanhong Huang
Microsoft Online Partner Support

Get Secure! ¨C www.microsoft.com/security
This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Chris Lovett]

You say the output from the server is a sequence of well formed XML
documents. Now by definition this means each document fragment is "variable
length", so the fact that you are attempting to call LoadXml() with partial
fragments in your 1024 byte buffer is completely wrong. LoadXml expects a
complete document fragment, you cannot call it repeatedly with partial
fragments.

So try this:


public class XmlStreamReader {
XmlTextReader reader;
XmlDocument doc;

public XmlStreamReader(string url) {
XmlNameTable nt = new NameTable();
XmlNamespaceManager nsMgr = new XmlNamespaceManager(nt);
XmlParserContext context = new XmlParserContext(nt, nsMgr, null,
XmlSpace.None);
WebRequest wr = WebRequest.Create(url);
Stream stream = wr.GetResponse().GetResponseStream();
this.reader = new XmlTextReader(stream, XmlNodeType.Element,
context);
this.doc = new XmlDocument();
}

public XmlElement ReadXmlElement() {
XmlNode node = null;
while ((node = doc.ReadNode(reader)) != null && !(node is
XmlElement)) {
}
return node as XmlElement;
}
}

and test it like this:

public class Test {
static void Main(string[] args) {
string url = args[0];
XmlStreamReader r = new XmlStreamReader(url);
XmlElement e;
while ((e = r.ReadXmlElement()) != null) {
Console.WriteLine(e.OuterXml);
}
return;
}
}

This might not work if the server injects an <?xml declaration and/or
<!DOCTYPE tag at the start of each document fragment...

Chris.

I have been using the following code (successfully) to read Xml formated
text packets from a TCP stream. The output from the server stream consists
of a sequence of well formed Xml documents written to the output stream.


We are willing to pay $ to any expert (e.g. MVP) consultant who has to time
to help us track down this problem.
(we will discuss rates if you can prove your expertise, and problem solving
approach).

The code below is for the client that is reading the stream.

All this worked well, as long as long delays were between each "packet". But
when multiple packets are sent in quick succession, it is dropping packets.
I have been trying to do this more "correctly" with using a XmlTextReader,
but with no luck.

E.g.

XmlTextReader r = new XmlTextReader();
reader.MoveToContent();
string s = reader.ReadOuterXml();
dgram.Load(s);

It basically hangs on the ReadOuterXml();

Here is the original "working" code:
--------------------------------------------

private void OrchTalk()
{
int bytesRead = 0;
XmlDocument dgram = new XmlDocument();

while (true)
{
byte[] buffer = new byte[1024]; // buffer for socket read stream
bytesRead = 0;
try { bytesRead = oStream.Read(buffer, 0, buffer.Length); }
catch (Exception e)
{
if (stateCode == Common.stateCodes.shutdown) {break;}
Tools.Debug(10, "OrchTalk: read error {0} ", e.ToString());
break;
}
if (bytesRead == 0)
{
if (this.stateCode == Common.stateCodes.shutdown)
{
Tools.Debug(10, "OrchTalk: Orchestrator died");
this.Disconnect();
return;
}
break;
}
else
{
string s = Tools.ByteToStr(buffer, 0);
try { dgram.LoadXml(s); }
catch (Exception e)
{ Tools.Debug(0, "OnReadComplete: {0} Runt packet: {1}", e.ToString(),
s); }
int endTime = Environment.TickCount;
this.inPacket = dgram.DocumentElement;
Tools.Debug(10, "packet: {0}", inPacket.OuterXml);
this.Dispatch();
int startTime = (int)this.stats["start"];
int elapsed = endTime - startTime;
this.stats["stop"] = endTime;
this.stats["time"] = elapsed;
this.stats["bytes"] = bytesRead;
this.stats["packetText"] = inPacket.OuterXml;
if (StationMaster.isConnected)
{
string outPacket = inPacket.OuterXml;
FlashComm.itself.Send(outPacket);
}
}
}
this.Disconnect();
}

 

[Jon Skeet [C# MVP]]

I have been using the following code (successfully) to read Xml formated
text packets from a TCP stream. The output from the server stream consists
of a sequence of well formed Xml documents written to the output stream.


We are willing to pay $ to any expert (e.g. MVP) consultant who has to time
to help us track down this problem.
(we will discuss rates if you can prove your expertise, and problem solving
approach).

The code below is for the client that is reading the stream.

All this worked well, as long as long delays were between each "packet". But
when multiple packets are sent in quick succession, it is dropping packets.
I have been trying to do this more "correctly" with using a XmlTextReader,
but with no luck.

<snip>

Do you have a short but complete pair of programs (one sender, one
received) which reliably demonstrates the problem?

Have you used a network analyser to check that the packets are being
sent correctly in the first place?

Your receiving code looks pretty odd though: you're ignoring the
bytesRead parameter after checking whether or not it's 0. What does
your Tools.ByteToStr method do? Does it try to find the first 0 byte
and assume that the "good data" ends there? Far better would be to use
an Encoder and GetChars, or just a straight StreamReader on top of the
network stream.

You also *seem* to be assuming that the whole document will come in a
single call to Read - you don't wait for the end of the stream in order
to get the whole document in a string before calling
XmlDocument.LoadXml.

What does your incoming stream look like? Is it meant to have several
documents in it? If so, you could have trouble if you don't have any
way of knowing in advance where one document will end and another one
start.

 

[Oleg Tkachenko]

This might not work if the server injects an <?xml declaration and/or
<!DOCTYPE tag at the start of each document fragment...

Happily XML external entities (which I believe are the same as XML
fragments) are not allowed to have Doctype. But they do allowed to have
text declaration like <?xml version="1.0" encoding="utf-8"?>
I think that's bad idea to use XmlTextReader to read a series of XML
fragments instead of single fragment, some delimiter should be used.

 

[Yechezkal Gutfreund]

1. Thank you Yan, Chris, Oleg and Jon - your points are well taken, and I
apologize for the lack of clarity

2. Yes, the design needs improvement, and the choice of mechanism are poor,
mainly due to my lack of familiarity with the various stream readers.

Lets cover a few of the system issues:

a. There is a central server, and about 10-20 PDA clients running Net CF.
For testing purposes, I am still at one client. So there is no massive data
overload, loss packets, etc. This problem occurs weather it is WiFi, GPRS,
or even on the same box. Communicaiton is via TCP for now. I we are planning
to change to UDP to optimize packet latency, recovery etc. So things look
odd, and now you know why. The code uses dgrams, but for the moment we are
going over a TCP stream.

b. Message are small, almost always less than 100 bytes.

c. We are not using binary or other seriel formatters because we are 1)
prototyping 2) need to run on slow speed PDAs

d. we are using GetBytes:
public static string ByteToStr(byte[] buffer, int start)
{
return System.Text.Encoding.ASCII.GetString(buffer, start,
buffer.Length);
}

e. The problem is most likely that the server uses a '\0' to deliminate
messages. Thus if I pause the server between each message (a debug pop up).
The NAGLE factor on the TCP link splirts the stream into two packets. But if
not then they may both come in one packet, and the GetString stops in the
middle of the buffer (packet) read.

f. Ok, one solution, is to run through the buffer a byte at a time and find
the null delimator (or better an EOD) and do the string one at a time. - But
remember this is a low cpu PDA. why run through the buffer twice? parse
twise. Once to split the packets, and the second to turn the string into an
XMLDoc?

g. TextReaderStream from URL (and all of that) is not meanigful. These are
raw TCP sockets, and we are going to UDP eventually.

Is there not some way to use the XmlStreamReaders to gobble up tokens and
even complete fragmets (.ReadOuterXml) or complete docs, so that only one
pass of the stream needs to be done?

The goal then is simple. Take a byte buffer, and extract (ONE at a time) xml
packets and stuff them into XmlDocuments. Assume that the buffer will be big
enough so that there will never be fragmentary documents, and that the
buffer is big enough for 10 messages, and there will never be more than 1 or
2 unread ones. So buffer overflow or splits are not a concern.


Is this clear?


--
==================================
Yechezkal Gutfreund
Chief Scientist
Kesser Technical Group, Inc.
==================================

 

[Jon Skeet [C# MVP]]

d. we are using GetBytes:
public static string ByteToStr(byte[] buffer, int start)
{
return System.Text.Encoding.ASCII.GetString(buffer, start,
buffer.Length);
}

In that case you're decoding a load of 0s which weren't actually
received. You need to specify how much to decode, at which point
there's no real point in having the extra method anyway.

e. The problem is most likely that the server uses a '\0' to deliminate
messages. Thus if I pause the server between each message (a debug pop up).
The NAGLE factor on the TCP link splirts the stream into two packets. But if
not then they may both come in one packet, and the GetString stops in the
middle of the buffer (packet) read.

Relying on packets doing exactly what you want sounds like a really bad
idea to me, to be honest - there's a reason why all of this is exposed
as a stream.

f. Ok, one solution, is to run through the buffer a byte at a time and find
the null delimator (or better an EOD) and do the string one at a time. - But
remember this is a low cpu PDA. why run through the buffer twice? parse
twise. Once to split the packets, and the second to turn the string into an
XMLDoc?

I don't think looking for a \0 really counts as full parsing - and
you're already running through the byte stream twice, once to decode
from bytes to characters. In fact, with your current code you're
decoding 1024 bytes every time, even if only 100 bytes actually contain
any text.

g. TextReaderStream from URL (and all of that) is not meanigful. These are
raw TCP sockets, and we are going to UDP eventually.

You've got a stream though, which means you can create a StreamReader.
When it goes to UDP, you'd have to rewrite stuff anyway.

Is there not some way to use the XmlStreamReaders to gobble up tokens and
even complete fragmets (.ReadOuterXml) or complete docs, so that only one
pass of the stream needs to be done?

I'm not *exactly* sure what you mean here, but I don't think so.

The goal then is simple. Take a byte buffer, and extract (ONE at a time) xml
packets and stuff them into XmlDocuments. Assume that the buffer will be big
enough so that there will never be fragmentary documents, and that the
buffer is big enough for 10 messages, and there will never be more than 1 or
2 unread ones. So buffer overflow or splits are not a concern.

Is this clear?

Well, if you're *really* sure you've got everything in a buffer, I'd
suggest scanning for your zero bytes, then decode each delimited
section using Encoding.ASCII.GetString (buffer, start, length) and then
feeding the results of that into XmlDocument.LoadXml.

 

[Yan-Hong Huang[MSFT]]

Hello Yechezkal,

Thanks for the detailed response.

Did you mean that you want XmlStreamReader to read complete docs for you
automatically? If so, I am afraid that it is not possible. That reader
class won't be so smart to do it for you. It is program's responsibility to
maintain a complete doc stream and feed it into a XmlStreamReader.

I think the original hang problem is that the reader didn't find complete
XML stream and so it got stuck in a loop when executing ReadOuterXml.

So my concerns is that:

1) How could you confirm that a whole XML doc is contained in a TCP packet?
Based on my experience, it is not guaranteed by TCP or UCP protocol. Even
when you are sending them in a buffer in server side, it doesn't mean you
could receive all of them in a single read execution in client side. ( I am
not 100% sure of it. So please correct me if I am wrong here )

2) How to tag a complete XML doc in a packet? You mentioned that the server
uses '/0' to separate XML documents. However, I think we need to search for
the first '0' in the stream first to tell that a real xml document's
beginning.

3) Is there any mechanism for the client to response to the server? So that
if one document is not received, the server will send it again.

Just my 2 cents.

Best regards,
Yanhong Huang
Microsoft Online Partner Support

Get Secure! ¨C www.microsoft.com/security
This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Yechezkal Gutfreund]

Thank you guys.

You see, I come from the olden days. Where we had 2-pass compilors. (e.g.
Lex and Yacc). The first pass tokenized the stream (or document) the second
did the syntactic analysis. Sometimes you needed 3 passes, for forward
declarations. (Does anyone teach this stuff anymore?).

Anyway, the point is well taken about the decode of the zero btyes.

I suppose I was getting lazy based on all the power that the .net framework
gives me, and was hopeing that I could just hand the stream to
XmlReadStream, and it could read and parse the stream, and stop when it came
to the logical end of the document (and leave the stream pointer at that
point).

Based on what you are telling me, what sounds the cleanest will be to PEEK
into the stream till I find the EOD (0-byte is probably too easy to corrupt
on wireless connections) then do the READ of the STREAM to the buffer up to
that point, decode that segment, and pass it to XmlDocument.LoadXml() for
the lexical parsing (tokenizing).

When we switch to UDP, since we know that packets will fit into the MTU for
wireless packets, we can avoid the complications that reading from a stream
are creating us. (sometimes TCP helps, sometimes it is better to get closer
to the iron).

 

[Yechezkal Gutfreund]

Uggh, No pointers into buffers, no peek() into the stream. C is much easier
to do this stuff. I also suspect that this byte by byte loop through the
very expensive CPU-wise for a PDA.

There are now three passes through the data. One looking for the EOD
delimiter, the second when it converts it to a string, the last when does a
lexical parse to Xml. The stream readers were really smart stream lexical
parsers, otherwise, what good really are they?


--
==================================
Yechezkal Gutfreund
Chief Scientist
Kesser Technical Group, Inc.
==================================



private void OrchTalk()
{
int bytesRead = 0;
byte[] buffer = new byte[8192];
int bIndex = 0;
bool eop = false; // end of packet
XmlDocument dgram = new XmlDocument();

while (true)
{
bytesRead = 0;
bIndex = 0;
eop = false;
while (!eop)
{
try { bytesRead = oStream.Read(buffer, bIndex, 1); }
catch (Exception e)
{
if (this.stateCode == Common.stateCodes.shutdown)
{
this.Disconnect();
return;
}
else
{
Tools.Debug(10, "OrchTalk: read error {0} ", e.ToString());
this.Shutdown();
return;
}
}
if (bytesRead == 0)
{
if (this.stateCode == Common.stateCodes.shutdown)
{
this.Disconnect();
return;
}
else
{
Tools.Debug(10, "OrchTalk: Orchestrator died");
this.Shutdown();
return;
}
}
eop = buffer[bIndex] == 0;
bIndex += bytesRead;
}
string s = Tools.ByteToStr(buffer, bIndex);
try { dgram.LoadXml(s); }
catch (Exception e)
{ Tools.Debug(0, "OnReadComplete: {0} Runt packet: {1}", e.ToString(),
s); }

 

[Yan-Hong Huang[MSFT]]

Hi Yechezkal,

If we could confirm that the stream contains a completed XML file and only
contains a XML file, the XML reader should be able to parse it. That is how
DOM works.

If there are any more questions, please feel free to post here.

Best regards,
Yanhong Huang
Microsoft Online Partner Support

Get Secure! ¨C www.microsoft.com/security
This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Yechezkal Gutfreund]

It is not a file, really. But a protocol packet. Since they are always under
100 bytes, they are guaranteed to arrive in one IP datagram, and therefore
there should be no problem in parsing them with the XML reader. The question
remains is how?

 

[Yan-Hong Huang[MSFT]]

Hi Yechezkal,

So it is better for us to say it as a stream, right?

I think we could use XmlDocument.Load, or XMLDocument.LoadXML to load that
string. However, we need to parse the data to one string first.

If there is any unclear, please feel free to post here.

Best regards,
Yanhong Huang
Microsoft Online Partner Support

Get Secure! ¨C www.microsoft.com/security
This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Jon Skeet [C# MVP]]

Uggh, No pointers into buffers, no peek() into the stream. C is much easier
to do this stuff.

C certainly makes it easy to do this stuff *badly*, yes.

I also suspect that this byte by byte loop through the
very expensive CPU-wise for a PDA.

You suspect? Have you timed it? Bear in mind that your previous version
of the code was *always* converting 1024 bytes into a string,
regardless of how many of those bytes were useful. Was the performance
unacceptable then?

There are now three passes through the data. One looking for the EOD
delimiter, the second when it converts it to a string, the last when does a
lexical parse to Xml. The stream readers were really smart stream lexical
parsers, otherwise, what good really are they?

The first of those passes should be very quick indeed. The second
should be pretty fast too. The third is needed whatever language you
were coding in (as would the second, unless you were assuming one-byte-
per-char to start with).

 

[Jon Skeet [C# MVP]]

It is not a file, really. But a protocol packet. Since they are always under
100 bytes, they are guaranteed to arrive in one IP datagram, and therefore
there should be no problem in parsing them with the XML reader. The question
remains is how?

Are they *really* guaranteed to arrive in one IP datagram? This seems a
dodgy assumption to make, I'm afraid - and an entirely unnecessary one
too.

 

[Yechezkal Gutfreund]

As I said, we will eventually go to UDP. You don't know the higher levels of
the protocol stack, and for real efficiency both stacks have to be designed
as one, otherwise you are repeating stuff at each level. So, given your
incomplete understanding of the problem, your remarks are on the ball. It is
only that you don't have the whole picture.

We prefer UDP, were we will be able to really guarantee the complete message
in a packet. (TCP only makes IP look like a stream, in reality, it is a
packet protocol, and these issues really come into play in wireless).
Bottom, line, you might think it is a stream, but in reality there are no
streams here, only packets.

We will probably keep the TCP layer on top of our reliable UDP layer, and
use the TCP when firewall isssues prevent the UDP. Even so, I like to keep
things efficient in TCP.

You can see the effect of tolerating things like 3 passes through the data
in multimedia protocols. Take Real. They prefer UDP, will fall back to TCP,
and at wost case HTTP over TCP. Guess what, you get a real performance hit
each time you fall back.

 

[Yechezkal Gutfreund]

So you are bassically telling me that doing a one pass to convert to string
encoding, and a second pass to do the LoadXml() to the document, is about
the same efficiency as the "one pass" of using he XmlStreamReader over the
NetworkStream? Frankly the documentation does not give me much insight.

But if you can answer this, then I would say your answers are helping
greatly.

(Besides which, I can't get the XmlTextReader to read the stream anyway
:-(. )

--
==================================
Yechezkal Gutfreund
Chief Scientist
Kesser Technical Group, Inc.
==================================

 

[Jon Skeet [C# MVP]]

As I said, we will eventually go to UDP. You don't know the higher levels of
the protocol stack, and for real efficiency both stacks have to be designed
as one, otherwise you are repeating stuff at each level. So, given your
incomplete understanding of the problem, your remarks are on the ball. It is
only that you don't have the whole picture.

We prefer UDP, were we will be able to really guarantee the complete message
in a packet. (TCP only makes IP look like a stream, in reality, it is a
packet protocol, and these issues really come into play in wireless).

Absolutely - and if you'd already been using UDP, I wouldn't have
mentioned it. When you can guarantee things about the packet size, that
gives a lot more flexibility on this issue. As you're using TCP, you
can't always make those assumptions, so it seems a bad idea to try to
do so.

Bottom, line, you might think it is a stream, but in reality there are no
streams here, only packets.

Well, that's true when you're using UDP - but you seemed to be making
assumptions when you were using TCP, which is dangerous.

We will probably keep the TCP layer on top of our reliable UDP layer, and
use the TCP when firewall isssues prevent the UDP. Even so, I like to keep
things efficient in TCP.

You can see the effect of tolerating things like 3 passes through the data
in multimedia protocols. Take Real. They prefer UDP, will fall back to TCP,
and at wost case HTTP over TCP. Guess what, you get a real performance hit
each time you fall back.

Sure, but I bet that's not due to just having to go through 100 bytes
three times. It's more due to the network than the load, I suspect. As
I said before, have you actually measured how long it takes to go
through those 100 bytes to find the null bytes, and then convert the
appropriate chunks to strings? Have you compared it with the time it
takes to build and XML document?

If you're really wanting to think about efficiency, I'd say that the
first two tasks are the least of your worries - go for some kind of
binary version of the XML document which will require far less parsing
etc. That's much more likely to save you processor time - but rather
than take my word for it, measure, measure, measure!

 

[Jon Skeet [C# MVP]]

So you are bassically telling me that doing a one pass to convert to string
encoding, and a second pass to do the LoadXml() to the document, is about
the same efficiency as the "one pass" of using he XmlStreamReader over the
NetworkStream?

Probably, yes. They're doing the same work - it's not like by using a
stream you avoid the conversion, it's just that the types of work are
interleaved instead of one chunk of converting and then one chunk of
parsing.

As ever: measure!

Frankly the documentation does not give me much insight.

But if you can answer this, then I would say your answers are helping
greatly.

(Besides which, I can't get the XmlTextReader to read the stream anyway
:-(. )

That's probably because your stream isn't really ending where it
should, it's just putting a 0 byte instead. What error message are you
getting?

 

[Yechezkal Gutfreund]

That's probably because your stream isn't really ending where it
should, it's just putting a 0 byte instead. What error message are you
getting?

It just blocks forever on the ReadOuterXML()

But yes, we will probably also go for a more compact transport protocol than
xml. Measure, measure is good advice. So is interate and re-factor (XP/agile
methodology). So we start with quick high level tools such as xml for
sending structured data, and leverage the .NET framework for all it is worth
to get something prototyp-able and then stuff our effort into the things 20%
of the system that is taking 80% of the time.

Thanks Jon and Yan. I think at this point it time, we certainly have a
working version (see my 3 pass code) and it will do till we can get the
detailed timing measurements.