XMLDocument and NetworkStream (VB.NET 2005)

[Sid Price]

Hello,
I have an application that needs to receive XML documents over a TCP/IP
connection. I have tried several approaches and so far have failed to find
one that I can get to work. The server is sending the XML correctly since I
can replace my XML receiver with a byte buffer and string builder and I see
the document. Here is the code that I am trying to make work:
Dim netStream As NetworkStream = tcpClient.GetStream()

If netStream.CanRead = True Then

Dim oData As XmlDocument = New XmlDocument

oData.Load(netStream) ' read the xml doc from the server ... blocking
call

The "Load" never returns. As I said if I replace the oData.Load call with
code to read the stream into a byte buffer the data is received.

Any help or pointers would be much appreciated,

Sid.

 

[Scott M.]

There's a good chance the XML you are attempting to load is not well-formed
and thus, the object is not being created.

-Scott

 

[Sid Price]

Hi Scott,
The server is my code too and I wrote the XML to a file as part of my
testing, it appears well formed and is opened by several XML apps I have
including XMLNotepad without complaint.
Sid.

 

[Scott M.]

Well, then only other possibility I see is that something is happening to
the XML to cause it to not be well-formed when you read it in. Have you
tried reading the stream into an XMLReader first?

Check out: http://msdn2.microsoft.com/en-us/library/e48zttz7.aspx for
details.

-Scott

 

[Sid Price]

Hello again Scott,
I have done a little more research on my problem and although I don't really
understand what is wrong with the transmitted document when I look at the
XMLDocument object's properties I do see something odd.

The Property "InnerXML" has the following:
"<?xml version="1.0"
encoding="UTF-8"?><RouteStatus><Devices><Device><Name>RMX_24</Name><Number>1</Number></Device></Devices><Routes
/></RouteStatus>"

I think this is well formed XML, any comments?

The Property "InnerText" has:
"RMX_241"
This is odd and if I select the XMLVisualizer I see the following:
The XML page cannot be displayed
Cannot view XML input using style sheet. Please correct the error and then
click the Refresh button, or try again later.
Invalid at the top level of the document. Error processing resource
'file:///C:/Documents and Settings/Sid Price/Local Sett...

RMX_241
^
Any comments would be appreciated.
Also, at the client end I have changed the code to read the stream into a
byte buffer,
convert it to string and then use XMLDocument.LoadXML to load my document. I
do
get an exception reported that the root of the document is not correctly
formed.
If I look at the string I see some odd characters before the XML.
"???<?xml version="1.0" encoding="UTF-8"?><RouteStatus> <Devices>
<Device> <Name>RMX_24</Name> <Number>1</Number> </Device>
</Devices> <Routes /></RouteStatus>
I don't know where these extra bytes are coming from, again any pointers
would be much appreciated.
Sid

 

[Scott M.]

The XML is well-formed (easy way to test is to just copy it to a text file,
save with an .xml extension and open in IE).

So, my next question is, what error are you getting when you use this code?

-Scott

 

[Jon Skeet [C# MVP]]

If I look at the string I see some odd characters before the XML.
"???<?xml version="1.0" encoding="UTF-8"?><RouteStatus> <Devices>
<Device> <Name>RMX_24</Name> <Number>1</Number> </Device>
</Devices> <Routes /></RouteStatus>
I don't know where these extra bytes are coming from, again any pointers
would be much appreciated.

They're almost certainly Byte-Order Marks.

I *think* I've seen an issue like this before, where the first packet
of data only contains a few bytes. I'll see if I can reproduce it
without network streams...

 

[Sid Price]

Here is the code I am using to receive the document:
Dim bytes(tcpClient.ReceiveBufferSize) As Byte



netStream.Read(bytes, 0, CInt(tcpClient.ReceiveBufferSize))

Dim returnData As String = System.Text.Encoding.ASCII.GetString(bytes)

Debug.WriteLine(returnData)

oData.LoadXml(returnData)

The exception from the LoadXML says "Data at the root level is invalid. Line
1, position 1."

As you will see in my previous post there are a few strange bytes at the
head of the document.

Sid.

 

[Sid Price]

Here is the code I am using to receive the document:
Dim bytes(tcpClient.ReceiveBufferSize) As Byte



netStream.Read(bytes, 0, CInt(tcpClient.ReceiveBufferSize))

Dim returnData As String = System.Text.Encoding.ASCII.GetString(bytes)

Debug.WriteLine(returnData)

oData.LoadXml(returnData)

The exception from the LoadXML says "Data at the root level is invalid. Line
1, position 1."

As you will see in my previous post there are a few strange bytes at the
head of the document.

Sid.

 

[Jon Skeet [C# MVP]]

Here is the code I am using to receive the document:
Dim bytes(tcpClient.ReceiveBufferSize) As Byte



netStream.Read(bytes, 0, CInt(tcpClient.ReceiveBufferSize))

Dim returnData As String = System.Text.Encoding.ASCII.GetString(bytes)

Debug.WriteLine(returnData)

oData.LoadXml(returnData)

The exception from the LoadXML says "Data at the root level is invalid. Line
1, position 1."

As you will see in my previous post there are a few strange bytes at the
head of the document.

That's not good code to receive an XML document:

1) It's using UTF-8, not ASCII. Don't do decoding yourself - let the
document loader do it.

2) There's no guarantee that Read will return the whole of the document
in a single call. If you're going to load it all first, you should loop
round, reading into a buffer until there's no more data. (An
alternative is to read from the NetworkStream and write to a
MemoryStream until you're done - then rewind the MemoryStream.)

 

[Sid Price]

That's not good code to receive an XML document:

1) It's using UTF-8, not ASCII. Don't do decoding yourself - let the
document loader do it.

2) There's no guarantee that Read will return the whole of the document
in a single call. If you're going to load it all first, you should loop
round, reading into a buffer until there's no more data. (An
alternative is to read from the NetworkStream and write to a
MemoryStream until you're done - then rewind the MemoryStream.)

The code I posted is not the code I wrote originally and posted in my first
message. I wrote this code to try and figure out why my original code was
not working. Here is what I started with:
Dim netStream As NetworkStream = tcpClient.GetStream()

If netStream.CanRead = True Then

Dim oData As XmlDocument = New XmlDocument

oData.Load(netStream)

With this code the "Load" never returns.

Sid.

 

[Jon Skeet [C# MVP]]

The code I posted is not the code I wrote originally and posted in my first
message. I wrote this code to try and figure out why my original code was
not working. Here is what I started with:
Dim netStream As NetworkStream = tcpClient.GetStream()

If netStream.CanRead = True Then

Dim oData As XmlDocument = New XmlDocument

oData.Load(netStream)

With this code the "Load" never returns.

Okay - so what's the protocol involved? Are you expecting the
connection to be closed after the whole document has been written? If
not, I suspect Load won't work - I'd at least *expect* it to read to
the end of the stream. If the stream never ends, that's quite possibly
the problem...

 

[Sid Price]

Okay - so what's the protocol involved? Are you expecting the
connection to be closed after the whole document has been written? If
not, I suspect Load won't work - I'd at least *expect* it to read to
the end of the stream. If the stream never ends, that's quite possibly
the problem...

Hi Jon,
That then is probably the issue, the stream indeed stays open. Can you
suggest a scheme that would enable me to send an XMLDocument to a stream and
receive it into new document? If possible can one send an EOF and have the
"Load" terminate?
You mentioned in another thread that you may have seen the strange bytes
that are at the head of the stream before, did you discover any information
about that?
Thanks so much for your help,
Sid.

 

[Jon Skeet [C# MVP]]

That then is probably the issue, the stream indeed stays open. Can you
suggest a scheme that would enable me to send an XMLDocument to a stream and
receive it into new document? If possible can one send an EOF and have the
"Load" terminate?

I'm not really sure what you mean. The reader would have to notice that
it's read a complete document, and stop at that point. What exactly is
the protocol you're using here?

You mentioned in another thread that you may have seen the strange bytes
that are at the head of the stream before, did you discover any information
about that?

The bytes are almost certainly a UTF-8 byte-order mark - nothing
particularly odd about that.

However, I *think* I've seen an issue where if only part of a byte-
order mark is read in the first read, things can get screwy. I haven't
tried to reproduce that though, and it sounds like it isn't your issue.

 

[Sid Price]

I'm not really sure what you mean. The reader would have to notice that
it's read a complete document, and stop at that point. What exactly is
the protocol you're using here?

I am not using any wrapper protocol around the XML document. The sender
writes the document to the stream and the reader reads it from the stream,
expecting only the XML document. So, if I understand correctly you are
saying that I need to add a protocol layer between the sender and the stream
that delimits the XML document, is that correct?
Thanks,
Sid.

 

[Jon Skeet [C# MVP]]

I am not using any wrapper protocol around the XML document.

No, in terms of the network connection. Are you talking to a web
server, an IRC server, a telnet server, an ssh server, etc? What's the
protocol of the network?

The sender
writes the document to the stream and the reader reads it from the stream,
expecting only the XML document. So, if I understand correctly you are
saying that I need to add a protocol layer between the sender and the stream
that delimits the XML document, is that correct?

Pretty much. I find the best way is to get the sender to prefix the
actual data with how many bytes of data they're going to send. The
receiver then knows when they're "done" just from the amount of data
received, without having to parse it until it's all ready.

 

[Sid Price]

No, in terms of the network connection. Are you talking to a web
server, an IRC server, a telnet server, an ssh server, etc? What's the
protocol of the network?

The protocol being used is TCP/IP, the server needs to send the XML to each
client that is logged in.

Pretty much. I find the best way is to get the sender to prefix the
actual data with how many bytes of data they're going to send. The
receiver then knows when they're "done" just from the amount of data
received, without having to parse it until it's all ready.

That's kind of where I was thinking I needed to go. Since I am generating
the class libray for both the client and server I can prepend the byte
count, read the stream into memory and when I have a full document load it
into into an XMLDocument object for the user. Many thanks for helping think
this through,
Sid.

 

[Sid Price]

That's kind of where I was thinking I needed to go. Since I am generating
the class libray for both the client and server I can prepend the byte
count, read the stream into memory and when I have a full document load it
into into an XMLDocument object for the user. Many thanks for helping
think this through,

Okay, so I have a new wrapper class that handles the sending of the document
to the stream by prepending the length of the document to the document and a
reader that uses the length to read the document into a memory stream and
then this is loaded into an XMLDocument object. I still have the same issue
of the three "rogue" bytes on the front of the document, these bytes cause
the XMLDocument object to throw an exception that the document does not
contain a root element.
Here is the sender:
'

' Send the passed doc to the passed stream

'

theDoc.Save(theData)

messageLength = theData.Length

'

' write the number of bytes in the message

' to the stream

theBuffer = Array.CreateInstance(GetType(Byte), 4)

'

theBuffer(0) = messageLength And &HFF

theBuffer(1) = (messageLength >> 8) And &HFF

theBuffer(2) = (messageLength >> 16) And &HFF

theBuffer(3) = (messageLength >> 24) And &HFF

theStream.Write(theBuffer, 0, 4)

'

' Now the data

'

theBuffer = Array.CreateInstance(GetType(Byte), theData.Length)

theBuffer = theData.ToArray()

theStream.Write(theBuffer, 0, theBuffer.Length)



And the reader

theBuffer = Array.CreateInstance(GetType(Byte), 4)

theStream.Read(theBuffer, 0, 4)

messageLength = theBuffer(0)

messageLength += (theBuffer(1) << 8)

messageLength += (theBuffer(2) << 16)

messageLength += (theBuffer(3) << 24)

Dim rxBytes As Int32 = 0

theBuffer = Array.CreateInstance(GetType(Byte), messageLength)

While (rxBytes <> messageLength)

rxBytes += theStream.Read(theBuffer, rxBytes, messageLength - rxBytes)

End While

theData.Write(theBuffer, 0, messageLength)

theDoc.Load(theData)

Seems that altough I now have better stream i/o code I still have the same
issue, any ideas?

Thanks,

Sid.

 

[Jon Skeet [C# MVP]]

The protocol being used is TCP/IP, the server needs to send the XML to each
client that is logged in.

Well, I'm sure there's some protocol involved above that level, even if
it's one you've created yourself (which it sounds like you are).

That's kind of where I was thinking I needed to go. Since I am generating
the class libray for both the client and server I can prepend the byte
count, read the stream into memory and when I have a full document load it
into into an XMLDocument object for the user. Many thanks for helping think
this through,

Sounds like the way to go. Best of luck.

 

[Jon Skeet [C# MVP]]

Okay, so I have a new wrapper class that handles the sending of the document
to the stream by prepending the length of the document to the document and a
reader that uses the length to read the document into a memory stream and
then this is loaded into an XMLDocument object. I still have the same issue
of the three "rogue" bytes on the front of the document, these bytes cause
the XMLDocument object to throw an exception that the document does not
contain a root element.
Here is the sender:

<snip>

1) BinaryWriter/BinaryReader and/or BitConverter are your friends

2) You're not not rewinding your MemoryStream on the reading side. Use
theData.Position = 0
before passing it to theDoc.Load