Huge xml files

[rickbear]

Hi Group,

I have an xml file wich contains a rootelement and a subelement with a
filename and a subelement with bindata.
In each xml file there is only one entry for each element. Like this:

<root>
<filename>somefile.exe</filename>
<bindata>bindata in base64 here</bindata>
</root>

I have to parse the content in the bindata element to a Websphere MQ
queue, so I read the xml file into memory with xmlserializer and
filestream and put it in the queue. This works pretty fine.

On the other side of the queue I get the queuedata and collects the
data as an object. This works very well also.

But now and then there are xml files with huge amount of bindata. When
they exceed about 350mb I get an out of memory error. So I don't know
how to handle this. My idea is to somehow break the bindata into
smaller peaces. But I do not know how to partially load the data of
one element (bindata). Is that possible and how?
With this problem solved - is there some way to tell the message queue
that this is only a portion of an element?

- rick -

 

[Ignacio Machin \( .NET/ C# MVP \)]

Hi,


I think this is more a question for a Websphere MQ group, it depend of how
they can manage huge amount of data.
You can always read part of the file (maybe you will have to treat it as a
text file instead of a XML).

 

[Jeroen Mostert]

Hi Group,

I have an xml file wich contains a rootelement and a subelement with a
filename and a subelement with bindata.
In each xml file there is only one entry for each element. Like this:

<root>
<filename>somefile.exe</filename>
<bindata>bindata in base64 here</bindata>
</root>

A spectacular misuse of XML for sure. This certainly doesn't beat putting
the actual files in a directory. Anyhoo...

I have to parse the content in the bindata element to a Websphere MQ
queue, so I read the xml file into memory with xmlserializer and
filestream and put it in the queue. This works pretty fine.

Use XmlReader.ReadContentAsBase64() to read the content in chunks instead.
It was designed for this.