2gb Limit

[Andrew Robinson]

My understanding is that strings and xml documents are limited to 2GB. Not
that this is a good practice, but I have a client that wants to parse /
convert (in memory) a document that is larger than 2GB. I don't think it can
be done even with XP or Vista 64. Am I correct? I assume that I can read
from a file that is larger than 2GB with a stream but I will need to read,
convert and write using much smaller chuncks of data.

Comments?

Thanks,

 

[Bruce Wood]

My understanding is that strings and xml documents are limited to 2GB. Not
that this is a good practice, but I have a client that wants to parse /
convert (in memory) a document that is larger than 2GB. I don't think it can
be done even with XP or Vista 64. Am I correct? I assume that I can read
from a file that is larger than 2GB with a stream but I will need to read,
convert and write using much smaller chuncks of data.

You should probably ask this question in the group

microsoft.public.dotnet.xml

or, better yet, search the archives there. That's where the XML gurus
hang out....

 

[Otis Mukinfus]

My understanding is that strings and xml documents are limited to 2GB. Not
that this is a good practice, but I have a client that wants to parse /
convert (in memory) a document that is larger than 2GB. I don't think it can
be done even with XP or Vista 64. Am I correct? I assume that I can read
from a file that is larger than 2GB with a stream but I will need to read,
convert and write using much smaller chuncks of data.

Comments?

Thanks,

You can do it as long as you don't try to load all of it at once. You're on
target with your last sentence. If the data is in lines of text it's a piece of
cake. If not exchange the word lines with chunks. Read a line, convert, save,
get next line and do it again until you reach the end. It's the way we did it
in the early times when all we had was 16 or 32k of memory.
Good luck with your project,

Otis Mukinfus
http://www.arltex.com
http://www.tomchilders.com

 

[chanmm]

One of the purposes of XML is transferring or stream the small amount of
data over the wire. I do not think the intention of XML is to handle
something like 1GB or 2GB of data. Use SQL server to handle such a volume
instead.

chanmm

 

[Bruce Wood]

One of the purposes of XML is transferring or stream the small amount of
data over the wire.

As you said: _one_ of the purposes of XML... not the _only_ purpose of
XML.

I do not think the intention of XML is to handle something like 1GB or 2GB of data.

In my experience, this is absolutely not true. XML is used for many
things, and one of them may be to transmit a large amount of data from
place to place. For example, it's perfectly reasonable to send a
product catalog from one company to another as an XML document, and
that could run into multi-gigabytes, especially if there are base-64
encoded images in the document.

Where I do agree with you is that XML is not a good format for storing
and maintaining large amounts of data. A database is much better suited
to that. However, there are people who disagree with me even on that
point.