CF Bug / Limitation in XMLTextWriter (OutOfMemory)

[Chris DiPierro]

It seems as if there's a hard limit in XMLTextWriter that's causing me a
headache.

If I create an XMLTextWriter from a StringWriter, once the string gets
to be about 2.5MB in length, the XMLTextWriter will OutOfMemory
exception. The simple code listed below causes this problem.

Anyone know of a workaround / solution?

private void button1_Click(object sender, EventArgs e)
{
System.IO.StringWriter sWriter = new System.IO.StringWriter();
System.Xml.XmlTextWriter xWriter = new System.Xml.XmlTextWriter(sWriter);
xWriter.WriteStartElement("Test");
string s = new string('A', 10000);
while (true)
{
xWriter.WriteStartElement("ABC");
xWriter.WriteString(s);
System.Diagnostics.Debug.WriteLine("Length: " +
sWriter.ToString().Length);
xWriter.WriteEndElement();
}
xWriter.WriteEndElement();
}

 

[Paul G. Tobey [eMVP]]

Seems like any single memory object that's 2.5MB (or is that 2.5Mcharacters,
which would be 5MB), is a non-starter on a little device like a Pocket PC.
Why do you want it in RAM, anyway? Why not write it to a file? At least
there's a chance that a multi-MB file can exist in the system.

Paul T.

 

[Chris DiPierro]

You're right that it's characters (so 5MB). As for why in RAM, I think
the counter to that is why not? The more we do with WM devices, the more
we want them to behave like other mobile devices (ie. Tablets, UMPCs),
so it's not atypical to be moving large amounts of data.

As for the specific issue, I probably will address it via file, but is
there some sort of known hard limit? In my case, I'm using the emulator,
and program memory indicates 65+ MB free when the error is thrown. To me
that seems unintuitive.

 

[Ilya Tumanov [MS]]

There's no "hard limit" in XmlTextWriter, you simply running out of memory.

Does not matter if you have even 1 gig of free program memory - there's a
limit of 32 MB of virtual memory per process and not all of it is initially
free.

--
Best regards,

Ilya

This posting is provided "AS IS" with no warranties, and confers no rights.

*** Want to find answers instantly? Here's how... ***

1. Go to
http://groups-beta.google.com/group/microsoft.public.dotnet.framework.compactframework?hl=en
2. Type your question in the text box near "Search this group" button.
3. Hit "Search this group" button.
4. Read answer(s).

 

[Paul G. Tobey [eMVP]]

These are little tiny devices, not modern PCs where you just dump every last
piece of data you can think of and who cares, there's another 512MB of RAM
and ample hard disk for use with virtual memory laying around anyway. You
are programming for an embedded system. It's not an embedded system of ten
years ago, when you'd be lucky to have a 32-bit processor, let alone
multi-MB flash disks, but it's still a small device and you have to program
it with its limitations in mind. I'm not saying that a 5MB XML file is out
of the question, just that trying to process a 5MB XML object in RAM is not
a smart way to start your work.

No, the limit will depend on the total number and size of allocations, both
in the system as a whole and the application which you are running. The
address space for your process is 32MB, regardless of how much RAM is in the
device, so you're presently using about 1/7 of the total *address space*
available to you for just this one object (imagine using 1/7 of the 2GB
address space of a process on your PC!). Not only that, but some of that
32MB address space is used for DLLs that you are using, etc. The lack of
virtual memory for data means that you can't be blissfully ignorant of
what's really going on under the covers, if you can even when virtual memory
is present.

Paul T.

 

[Chris DiPierro]

I understand that there are ways around it, but I think we have
differing opinions here. I can in fact allocate 300MB or so in a desktop
application w/o a problem (1/7th of my address space), so I guess I'd
assume a platform being presented as an enterprise-capable one would be
able to do the same. But even so, let's be realistic, in this day and
age we're not moving 300MB data files very often, but 5MB is hardly out
of the realm of realism.

I've been doing embedded system development for the better part of 10
years now, so I guess I was just expecting too much of these modern
platforms.

In any event, thank you for the informative reply.

 

[Robert Simpson]

Your desktop can page memory to and from disk as needed in order to fulfill
an allocation request. What medium do you suppose CE can page to?

5mb may not seem like much, but your app isn't the only one installed on the
device, and may not be the only one running. Best to play nice and code for
the hardware limitations of the platform rather than assume that any
service, memory and capability is at your app's sole disposal. When coding
for CE you're going to have to make conscious efforts to limit the memory
and resources you allocate, and code for the fact that an allocation request
could fail.

Robert

I understand that there are ways around it, but I think we have differing
opinions here. I can in fact allocate 300MB or so in a desktop application
w/o a problem (1/7th of my address space), so I guess I'd assume a platform
being presented as an enterprise-capable one would be able to do the same.
But even so, let's be realistic, in this day and age we're not moving 300MB
data files very often, but 5MB is hardly out of the realm of realism.

I've been doing embedded system development for the better part of 10
years now, so I guess I was just expecting too much of these modern
platforms.

In any event, thank you for the informative reply.

These are little tiny devices, not modern PCs where you just dump every
last piece of data you can think of and who cares, there's another 512MB
of RAM and ample hard disk for use with virtual memory laying around
anyway. You are programming for an embedded system. It's not an
embedded system of ten years ago, when you'd be lucky to have a 32-bit
processor, let alone multi-MB flash disks, but it's still a small device
and you have to program it with its limitations in mind. I'm not saying
that a 5MB XML file is out of the question, just that trying to process a
5MB XML object in RAM is not a smart way to start your work.

No, the limit will depend on the total number and size of allocations,
both in the system as a whole and the application which you are running.
The address space for your process is 32MB, regardless of how much RAM is
in the device, so you're presently using about 1/7 of the total *address
space* available to you for just this one object (imagine using 1/7 of
the 2GB address space of a process on your PC!). Not only that, but some
of that 32MB address space is used for DLLs that you are using, etc. The
lack of virtual memory for data means that you can't be blissfully
ignorant of what's really going on under the covers, if you can even when
virtual memory is present.

Paul T.

You're right that it's characters (so 5MB). As for why in RAM, I think
the counter to that is why not? The more we do with WM devices, the more
we want them to behave like other mobile devices (ie. Tablets, UMPCs),
so it's not atypical to be moving large amounts of data.

As for the specific issue, I probably will address it via file, but is
there some sort of known hard limit? In my case, I'm using the emulator,
and program memory indicates 65+ MB free when the error is thrown. To me
that seems unintuitive.


Paul G. Tobey [eMVP] wrote:
Seems like any single memory object that's 2.5MB (or is that
2.5Mcharacters, which would be 5MB), is a non-starter on a little
device like a Pocket PC. Why do you want it in RAM, anyway? Why not
write it to a file? At least there's a chance that a multi-MB file can
exist in the system.

Paul T.

It seems as if there's a hard limit in XMLTextWriter that's causing me
a headache.

If I create an XMLTextWriter from a StringWriter, once the string gets
to be about 2.5MB in length, the XMLTextWriter will OutOfMemory
exception. The simple code listed below causes this problem.

Anyone know of a workaround / solution?

private void button1_Click(object sender, EventArgs e)
{
System.IO.StringWriter sWriter = new System.IO.StringWriter();
System.Xml.XmlTextWriter xWriter = new
System.Xml.XmlTextWriter(sWriter);
xWriter.WriteStartElement("Test");
string s = new string('A', 10000);
while (true)
{
xWriter.WriteStartElement("ABC");
xWriter.WriteString(s);
System.Diagnostics.Debug.WriteLine("Length: " +
sWriter.ToString().Length);
xWriter.WriteEndElement();
}
xWriter.WriteEndElement();
}

 

[Guest]

I think the point is that 10MB of data in in an OS that has a 32MB process
space is out of the realm of realism. So that's actually nealy 1/3 of your
address space. And depending on your target device, it may already have a
huge chunk of it gone (the Phone Edition devices classically are misdesigned
by the OEM in this respect). While CE has a lot more to offer than a PIC,
it's still a long way from XP, which UMPCs and Tablets use, in both OS
resource availability as well as physical hardware resources.


--
Chris Tacke
OpenNETCF Consulting
Managed Code in the Embedded World
www.opennetcf.com
--

I understand that there are ways around it, but I think we have differing
opinions here. I can in fact allocate 300MB or so in a desktop application
w/o a problem (1/7th of my address space), so I guess I'd assume a platform
being presented as an enterprise-capable one would be able to do the same.
But even so, let's be realistic, in this day and age we're not moving 300MB
data files very often, but 5MB is hardly out of the realm of realism.

I've been doing embedded system development for the better part of 10
years now, so I guess I was just expecting too much of these modern
platforms.

In any event, thank you for the informative reply.

These are little tiny devices, not modern PCs where you just dump every
last piece of data you can think of and who cares, there's another 512MB
of RAM and ample hard disk for use with virtual memory laying around
anyway. You are programming for an embedded system. It's not an
embedded system of ten years ago, when you'd be lucky to have a 32-bit
processor, let alone multi-MB flash disks, but it's still a small device
and you have to program it with its limitations in mind. I'm not saying
that a 5MB XML file is out of the question, just that trying to process a
5MB XML object in RAM is not a smart way to start your work.

No, the limit will depend on the total number and size of allocations,
both in the system as a whole and the application which you are running.
The address space for your process is 32MB, regardless of how much RAM is
in the device, so you're presently using about 1/7 of the total *address
space* available to you for just this one object (imagine using 1/7 of
the 2GB address space of a process on your PC!). Not only that, but some
of that 32MB address space is used for DLLs that you are using, etc. The
lack of virtual memory for data means that you can't be blissfully
ignorant of what's really going on under the covers, if you can even when
virtual memory is present.

Paul T.

You're right that it's characters (so 5MB). As for why in RAM, I think
the counter to that is why not? The more we do with WM devices, the more
we want them to behave like other mobile devices (ie. Tablets, UMPCs),
so it's not atypical to be moving large amounts of data.

As for the specific issue, I probably will address it via file, but is
there some sort of known hard limit? In my case, I'm using the emulator,
and program memory indicates 65+ MB free when the error is thrown. To me
that seems unintuitive.


Paul G. Tobey [eMVP] wrote:
Seems like any single memory object that's 2.5MB (or is that
2.5Mcharacters, which would be 5MB), is a non-starter on a little
device like a Pocket PC. Why do you want it in RAM, anyway? Why not
write it to a file? At least there's a chance that a multi-MB file can
exist in the system.

Paul T.

It seems as if there's a hard limit in XMLTextWriter that's causing me
a headache.

If I create an XMLTextWriter from a StringWriter, once the string gets
to be about 2.5MB in length, the XMLTextWriter will OutOfMemory
exception. The simple code listed below causes this problem.

Anyone know of a workaround / solution?

private void button1_Click(object sender, EventArgs e)
{
System.IO.StringWriter sWriter = new System.IO.StringWriter();
System.Xml.XmlTextWriter xWriter = new
System.Xml.XmlTextWriter(sWriter);
xWriter.WriteStartElement("Test");
string s = new string('A', 10000);
while (true)
{
xWriter.WriteStartElement("ABC");
xWriter.WriteString(s);
System.Diagnostics.Debug.WriteLine("Length: " +
sWriter.ToString().Length);
xWriter.WriteEndElement();
}
xWriter.WriteEndElement();
}

 

[Paul G. Tobey [eMVP]]

Three words: paged virtual memory. CE doesn't have it.

Paul T.

I understand that there are ways around it, but I think we have differing
opinions here. I can in fact allocate 300MB or so in a desktop application
w/o a problem (1/7th of my address space), so I guess I'd assume a platform
being presented as an enterprise-capable one would be able to do the same.
But even so, let's be realistic, in this day and age we're not moving 300MB
data files very often, but 5MB is hardly out of the realm of realism.

I've been doing embedded system development for the better part of 10
years now, so I guess I was just expecting too much of these modern
platforms.

In any event, thank you for the informative reply.

These are little tiny devices, not modern PCs where you just dump every
last piece of data you can think of and who cares, there's another 512MB
of RAM and ample hard disk for use with virtual memory laying around
anyway. You are programming for an embedded system. It's not an
embedded system of ten years ago, when you'd be lucky to have a 32-bit
processor, let alone multi-MB flash disks, but it's still a small device
and you have to program it with its limitations in mind. I'm not saying
that a 5MB XML file is out of the question, just that trying to process a
5MB XML object in RAM is not a smart way to start your work.

No, the limit will depend on the total number and size of allocations,
both in the system as a whole and the application which you are running.
The address space for your process is 32MB, regardless of how much RAM is
in the device, so you're presently using about 1/7 of the total *address
space* available to you for just this one object (imagine using 1/7 of
the 2GB address space of a process on your PC!). Not only that, but some
of that 32MB address space is used for DLLs that you are using, etc. The
lack of virtual memory for data means that you can't be blissfully
ignorant of what's really going on under the covers, if you can even when
virtual memory is present.

Paul T.

You're right that it's characters (so 5MB). As for why in RAM, I think
the counter to that is why not? The more we do with WM devices, the more
we want them to behave like other mobile devices (ie. Tablets, UMPCs),
so it's not atypical to be moving large amounts of data.

As for the specific issue, I probably will address it via file, but is
there some sort of known hard limit? In my case, I'm using the emulator,
and program memory indicates 65+ MB free when the error is thrown. To me
that seems unintuitive.


Paul G. Tobey [eMVP] wrote:
Seems like any single memory object that's 2.5MB (or is that
2.5Mcharacters, which would be 5MB), is a non-starter on a little
device like a Pocket PC. Why do you want it in RAM, anyway? Why not
write it to a file? At least there's a chance that a multi-MB file can
exist in the system.

Paul T.

It seems as if there's a hard limit in XMLTextWriter that's causing me
a headache.

If I create an XMLTextWriter from a StringWriter, once the string gets
to be about 2.5MB in length, the XMLTextWriter will OutOfMemory
exception. The simple code listed below causes this problem.

Anyone know of a workaround / solution?

private void button1_Click(object sender, EventArgs e)
{
System.IO.StringWriter sWriter = new System.IO.StringWriter();
System.Xml.XmlTextWriter xWriter = new
System.Xml.XmlTextWriter(sWriter);
xWriter.WriteStartElement("Test");
string s = new string('A', 10000);
while (true)
{
xWriter.WriteStartElement("ABC");
xWriter.WriteString(s);
System.Diagnostics.Debug.WriteLine("Length: " +
sWriter.ToString().Length);
xWriter.WriteEndElement();
}
xWriter.WriteEndElement();
}