maximum string length in c# and .net

<fawcett@gmail.com> wrote:
> The reason I find it interesting is that the practical limit for
> default StringBuilder and string behavior is much smaller than the
> theoretical limit. So, a 100Mb string with default handling can
> generate much more massive memory swings. Understanding why the vm size
> drastically exceeds the size of strings is important. In .net, if you
> have a string in the megabyte range, you'll generate a lot of objects
> that end up on the large object heap as residue as you grow the string.

That has nothing to do with the maximum size of strings though - it has
everything to do with the fact that strings are (publically) immutable.

Using StringBuilder instead of string concatenation to catenate several
strings (particularly when the number is variable) is a well known
optimisation (unfortunately less well understood). The benefits in
appropriate situations would be exactly the same even if strings had a
much larger or smaller theoretical maximum size.

--
Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

Jon,

The complete independence of the practical string limit from the
theoretical is precisely what I find so interesting. Also note that I
the program ran out of memory at widely differing string lengths using
different initializations for the stringbuilder. It wasn't just
comparing string concatenation to string building, but string building
under different initializations.

It is very difficult to intuit the cause of out of memory errors. The
strings in question were large but not absurdly large. Honestly, even
though it is clearly inefficient, I wouldn't expect an out of memory
from growing a stringBuilder to 325M or so. My first thought was, well
maybe the string max isn't very big. So that is where my questions
started.

The experiments show that the repeated doubling of the char array just
savages the memory manager. This limits the practical limit of the
string created by a stringbuilder to 1/3rd of its theoretical max.

So, the way you initialize the string builder is far more important
than the length of the string in terms of the memory management.


thanks,
fawce

<fawcett@gmail.com> wrote in message
news:1147389593.465324.149010@j73g2000cwa.googlegroups.com...
| Jon,
|
| The complete independence of the practical string limit from the
| theoretical is precisely what I find so interesting. Also note that I
| the program ran out of memory at widely differing string lengths using
| different initializations for the stringbuilder. It wasn't just
| comparing string concatenation to string building, but string building
| under different initializations.
|
| It is very difficult to intuit the cause of out of memory errors. The
| strings in question were large but not absurdly large. Honestly, even
| though it is clearly inefficient, I wouldn't expect an out of memory
| from growing a stringBuilder to 325M or so. My first thought was, well
| maybe the string max isn't very big. So that is where my questions
| started.
|
| The experiments show that the repeated doubling of the char array just
| savages the memory manager. This limits the practical limit of the
| string created by a stringbuilder to 1/3rd of its theoretical max.
|
| So, the way you initialize the string builder is far more important
| than the length of the string in terms of the memory management.
|
|
| thanks,
| fawce
|

Note that creating a SB with a capacity of > 750M characters (1.5 GB) may
well be possible when done very early in the process, this is what you
should do if ever you need to process very large strings, pre-allocate the
SB with a capacity large enough to hold the largest string you expect, don't
let SB extend dynamically until you run OOM.

Willy.