Large Datasets

[Bernie Hunt]

Has anyone seen any documentation or white papers with suggestions on how
to handle large datasets. My task is to read in 10K records and then step
through each one processing it's data. The processing involved fetching 3
other records that match the current, manipulate the data and then write
out 5 records to a different database.

DataReader is out of the question, because this whole precessing time could
take 10 or 20 minutes, so it looks like a DataSet. My concern is what
happens if a DataSet exceeds memory. Is it cached on disk in the swapfile
or does .Net handle it's temp storage. I'd love to find some kind of
guideline on how large can a DataSet get without causing calaterail damage
in the environment.

Thanks for any guidance.

Bernie

 

[Miha Markic [MVP C#]]

Hi Bernie,

Is server processing an option for you? (stored procedure or something)
What databaes are we talking?

 

[Cor Ligthert [MVP]]

Bernie,

A dataset can be as large as the program leaves enough memory for it. Where
the program can be AFAIK not larger than 800Mb.

I hope this helps

Cor

 

[David Browne]

Has anyone seen any documentation or white papers with suggestions on how
to handle large datasets. My task is to read in 10K records and then step
through each one processing it's data. The processing involved fetching 3
other records that match the current, manipulate the data and then write
out 5 records to a different database.

DataReader is out of the question, because this whole precessing time
could
take 10 or 20 minutes, so it looks like a DataSet. My concern is what
happens if a DataSet exceeds memory. Is it cached on disk in the swapfile
or does .Net handle it's temp storage. I'd love to find some kind of
guideline on how large can a DataSet get without causing calaterail damage
in the environment.

10,000 rows in a DataSet really isn't all that much. Just try it.

David

 

[Miha Markic [MVP C#]]

Hi Cor,

Bernie,

A dataset can be as large as the program leaves enough memory for it.
Where the program can be AFAIK not larger than 800Mb.

Where did you get this number from? Voodoo?

 

[Cor Ligthert [MVP]]

Miha Markic [MVP C#]

Where did you get this number from? Voodoo?

--

What do you mean?

Cor

 

[Miha Markic [MVP C#]]

I mean where do you get the 800Mb limit from.

--
Miha Markic [MVP C#]
RightHand .NET consulting & development www.rthand.com
Blog: http://cs.rthand.com/blogs/blog_with_righthand/

Miha Markic [MVP C#]
Where did you get this number from? Voodoo?

--

What do you mean?

Cor

 

[Cor Ligthert [MVP]]

I mean where do you get the 800Mb limit from.
From Herfried K. Wagner who gave this link

http://blogs.msdn.com/tess/archive/2005/11/25/496898.aspx

Cor

 

[Miha Markic [MVP C#]]

I could discuss here, but there are 64 bit CPUs out there and there is .net
2 that manages memory better.
Ah, and if I recall correctly, there is a switch that lets you use more than
2GB memory for user appsin 32bit windows, too.
And 800Mb is not accurate - it is just an assumption.

 

[Cor Ligthert [MVP]]

I could discuss here, but there are 64 bit CPUs out there and there is .net
2 that manages memory better.
Ah, and if I recall correctly, there is a switch that lets you use more
than
2GB memory for user appsin 32bit windows, too.
And 800Mb is not accurate - it is just an assumption.

Did I write something else?

Although the by you mentioned switch (the name is 3Gb switch) seems not
affect this behaviour it stays aprox 800Mb per process.

Cor

 

[Miha Markic [MVP C#]]

Did I write something else?

" Where the program can be AFAIK not larger than 800Mb."

Note: 64bit CPUs can address terabytes of memory.

 

[Cor Ligthert [MVP]]

" Where the program can be AFAIK not larger than 800Mb."

Note: 64bit CPUs can address terabytes of memory.

Yes I wrote As Far As I Know.

Can you than give it for a 64bit CPU. Because this is not a memory behaviour
but an OS behaviour and than even as it is written with the layer Net 1.x.

However as you can enlighten me with some more precise figurs for a 64bit XP
OS with Net 2.0 and/or Vista than I will be delighted.

Cor

 

[Jeff Clausius]

Cor:

I did some extensive testing on this a couple of years back with the 1.1
..Net Framework and console/web services. The /3GB switch *might* affect
the amount of memory a .Net application can use. It really depends on
the flavor of Operating System.


For example, a test app on 32-bit Windows XP or Windows 2003 Standard
Server (using /3GB) will only allocate an additional 900MB - 1200 MB of
memory. But on 32-bit Windows 2003 Enterprise Server, using the /3GB
switch will provide an application with the additional memory. Note, the
/3GB switch is a double edged sword as it will limit any OS processes max
memory size to use 1GB.


Some useful info from the original thread (watch word wrap) -

http://groups.google.com/group/microsoft.public.dotnet.framework.aspnet.w
ebservices/tree/browse_frm/thread/cf6567aa5b6ee5d8/265abc31aa209df0?rnum=
1&hl=en&q=%2F3gb+clausius&_done=%2Fgroup%
2Fmicrosoft.public.dotnet.framework.aspnet.webservices%2Fbrowse_frm%
2Fthread%2Fcf6567aa5b6ee5d8%2F265abc31aa209df0%3Ftvc%3D1%26q%3D%
2F3gb+clausius%26hl%3Den%26#doc_265abc31aa209df0


Jeff Clausius
SourceGear

 

[Cor Ligthert [MVP]]

Jeff,

Thanks,

Cor

Cor:

I did some extensive testing on this a couple of years back with the 1.1
.Net Framework and console/web services. The /3GB switch *might* affect
the amount of memory a .Net application can use. It really depends on
the flavor of Operating System.


For example, a test app on 32-bit Windows XP or Windows 2003 Standard
Server (using /3GB) will only allocate an additional 900MB - 1200 MB of
memory. But on 32-bit Windows 2003 Enterprise Server, using the /3GB
switch will provide an application with the additional memory. Note, the
/3GB switch is a double edged sword as it will limit any OS processes max
memory size to use 1GB.


Some useful info from the original thread (watch word wrap) -

http://groups.google.com/group/microsoft.public.dotnet.framework.aspnet.w
ebservices/tree/browse_frm/thread/cf6567aa5b6ee5d8/265abc31aa209df0?rnum=
1&hl=en&q=%2F3gb+clausius&_done=%2Fgroup%
2Fmicrosoft.public.dotnet.framework.aspnet.webservices%2Fbrowse_frm%
2Fthread%2Fcf6567aa5b6ee5d8%2F265abc31aa209df0%3Ftvc%3D1%26q%3D%
2F3gb+clausius%26hl%3Den%26#doc_265abc31aa209df0


Jeff Clausius
SourceGear

 

[Bernie Hunt]

No, There are two applications where this applies. First the originating
database is an ODBC connection to a Faircom database. The vast majority of
the customers using this will be running XP Pro with no server.

The second application is for data conversion where the originating
database could be just about anything, Pervasive, Advantage and dBase are
the most common.

Bernie