Handling very huge data

  • Thread starter Thread starter Guest
  • Start date Start date
G

Guest

Hi All

Wish all of u Happy New Year

I am developing a application where in I retrieve data from Database which
can go up millions records, do some processing on that and update the same in
single transaction,
now with limited RAM say 500MB how should I handle this data (I know one
solution might be paging) so that my ASP.NET process does not fall short of
memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
recycled because memory consumption exceeded the 305 MB (60 percent of
available RAM). "

Regards
Makarand
 
Did you consider using a stored procedure for processing the data?
 
This much data is not a good candidate for putting in a dataset or the likes
of dataset. Keep it in the database for processing if you can.
What is the compelling reason that you must pull it out of the db to work on
so much data?

- Sahil Malik
http://dotnetjunkies.com/weblog/sahilmalik
 
Makarand,

It sounds to me like you have to real options here.

1) Chunk thru the data. I would choose this if there is some specific
business logic that you cannot perform directly on the database layer.

2) Process the Data in the Database. I would choose this if you can
perform the logic on the database layer. If you can go this route it
will probably be the most efficient.

HTH,
-eric
 
Sahil
Thanx for prompt reply,
Thing is I get the data from database server, add feed this data to MS
RuleEngine for applying Business Rules and after that I take same data and
update it in database.
So I don't c any scope of doing all these things in database.
Hope this time I am more clear on my problem

Makarand
 
Miha
Thanx for prompt reply,
Thing is I get the data from database server, add feed this data to MS
RuleEngine for applying Business Rules and after that I take same data and
update it in database.
So I don't c any scope of doing all these things in database.
Hope this time I am more clear on my problem

Makarand


Miha Markic said:
Did you consider using a stored procedure for processing the data?

--
Miha Markic [MVP C#] - RightHand .NET consulting & development
SLODUG - Slovene Developer Users Group
www.rthand.com

Makarand Keer said:
Hi All

Wish all of u Happy New Year

I am developing a application where in I retrieve data from Database which
can go up millions records, do some processing on that and update the same
in
single transaction,
now with limited RAM say 500MB how should I handle this data (I know one
solution might be paging) so that my ASP.NET process does not fall short
of
memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
recycled because memory consumption exceeded the 305 MB (60 percent of
available RAM). "

Regards
Makarand
Click to expand...
Click to expand...
 
I'd recommend getting peices of the data rather than millions of rows
together. Maybe in another 4-5 years when we have terabytes of RAM and 64
bit CPUs my suggestion will change, but today this is what I'd recommend.

- Sahil Malik
http://dotnetjunkies.com/weblog/sahilmalik
 
Hi,

try using API server side cursors that would fetch you data in batches, so
that your app. will not block for that period, which greatly improves user
experience.

regards
Joyjit
 
You must log in or register to reply here.

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

aspnet_wp.exe Problem 8
aspnet_wp.exe was recycled because memory consumption exceeded the 1
ASP.NET choke when trying to delete cookies 3
aspnet_wp.exe was recycled because memory consumption exceeded 1
asp.net recycled 2
Memory problem with aspnet_wp.exe 1
Server Application Unavailable ?? 4
aspnet_wp recycled 4

Back
Top