Data Access Tier with Large Result Set

[dm1608]

I'm wondering what the best practice should be when processing a large
result set within a Data tier. Basically, I have a C# app that runs a query
and loops thru over 500K rows and does a few things per row. I would like
to implement a DAL, but I think this would be stupid considering I'd have to
populate a generic collection with all the data, then pass from DAL to BAL.

How to folks handle this in the real world? Running multiple smaller
queries really isn't an option.

 

[Michael Nemtsev]

Hello dm1608,

You are relly need to do it in one DataSet/collection? You can apply async
scheme and send data by blocks.
BTW, if u need to send it bulky - zip it and send


d> I'm wondering what the best practice should be when processing a
d> large result set within a Data tier. Basically, I have a C# app that
d> runs a query and loops thru over 500K rows and does a few things per
d> row. I would like to implement a DAL, but I think this would be
d> stupid considering I'd have to populate a generic collection with all
d> the data, then pass from DAL to BAL.
d>
d> How to folks handle this in the real world? Running multiple
d> smaller queries really isn't an option.
d>
---
WBR,
Michael Nemtsev :: blog: http://spaces.msn.com/laflour

"At times one remains faithful to a cause only because its opponents do not
cease to be insipid." (c) Friedrich Nietzsch

 

[William Stacey [MVP]]

Need to pass the whole (i.e. 500K) generic collection back to client? Or is
this client direct to sql?

--
William Stacey [MVP]

| I'm wondering what the best practice should be when processing a large
| result set within a Data tier. Basically, I have a C# app that runs a
query
| and loops thru over 500K rows and does a few things per row. I would like
| to implement a DAL, but I think this would be stupid considering I'd have
to
| populate a generic collection with all the data, then pass from DAL to
BAL.
|
| How to folks handle this in the real world? Running multiple smaller
| queries really isn't an option.
|
|
|

 

[dm1608]

Well, I actually plan on running my utility app on the server directory
where SQL Server is installed. However, there would be nothing to stop me
from running it from another workstation -- other than poor performance.