Breaking up a Dataset to send over Remoting

[brian.twardzik]

Hey there,

I've been working on a project which requires me to broadcast a large
volume of information to various subscriber applications. I've been
using an event based architecture combined with using remote-
asynchronous-delegate invocation. This works fine for keeping the GUI
events all in order, however, when it comes to actually sending large
amounts of data I am finding the memory consumption unreasonably high.

For example, I query a database on a 'Provider' application. The
resulting dataset is then given to a 'Subscriber' object through a
remote call.

First, it seems dumping the SQL results into a dataset object consumes
a great deal of memory in the first place.

ie:
int numEffected = myAdapter.Fill(dataTable);

Now, I want to break up the dataset objects I send into smaller pieces
and then reintegrate them on the subscriber end. I know there is an
overloaded version of Fill that allows one to set the startrecord, but
I'm curious if it will ensure a contiguous record.

Can I request:

myAdapter.Fill(0, 4000, dataTable)
myAdapter.Fill(4000, 8000, dataTable)

And not miss anything?

 

[shashank kadge]

Hey there,

I've been working on a project which requires me to broadcast a large
volume of information to various subscriber applications. I've been
using an event based architecture combined with using remote-
asynchronous-delegate invocation. This works fine for keeping the GUI
events all in order, however, when it comes to actually sending large
amounts of data I am finding the memory consumption unreasonably high.

For example, I query a database on a 'Provider' application. The
resulting dataset is then given to a 'Subscriber' object through a
remote call.

First, it seems dumping the SQL results into a dataset object consumes
a great deal of memory in the first place.

ie:
int numEffected = myAdapter.Fill(dataTable);

Now, I want to break up the dataset objects I send into smaller pieces
and then reintegrate them on the subscriber end. I know there is an
overloaded version of Fill that allows one to set the startrecord, but
I'm curious if it will ensure a contiguous record.

Can I request:

myAdapter.Fill(0, 4000, dataTable)
myAdapter.Fill(4000, 8000, dataTable)

And not miss anything?

should work. i tried for small numbers n it did not give any problem
with a SQL database.
Only thing is its
adapter.fill(DSone,0,4000,"Table")
adapter.fill(DStwo,4000,8000,"Table")

-
shashank kadge

 

[brian.twardzik]

should work. i tried for small numbers n it did not give any problem
with a SQL database.
Only thing is its
adapter.fill(DSone,0,4000,"Table")
adapter.fill(DStwo,4000,8000,"Table")

-
shashank kadge

Unless you do the following beforehand:

adpt.SelectCommand = new SqlCommand("SELECT bar FROM foo",
_connection);

My concern is that:

1) I suspect the datasource is a cached query result at the database,
no idea how long it will remain consistant D:
2) I need to break up the dataset into chunks because loading the
entire query result causes too much memory consumption

In order to iterate through the pieces with the above fill command, I
need to know the total number of rows of the result set without
actually loading the results into a dataset.

*boggles* anyone have any ideas?

 

[shashank kadge]

Unless you do the following beforehand:

adpt.SelectCommand = new SqlCommand("SELECT bar FROM foo",
_connection);

My concern is that:

1) I suspect the datasource is a cached query result at the database,
no idea how long it will remain consistant D:
2) I need to break up the dataset into chunks because loading the
entire query result causes too much memory consumption

In order to iterate through the pieces with the above fill command, I
need to know the total number of rows of the result set without
actually loading the results into a dataset.

*boggles* anyone have any ideas?- Hide quoted text -

- Show quoted text -

Select count(*) would give the record count.

-
shashank kadge