Background reading of large array of records

[Claire]

Hi,
I'm writing an internal mail system module for my project.
There may be several thousand 'users' on site, and I need the whole list for
filtering, so I decided to load up the users in the main form of the module
in a background thread
I used to read in this list the first time it was required, but as there's
usually a "dead" period between opening the main form and actually needing
the list it seemed a good idea to do it this way to speed things up.
Is that bad practice?

I also dont like what Ive done in the following.
"while (_ReadUsersThread != null)Thread.Sleep(0);"
I don't THINK that _ReadUsersThread could remain viable forever as it's set
to null on exit of the thread method, and this is one place I'd need a
synchronous read of the database anyway, but I think while loops like that
are bad.
I also think that I should be using a Lock when I access _ReadUsersThread,
but not sure what I should lock.
Would someone comment please on a better way of doing it?

private void MessageCentreInboxForm_Load(object sender, System.EventArgs e)
{
_ReadUsersThread = new Thread(new ThreadStart(ReadUsersThreadMethod));
_ReadUsersThread.IsBackground = true;
_ReadUsersThread.Start();
}

// property getter for AllUsers

private tableUsers[] AllUsers
{
get
{
string fnName = "AllUsers:get";
// Wait for _ReadUsersThread to finish if it's
// already reading from the database
while (_ReadUsersThread != null)Thread.Sleep(0);
if (_AllUsers == null)
{
try
{
_AllUsers = _database.ReadUsers(tableUsers.UserType.User,"",false);
}
catch(Exception e)
{
LogException(fnName, e);
frmWarning.ShowError(this, e);
}
}
return _AllUsers;
}
}


private void ReadUsersThreadMethod()
{
string fnName = "ReadUsersThreadMethod";
try
{
tableUsers[] allusers =
_database.ReadUsers(tableUsers.UserType.User,"",false);
_AllUsers = allusers;
}
catch(Exception e)
{
LogException(fnName, e);
Invoke(new OnThreadExceptionDelegate(OnThreadException), new object[]
{e});
}
finally
{
_ReadUsersThread = null;
}
}// function

 

[Guest]

Hi,
I'm writing an internal mail system module for my project.
There may be several thousand 'users' on site, and I need the whole list for
filtering, so I decided to load up the users in the main form of the module
in a background thread
I used to read in this list the first time it was required, but as there's
usually a "dead" period between opening the main form and actually needing
the list it seemed a good idea to do it this way to speed things up.
Is that bad practice?

I also dont like what Ive done in the following.
"while (_ReadUsersThread != null)Thread.Sleep(0);"
I don't THINK that _ReadUsersThread could remain viable forever as it's set
to null on exit of the thread method, and this is one place I'd need a
synchronous read of the database anyway, but I think while loops like that
are bad.
I also think that I should be using a Lock when I access _ReadUsersThread,
but not sure what I should lock.
Would someone comment please on a better way of doing it?

private void MessageCentreInboxForm_Load(object sender, System.EventArgs e)
{
_ReadUsersThread = new Thread(new ThreadStart(ReadUsersThreadMethod));
_ReadUsersThread.IsBackground = true;
_ReadUsersThread.Start();
}

// property getter for AllUsers

private tableUsers[] AllUsers
{
get
{
string fnName = "AllUsers:get";
// Wait for _ReadUsersThread to finish if it's
// already reading from the database
while (_ReadUsersThread != null)Thread.Sleep(0);
if (_AllUsers == null)
{
try
{
_AllUsers = _database.ReadUsers(tableUsers.UserType.User,"",false);
}
catch(Exception e)
{
LogException(fnName, e);
frmWarning.ShowError(this, e);
}
}
return _AllUsers;
}
}


private void ReadUsersThreadMethod()
{
string fnName = "ReadUsersThreadMethod";
try
{
tableUsers[] allusers =
_database.ReadUsers(tableUsers.UserType.User,"",false);
_AllUsers = allusers;
}
catch(Exception e)
{
LogException(fnName, e);
Invoke(new OnThreadExceptionDelegate(OnThreadException), new object[]
{e});
}
finally
{
_ReadUsersThread = null;
}
}// function

Hello Claire,

i read the code and ther a lot to improve in the cod, first lets start with
architecture, do we want the system to load the user manually (property get )
or useing cache memory loading the user, listen to event of user changes in
the data base and update the user cache storage, the code try to solve some
of the issue, but there is no listen or caching mecahnisem (it should be
simple cache), also try to figure out whther you can listen to the data bse
changes and reflect the changes to the cache.

Second.
'While' its a bad code, commonly used in naive script to eait to somethings,
there are some ways to handle this, In case you will implement a simple
cache, so have a static sync object will solve this, so in case the static
sync object is not locked, so check the cache and get the data, saperate the
data implementation from the using the data.
Another way, use monitor, like lock.
Another way, wait call back, the event method, in case the thread complete
you can listen to event (from the cahce) and read the data.
There alot of ways to implement this situation, check the wab (lot of
examples).

Its bad to write this, because its a hell gate, after long time you can
forget this code and you can get bugs whcih are very hard to find occurred by
the sleep(0).
------- while (_ReadUsersThread != null) Thread.Sleep(0); ---------

Hope it will help to you.

Yaron karni
http://dotnetbible.blogspot.com/

 

[Claire]

Hello Claire,

i read the code and ther a lot to improve in the cod, first lets start
with
architecture, do we want the system to load the user manually (property
get )
or useing cache memory loading the user, listen to event of user changes
in
the data base and update the user cache storage, the code try to solve
some
of the issue, but there is no listen or caching mecahnisem (it should be
simple cache), also try to figure out whther you can listen to the data
bse
changes and reflect the changes to the cache.

Hello Yaron
I'd love to do something like this for the project, keep a cache of users
somewhere.
Would u point me in the general direction of what I should be looking at to
"listen" for changes on a database. This is the first database application
I've ever written.
I'm using ADO ( IDbConnection,IDbDataAdapter etc) , desktop application.
(Must work with MySQL 5.0 and/or Microsoft SQL server engines)

 

[Guest]

You may loook at the following link
http://www.codeproject.com/dotnet/usingado3.asp

hope it will help, in our company we develop a platform called AIS
(www.attunity.com) which have the ability (one on many abilities) to notify
on every data base data changed.

 

[Peter Duniho]

[...]
I used to read in this list the first time it was required, but as there's
usually a "dead" period between opening the main form and actually needing
the list it seemed a good idea to do it this way to speed things up.
Is that bad practice?

No. It's a fine practice. Good, even.

I also dont like what Ive done in the following.
"while (_ReadUsersThread != null)Thread.Sleep(0);"
I don't THINK that _ReadUsersThread could remain viable forever as it's set
to null on exit of the thread method, and this is one place I'd need a
synchronous read of the database anyway, but I think while loops like that
are bad.

Well, the main reason the loop is bad is that it means that the AllUsers
method will now block until the thread exits. There are other thing bad
about that sort of thing, but the blocking basically negates whatever
benefit you had to putting the work in a background thread in the first
place.

Yes, it's slightly better than just calling the code directly, since at
least you can get it going ahead of time. But you ought to have a
design that does not make the main UI thread block at all.

I also think that I should be using a Lock when I access _ReadUsersThread,
but not sure what I should lock.
Would someone comment please on a better way of doing it?

You should not have to look at the thread instance value at all.

On your main form, you should have some logic that does something
appropriate until the data is available. Display a placeholder value or
some sort of "not ready" label until the background thread is done.

For the background thread itself, my suggestion is that you use the
BackgroundWorker class, and make sure you use it as intended. That is,
subscribe to the events it contains for the purpose of notification of
its progress and completion. This will ensure that all of your handling
of the thread's state still happens on your main UI thread, avoiding any
need to lock or otherwise synchronize the data.

The synchronization will happen naturally by you ensuring that you only
ever access things on the main thread when they are known to be valid
and not in use by the background thread. Use the BackgroundWorker's
RunWorkerCompleted event to set state that tells you what's valid and
what's not and when you might be able to update the UI to reflect the
completion of the background worker. This means, for one, being able to
handle the AllUsers property returning null, and doing something
appropriate with that null value (like displaying the placeholder, as
mentioned above).

Pete