Suggestion

[Peter]

I have Windows Service application which creates reports. It's a
multithreaded application which scans database every several seconds and
retrieves records from a database and than spans a new thread for each
record which creates a report. I run this application on 3 different
servers and each server does a specific type of records. What I would like
to do is to have all servers processing any type of record, but I want only
one program processing any one record. My question is how do I keep
multiple servers from processing the same records. I've tried to place a
flag in the database as 1 = processing and 0 = is not, but database it not
fast enough, caching on the database prevents reading of the latest status.
How do I communicate between all servers. I am not looking for specific
code just an idea or pointers on how to communicate between all servers
before processing a record. I don't want to have a master server because
it's a one point of failure.


Thank You


Peter

 

[Ken Foskey]

I keep multiple servers from processing the same records. I've tried to
place a flag in the database as 1 = processing and 0 = is not, but
database it not fast enough, caching on the database prevents reading of
the latest status. How do I communicate between all servers. I am not
looking for specific code just an idea or pointers on how to communicate
between all servers before processing a record. I don't want to have a
master server because it's a one point of failure.

Implement a socket server that monitors the database and hands out the
identifiers to the clients. Clients start and connect to find out next
record, if there is none then the server blocks the client reports until
there is something to do. You can start as many clients as you want and
all the server does is monitor the database and cycle through the queue
of requests handing them out to work.

Ken

 

[Peter]

Thanks Ken for your help!

But if the socket server goes down all of the clients are down - single
point of failure.

 

[Ken Foskey]

Thanks Ken for your help!

But if the socket server goes down all of the clients are down - single
point of failure.

Use UDP and a broadcast, have all clients monitor the same UDP
broadcast socket. No lag time. Each one advertises which ones they pick
up.

Servers should advertise completed ones. If there is no completed and a
server is idle it should query old incomplete jobs using the same
broadcast and if there is no response pick that one up and start working
on it again (recoverability).

Note that UDP is not reliable and two could start work on exactly the
same piece at the same time. So you have to handle conflicts still, it
just reduces the probability.

I am wondering how you don't have a central point of failure with a
Database anyway?

Ken

 

[Leon Lambert]

How are you handling redundancy on the database? This mechanism could
also handle switch over to a redundant data manager. If you don't have a
redundant database then you already have a single point of failure and
the whole topic is mute.

 

[Peter Morris]

DB is not fast enough?

01: Select all rows with state = 0

02:

while (move to next row)
{
try
{
start transaction
update the status to 01
commit transaction
process it
}
catch (the optimistic locking exception you experience)
{
roll back transaction
}
}



Pete

 

[Peter]

Thanks Peter - you have convinced me!

I do have a thread pool, but it's more than number of CPUs, the reason for
that is this program runs Crystal Reports, the reports access DB2 database
on AIX server, one report can take few seconds to retreive data and next one
could take few minutes, the CPU is idle when Crystal Reports is retreiving
the data. I agree the program should manage the threads based on the
workload, but in this case it's impossible. For example the program starts a
thread and asks database for the data, when this occurs the CPU drops down
almost to zero, so my program looks at the CPU and says there is still
plenty of CPU so I will span another thread, and so on and so on, then each
thread returns data from the database and CPU goes up 100%. Same thing
could happen the other way, my program looks at the CPU and it's at 100% so
it would not span anymore threads, but it could be spanning new threads
because it takes few seconds to retreive data from the database.

 

[Peter]

I really don't care if the database is point of failure, if database is down
the whole company is down, if my program is down users are screaming
"where's my report!" - It's kind of like we don't carry a spare engine, but
we do carry spare tire and the engine is a single point of failure so why do
we carry the spare tire.

I don't need redundancy in a true sense "if one process goes down there's
another one waiting in the wings", what I need is multiple programs
processing data from the same pool, if one goes down the process is still
continuing, although slower and also ability to add more servers to the pool
if need arises.

Thanks Lean for pointing the possible problems, I do appreciate it - it
makes you think.

 

[Peter]

Thanks Ken for your advice I think I'll go with your suggestion.

I don't want to overstay my welcome, but do you know of any source code
examples or reading material?

Yes database is a single point of failure, but that's not relevant in my
case, if the database goes down the whole company is down, if my program
goes down the users will be screaming "where is my report!", so I have to
take care of my piece of process first.

 

[Ken Foskey]

Thanks Ken for your advice I think I'll go with your suggestion.

I don't want to overstay my welcome, but do you know of any source code
examples or reading material?

I have something that does the broadcast and another that monitors it.
Unfortunately what I have done is totally different to what you need.

Ken

loose rmove. if you want to go offline.