Asynchronous I/O or multiple threads?

  • Thread starter Thread starter Quiet Man
  • Start date Start date
Q

Quiet Man

Hi all,

I'm designing a fairly simple service that will run on W2K/SP4 and W2K3
servers. It's job is to be a very specialized database server that listens
on a given IP address / TCP port and handles multiple connections. Client
programs will make a connection and pass text strings to the service, which
will then return a value for each of the strings. It's unlikely the service
will ever have more than 100 simultaneous connections and most of what it
will do is a lookup in a table for each string. We've considered MSDE for
this and for a number of reasons it's impractical for us to use it.

In researching various ways of handling simultaneous connections I've seen
examples using multithreading and others using async I/O. I'd like the
program to scale well but I don't want to overwhelm the server. I'm looking
for guidance on choosing between the two methods.

Alternately, I'd be happy to use a commercial database server if I could
find one that can be distributed easily and royalty free and which doesn't
cost too much. Something like the VistaDB database engine would be ideal but
it can't run as a service.

Thoughts / suggestions?
 
Quiet Man said:
I'm designing a fairly simple service that will run on W2K/SP4 and W2K3
servers. It's job is to be a very specialized database server that listens
on a given IP address / TCP port and handles multiple connections. Client
programs will make a connection and pass text strings to the service, which
will then return a value for each of the strings. It's unlikely the service
will ever have more than 100 simultaneous connections and most of what it
will do is a lookup in a table for each string. We've considered MSDE for
this and for a number of reasons it's impractical for us to use it.

In researching various ways of handling simultaneous connections I've seen
examples using multithreading and others using async I/O. I'd like the
program to scale well but I don't want to overwhelm the server. I'm looking
for guidance on choosing between the two methods.

Alternately, I'd be happy to use a commercial database server if I could
find one that can be distributed easily and royalty free and which doesn't
cost too much. Something like the VistaDB database engine would be ideal but
it can't run as a service.

Thoughts / suggestions?
Click to expand...

Given that MSDE is the most obvious choice IMO, it would help if you
could state the reasons you have for not using it.

In terms of handling simultaneous connections, asynchronous IO is
likely to scale better, but be warned that you'll end up using system
threadpool threads automatically - you need to be aware of this, as if
you start doing other things which require the threadpool within
threads that are already running in the pool, you can easily deadlock
(by having all the threadpool threads waiting until another threadpool
thread is available before they can do any more work).
 
Quiet Man said:
I'm designing a fairly simple service that will run on W2K/SP4 and W2K3
servers. It's job is to be a very specialized database server that listens
on a given IP address / TCP port and handles multiple connections. Client
programs will make a connection and pass text strings to the service, which
will then return a value for each of the strings. It's unlikely the service
will ever have more than 100 simultaneous connections and most of what it
will do is a lookup in a table for each string. We've considered MSDE for
this and for a number of reasons it's impractical for us to use it.

In researching various ways of handling simultaneous connections I've seen
examples using multithreading and others using async I/O. I'd like the
program to scale well but I don't want to overwhelm the server. I'm looking
for guidance on choosing between the two methods.

Alternately, I'd be happy to use a commercial database server if I could
find one that can be distributed easily and royalty free and which doesn't
cost too much. Something like the VistaDB database engine would be ideal but
it can't run as a service.

Thoughts / suggestions?
Click to expand...

Given that MSDE is the most obvious choice IMO, it would help if you
could state the reasons you have for not using it.

In terms of handling simultaneous connections, asynchronous IO is
likely to scale better, but be warned that you'll end up using system
threadpool threads automatically - you need to be aware of this, as if
you start doing other things which require the threadpool within
threads that are already running in the pool, you can easily deadlock
(by having all the threadpool threads waiting until another threadpool
thread is available before they can do any more work).

--
Jon Skeet - <[email protected]>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

----------------

Thanks for the reply Jon.

Regarding MSDE - one big reason is the complexity of deployment. This app
will be installed on servers that may or may not already have MSDE 1.0, MSDE
2000, MSDE 2000A, SQL 7 or one of the SQL 2000 engines (personal, developer,
standard, enterprise) installed already for use by other apps - this makes
the deployment process / installation program a bear. Another reason is
size - the rest of the app is looking like about 4 MB of code and the MSDE
is 45MB or so. Lessor reasons include the throughput uncertainty that comes
with the workgroup governor (which has bitten us in the past) and resistance
on the part of some network / server admins due to past security issues with
the SQL 2000 engine. Still lessor reasons are the thought that MSDE is
overkill and that a specialized app will have a smaller memory footprint and
possibly be quicker.

All that being said I do like the MSDE and have used it successfully in the
past.

Thanks for the async I/O scalability note. I also appreciate your pointing
out the threadpool consideration. I'm hoping it won't be a problem in that
99% of what this service will be doing is looking up values in a global
table in read-only mode. The table will peak at about 300,000 string / value
pairs, average string length of 8 characters. The data will be refreshed
once per day or less. I may have to look into limiting the number of
connections that the service will accept.

Quiet Man
 
Hi,

I may have missed the point here, but why do you need a database at all for
such a simple task ?

Cheers

Doug Forster
 
100 threads should be not problem. Some use <= 1000 threads without issue
(however 1000 is pushing it imo.) You could fire up new threads upto to
some max, say 500, then block new requests with server busy reply. You
could also use your own thread pool (many existing example out there such as
Codeproject) with some max and have ready threads ready to go. These
servers are much easier to create and reason about. Async servers may scale
better, but are harder to code and get right and have the potential thread
pool blocking issue. If you have like 10,000 active requests, then async is
the option, but if you really have that many then the server will melt
anyway or be so slow that almost nothing gets done. So your back to
factoring out load between other servers - which you could do with either
approach. Therefore, the threaded server may start to look better again.
If your not blocking alot in your processing piece (i.e. just doing cached
table lookups) then potentially you could service all requests with *one*
thread using a queued approach. Queue up all requests and service the queue
with one tight server loop and send the reply. Your replies could be put
into yet another queue that is serviced by one-to-N threads (say your own
thread pool or the system thread pool) so that one reply can't block the
server. For that matter, your receiver side could be 1-N threads also that
waits for and builds the complete requests - so one slow client does not do
a DoS on your server. Each Client-Listener thread would put completed
request into the server IN-Q. The one server thread reads from IN-Q,
processes, and writes reply to OUT-Q. The Out thread pool handles the
socket sends and socket closing (UDP may work better for this single
request/reply deal). Even with TCP, this would work well for simple
request/reply where a reply signals the end of the session. I tend to favor
the threaded server or queued server approach (others may vary.) hth
 
Thanks for your thoughts William. It helps to have some guidelines regarding
what a reasonable number of threads is, etc. I obviously haven't done a
great deal of multithreaded programming.

I'm comfortable with limiting the service to accepting 100 simultaneous
connections. A typical load on the server will be less than 1000 connections
per hour; each connection will pass (in one big lump) an average of 200 text
strings that will be looked up. If I go with a multithreaded approach my
current thought is that each thread will take one of these connections,
lookup values for each of the text strings, return all the values and at
that point terminate the connection (so yes, the reply indicates the end of
the session). The thread would then be put back into the thread pool for
reuse. All the table data (probably 300,000 rows) will stay in RAM so
lookups should be extremely quick.

I might be able to pull this off with a single threaded app but it seems
better to plan for future growth now. I haven't done any performance testing
yet but I can't imagine that the actual lookup process will take much time -
I'm guessing that most of the time will be occupied by making the connection
and transferring the data between the client and the server.

I'll have to study up on the response queue idea.

Again, thanks. You've given me plenty to think about.

------------------------------

100 threads should be not problem. Some use <= 1000 threads without issue
(however 1000 is pushing it imo.) You could fire up new threads upto to
some max, say 500, then block new requests with server busy reply. You
could also use your own thread pool (many existing example out there such as
Codeproject) with some max and have ready threads ready to go. These
servers are much easier to create and reason about. Async servers may scale
better, but are harder to code and get right and have the potential thread
pool blocking issue. If you have like 10,000 active requests, then async is
the option, but if you really have that many then the server will melt
anyway or be so slow that almost nothing gets done. So your back to
factoring out load between other servers - which you could do with either
approach. Therefore, the threaded server may start to look better again.
If your not blocking alot in your processing piece (i.e. just doing cached
table lookups) then potentially you could service all requests with *one*
thread using a queued approach. Queue up all requests and service the queue
with one tight server loop and send the reply. Your replies could be put
into yet another queue that is serviced by one-to-N threads (say your own
thread pool or the system thread pool) so that one reply can't block the
server. For that matter, your receiver side could be 1-N threads also that
waits for and builds the complete requests - so one slow client does not do
a DoS on your server. Each Client-Listener thread would put completed
request into the server IN-Q. The one server thread reads from IN-Q,
processes, and writes reply to OUT-Q. The Out thread pool handles the
socket sends and socket closing (UDP may work better for this single
request/reply deal). Even with TCP, this would work well for simple
request/reply where a reply signals the end of the session. I tend to favor
the threaded server or queued server approach (others may vary.) hth
 
Just got a chance to check the group again. Many thanks! I'll be examining
the code and trying it out as well tomorrow.

Sound good. Here is a little tcp server and client (see attached zip of
both projects) I just worked up for a simple example. I use XmlSerializer
to serialize a type to send and receive. This is overkill if you just need
to send one string and return one string, but is helpful if you needs are
more then just passing one string. Just got it to the point of working
without a lot of debugging or error handling, but should be a good start.
The main server loop is below:

private void ProcessStart()
{
listener = new TcpListener(lep);
listener.Start();
Socket s = null;

while( IsStarted )
{
try
{
s = listener.AcceptSocket();
}
catch(System.Net.Sockets.SocketException sx)
{
if ( ! IsStarted )
break; // We Stopped server.
Console.WriteLine("Server Error:"+sx.Message);
isStarted = false;
}
MyWorker newWorker = new MyWorker(s, this);
newWorker.Start();
Console.WriteLine("Server received new request.");
}
}

--
William Stacey, MVP
http://mvp.support.microsoft.com

Quiet Man said:
Thanks for your thoughts William. It helps to have some guidelines regarding
what a reasonable number of threads is, etc. I obviously haven't done a
great deal of multithreaded programming.
Click to expand...
....
 
Back
Top