Multiple worker threads/objects pulling from the same MSMQ queue

[Jeff L.]

I'll try to keep this short and sweet...

I have a message queue that holds data received from multiple clients.
For each message in the queue, I need to do some processing that
involves communicating with a web service and then dump the data into
a SQL server table. Also, all work is done within an MSMQ transaction
so that the message stays in the queue if something goes wrong (can't
communicate with SQL or something, for example). So:

1. Begin MSMQ transaction.
2. Pull message.
3. Parse data object.
4. Pull additional info from web service.
5. Run stored proc to insert data into SQL server.
6. End MSMQ transaction.

The problem I'm having is that the data can possibly come in faster
than I can process it. It's not super time critical, but if it loads
up it will become a serious problem. The current process takes about
300ms to complete.

I tried creating a pool of threads (not ThreadPool...don't want to go
there due to past issues with WebRequest/WebResponse). This did not
work...it seemed like the more threads I created, the slower the
process became (2 threads = 700ms, 10 threads = 3000ms, 25 threads =
9000ms).

Any suggestions as to how I can have multiple threads/objects working
from the same message queue while still utilizing transactions? Maybe
I just need a fresh brain to set me straight on this issue.
Thanks for any help you can offer!

 

[AlexS]

Wouldn't it be much simpler to take message from queue, process it in
separate thread or process and if any of processing steps fails - to post it
back into queue? Without extending transaction across MQ, web and SQL?

I think you are missing the point with MQ transactions. They are for
transacting delivery of complex messages (like several interrelated or
several recipients) - not for transacting web or SQL processing. Web service
should have its own transacting capabilities. SQL has its own.

I would consider to have separate worker threads which do steps 2-3,
separate workers for web service (which by definition might be very slow and
unreliable) and separate workers for SQL. Worker threads might send each
other information through MQ, internal queues or even through SQL tables if
you are really worried about persistence and tracing.

HTH
Alex

 

[Jeff L.]

Thanks for the reply, Alex...that makes sense. The reason I was using
transactions was just so I could roll the message back into the queue
in case something went wrong. Receiving and reposting works just as
well (rewrote that part this morning), so I guess I really don't need
MSMQ transactions.

I did some more testing last night and it seems that the problem area
in my code is the web service call. The more requests I throw at it,
the slower things get. There isn't any reason for the slowdown since
the box that the web service runs on is much more powerful than it
needs to be...I'm thinking something is going wrong in my call to
retrieve the XML document. I guess I'll have to start digging into
this new problem now that my old one is resolved.