Application design

[David]

Hi,

This question could be very subjective but I am after a way to design
this...

I am developing in C#, .NET 2.0

Basically, I have data coming in from a mobile phone. This is sending a file
to a webservice. The webservice will save the file to local disk and also
send / receive messages from the mobile phone. This part is working fine,
though wether I should have a multi-thread on this, I don't know.


Once the file has been saved, I need to then work on it. The process can
become quite intensive. Also, each file can come from a different customer
and each customer will have a 'priority' preference, for example, some files
have to be acted upon immediately (due to them being time critical) where
others that are not so critical can wait.


My thoughts to handle this is a windows service with a filewatcher to watch
the webservice receiving folder. Then, depending on the file (customer),
react to it...

I guess I should be doing threading here (I have not really done any
threading before...) but I do have to bear in mind priority files. Also, if
there are for example, 3 files from one phone, they have to be handled
chronologically.

We could be talking many thousands of files per day that need to be handled
(both for the webservice and windows service). Also, there could be varying
outputs, for example, some customers may just want a raw processed file
(XML), others may want an image while others may want a PDF. Also, any / all
of these could be selected.

Finally, the file can be delivered either back to the phone, via email, via
FTP, HTTP to another web server or any other common communications
technology. (Again, any / all options could be selected.)

I also have to make the application work on mulitple servers, though that
probably won't be such an issue unless I have a common fileserver to store
the incoming files and the filewatcher service on two servers both attempt
to collect the same file.

Ideas on how I should approach this would be very much appreciated.


--
Best regards,
Dave Colliver.
http://www.AshfieldFOCUS.com
~~
http://www.FOCUSPortals.com - Local franchises available

 

[sloan]

This is a deprecated application block, but you might draw some ideas from
it:

http://msdn.microsoft.com/en-us/library/ms998466.aspx


I would create WCF code, and expose it as a WebService.
http://sholliday.spaces.live.com/Blog/cns!A68482B9628A842A!158.entry
This would give you greater flexibility in the future.....instead of hard
coding up a WebService now.

.......

//> Finally, the file can be delivered either back to the phone, via email,
via

FTP, HTTP to another web server or any other common communications
technology. (Again, any / all options could be selected.)//

Do you have to send them the file? Or can you send them a link (http url)
to the file?

We have a scenario for reports. The person would request a report.
Sometimes it took 20 minutes to create it.
We'd send them a link to the (final output) as a URL.

I would also use the "double guid" system for filenames.

string directoryName = Guid.NewGuid().ToString("N");
string fileName = Guid.NewGuid().ToString("N");
string ext = ".xml";

string physicalFullName = "C:\inetpub\wwwroot\" + directoryName + "\" +
fileName + ext);
string urlName = "http://mymachine/" + directoryName + fileName + ext;

File.Save (physicalFullName ); // pseudo code

Email.Send ("You have a file waiting for you at: " + urlName );// pseudo
code

 

[David]

Hi,

Thanks Sloan, Peter and Patrice. Definately food for thought.

Patrice, can you expand on what you mean?


The system has to work with a plugin architecture and be scalable. For
example, it will also be database driven and I need to be able to use
different databases. This I can easily achieve by using a layered approach,
something I haven't mentioned yet though is that it has to be able to work
from a clustered perspective as well (such as a clustered DB server or
clustered web server etc.) However, I don't think that affects this part of
the structure.


--
Best regards,
Dave Colliver.
http://www.AshfieldFOCUS.com
~~
http://www.FOCUSPortals.com - Local franchises available

 

[wdudek]

This could add a whole new level of complication to your system, but we
handled a similiar priority requirement using a messaging system. In out case
we used SQL Server's Service Broker, but I'm sure what we did could be
replicated with anyone. We have 2 queues, one is a low priority queue and has
one reader (think thread) at the end of it. The other is a high priority
queue and has several readers at the end so that it can process more data.
The application must determine which messages are high and low priority, but
after that most of the multi threading aspect is handled by the messaging
system. This also allows us to dynamically increase out processing
capabilities when the need arises by increasing the number of readers on the
queue. Another benefit of this is that the messages are garaunteed to be
delivered once they are in the queue.
On the other hand in an environment that does not already use messaging
this introduces allot of new pieces that can and will break and will need to
be monitored.

Bill

[...]
Once the file has been saved, I need to then work on it. The process can
become quite intensive. Also, each file can come from a different
customer
and each customer will have a 'priority' preference, for example, some
files
have to be acted upon immediately (due to them being time critical) where
others that are not so critical can wait.

My thoughts to handle this is a windows service with a filewatcher to
watch
the webservice receiving folder. Then, depending on the file (customer),
react to it...

I guess I should be doing threading here (I have not really done any
threading before...) but I do have to bear in mind priority files. Also,
if
there are for example, 3 files from one phone, they have to be handled
chronologically.

[...]
Ideas on how I should approach this would be very much appreciated.

The question is very broad, and this is probably not the best forum for a
deep discussion about your options.

That said, some thoughts about what you've posted:

-- Don't use FileSystemWatcher unless you really have to. It's a
useful class in certain scenarios, but it isn't 100% reliable. If file
system changes happen too rapidly for your own code to keep up,
notifications may be discarded. It's extra work to make it reliable --
basically you wind up having to do your own scans to double-check that you
haven't missed anything -- and if you do that, then the apparent
simplicity of using FileSystemWatcher winds up negated.

Since you appear to have control over the code that's receiving the files,
you will be much better off just managing your own queue of work based on
what files you know you've received, rather than relying on the file
system as your mediator between code modules.

-- The prioritization feature has the potential for really causing
problems. It's not really clear from your description what prioritization
would be used, but hopefully it's something relatively simple. If it's
okay for starvation to occur, then a simple ordered queue is probably a
good bet. One thing you're vague about is whether priorities are for
manging files for a specific user only, or whether users themselves have
individual priorities that affect which user's files get processed first.

A possible fallback to you implementing all the prioritization logic
yourself would be to dedicate individual threads to each priority class
you have, setting the thread priorities appropriately. I almost always
recommend _against_ setting thread priorities, but in this case, as long
as you can categorize your priorities in a reasonably few number of
groups, it might actually work in your favor to set the thread priorities
and then let Windows do the heavy lifting for you. Its thread scheduler
has lots of little details in it to ensure that prioritization happens
correctly without starving work and while maintaining a consistent flow.

Other than that, I'd say that without a much more detailed description of
your architecture,it would be difficult or impossible to offer any really
useful design advice.

Pete

 

[David]

Thanks Patrice,

I have never done a queue based system (and also done very little in the way
of generating events, which leads me onto a new thread that I will ask in a
short while.).

Would you be able to do a quick mock up so that I can understand the
principle (or point me in the direction)?

Thanks.

--
Best regards,
Dave Colliver.
http://www.AshfieldFOCUS.com
~~
http://www.FOCUSPortals.com - Local franchises available

 

[David]

Cool. That's going into my favourites...

Cheers Patrice.

--
Best regards,
Dave Colliver.
http://www.AshfieldFOCUS.com
~~
http://www.FOCUSPortals.com - Local franchises available

 

[David]

Hi Patrice,

Gone through the whole article and written the sample app. Towards the end
of article is a sentence that says to download the WROXTester project, but
no link to it. I have done a google for wroxtester but cannot find it.

Any ideas on the other side of this story? It looks like the app I have
written from this site is the receiving side of the queue. I also need to
see the sending side.

This queueing system could be the answer to a lot of the design questions...
it sounds like we can use this to help us scale the app, say for example, we
find one of our services is getting bogged down (being process intensive),
we can quite quickly add in another server to handle some of the processing.
This is quite cool.
--
Best regards,
Dave Colliver.
http://www.AshfieldFOCUS.com
~~
http://www.FOCUSPortals.com - Local franchises available