Time Critical Process in .NET

[Charles Law]

I am wading through it all now

There are about 30 classes that can raise events, each with their own custom
EventArgs class. At the moment I use New each time I raise an event. I
should add that each task in my critical process does not use 30 classes.
Each task uses just one of any of these 30 class objects. The class objects
are pre-loaded, and are not therefore created on-the-fly.

Incidentally, I used PerfMon to look at what was happening with the GC, and
when I added a GC.Collect into one of my tasks, I saw the % time in GC value
increase. Weird, but not entirely unexpected?

Cheers.

Charles

 

[Cor Ligthert]

Charles,

Sometimes starting completely new (for the problem part of course) will give
a huge amount of time wining. The only problem is that you can only say that
afterwards when you did not do that.

Just a thought in with I wanted you to deal.

Cor

 

[Jon Skeet [C# MVP]]

From the MSDN documentation for *Control.BeginInvoke*
"Note The BeginInvoke method calls the specified delegate back on a
different thread pool thread. You should not block a thread pool thread for
any length of time."

Yes, that's just a bug in the documentation though. I reported it to MS
a while ago - hopefully it'll be fixed by the time .NET 2.0 ships.

I suspect what happens (indeed it seems to me that it HAS to happen this
way) behind the scenes is that WorkerThread calls BeginInvoke, BeginInvoke
spawns thread to queue the delegate call, Execution returns to WorkerThread,
The New thread "queues" the call that will eventually occur on the UI thread.
Makes sense to me. ;-)

Unlikely, IMO. I suspect it's more likely that it uses a Win32
PostMessage or something similar. I don't see any reason to get an
extra thread involved.

 

[Jay B. Harlow [MVP - Outlook]]

CMM,

From the MSDN documentation for *Control.BeginInvoke*
"Note The BeginInvoke method calls the specified delegate back on a
different thread pool thread. You should not block a thread pool thread
for
any length of time."

IMHO that sounds like a serious documentation error! I will check with MS to
make sure. I would expect the paragraph to read something like: "Note The
BeginInvoke method calls the specified delegate back on the control's UI
thread. You should not block the control's UI thread for any length of
time."

The first paragraph is the important one. "Executes the specified delegate
asynchronously on the thread that the control's underlying handle was
created on."

Think very carefully about it. If the purpose of Control.BeginInvoke is to
"execute on the thread that the control was created on", why would a third
thread from the thread pool be involved? Ergo the statement you quoted is
highly suspect and I will check with MS on it!

I suspect what happens (indeed it seems to me that it HAS to happen this
way) behind the scenes is that WorkerThread calls BeginInvoke, BeginInvoke
spawns thread to queue the delegate call, Execution returns to
WorkerThread,
The New thread "queues" the call that will eventually occur on the UI
thread.
Makes sense to me. ;-)

Sounds like you are double queuing the request, which IMHO is one queuing
too many. In other words it makes more sense when the worker thread calls
BeginInvoke, BeginInvoke simply queues the request for the main thread to
execute it. Rather then have the worker thread, queue a request for the
thread pool, then the thread pool simply queues the request for the UI
thread.

I don't have the Rotor source loaded, however using ILDASM & Rotor,
Control.BeginInvoke calls Control.MarshaledInvoke, which registers a private
windows message and calls the win32 GetWindowThreadProcessId & PostMessage
APIs. I don't see any reference to the Thread Pool in the implementation of
Control.BeginInvoke...

I will post something to MS about the documentation...

Hope this helps
Jay

From the MSDN documentation for *Control.BeginInvoke*
"Note The BeginInvoke method calls the specified delegate back on a
different thread pool thread. You should not block a thread pool thread
for
any length of time."

I suspect what happens (indeed it seems to me that it HAS to happen this
way) behind the scenes is that WorkerThread calls BeginInvoke, BeginInvoke
spawns thread to queue the delegate call, Execution returns to
WorkerThread,
The New thread "queues" the call that will eventually occur on the UI
thread.
Makes sense to me. ;-)

Having said that, I agree with your 80/20 characterization. It seems to me
that the chances of his bottleneck being the event model is low compared
to
other possibilities.

<<snip>>

 

[Jay B. Harlow [MVP - Outlook]]

Jon,

Yes, that's just a bug in the documentation though. I reported it to MS
a while ago - hopefully it'll be fixed by the time .NET 2.0 ships.

I was just noticed that .NET 2.0 beta docs currently contains the same bug.

http://msdn2.microsoft.com/library/0b1bf3y3.aspx

If you already submitted a bug report on it, I don't think I need to...

Thanks
Jay

 

[Jay B. Harlow [MVP - Outlook]]

Charles,

There are about 30 classes that can raise events, each with their own
custom

During the critical thread how many custom EventArgs & other objects are you
creating?

Incidentally, I used PerfMon to look at what was happening with the GC,
and when I added a GC.Collect into one of my tasks, I saw the % time in GC
value increase. Weird, but not entirely unexpected?

Yes calling GC.Collect will cause the GC to behave "badly", for details see
Rule #1 at:

You may want to use CLR Profiler instead of PerfMon to get a better feel of
what is happening with your objects during the critical thread.
Unfortunately I have not used CLR Profiler enough to get a good idea of how
to use it only for the critical thread section of your code...

Hope this helps
Jay

 

[Guest]

Post any responses from MS to the group. Interesting topic. G'luck!

 

[Charles Law]

I probably create about five custom EventArgs for each task. Although there
is only one directly per task, the associated processes generate several
more. That said, there are no handlers sinking the events raised by these
associated processes, so the custom EventArgs will not, in most cases, be
created.

I'll have a look at the CLR profiler, and see if I can get it to do what
PerfMon was doing.

Charles

 

[Guest]

Think very carefully about it. If the purpose of Control.BeginInvoke is to

"execute on the thread that the control was created on", why would a third
thread from the thread pool be involved?

To return execution to the caller of as quickly as possible. If BeginInvoke
were truly asynchronous, why should the caller have to wait around for the
method to do whatever work it has to do like prepare and post a message via
the message pump (or whatever it does to queue the control call) and then
finally return execution to the caller. Although... perhaps it is more
expensive to pick up a new thread for this purpose. IMO, picking up threads
from the threadpool isn't as fast as it should be (sometimes taking a good
500ms or more). I'm not saying I'm correct about BeginInvoke... just thinking
out loud about the possibilities.

 

[Jay B. Harlow [MVP - Outlook]]

CMM,

To return execution to the caller of as quickly as possible. Huh?

If BeginInvoke
were truly asynchronous,

Control.BeginInvoke is truly asynchronous! As is Delegate.BeginInvoke. Its
simply that they are implemented is different.

why should the caller have to wait around for the
method to do whatever work it has to do like prepare

Because the delegate & parameters that are needed needs to be "copied" from
the worker thread to the UI thread. I hope you agree that even
Delegate.BeginInvoke needs to "copy" the delegate & parameters from the
worker thread to the thread pool thread. This "copying" can be called
marshalling . In both cases the delegate & parameters need to be marshaled
from one thread to the other.

NOTE: Unless you are crossing an app domain, the marshalling here only needs
to save a reference to the delegate & parameters to be sent so the other
thread can use them. When you cross an app domain the objects may need to be
serialized & deserialized on the other side...

and post a message via
the message pump

The message pump itself runs on the UI thread, it is the Application.Run
call that is either directly or indirectly called when you start your
application. I would expect Thread Pool threads to have something very
similar to a message pump, however rather being based on Win32 message
queues (PostMessage, GetMessage apis) that is uses a System.Collection.Queue
object.

Hope this helps
Jay

 

[Guest]

Charles,

Before you start changing your messaging architecture, I would be interested
in what you saw in Performance Monitor (discussed earlier). I have
definitely seen some latency in the past that I attributed to garbage
collection in .NET, although not in the magnitude you describe. Just how
much data are you allocating / de-allocating in your computation process?

Also, does your thread catch exceptions on a regular basis? I have seen
some very noticeable execution lag I had attributed to catching exceptions,
but I was working on a simple UI at the time and did not investigate it.

Finally, I have not used an RTF control in .NET yet, does it work by
assigning a string? I.e.:

rtfThing.Text += sNewLine;

If so, and you have a lot of updates, this would certainly be a likely
culprit. The .NET string type is immutable, so the entire previous value of
the string is copied to the new string every time you make a change.

Hope this helps... keep us updated.

 

[Charles Law]

Hi John

Thanks for the thoughts. I won't get chance to run PerfMon again until
Thursday (I have to go on site to do it), but I'll come back then. You have
got me thinking about the string thing though. The rich text box has an
Append type of method, but I don't know what it does internally. It might be
quicker to maintain the string outside the control with a string builder and
write it to the control fresh every time. The problem with that though is
the colouring of parts of the text. Hmm ... I'll have to give this some more
thought.

Cheers.

Charles

 

[Charles Law]

I can't remember where I got up to with this thread, so this is a general
update on the problem.

I ran the application again this morning, against the real equipment. The
time critical parts of the process now seem to be more stable, after
implementing some of the suggestions, but at the severe detriment to the UI.
As mentioned before, the background process raises events when it wants to
notify the UI that something has happened, and the event handler marshals
the event onto the UI thread, using a delegate and BeginInvoke, and appends
the message to a rich text box.

The problem is now that the screen appears to be updating at a reasonable
pace, but it is clearly lagging a long way behind the messages being
supplied to it. At its worst, the background process finished, and it took
over five minutes for the screen to catch up. FIVE MINUTES! I think I have
got my pattern wrong here. I don't mind the screen lagging slightly behind,
but five minutes is just silly.

Going back to basics, I want to (need to) capture the messages passed in all
the events that the background thread raises, and I want to maintain a
display of these messages. Some messages will be coloured red to make them
stand out to the user, which is why I have used a rich text box.

Is there a better pattern for this type of application?

Incidentally, I ran this on a 3.0 GHz P4 Dell laptop, not the slower
Celeron m/c, so it will be much worse on the intended target m/c.

Charles

 

[Jay B. Harlow [MVP - Outlook]]

Charles,
What was the extent of your changes? What exactly were your changes? Can you
apply an individual change at a time to isolate which one is causing the
problem.

Can you use CLR Profiler or another profiler to isolate *exactly* where the
slow down is? As other wise we are all just guessing as to where the problem
lies!

I saw a question the other day that I though might apply here. Is the slow
down occurring on the first Exception that occurs in your app? Remember for
apps that run under the debugger, the first exception usually takes about 2
seconds...

Is the JIT compiler loading & compiling code that causes the slow down?

want to maintain a display of these messages. Some messages will be
coloured red to make them stand out to the user, which is why I have used
a rich text box.

Have you considered an Owner Draw ListBox, a ListView or other control, that
is a little more "light weight" then a RichTextBox?

For example ListViewItem has a ForeColor property on it...

Hope this helps
Jay

 

[Charles Law]

Hi Jay

One of the changes I made was to stop exceptions being raised for events
that could occur as a normal part of the background process. I was throwing
an exception to indicate a process step failure, whereas I now always return
a Result class that indicates the state. That said, these failures did not
occur very often and, all being well, won't happen at all, but it is now
neater this way.

I also raised the priority of the worker thread to Highest.

I haven't got to grips with the CLR Profiler yet; perhaps I should
Intuitively, I feel that the problem is that the worker thread is now taking
all the time it needs, and the UI thread is getting only a small amount of
time. If the worker thread is performing its tasks at full tilt it will
raise an event every 50 ms or so, with longer gaps here and there. I just
don't think that the UI can keep up. Messages appear on-screen at a goodly
rate, but as the rich text box gets fuller it appears to get slower. When
the whole process was over I saved the contents of the rich text box and
opened it in word. The file is 2 Mb long, and converts to 319 A4 pages.

As far as applying a change at a time, I am somewhat hampered by access to
equipment, and although I can test fairly well off-site, I don't get the
same problems because I can only execute a small number of the processes
properly.

Have you considered an Owner Draw ListBox, a ListView or other control,
that is a little more "light weight" then a RichTextBox?

I did think about this but thought that it would actually be slower. I
thought that as the rich text box is free text it should be quicker to tack
stuff on the end, whereas I imagine that the listview will create an object
for each line and would therefore take longer and be more memory hungry. Now
you suggest it though I think I will try it. In fact, if it is quicker it
will probably make the management and navigation of the information easier.

Charles

 

[Cor Ligthert]

Charles,

I still opt for the queue sample I gave you. Don't think about that as an
English queue.

In this case you put from every thread what is received as soon as possible.
Your mainthread is getting it out in the tempo he can process it.

I really think that this can make your processes more indipendent from
delays (although I agree that it never can solve that you cannot ring the
Big Ben 5 minutes over the hour).

There was as well an option for a stack in this thread, however the only
difference between a stack and a queue is that with a stack it goes last in
first out while the queue goes first in first out. (Still a little bit
English)

I think I wrote all the time queue well (it stays a terrible word)



Cor

 

[Charles Law]

Hi Cor

The reason I haven't tried the queue idea is that it seems to me to be, in
essence, what is happening when I use BeginInvoke.

With BeginInvoke, isn't it the case that the update gets queued until the UI
has time to put the message on the screen? In my scenario, the UI doesn't
get the time to do it very often, and when it does it is not quick enough.

Charles

 

[Cor Ligthert]

Charles,

Are we talking about the same, a queue is a datastore in memory, that can be
feed from thread a,b,c,d,e,f, and read by thread y where it deletes every
time one from the bottom.

Therefore makes in my opinion a very smooth process possible. I have used
this forever long ago in past when that had other names. (It was nice to
make asynchronous database and datacom controllers, now named multi tier
applications).

Cor

 

[Charles Law]

I think we are talking about the same thing. My doubt is whether it will be
an improvement on using BeginInvoke.

Charles

 

[Jay B. Harlow [MVP - Outlook]]

Charles,
As I mentioned before Control.BeginInvoke uses the Win32 PostMessage API.
(you can use ILDASM or Reflector to verify)

The Win32 PostMessage places messages in the Win32 Message Queue.
http://msdn.microsoft.com/library/d...agesandmessagequeuesfunctions/postmessage.asp


Rolling your own queuing verses BeginInvoke may or may not help. Before I
attempted rolling my own as Cor suggests. I would want to be certain that is
the aspect of the entire process that is hurting performance. The 80/20
rule!

Hope this helps
Jay