How to Make a .NET Timer Always Fire at the Right Time

[Charles]

This is a follow up to an earlier post, about a Threading.Timer that
occasionally fired at odd times. In that case I discovered that low memory
meant that the machine 'froze' intermittently and a timer callback could
fire after 30 seconds instead of every 10 seconds as intended.

I now find that if the machine becomes preoccupied with another task, I get
the same effect. This is a very bad state of affairs, as I can no longer
rely on my 10 second tick occurring every 10 seconds.

I need to have a reliable 10 second timer, such that an event happens every
10 seconds. It's no good if I get two events after 20 seconds, I need one
every 10 seconds.

How is this possible to guarantee in .NET? The app is running on Windows
Server 2003 x64.

TIA

Charles

 

[Family Tree Mike]

This is a follow up to an earlier post, about a Threading.Timer that
occasionally fired at odd times. In that case I discovered that low
memory meant that the machine 'froze' intermittently and a timer
callback could fire after 30 seconds instead of every 10 seconds as
intended.

I now find that if the machine becomes preoccupied with another task, I
get the same effect. This is a very bad state of affairs, as I can no
longer rely on my 10 second tick occurring every 10 seconds.

I need to have a reliable 10 second timer, such that an event happens
every 10 seconds. It's no good if I get two events after 20 seconds, I
need one every 10 seconds.

How is this possible to guarantee in .NET? The app is running on Windows
Server 2003 x64.

TIA

Charles

If the task is that critical that it _must_ occur every 10 seconds, then
I would dedicate a machine to that task. If that task is it's job, then
it does nothing else.

It is impossible for me to know your situation, but is there a way to
"throttle" the other process that is grabbing the CPU attention? That
may be an option to look into.

 

[Patrice]

Some more details could help but this is possible only for a real time OS
(http://en.wikipedia.org/wiki/Real-time_operating_system)... Also a common
misuse is to to use a timer to measure the time, if applicable you could get
this timer fire with a possible margin and then get the system time date to
get the actual time you are called.

We can't say much more wihtout knwoing what is the critical task that needs
to be done...

 

[Jeroen Mostert]

This is a follow up to an earlier post, about a Threading.Timer that
occasionally fired at odd times. In that case I discovered that low
memory meant that the machine 'froze' intermittently and a timer
callback could fire after 30 seconds instead of every 10 seconds as
intended.

I now find that if the machine becomes preoccupied with another task, I
get the same effect. This is a very bad state of affairs, as I can no
longer rely on my 10 second tick occurring every 10 seconds.

I need to have a reliable 10 second timer, such that an event happens
every 10 seconds. It's no good if I get two events after 20 seconds, I
need one every 10 seconds.

How is this possible to guarantee in .NET? The app is running on Windows
Server 2003 x64.

You can't really do this with managed code only, because all managed timing
mechanisms that aren't WM_TIMER rely on a hand-rolled thread pool which is
very much subject to preemption.

Technically speaking, as others have pointed out, you can't do it in
unmanaged code either, as it requires a real-time OS to do tasks in hard
real-time. That said, unmanaged code *can* get much better accuracy in fixed
timing, because you can use the OS mechanisms. But there is still no cure
for a system that's completely locked up with I/O -- you can't just steal
time from the interrupt handlers.

Doing something every 10 seconds doesn't require high resolution, so an
option is to use a thread running at high or even realtime priority that
sleeps for 500 ms and checks how much time has passed every time it wakes
up. This is still subject to preemption, so how well it works depends.

Another option is to P/Invoke to CreateWaitableTimer() or
CreateTimerQueueTimer() using one of the the WT_EXECUTEIN* flags. I have no
idea how well this holds up when the system is busy, but it ought to do
better than .NET's own thread pool. Getting the interop right can be tricky,
though.

Many people still use the multimedia timer functions (timeSetEvent() and the
like) despite these being nominally obsoleted, as they still offer higher
accuracy -- but with a 10 second period you're not likely to need them.

 

[Charles]

Hi Mike

Thanks for the quick reply. It's not critical that it is exactly 10 seconds.
12 seconds would do, or even 15 seconds occasionally. But what I can't have
is it being 30 seconds between timer events, and that is the worst cases I
have seen. I have to send a message over TCP/IP about every 10 seconds in
order to keep a connection open, otherwise the other end shuts up shop and
goes away.

Charles

 

[Charles]

Hi Patrice

I have just replied to Mike about what I am doing, but essentially I am
sending a heartbeat over TCP/IP and if it doesn't get sent in time then the
other end thinks I've died and it abandons the connection.

Charles

 

[Charles]

Hi Jeroen

I understand better now from your explanation, thanks. I might give the
P/Invoke method a try as I've used this before for other comms projects, so
I'm reasonably familiar with getting it working. I presume if I switched
from a Threading timer to a System.Timers.Timer I would be no better off?

Cheers

Charles

 

[Patrice]

I have just replied to Mike about what I am doing, but essentially I am

sending a heartbeat over TCP/IP and if it doesn't get sent in time then
the other end thinks I've died and it abandons the connection.

I'm not a TCP expert but it really looks like the "TCP keepalive" feature.
Have you tried http://msdn.microsoft.com/en-us/library/e160993d.aspx with
the keepalive option ? Also it looks really low ? Is this is third party
application server side ?

 

[Charles]

Yes, it's third-party, so I have no control over it, unfortunately.

Charles

 

[Jeroen Mostert]

I presume if I switched from a Threading timer to a System.Timers.Timer I
would be no better off?

You presume correctly. System.Timers.Timer is in fact a wrapper around
System.Threading.Timer with a slightly more conventional API. Except for
System.Windows.Forms timer, which is based on SetTimer() and WM_TIMER, all
managed timers rely on the managed thread pool, which is almost but not
quite like the native OS thread pool.

 

[Jeroen Mostert]

I'm not a TCP expert but it really looks like the "TCP keepalive"
feature.

I'm not a TCP expert either, but I do know TCP keepalives are usually more
trouble than they're worth. You have to change the default keepalive time
(by default it's two hours, which is useless), and even then all the
mechanism does is send a number of probes with 1-second intervals to probe
the other side for connectivity. The packets sent this way are just 0-byte
data packets, which the other side will not necessarily respond correctly to
("responding" in this case is just receiving on the socket and not crashing
on the zero-byte packet; the network layer will take care of the ACK).

It's almost always a better idea to implement an explicit keepalive
mechanism in whatever protocol you're using than to rely on the TCP
keepalive. That's assuming your protocol actually needs keepalives, as many
people forget that TCP's ability to keep a connection open without traffic
is a feature, not a bug.

 

[Charles]

I've been wondering how I might increase the priority of the the thread that
the timer runs on, but if it uses the thread pool then I don't imagine I can
increase the priority? If that's the case, then perhaps that's another
reason for going for a waitable timer, as the callback runs on the same
thread on which the timer was created and I could boost the priority of that
thread. Does that sound reasonable?

Charles

 

[Patrice]

That's assuming your protocol actually needs keepalives, as many people

forget that TCP's ability to keep a connection open without traffic is a
feature, not a bug.

It begins to be OT respective with the OP initial demand by may I ask what
you think about the connection being closed after 10 s ? Is this something
usual ?

TIA

 

[Jeroen Mostert]

It begins to be OT respective with the OP initial demand by may I ask
what you think about the connection being closed after 10 s ? Is this
something usual ?

I have worked with setups where the server side closed the connection after
just 20 seconds of inactivity, because it was more important for the
connection to be "known good" even at short intervals than it was to be
mindful of the bandwidth wasted this way.

I'm not saying this is a good way of doing things (using short transmission
timeouts is probably a better idea), but I don't think it's unusual either.

 

[Jeroen Mostert]

I've been wondering how I might increase the priority of the the thread
that the timer runs on, but if it uses the thread pool then I don't
imagine I can increase the priority?

No, you can't. Well, to be precise, you can, but there's no thread affinity
for the timer, so you'll just be boosting the priority of some random thread
that's handling the timer callback.

I don't recall if it's possible for the thread that sets the timer to get
the callback if the interval is sufficiently short, but in any case that
won't apply here since your interval is in the range of seconds.

If that's the case, then perhaps that's another reason for going for a
waitable timer, as the callback runs on the same thread on which the
timer was created and I could boost the priority of that thread. Does
that sound reasonable?

Yes, in principle. There's one caveat with waitable timers, though: they use
APCs to signal completion. This is slightly tricky because the CLR also uses
APCs to communicate with threads (Thread.Interrupt() and Thread.Abort(),
among others, queue an APC on the thread). I recall having some problems
using APCs in managed threads, but nothing specific; it's probably the
general trouble of having to be careful in callbacks from unmanaged code,
especially something as low-level as APCs. I do recall you can use

try {
Thread.Sleep(Timeout.Infinite);
} catch (InterruptedException) {
return; // asked to stop
}

And then call Thread.Interrupt() when your thread should stop.
Thread.Sleep() uses SleepEx() under the covers and will properly respond to
the CLR signaling APCs.

I'd try timer queues first. By default these just use worker threads, so no
mucking around with APCs. They don't allow for boosting priority but they
don't use the .NET thread pool either, so they might be less susceptible to
preemption under load. I have no idea if this is actually the case, but
you'll have to thoroughly test whatever solution you pick anyway, so you
might as well start with the easier ones.

Speaking of which, you really are better off dedicating a machine that's
guaranteed to have low loads to this task, like Mike suggested. If you
can isolate the processes that are ruining it for the rest, you could try
looking into job objects to limit the memory and CPU they're assigned.
Keeping your program simple and not overly reliant on interesting low-level
techniques is certainly worth some effort.

 

[Jesse Houwing]

* Charles wrote, On 1-11-2009 17:59:

Hi Mike

Thanks for the quick reply. It's not critical that it is exactly 10
seconds. 12 seconds would do, or even 15 seconds occasionally. But what
I can't have is it being 30 seconds between timer events, and that is
the worst cases I have seen. I have to send a message over TCP/IP about
every 10 seconds in order to keep a connection open, otherwise the other
end shuts up shop and goes away.

Isn't it possible to make that connection more reliable (by leaving it
open longer), or wouldn't it just be possible to switch to UDP instead
and forget the notion of a connection altogether?

Jesse

 

[Charles]

Hi Jesse

Because it's third-party, I'm at their mercy. This is just the protocol I
have to accommodate.

Charles

 

[James Hahn]

If your timer can't respond within, say 5s of the nominated interval, then
surely that indicates that the machine is not responsive, and that is
exactly why the server needs that acknowledgement. If you manage to force
the keepalive even when the machine is bogged down like this, surely you
will risk corrupting the connection because the server is being fooled into
thinking that the connection is active when it isn't.

 

[Charles]

Hi James

Actually, you have a point there. I hadn't thought of it like that. I should
perhaps put more effort into re-establishing the connection when the machine
unfreezes.

On the other side though, the connection is important enough to keep going
at all costs, so if I could make my task higher priority then it would be ok
if other processes suffered slightly.

Charles

 

[Charles]

It's not really practical to dedicate a machine to this task, and in any
case, if I could assign an appropriate level of priority to the task then
any other tasks would get plenty of time to operate.

The thing that currently seems to upset it is file transfers over RDP. I
occasionally need to copy a file to the machine over RDP, which can take
anything from 1 minute to 30 minutes. There are times in this window when
the machine appears to hang, but I imagine it is just concentrating really
hard on the file transfer. Anyway, that's when my timer goes astray. Giving
a regular 10 second slot to my task would not hinder the file transfer in
any way, and even if it did, I could live with it. It's just a case of
priorities, as it were.

Charles