Drastic slow-down in the execution time of my program

[Marcelo R]

I am not sure this is the best place to ask this; suggestions of alternative
forums will be greatly appreciated.

I am at my wits' end trying to understand a problem that I have with my
software and/or hardware and/or Windows. My company's software runs on an
industrial PC with a framegrabber board and some other hardware. The OS is
Windows XP SP2 and the compiler is Visual C++ 6.0. We noticed that sometimes
the software is MUCH slower than normal. Instead of acquiring and processing
20 video frames per second, it can take more than one second per frame. If I
exit the program and start it again, everything is normal. The program can
run for hours without any problem. If I exit and restart it again, it might
right away run very slowly.

My initial suspicion was that there was something wrong with the
framegrabber board or drivers. That is definitely not the case though, as I
wrote a test program that doesn't use the framegrabber at all and the
slowdown still happens. My test program is fairly simple: it runs 2000
iterations of a complex calculation and it measures the time (using the high
resolution timer) that each iteration takes. At the end it saves to a file
the total time that it took and the minimum and maximum time that each
iteration took. I call the program from inside a loop in a batch file. The
results are very strange. The program will run dozens of times with very
consistent results (for example total time 29 seconds, shortest iteration 14
ms, longest 17 ms) and then suddenly it will slow down. One run will total
640 seconds, with a minimum and maximum of 320 and 322 ms, then the next will
take nearly half an hour, with each iteration taking between 839 and 841 ms.
I would understand if some iterations within each run would take much longer
- that would be an indication that Windows is busy doing something else. But
that's not what's happening. Either the program starts correctly, in which
case all iterations will run in around 15 ms (that seems to be true
regardless of the number of iterations, two thousand or ten million), or
right from the first iteration every single one will take much longer. And
the minimum and maximum execution time for each iteration are always close.
The speed is either constantly normal, constantly slow or constantly VERY
slow.

To make matters more mysterious, the same thing does NOT happen when I run
the exact same batch file on my desktop or on my laptop or on a few other
different computers. On the industrial computers that my company uses,
however, it happens very consistently. I have tried running it on one of
these computers after doing a fresh Windows XP install, before installing any
other drivers and without connecting to the internet even once (so definitely
no viruses there). Same thing.

Any ideas or suggestions of how to go about troubleshooting this thing? I
have been working with C for twenty years, but I am really not all that
familiar with Windows' innards.

 

[Carl Daniel [VC++ MVP]]

I am not sure this is the best place to ask this; suggestions of
alternative forums will be greatly appreciated.

you might try microsoft.public.wi32.programmer.kernel. It's not quite
on-topic there either, but it's a good audience.

[detailed description elided]

Any ideas or suggestions of how to go about troubleshooting this
thing? I have been working with C for twenty years, but I am really
not all that familiar with Windows' innards.

You've definitely got a head-scratcher there, and it sounds like you've
already ruled out most of the usual suspects.

A couple of ideas: What if you make your test program do nothing at all -
does it still sometimes take a long time to run?
Can you make your test program write out incremental progress so that you
can see where the slowdown is occurring (i.e. all at the start, evenly
spread, etc). You said you've tried this on a virgin install of XPSP2 with
no divers - is the frame grabber hardware already present though? Any other
hardware devices beyond disk, memory and human interface that you can remove
from the system for testing?

Misbehaving network hardware can sometimes cause all sorts of odd pauses, as
can misbehaving IDE/ATA interfaced storage devices. I couldn't tell for
sure from your original post, but I'm assuming that you've reproduced this
behavior on several of your industrial PCs while being unable to reproduce
it anywhere else, yes?

-cd

 

[Marcelo R]

Thanks Carl.

Can you make your test program write out incremental progress so that you
can see where the slowdown is occurring (i.e. all at the start, evenly
spread, etc).

It's evenly spread. Remember how I said that I run the program from inside a
loop in a batch file, and that each time the program runs it does 2000
iterations of a complex calculation? I can have 40 consecutive runs where
every single iteration takes between 14 and 17 ms, and then on the next run
every iteration will take between 206 and 208 ms - more than ten times
slower. Then the next run is back to normal.

You said you've tried this on a virgin install of XPSP2 with
no divers - is the frame grabber hardware already present though? Any other
hardware devices beyond disk, memory and human interface that you can remove
from the system for testing?

I removed the frame grabber. The motherboard has an integrated network
connection but there is nothing hooked up to it. There is no other hardware
to be removed.

Misbehaving network hardware can sometimes cause all sorts of odd pauses, as
can misbehaving IDE/ATA interfaced storage devices. I couldn't tell for
sure from your original post, but I'm assuming that you've reproduced this
behavior on several of your industrial PCs while being unable to reproduce
it anywhere else, yes?

That is correct. I have seen the same thing on every one of the industrial
PCs where I tried it, and not on any of the half a dozen desktops and laptops
where I tried it.

I have written to the manufacturer of the industrial PC and I am waiting for
an answer.

One more thing that adds to the mystery: a colleague suggested that Windows
was slowing down the clock rate for some reason. So I ran two programs
simultaneously: one from a loop in a batch file and the other in a single
execution that would take hours to complete. At the times when the first one
was running very slowly, the second one was still running normally. So it's
not the whole CPU that slows down.

 

[Giovanni Dicanio]

the compiler is Visual C++ 6.0.
[...]
I
wrote a test program that doesn't use the framegrabber at all and the
slowdown still happens.
[...]
Any ideas or suggestions of how to go about troubleshooting this thing?

In additon to Carl's detailed answer, I would like to suggest you to rebuild
your test code with a more modern compiler, like VS2005 or 2008.
You can find the Express Edition as freely available download.

I think that more modern C/C++ compilers can produce higher-quality binary
code.

Giovanni

 

[Carl Daniel [VC++ MVP]]

Thanks Carl.


It's evenly spread. Remember how I said that I run the program from
inside a loop in a batch file, and that each time the program runs it
does 2000 iterations of a complex calculation? I can have 40
consecutive runs where every single iteration takes between 14 and 17
ms, and then on the next run every iteration will take between 206
and 208 ms - more than ten times slower. Then the next run is back to
normal.

OK - Yes, you did describe that clearly, I didn't read clearly enough! (I
wasn't sure if you were measruing total program invocation time or the
individual calculation time).

Next thought: Does the calculation involve memory allocation? Are there
multiple threads involved? I'm wondering if there's some kind of
pathological cache behavior going on that causes the cache hit ratio to
suffer markedly.

That is correct. I have seen the same thing on every one of the
industrial PCs where I tried it, and not on any of the half a dozen
desktops and laptops where I tried it.

I have written to the manufacturer of the industrial PC and I am
waiting for an answer.

One more thing that adds to the mystery: a colleague suggested that
Windows was slowing down the clock rate for some reason. So I ran two
programs simultaneously: one from a loop in a batch file and the
other in a single execution that would take hours to complete. At the
times when the first one was running very slowly, the second one was
still running normally. So it's not the whole CPU that slows down.

That is interesting - in fact, bordering on the inexplicable. I wonder
again if some kind of pathological cache-hit case could explain that - the
slow instance is getting a bad cache hit ratio while the other is doing
alright. It'd be an extraordinary circumstance, but it might fit.

What's the CPU in the industrial PC? Have you tried any other machines with
the same CPU?

-cd

 

[Marcelo R]

Thanks again. The test program is a single-thread console application. At
the beginning it opens a file and reads its contents into an array of 768 K
bytes. The file is closed and then the calculations are done on this array.
No extra memory is allocated.

I think you might be on to something with the cache idea. The computers have
only 512 MB of RAM. I added another 512 MB to one of them and it has been
running without slowing down for a few hours. Still early to say, but maybe
that's a solution.

The CPU is a Pentium 4 2.8GHz. I am not sure any of the other computers that
I tested has the same CPU.

 

[Marcelo R]

Grazie Giovanni. I did recompile with VS2005 and the only difference it made
is that the normal runs are now taking 16 seconds instead of 19 - so a good
performance improvement there - but the slow-down problem still happens
(seemingly) at random.

the compiler is Visual C++ 6.0.
[...]
I
wrote a test program that doesn't use the framegrabber at all and the
slowdown still happens.
[...]
Any ideas or suggestions of how to go about troubleshooting this thing?

In additon to Carl's detailed answer, I would like to suggest you to rebuild
your test code with a more modern compiler, like VS2005 or 2008.
You can find the Express Edition as freely available download.

I think that more modern C/C++ compilers can produce higher-quality binary
code.

Giovanni

 

[Carl Daniel [VC++ MVP]]

Thanks again. The test program is a single-thread console
application. At the beginning it opens a file and reads its contents
into an array of 768 K bytes. The file is closed and then the
calculations are done on this array. No extra memory is allocated.

I think you might be on to something with the cache idea. The
computers have only 512 MB of RAM. I added another 512 MB to one of
them and it has been running without slowing down for a few hours.
Still early to say, but maybe that's a solution.

Interesting. Rather than cache, I'd wonder if you're experiencing some kind
of virtual memory thrashing. Try using Perfmon to monitor the hard page
fault rate (Object: Memory, Counter: Page Reads/Sec). Adding memory is
unlikely to significantly affect cache alignment, but it can absolutely
affect virtual memory layout. For what you describe, 512MB sounds like it
ought to be plenty, but if adding RAM improved it, I'd be highly suspicious
of thrashing.

The CPU is a Pentium 4 2.8GHz. I am not sure any of the other
computers that I tested has the same CPU.

Well, that's an ordinary enough CPU, so it doesn't sound like there's
anything to worry bout there.

-cd

 

[Marcelo R]

Actually adding RAM seems to have reduced the frequency of the problem but
did not eliminate it.

I tried PerfMon and I didn't see a difference in the Page Reads/Sec count
during "normal" and "slow" runs. It was my first time using PerfMon, so I
might not have done it right.

Do you know of any other counter that PerfMon monitors that might give a
clue to what is going on?

Thanks again.

 

[Ben Voigt [C++ MVP]]

Actually adding RAM seems to have reduced the frequency of the problem but
did not eliminate it.

I tried PerfMon and I didn't see a difference in the Page Reads/Sec count
during "normal" and "slow" runs. It was my first time using PerfMon, so I
might not have done it right.

Do you know of any other counter that PerfMon monitors that might give a
clue to what is going on?

I would suggest using an IP sampling profiler that collects call stacks.
That'll find out what part of your code is getting stuck.

Thanks again.



__________ Information from ESET NOD32 Antivirus, version of virus
signature database 4212 (20090703) __________

The message was checked by ESET NOD32 Antivirus.

http://www.eset.com

__________ Information from ESET NOD32 Antivirus, version of virus signature database 4212 (20090703) __________

The message was checked by ESET NOD32 Antivirus.

http://www.eset.com

 

[SvenC]

Hi Marcelo,

Actually adding RAM seems to have reduced the frequency of the problem but
did not eliminate it.

I tried PerfMon and I didn't see a difference in the Page Reads/Sec count
during "normal" and "slow" runs. It was my first time using PerfMon, so I
might not have done it right.

Do you know of any other counter that PerfMon monitors that might give a
clue to what is going on?

For a quick look you can also take Task Manager (press ctrl+shift+esc) and
click on the processes tab. What happens to the memory column, does the
value increase for every run? You migh use View->Select columns to add
the column for virtual memory and look as that as well.

It those values only go up and never down, than you do have a memory
leak. Check your allocation and deallocation calls.