service soemtime will get slower -- what to watch

[Ryan Liu]

Hi,

I use C# wrote an Client/Server application. In production environment, will
be 130 clients (Windows XP) connect to a Server (Windows 2000/2003 Server)
thought TCP/IP socket in a local 100M LAN. "Server" is running as a Windows
service. The is one thread running for one clinet in the server.

Sometime the user tells me the Sever will be slow after it runs for 1-2
days.

Usually what will be the cause? Managed code won't have memory leaking,
right? And how efficient is the garbage collection in .NET? Should I force
garbage collection once a while in my code?

And if I want to use the performance counter to check the problem, what
usually are the objects/parameters I should look at? How many threads are
running for my service? Memory used by the service or each server thread in
my service? Thread idle time ..? I just see too many things in the
performance counter.

And if I write code to sample those performance parameters, will it make the
service runs even slower?

Thanks!

Ryan

 

[Willy Denoyette [MVP]]

Hi,

I use C# wrote an Client/Server application. In production environment,
will be 130 clients (Windows XP) connect to a Server (Windows 2000/2003
Server) thought TCP/IP socket in a local 100M LAN. "Server" is running as
a Windows service. The is one thread running for one clinet in the server.

Keep in mind that doing so, you are wasting a lot of memory resources for
the thread's stack (1MB per thread).Better is to apply an asynchronous
design pattern.

Sometime the user tells me the Sever will be slow after it runs for 1-2
days.

Probably caused by a memory 'leak', which causes higher memory pressure and
resulting in paging after a long period of time.

Usually what will be the cause? Managed code won't have memory leaking,

Managed code can have 'leaks' but that's the result of a 'bug' in your code
or a design flaw, but (usable) memory can also get scarce because of
fragmentation of the GC heap, especially on V1.1 (better move to V2) in
combination with socket based communications using large buffers (taken from
the Large Object Heap).

right? And how efficient is the garbage collection in .NET? Should I force
garbage collection once a while in my code?

No, forcing a GC won't help, what you have to do is measure first...before
you can take some corrective actions.

And if I want to use the performance counter to check the problem, what
usually are the objects/parameters I should look at? How many threads are
running for my service? Memory used by the service or each server thread
in my service? Thread idle time ..? I just see too many things in the
performance counter.

Use Perfmon to check the CLR preformance counters, especially you should
keep an eye on the GC memory counters like the LOH size and the Gen2 size
and the process (your service) Memory counters like the "Private Bytes" and
"Virtual Bytes" should stay stable (within some bounds) after a while.

And if I write code to sample those performance parameters, will it make
the service runs even slower?

There is no need to write something yourself, Perfmon was invented for this.

Willy.

 

[Nicholas Paldino [.NET/C# MVP]]

Ryan,

Well, if you do some sort of monitoring, it will make your server
slower. By how much is impossible to say, depending on the method you use
to monitor the performance of your server. The simple act of monitoring
anything will have the effect of changing what you are monitoring.

I would say that forcing a GC is a bad idea, since it will typically do
a good job of that for you. There are exceptions to this though, but they
are rare. In a server environment, or any environment where you have a
consistent, almost predictable use, it's definitely a bad idea.

As for not having memory leaks, you are right, managed code doesn't have
memory leaks, but that doesn't mean you don't have to pay attention to what
you are using. Anything that implements IDisposable definitely needs
special attention, as you need to make sure you call Dispose (or you should,
at least) on those instances when you are done, as the IDisposable interface
indicates that the instance in question is holding onto something which
should be disposed of when done (as opposed to waiting around for it to be
disposed of, like file handles, sockets, database connections, etc, etc).

Of course, the most important thing to do is really look at what your
server is doing. I know that sounds broad, but you didn't indicate anything
about what your server does, so it's impossible for anyone to comment on
what might be making it slow. Are you executing a massive stored procedure
which processes five billion rows for each of your 100 clients? Are you
writing out 20GB of files for each client at the same time? There are many
things that could indicate performance problems, but you can't even begin to
diagnose them unless you know what the app is doing (or supposed to do) in
the first place.

That being said, can you shed some more light on what your app does?

 

[Ryan Liu]

Nicholas,

Thanks a lot for your help.

Things server does mainly including:

1: Passing very short string messages from one client to another, most
time
it does not parse it, just route it;

2: sync some client operation -- only one client can get an item from
database to work on it; And quota control -- only limited number of one type
of items can be taken out from db for all clients.

3: Client write result directly to the database, but notify server the item
is done.

4: Call database stored procedures that might updates lots of db records
(but just 1-2 time a day ); Parse some XML files (< 1M) once a while.

5: Write log files, one file per day for all clients. Most time I only
write log when there are things go wrong.

The database is running on the same machine as the server.

Thanks again!
Ryan

Ryan,

Well, if you do some sort of monitoring, it will make your server
slower. By how much is impossible to say, depending on the method you use
to monitor the performance of your server. The simple act of monitoring
anything will have the effect of changing what you are monitoring.

I would say that forcing a GC is a bad idea, since it will typically do
a good job of that for you. There are exceptions to this though, but they
are rare. In a server environment, or any environment where you have a
consistent, almost predictable use, it's definitely a bad idea.

As for not having memory leaks, you are right, managed code doesn't
have memory leaks, but that doesn't mean you don't have to pay attention
to what you are using. Anything that implements IDisposable definitely
needs special attention, as you need to make sure you call Dispose (or you
should, at least) on those instances when you are done, as the IDisposable
interface indicates that the instance in question is holding onto
something which should be disposed of when done (as opposed to waiting
around for it to be disposed of, like file handles, sockets, database
connections, etc, etc).

Of course, the most important thing to do is really look at what your
server is doing. I know that sounds broad, but you didn't indicate
anything about what your server does, so it's impossible for anyone to
comment on what might be making it slow. Are you executing a massive
stored procedure which processes five billion rows for each of your 100
clients? Are you writing out 20GB of files for each client at the same
time? There are many things that could indicate performance problems, but
you can't even begin to diagnose them unless you know what the app is
doing (or supposed to do) in the first place.

That being said, can you shed some more light on what your app does?


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

Hi,

I use C# wrote an Client/Server application. In production environment,
will be 130 clients (Windows XP) connect to a Server (Windows 2000/2003
Server) thought TCP/IP socket in a local 100M LAN. "Server" is running as
a Windows service. The is one thread running for one clinet in the
server.

Sometime the user tells me the Sever will be slow after it runs for 1-2
days.

Usually what will be the cause? Managed code won't have memory leaking,
right? And how efficient is the garbage collection in .NET? Should I
force garbage collection once a while in my code?

And if I want to use the performance counter to check the problem, what
usually are the objects/parameters I should look at? How many threads are
running for my service? Memory used by the service or each server thread
in my service? Thread idle time ..? I just see too many things in the
performance counter.

And if I write code to sample those performance parameters, will it make
the service runs even slower?

Thanks!

Ryan

 

[Ignacio Machin \( .NET/ C# MVP \)]

Hi,

Hi,

I use C# wrote an Client/Server application. In production environment,
will be 130 clients (Windows XP) connect to a Server (Windows 2000/2003
Server) thought TCP/IP socket in a local 100M LAN. "Server" is running as
a Windows service. The is one thread running for one clinet in the server.

How you know how many clients are running at a given time?
Do you close your connections after you transfer data?

Sometime the user tells me the Sever will be slow after it runs for 1-2
days.

It sound like you are not closing the connections. Do a netstat -an and see
how many open connections you have.

 

[Ryan Liu]

Hi,





How you know how many clients are running at a given time?
Do you close your connections after you transfer data?


It sound like you are not closing the connections. Do a netstat -an and
see how many open connections you have.

The connectoins will be closed probably at the end of day, when all the
operators go home. Otherwise, yes, they remain open since operators are
continously work on it. I am using a statefull connection.

Thanks,

 

[Ignacio Machin \( .NET/ C# MVP \)]

Hi,

How many open connections there are?
I bet that after a couple of days you have a BIG number of connections open
and that is the problem you are seeing.

Can you use a poll method instead? The client connect to the server in
regular intervals and send/receive messages?
What about using UDP?
IIRC all the IM systems use UDP for this same reason

 

[Ryan Liu]

Hi,

How many open connections there are?
I bet that after a couple of days you have a BIG number of connections
open and that is the problem you are seeing.

Can you use a poll method instead? The client connect to the server in
regular intervals and send/receive messages?
What about using UDP?
IIRC all the IM systems use UDP for this same reason

Machin,

Not just client initiative the conversation, sometime the srever will
actively send data to client,and then client response as well.

And the conversation between client and server has to be "real time". I am
afraid poll mechanism is not handling it quick enough.

And I heard UDP is not as reliable as TCP. Maybe LAN enviorment is not a
problem. Also I use "statefull connectoin" to make my programming easy.

All clients will disconnect from the server at the end of day.

Thanks,
Ryan

 

[Ignacio Machin \( .NET/ C# MVP \)]

Hi,

And I heard UDP is not as reliable as TCP. Maybe LAN enviorment is not a
problem. Also I use "statefull connectoin" to make my programming easy.

You could use UDP to send a "ping" to the other party and then a TCP
connection is established and closed when the data is sent.

All clients will disconnect from the server at the end of day.

Are you sure of that?
More importantly, are you sure that teh server close all the connections?
Most probably you are handling each connection in a separated thread, are
all those threads being recicled and the instances of objects they use
returned?
How big is the memory consuption of the server?

 

[Ryan Liu]

Hi,



You could use UDP to send a "ping" to the other party and then a TCP
connection is established and closed when the data is sent.

Thanks fot the suggestion.

Are you sure of that?

Yes, it is the rule of my client company.

More importantly, are you sure that teh server close all the connections?
Most probably you are handling each connection in a separated thread, are
all those threads being recicled and the instances of objects they use
returned?
How big is the memory consuption of the server?

I don't know extractly, but seems it is not a lot. The server is 800 miles
away from me and most time it OK. I will try to log it when it happens next
time.

Thanks,
Ryan