Large virtual memory, small working set

[Adam Clauss]

We have an application which is fairly network intensive. During some
stress testing, we had it setup to open approximately 300-400 TCP
connections (outbound connections, we are a TCP client). Almost immediately
upon starting and loading configuration information, the application starts
opening up connections and the memory usage immediately skyrockets. With
slightly higher numbers of connections, the application actually begins
throwing out of memory exceptions; we tuned the number down slightly so we
could troubleshoot it without the process crashing.

The strange thing is - only the virtual memory (as described in Task
Manager) increases this drastic. After running for 30-45 seconds (and all
the connections are open), the Mem Usage (working set) is roughly 200-300mb,
which is more or less what we expected. What is unexpected is the virtual
memory size, which crosses the 1gb mark.

We attached the memory profiler from Visual Studio, but we simply cannot
account for that volume of memory anywhere. If we add up things, we can
typically justify the working set, but nothing in the gigabyte quantity.

Any thoughts on what type of things would cause the virtual memory to shoot
up so high? Is it some sort of side-effect from having that number of open
TCP connections that we are stuck with?

Been banging our head against the wall on how to approach this one,
appreciate any comments.

Thanks!

 

[Adam Clauss]

Hey mate,

Have you tried setting up some traces and/or performance counters instead
of a memory profiler in VS?

What types of counters? We've tried "manual" tracing - pausing in
breakpoints at various points of execution to check the sizes of Lists,
Dictionaries, etc, nothing unexpected came up.

What does your application do? If you're using non-managed resources, are
you disposing them correctly?

We're basically opening all these connections and receiving data. We get
the bytes, parse it out (various protocols in use), and periodically (~60
seconds or so typically) perform a set of calculations on the received data.
Pretty much everything is straight managed objects (classes which parse out
the bytes and expose properties to represent the results, Lists,
Dictionaries, etc). As close to an unmanaged resource as we get is the
sockets themselves.

There are a somewhat higher number of threads than some of our other
applications. Right now, we have a thread-per connection processing data as
it gets received off a socket, but those threads tend to spend most of their
life in a Monitor.Wait() waiting for data to be put into a queue from the
socket code.