a leaky program...

  • Thread starter Thread starter Guest
  • Start date Start date
G

Guest

Hi,

I have a C# program that in principle should only use a constant amount of
memory to execute (with periodic garbage collection) and in fact does so when
executed in Mono under either Windows or Linux. However, with Microsoft's
own .NET execution environment, the program rapidly takes up more and more
memory until it dies with an out-of-memory exception.

I post an example code below, in the hope that someone with more knowledge
of C#/.NET could enlighten me on some possible causes of such leaky behavior.

Thanks.

------------------------------------------------------------------------------------------------

using System;
using System.Data.SqlClient;
using System.Security.Cryptography;

/*
* Prerequisites:
* - On (local), need to have DB named "test_db" with table named "gotera",
* or adjust connection and query strings to match own settings.
* - The table can have any schema but needs to have sufficient number of
* rows (say, 1000000) for the program to run for more than a few seconds.
*
* No leaky behavior is observed if:
* - The [STAThread] tag is removed;
* - The MD5 construction is removed; or
* - The enumeration of DB table rows is replaced with a simple loop
* such as "for (int i=0; i<100000; ++i) { ... }".
*/

class Leaky
{
[STAThread]
static int Main() {
long n_RowCnt = 0;
SqlConnection conn = new
SqlConnection(@"server=(local);database=test_db;trusted_connection=yes");
try {
conn.Open();
SqlCommand cmd = new SqlCommand("select * from gotera", conn);
SqlDataReader reader = cmd.ExecuteReader();
while (reader.Read()) {
++n_RowCnt;
MD5CryptoServiceProvider md5 = new MD5CryptoServiceProvider();
}
}
catch (Exception ex) {
System.Console.WriteLine(ex.Message);
}
finally {
conn.Close();
}
System.Console.WriteLine("Counted {0} rows.", n_RowCnt);
return 0;
}
}
 
Casual Reader said:
I have a C# program that in principle should only use a constant amount of
memory to execute (with periodic garbage collection) and in fact does so when
executed in Mono under either Windows or Linux. However, with Microsoft's
own .NET execution environment, the program rapidly takes up more and more
memory until it dies with an out-of-memory exception.

I post an example code below, in the hope that someone with more knowledge
of C#/.NET could enlighten me on some possible causes of such leaky behavior.
Click to expand...

One thing you're not doing is disposing of either the command (which
shouldn't be causing a leak as there's only one of them) or the
MD5CryptoServiceProvider. I'm surprised it's causing a leak, but it's
something to try...
 
Casual Reader said:
Hi,

I have a C# program that in principle should only use a constant amount of
memory to execute (with periodic garbage collection) and in fact does so
when
executed in Mono under either Windows or Linux. However, with Microsoft's
own .NET execution environment, the program rapidly takes up more and more
memory until it dies with an out-of-memory exception.

I post an example code below, in the hope that someone with more knowledge
of C#/.NET could enlighten me on some possible causes of such leaky
behavior.

Thanks.

------------------------------------------------------------------------------------------------

using System;
using System.Data.SqlClient;
using System.Security.Cryptography;

/*
* Prerequisites:
* - On (local), need to have DB named "test_db" with table named "gotera",
* or adjust connection and query strings to match own settings.
* - The table can have any schema but needs to have sufficient number of
* rows (say, 1000000) for the program to run for more than a few
seconds.
*
* No leaky behavior is observed if:
* - The [STAThread] tag is removed;
* - The MD5 construction is removed; or
* - The enumeration of DB table rows is replaced with a simple loop
* such as "for (int i=0; i<100000; ++i) { ... }".
*/

class Leaky
{
[STAThread]
static int Main() {
long n_RowCnt = 0;
SqlConnection conn = new
SqlConnection(@"server=(local);database=test_db;trusted_connection=yes");
try {
conn.Open();
SqlCommand cmd = new SqlCommand("select * from gotera", conn);
SqlDataReader reader = cmd.ExecuteReader();
while (reader.Read()) {
++n_RowCnt;
MD5CryptoServiceProvider md5 = new MD5CryptoServiceProvider();
}
}
catch (Exception ex) {
System.Console.WriteLine(ex.Message);
}
finally {
conn.Close();
}
System.Console.WriteLine("Counted {0} rows.", n_RowCnt);
return 0;
}
}
Click to expand...


The reason relates to the apartment context your main thread runs in, Mono
doesn't have any apartment concept, that's the reason why you don't se this
happen.

What's happening is the following, your MD5CryptoServiceProvider is never
disposed of, so you are relying on the finalizer to clean up the unmanaged
resources when he runs MD5CryptoServiceProvider Finalize method.
Now, whenever you specify [STAThread] on "Main", your thread enters a STA ,
however, the finalizer thread, who runs in a MTA, can't directly call
Finalize without marshaling the call between both apartments/threads.
In order to successfully marshal a call, an STA thread must pump messages
f.i by calling DoEvents(), failing to do so will block the finalizer.

There are two possible options to solve this:

1. Call DoEvents(),
while (reader.Read()) {
DoEvents();
..

or much better...

2. Remove the STAThread attribute AND wrap your
MD5CryptoServiceProvider in a using block like this/

using(MD5CryptoServiceProvider md5 = ...)
{
. .
}

Not sure why you need another instance of md5 for each iteration though.

Willy.
 
Hi,

Thanks to both Jon Skeet and Willy Denoyette for contributing their insights
to my problem.

As I understand the comments of both people, it is a bad interaction between
"STA" and "MTA" that exists between my program and the execution environment;
one remedy is to explicitly dispose of objects that are no longer required.

In response to a comment re: why a new MD5 object was instantiated per
iteration, I found very little documentation on the proper usage of MD5
objects in .NET when I wrote the program. The one example that I found was
Microsoft Knowledgebase Article 307020; bullet point 6 of that article says
to use a new MD5 object for each new MD5 that you want to compute.

Finally, one nit-picking comment that deserves no response: When I was
investigating why my program was leaking memory, I narrowed down the problem
to an interplay between three things in my program: the [STAThread] tag, the
repeated instantiation of the MD5 object, and the enumeration of rows of a DB
table. If I removed any one of the three, I would get no leaky behavior;
that's why I posted the example code. But I've already learned a few things
from the two MVPs, and I'm content for now...

Thanks and regards.


Willy Denoyette said:
Casual Reader said:
Hi,

I have a C# program that in principle should only use a constant amount of
memory to execute (with periodic garbage collection) and in fact does so
when
executed in Mono under either Windows or Linux. However, with Microsoft's
own .NET execution environment, the program rapidly takes up more and more
memory until it dies with an out-of-memory exception.

I post an example code below, in the hope that someone with more knowledge
of C#/.NET could enlighten me on some possible causes of such leaky
behavior.

Thanks.

------------------------------------------------------------------------------------------------

using System;
using System.Data.SqlClient;
using System.Security.Cryptography;

/*
* Prerequisites:
* - On (local), need to have DB named "test_db" with table named "gotera",
* or adjust connection and query strings to match own settings.
* - The table can have any schema but needs to have sufficient number of
* rows (say, 1000000) for the program to run for more than a few
seconds.
*
* No leaky behavior is observed if:
* - The [STAThread] tag is removed;
* - The MD5 construction is removed; or
* - The enumeration of DB table rows is replaced with a simple loop
* such as "for (int i=0; i<100000; ++i) { ... }".
*/

class Leaky
{
[STAThread]
static int Main() {
long n_RowCnt = 0;
SqlConnection conn = new
SqlConnection(@"server=(local);database=test_db;trusted_connection=yes");
try {
conn.Open();
SqlCommand cmd = new SqlCommand("select * from gotera", conn);
SqlDataReader reader = cmd.ExecuteReader();
while (reader.Read()) {
++n_RowCnt;
MD5CryptoServiceProvider md5 = new MD5CryptoServiceProvider();
}
}
catch (Exception ex) {
System.Console.WriteLine(ex.Message);
}
finally {
conn.Close();
}
System.Console.WriteLine("Counted {0} rows.", n_RowCnt);
return 0;
}
}
Click to expand...


The reason relates to the apartment context your main thread runs in, Mono
doesn't have any apartment concept, that's the reason why you don't se this
happen.

What's happening is the following, your MD5CryptoServiceProvider is never
disposed of, so you are relying on the finalizer to clean up the unmanaged
resources when he runs MD5CryptoServiceProvider Finalize method.
Now, whenever you specify [STAThread] on "Main", your thread enters a STA ,
however, the finalizer thread, who runs in a MTA, can't directly call
Finalize without marshaling the call between both apartments/threads.
In order to successfully marshal a call, an STA thread must pump messages
f.i by calling DoEvents(), failing to do so will block the finalizer.

There are two possible options to solve this:

1. Call DoEvents(),
while (reader.Read()) {
DoEvents();
..

or much better...

2. Remove the STAThread attribute AND wrap your
MD5CryptoServiceProvider in a using block like this/

using(MD5CryptoServiceProvider md5 = ...)
{
. .
}

Not sure why you need another instance of md5 for each iteration though.

Willy.
Click to expand...
 
Inline **

Willy.

Casual Reader said:
Hi,

Thanks to both Jon Skeet and Willy Denoyette for contributing their
insights
to my problem.

As I understand the comments of both people, it is a bad interaction
between
"STA" and "MTA" that exists between my program and the execution
environment;
one remedy is to explicitly dispose of objects that are no longer
required.
Click to expand...

** The real source of the problem here is - a failure to pump messages on a
STA thread - which makes it impossible for the Finalizer thread, that runs
in a MTA, to marshal the calls to the Finalize method on the STA thread.
Therefore, I suggest you only apply STAThread on a Windows (Forms) program
"Main" entry. In all other cases, if you need a STA thread for COM interop,
make sure you pump the message queue.
In response to a comment re: why a new MD5 object was instantiated per
iteration, I found very little documentation on the proper usage of MD5
objects in .NET when I wrote the program. The one example that I found
was
Microsoft Knowledgebase Article 307020; bullet point 6 of that article
says
to use a new MD5 object for each new MD5 that you want to compute.
Click to expand...
** That's right.
Finally, one nit-picking comment that deserves no response: When I was
investigating why my program was leaking memory, I narrowed down the
problem
to an interplay between three things in my program: the [STAThread] tag,
the
repeated instantiation of the MD5 object, and the enumeration of rows of a
DB
table. If I removed any one of the three, I would get no leaky behavior;
that's why I posted the example code. But I've already learned a few
things
from the two MVPs, and I'm content for now...

Thanks and regards.


Willy Denoyette said:
Casual Reader said:
Hi,

I have a C# program that in principle should only use a constant amount
of
memory to execute (with periodic garbage collection) and in fact does
so
when
executed in Mono under either Windows or Linux. However, with
Microsoft's
own .NET execution environment, the program rapidly takes up more and
more
memory until it dies with an out-of-memory exception.

I post an example code below, in the hope that someone with more
knowledge
of C#/.NET could enlighten me on some possible causes of such leaky
behavior.

Thanks.

------------------------------------------------------------------------------------------------

using System;
using System.Data.SqlClient;
using System.Security.Cryptography;

/*
* Prerequisites:
* - On (local), need to have DB named "test_db" with table named
"gotera",
* or adjust connection and query strings to match own settings.
* - The table can have any schema but needs to have sufficient number
of
* rows (say, 1000000) for the program to run for more than a few
seconds.
*
* No leaky behavior is observed if:
* - The [STAThread] tag is removed;
* - The MD5 construction is removed; or
* - The enumeration of DB table rows is replaced with a simple loop
* such as "for (int i=0; i<100000; ++i) { ... }".
*/

class Leaky
{
[STAThread]
static int Main() {
long n_RowCnt = 0;
SqlConnection conn = new
SqlConnection(@"server=(local);database=test_db;trusted_connection=yes");
try {
conn.Open();
SqlCommand cmd = new SqlCommand("select * from gotera", conn);
SqlDataReader reader = cmd.ExecuteReader();
while (reader.Read()) {
++n_RowCnt;
MD5CryptoServiceProvider md5 = new MD5CryptoServiceProvider();
}
}
catch (Exception ex) {
System.Console.WriteLine(ex.Message);
}
finally {
conn.Close();
}
System.Console.WriteLine("Counted {0} rows.", n_RowCnt);
return 0;
}
}
Click to expand...


The reason relates to the apartment context your main thread runs in,
Mono
doesn't have any apartment concept, that's the reason why you don't se
this
happen.

What's happening is the following, your MD5CryptoServiceProvider is never
disposed of, so you are relying on the finalizer to clean up the
unmanaged
resources when he runs MD5CryptoServiceProvider Finalize method.
Now, whenever you specify [STAThread] on "Main", your thread enters a STA
,
however, the finalizer thread, who runs in a MTA, can't directly call
Finalize without marshaling the call between both apartments/threads.
In order to successfully marshal a call, an STA thread must pump messages
f.i by calling DoEvents(), failing to do so will block the finalizer.

There are two possible options to solve this:

1. Call DoEvents(),
while (reader.Read()) {
DoEvents();
..

or much better...

2. Remove the STAThread attribute AND wrap your
MD5CryptoServiceProvider in a using block like this/

using(MD5CryptoServiceProvider md5 = ...)
{
. .
}

Not sure why you need another instance of md5 for each iteration though.

Willy.
Click to expand...
Click to expand...
 
Back
Top