Elusive randomly occuring exceptions

J

Jeff Shepler

This is probably not the right newsgroup for this, but this is the
only one I read and there are a lot of smart people that "live" here.
Please don't berate me if you think this should have been posted in
another newsgroup.

The web application I'm currently working on will (seemingly) randomly
throw an exception when opening a database connection. Not always in
the same place. Not always the same SqlConnection object. Not always
the same anything. The application could run fine for hours or it
could crash within a couple of requests of the last one.


Here is the most common exception thrown:

[COMException (0x80070006): The handle is invalid. (Exception from
HRESULT: 0x80070006 (E_HANDLE))]

System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32
errorCode, IntPtr errorInfo) +0
System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(Int32
errorCode) +34
System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection
owningObject) +636

System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection
owningConnection) +82

System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection
outerConnection, DbConnectionFactory connectionFactory) +105
System.Data.SqlClient.SqlConnection.Open() +111
System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset,
DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String
srcTable, IDbCommand command, CommandBehavior behavior) +121
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32
startRecord, Int32 maxRecords, String srcTable, IDbCommand command,
CommandBehavior behavior) +137
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet) +86

What handle????


Sometimes, I get a SemephoreFullException instead:

[SemaphoreFullException: Adding the given count to the semaphore would
cause it to exceed its maximum count.]
System.Threading.Semaphore.Release(Int32 releaseCount) +1853063

System.Data.ProviderBase.DbConnectionPool.PutNewObject(DbConnectionInternal
obj) +54

System.Data.ProviderBase.DbConnectionPool.DeactivateObject(DbConnectionInternal
obj) +228

System.Data.ProviderBase.DbConnectionPool.PutObject(DbConnectionInternal
obj, Object owningObject) +265

System.Data.ProviderBase.DbConnectionInternal.CloseConnection(DbConnection
owningObject, DbConnectionFactory connectionFactory) +97
System.Data.SqlClient.SqlConnection.Close() +117
System.Data.Common.DbDataAdapter.QuietClose(IDbConnection
connection, ConnectionState originalState) +13
System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset,
DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String
srcTable, IDbCommand command, CommandBehavior behavior) +285
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32
startRecord, Int32 maxRecords, String srcTable, IDbCommand command,
CommandBehavior behavior) +137
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet) +86


Or sometimes I get:

[SqlException (0x80131904): A transport-level error has occurred when
sending the request to the server. (provider: TCP Provider, error: 0 -
An operation was attempted on something that is not a socket.)]
System.Data.SqlClient.SqlConnection.OnError(SqlException exception,
Boolean breakConnection) +857370
System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
exception, Boolean breakConnection) +734982

System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject
stateObj) +188
System.Data.SqlClient.TdsParserStateObject.ThrowExceptionAndWarning()
+13
System.Data.SqlClient.TdsParserStateObject.WriteSni() +638049
System.Data.SqlClient.TdsParserStateObject.WritePacket(Byte
flushMode) +265
System.Data.SqlClient.TdsParserStateObject.ExecuteFlush() +51
System.Data.SqlClient.TdsParser.TdsExecuteRPC(_SqlRPC[] rpcArray,
Int32 timeout, Boolean inSchema, SqlNotificationRequest
notificationRequest, TdsParserStateObject stateObj) +3977
System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior
cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean
async) +886
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior
cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String
method, DbAsyncResult result) +132
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior
cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String
method) +32
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior
behavior, String method) +122
System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior
behavior) +12

System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader(CommandBehavior
behavior) +7
System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset,
DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String
srcTable, IDbCommand command, CommandBehavior behavior) +141
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32
startRecord, Int32 maxRecords, String srcTable, IDbCommand command,
CommandBehavior behavior) +137
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet) +86



After I get any of those exceptions, any further requests that attempt
to open the connection results IIS requesting login credentials - none
of which work (including administrator) and resuts in:

[UnauthorizedAccessException: Access is denied. (Exception from
HRESULT: 0x80070005 (E_ACCESSDENIED))]

System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32
errorCode, IntPtr errorInfo) +0
System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(Int32
errorCode) +34
System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection
owningObject) +636

System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection
owningConnection) +82

System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection
outerConnection, DbConnectionFactory connectionFactory) +105
System.Data.SqlClient.SqlConnection.Open() +111
System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset,
DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String
srcTable, IDbCommand command, CommandBehavior behavior) +121
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32
startRecord, Int32 maxRecords, String srcTable, IDbCommand command,
CommandBehavior behavior) +137
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet) +86


This makes even less sense to me than the Invalid Handle exception.
The connection uses a SQL login, not windows auth/trusted connection.
Why, after any of the first exceptions, is the login not valid? Why is
IIS prompting for windows credentials?



This happens on both an XP/IIS5 develeopment box and a 2003/IIS6
production box. The application has it's own application pool on the
production box for now while I'm testing.

The IIS processes (aspnet on IIS5, w3wp on IIS6) will eventually
crash, but not immediately. Letting the process crash or manually
ending the process or recycling the application pool or restarting the
web service will clear the problem and all will be fine for an unknown
amount of time (could be the next request, could be the next day).

I don't know what to do. I don't know how to troubleshoot this. Days
of googling and searching forums has not helped me.

Please help!
 
G

Guest

Jeff,
This looks like some sort of Connection Pool error, Connection Pooling is
based on COM+ "under the hood". Can you post some sample code illustrating
how your SqlConnections are being opened and closed? Is the connection string
the same for all connections?
Peter

--
Co-founder, Eggheadcafe.com developer portal:
http://www.eggheadcafe.com
UnBlog:
http://petesbloggerama.blogspot.com




Jeff Shepler said:
This is probably not the right newsgroup for this, but this is the
only one I read and there are a lot of smart people that "live" here.
Please don't berate me if you think this should have been posted in
another newsgroup.

The web application I'm currently working on will (seemingly) randomly
throw an exception when opening a database connection. Not always in
the same place. Not always the same SqlConnection object. Not always
the same anything. The application could run fine for hours or it
could crash within a couple of requests of the last one.


Here is the most common exception thrown:

[COMException (0x80070006): The handle is invalid. (Exception from
HRESULT: 0x80070006 (E_HANDLE))]

System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32
errorCode, IntPtr errorInfo) +0
System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(Int32
errorCode) +34
System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection
owningObject) +636

System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection
owningConnection) +82

System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection
outerConnection, DbConnectionFactory connectionFactory) +105
System.Data.SqlClient.SqlConnection.Open() +111
System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset,
DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String
srcTable, IDbCommand command, CommandBehavior behavior) +121
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32
startRecord, Int32 maxRecords, String srcTable, IDbCommand command,
CommandBehavior behavior) +137
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet) +86

What handle????


Sometimes, I get a SemephoreFullException instead:

[SemaphoreFullException: Adding the given count to the semaphore would
cause it to exceed its maximum count.]
System.Threading.Semaphore.Release(Int32 releaseCount) +1853063

System.Data.ProviderBase.DbConnectionPool.PutNewObject(DbConnectionInternal
obj) +54

System.Data.ProviderBase.DbConnectionPool.DeactivateObject(DbConnectionInternal
obj) +228

System.Data.ProviderBase.DbConnectionPool.PutObject(DbConnectionInternal
obj, Object owningObject) +265

System.Data.ProviderBase.DbConnectionInternal.CloseConnection(DbConnection
owningObject, DbConnectionFactory connectionFactory) +97
System.Data.SqlClient.SqlConnection.Close() +117
System.Data.Common.DbDataAdapter.QuietClose(IDbConnection
connection, ConnectionState originalState) +13
System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset,
DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String
srcTable, IDbCommand command, CommandBehavior behavior) +285
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32
startRecord, Int32 maxRecords, String srcTable, IDbCommand command,
CommandBehavior behavior) +137
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet) +86


Or sometimes I get:

[SqlException (0x80131904): A transport-level error has occurred when
sending the request to the server. (provider: TCP Provider, error: 0 -
An operation was attempted on something that is not a socket.)]
System.Data.SqlClient.SqlConnection.OnError(SqlException exception,
Boolean breakConnection) +857370
System.Data.SqlClient.SqlInternalConnection.OnError(SqlException
exception, Boolean breakConnection) +734982

System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject
stateObj) +188
System.Data.SqlClient.TdsParserStateObject.ThrowExceptionAndWarning()
+13
System.Data.SqlClient.TdsParserStateObject.WriteSni() +638049
System.Data.SqlClient.TdsParserStateObject.WritePacket(Byte
flushMode) +265
System.Data.SqlClient.TdsParserStateObject.ExecuteFlush() +51
System.Data.SqlClient.TdsParser.TdsExecuteRPC(_SqlRPC[] rpcArray,
Int32 timeout, Boolean inSchema, SqlNotificationRequest
notificationRequest, TdsParserStateObject stateObj) +3977
System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior
cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean
async) +886
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior
cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String
method, DbAsyncResult result) +132
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior
cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String
method) +32
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior
behavior, String method) +122
System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior
behavior) +12

System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader(CommandBehavior
behavior) +7
System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset,
DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String
srcTable, IDbCommand command, CommandBehavior behavior) +141
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32
startRecord, Int32 maxRecords, String srcTable, IDbCommand command,
CommandBehavior behavior) +137
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet) +86



After I get any of those exceptions, any further requests that attempt
to open the connection results IIS requesting login credentials - none
of which work (including administrator) and resuts in:

[UnauthorizedAccessException: Access is denied. (Exception from
HRESULT: 0x80070005 (E_ACCESSDENIED))]

System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32
errorCode, IntPtr errorInfo) +0
System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(Int32
errorCode) +34
System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection
owningObject) +636

System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection
owningConnection) +82

System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection
outerConnection, DbConnectionFactory connectionFactory) +105
System.Data.SqlClient.SqlConnection.Open() +111
System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset,
DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String
srcTable, IDbCommand command, CommandBehavior behavior) +121
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32
startRecord, Int32 maxRecords, String srcTable, IDbCommand command,
CommandBehavior behavior) +137
System.Data.Common.DbDataAdapter.Fill(DataSet dataSet) +86


This makes even less sense to me than the Invalid Handle exception.
The connection uses a SQL login, not windows auth/trusted connection.
Why, after any of the first exceptions, is the login not valid? Why is
IIS prompting for windows credentials?



This happens on both an XP/IIS5 develeopment box and a 2003/IIS6
production box. The application has it's own application pool on the
production box for now while I'm testing.

The IIS processes (aspnet on IIS5, w3wp on IIS6) will eventually
crash, but not immediately. Letting the process crash or manually
ending the process or recycling the application pool or restarting the
web service will clear the problem and all will be fine for an unknown
amount of time (could be the next request, could be the next day).

I don't know what to do. I don't know how to troubleshoot this. Days
of googling and searching forums has not helped me.

Please help!
Click to expand...
 
T

Tom Spink

Jeff said:
This is probably not the right newsgroup for this, but this is the
only one I read and there are a lot of smart people that "live" here.
Please don't berate me if you think this should have been posted in
another newsgroup.

The web application I'm currently working on will (seemingly) randomly
throw an exception when opening a database connection. Not always in
the same place. Not always the same SqlConnection object. Not always
the same anything. The application could run fine for hours or it
could crash within a couple of requests of the last one.
Click to expand...

<snippedy-doo-dah>

Hi Jeff,

Sounds like you've got some wierd heisenbug here. Are you using threading
or asynchronous operations in your code anywhere?
 
M

Marc Gravell

Also what security context is the app-pool runing under, and is IIS using
impersonation (based on the web-user account) - as I /believe/ the pool is
split by NT identity as well as connection string (even if not using
integrated security).

Marc
 
J

Jeff Shepler

I don't have access to the code right now (Happy 4th!!), but I did
write a wrapper around the SqlConnection object for 2 purposes. One is
to use the customer connection string manager I wrote and the other is
to hide the connection properties and provide a facade for executing
queries.

As a result, the same connection object is used for all methods the
consumer uses. Originally, I kept the connection open for the duration
of the object - which was only as long as a request being processed -
thinking that it would improve performance. However, I read somehwere
(duing my searching for answers to these expections) that implied that
it wasn't a good idea to do so. So, now I open/close when needed.

If I understood what I read, just because I have a SqlConnection
object doesn't mean that I have the same connection from the pool. The
connection is aquired on .Open(), correct? Regardless, it didn't fix
the errors I was having and I did not notice any improvement in speed.

No, the connection string is not the same for all connections. The
application uses 3 databases, 2 of which are on the same server. All
have different login credentials.


From memory, here's approximately what I'm doing:

public class ManagedConnection
{
private SqlConnection conn;

public ManagedCollection(string dbAlias, string appName)
{
conn = new
SqlConnection(ConnectionStringManager.GetConnectionString(dbAlias) +
";app=" + appName);
}

public DataSet Execute_Procedure_DataSet(string spName,
List<SqlParameter> Parameters)
{
SqlCommand cmd;
SqlDataAdapter sda;
DataSet ds = new DataSet();

cmd = Connection.CreateCommand();
cmd.CommandType = CommandType.StoredProcedure;
cmd.CommandText = spName;
foreach(SqlParameter p in Parameters)
cmd.Parameters.Add(p);

sda = new SqlDataAdapter(cmd);

try
{
sda.Fill(ds);
}
catch(Exception ex)
{
throw new Exception(....customized error here showing
query and parameters for easy runtime debugging...);
}

return ds;
}
}

There are many other methods for dealing with adhoc queries,
transactional queries, queries with no result, queries with a single
result (scalar) and queries that return a value. They all basically do
the same thing.
 
J

Jeff Shepler

I did change the security context to LocalSystem because of the
problem with the worker process crashing on every recyle.

http://blogs.msdn.com/david.wang/ar...een_a_WebEngine_dll_crash_with_ASPNet_20.aspx


It's been a while since I've researched the bug. Maybe it's been fixed
by some Windows Update since then. I'll set it back to network and try
again. Or would you suggest something different?


IIS is set to integrated auth, yes. I'm not sure I follow "the pool is
split by NT identity ...". Are you saying that if user1 and user2 ask
for a connection (is the same app domain) using the exact same
connection string, they would get two different connections?
Interesting if true, but I'm not really there yet. I've been the only
one using this app so far (still under development).
 
J

Jeff Shepler

No.

After I started getting the errors, I re-thought my managed connection
object and decided to make it thread-safe by locking access to the
SqlConnection object. Just in case I (or someone else) would want to
use the object in a threaded environment or as a singleton in a web
app.

However, it didn't fix the problems and I read somewhere (can't find
it right now) that it wasn't a good idea to do that - it was better to
let the connection pool handle concurrency (???). So, I removed it.

It might become a concern once this thing is to go live as, being a
web app, some of my singletons could very well (and probably will) be
accessed from multiple threads.

I'm relatively new to threading. I've done a few things with various
successes. I don't think I have a full understanding though - it
hasn't "clicked" for me yet.
 
M

Marc Gravell

Are you saying that if user1 and user2 ask
for a connection (is the same app domain) using the exact same
connection string, they would get two different connections?
Click to expand...

If IIS is configured to switch into the user's NT identity, then my
understanding is that yes: each user would get their own pool; the pool
maximum would probably kick in quite soon such that you don't see much
benefit from pooling, but I was mainly asking to see if you were drowning in
the connection pool. Many IIS configurations use the trusted sub-system
model, with some form of access-control at the boundary, then a single NT
identity (e.g. "NETWORK SERVICE", etc) performing all access within the
system; as I understand it this will allow the pool to be shared across all
users.

I'm still not sure that this would explain the level of errors that you are
seeing; are you /sure/ the connections are closed and disposed
appropriately?

Marc
 
J

Jeff Shepler

I do close, but not dispose. The class doesn't know how many times a
method is going to be called. I didn't want to create a new
SqlConnection instance for each method. I guess I could implement a
destructor in my facade class that would call conn.Dispose(). While
that might be preferred, I don't think it helps with these exceptions
that I'm getting.

However, I believe that if the pool was running out of connections,
I'd get a the timeout exception. Something like: Error connecting:
Timeout expired. The timeout period elapsed prior to obtaining a
connection from the pool. This may have occurred because all pooled
connections were in use and max pool size was reached.
 
J

Jeff Shepler

I'm having a problem understanding this "handle" thing.

You say the connection pooling is based on COM "under the hood". I can
accept that.. No problem.. Kinda obvious given the stack trace.. What
I don't understand is how how I use objects affects "under the hood".
I mean, I just call SqlDataAdapter.Fill(DataSet), for example, and the
exception gets thrown. The code that deals with connection pooling
isn't under my control. The framework is creating this handle, not me.
So, if the handle is invalid, then the framework is doing something
wrong???


The only thing I do to affect it is the connection string, right? If
the connection string got corrupted somehow, wouldn't one of the
objects in the chain (SqlClient.SqlConnection,
ProviderBase.DbConnectionFactory, ProviderBase.DbConnectionPool) catch
it or is it only checked on creation?


I'll spend time investigating the connection string to see if it's
getting munged somewhere along the line.
 
G

Guest

Close is fine without Dispose. In fact, before a hotfix in 1.1, it was the
only way to return the connection to the pool. However, I'd consider using
the "Best practices"Pattern: Open a connection immediately before the code
the uses it, and close it immediately thereafter. With pooling most of the
overhead of connection - related semantics is alleviated.
Peter

--
Co-founder, Eggheadcafe.com developer portal:
http://www.eggheadcafe.com
UnBlog:
http://petesbloggerama.blogspot.com
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

How do I best fix this error 4
Transaction Error on None-Transaction Query 16
C# 2.0 + MS Access == Error 5
Reading Excel File 3
Confusing error message 3
Trying to debug an exception 4
access to mysql over csharp 1
Simple database class error 5

Top