Internal connection fatal error

[Max Metral]

I've seen other posts about such errors, but never a solution. It seems to
appear randomly under load. We are pretty vigilant about "using"
IDataReader's and Connections, so it's pretty unlikely that we're doing
something outwardly wrong that would excuse this sort of behavior. The
stack is:

TARGETSITE: System.Data.SqlClient.SqlDataReader
ExecuteReader(System.Data.CommandBehavior,
System.Data.SqlClient.RunBehavior, Boolean)

STACKTRACE: at
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior cmdBehavior,
RunBehavior runBehavior, Boolean returnStream)
at System.Data.SqlClient.SqlCommand.ExecuteReader()
at Web.MyPage.Page_Load(Object sender, EventArgs e)
<removed by me>

Perhaps related, we also see random errors like this about columns we KNOW
exist in recordsets that are being returned:

MESSAGE: EventId
SOURCE: System.Data
TARGETSITE: Int32 GetOrdinal(System.String)
STACKTRACE: at System.Data.Common.FieldNameLookup.GetOrdinal(String
fieldName)
at System.Data.SqlClient.SqlDataReader.GetOrdinal(String name)
at MyObject..ctor(IDataReader eventReader, EventData eventData)
<removed by me>

This is definitely unacceptable behavior from ADO, SQL Server, or both... No
code that an end user writes in a managed language should be able to create
this kind of untraceable havoc.

Any thoughts?

 

[Max Metral]

The words "sucks" and "SQL Server" are starting a slow walk towards each
other. For *now*, rebooting the db machine seems to have eradicated these
errors, although the load may not have reached similar heights since it's a
weekend.

 

[William \(Bill\) Vaughn]

Turn on Perfmon and watch the connection pool. Is it constantly growing over
time or does it flatten out after a few minutes under load? Is the network
stable? Are you connecting over a LAN, WAN or web? If you lose a connection
due to IO problems on the wire, you can get this type of exception.

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________

 

[Max Metral]

LAN connection, which isn't especially loaded (.27% according to perfmon).
The pool stabilizes at about 80 connections under heavy load, although I
just got one of these errors with 10 in the pool. I also wonder whether
it's some sort of "dirty" connection in the pool, is this possible? i.e.
something like:

1) Expensive query in web page
2) user hits stop, recordset still "in" the connection
3) connection goes back in pool
4) another thread gets conn
5) conn is in weird state, all bets off (also would explain errors like
missing columns, sort of)

I realize this kind of diagnosis is indistinguishable from guessing, but you
never know

 

[William \(Bill\) Vaughn]

80 connections in the pool tells me this is a heavily loaded system. Saying
that means that it's being asked to process more work than it can handle.
Think of the connection pool (number of connections in the pool) as the
system tachometer. 100 connections is "overrev". Even a heavily loaded
system should make do with far fewer connections in the pool.
Yes, if an operation fails at the server, the connection can be broken. This
is not that uncommon so your error handlers need to be prepared for it. If
the connection breaks, the connection is unusable--but the pooler will still
try to reuse it (once). Your app gets an exception when it tries to use a
broken connection, but the pooler tears it down. In ADO 2.0, the whole pool
is tossed if the pooler thinks the server has crashed and you can choose to
toss the pool on your own.
When an application "inherits" a connection, it's cleared (reset) of its
"Environment" settings so you should not have one instance inherit "stuff"
from another--UNLESS you disabled the reset mechanism to get better
performance.

hth

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________

 

[Angel Saenz-Badillos[MS]]

Max,
I believe from your description that you are running into Q814114. Please
contact PSS directly for the fix
http://support.microsoft.com/default.aspx?scid=/support/contact/default.asp
(this is a free call).

The bug addressed by this QFE is almost exactly what you describe, a dirty
connection gets in the pool. This only repros when the network you are using
is high traffic or high latency, this forces the network libraries to
include multiple TDS Packets per read. Normally on connection close we clean
the data waiting on the connection, there is a bug where we fail to clean up
multiple packet TDS under certain conditions.

I apologize for the obvious pain this is causing, let me know if this does
not fix your problem.
--
Angel Saenz-Badillos [MS] Managed Providers
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging about ADO.NET: http://weblogs.asp.net/angelsb/

 

[William \(Bill\) Vaughn]

Angel, I don't get any hits on Q814114.

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________

Max,
I believe from your description that you are running into Q814114. Please
contact PSS directly for the fix
http://support.microsoft.com/default.aspx?scid=/support/contact/default.asp
(this is a free call).

The bug addressed by this QFE is almost exactly what you describe, a dirty
connection gets in the pool. This only repros when the network you are using
is high traffic or high latency, this forces the network libraries to
include multiple TDS Packets per read. Normally on connection close we clean
the data waiting on the connection, there is a bug where we fail to clean up
multiple packet TDS under certain conditions.

I apologize for the obvious pain this is causing, let me know if this does
not fix your problem.
--
Angel Saenz-Badillos [MS] Managed Providers
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging about ADO.NET: http://weblogs.asp.net/angelsb/

LAN connection, which isn't especially loaded (.27% according to perfmon).
The pool stabilizes at about 80 connections under heavy load, although I
just got one of these errors with 10 in the pool. I also wonder whether
it's some sort of "dirty" connection in the pool, is this possible? i.e.
something like:

1) Expensive query in web page
2) user hits stop, recordset still "in" the connection
3) connection goes back in pool
4) another thread gets conn
5) conn is in weird state, all bets off (also would explain errors like
missing columns, sort of)

I realize this kind of diagnosis is indistinguishable from guessing, but you
never know

growing
over we
KNOW

 

[Angel Saenz-Badillos[MS]]

Bill,
I apologize, I know next to nothing about how KBs are handled. Like you I
cannot find any information on this error. I will try to hunt down what
happened to the KB article for this.

--
Angel Saenz-Badillos [MS] Managed Providers
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging about ADO.NET: http://weblogs.asp.net/angelsb/

Angel, I don't get any hits on Q814114.

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________

Max,
I believe from your description that you are running into Q814114. Please
contact PSS directly for the fix

http://support.microsoft.com/default.aspx?scid=/support/contact/default.asp

(this is a free call).

The bug addressed by this QFE is almost exactly what you describe, a dirty
connection gets in the pool. This only repros when the network you are using
is high traffic or high latency, this forces the network libraries to
include multiple TDS Packets per read. Normally on connection close we clean
the data waiting on the connection, there is a bug where we fail to

clean

up
multiple packet TDS under certain conditions.

I apologize for the obvious pain this is causing, let me know if this does
not fix your problem.
--
Angel Saenz-Badillos [MS] Managed Providers
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging about ADO.NET: http://weblogs.asp.net/angelsb/

LAN connection, which isn't especially loaded (.27% according to perfmon).
The pool stabilizes at about 80 connections under heavy load, although I
just got one of these errors with 10 in the pool. I also wonder whether
it's some sort of "dirty" connection in the pool, is this possible? i.e.
something like:

1) Expensive query in web page
2) user hits stop, recordset still "in" the connection
3) connection goes back in pool
4) another thread gets conn
5) conn is in weird state, all bets off (also would explain errors like
missing columns, sort of)

I realize this kind of diagnosis is indistinguishable from guessing,

but
you

never know

Turn on Perfmon and watch the connection pool. Is it constantly growing
over
time or does it flatten out after a few minutes under load? Is the network
stable? Are you connecting over a LAN, WAN or web? If you lose a
connection
due to IO problems on the wire, you can get this type of exception.

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no
rights.
__________________________________

I've seen other posts about such errors, but never a solution. It seems
to
appear randomly under load. We are pretty vigilant about "using"
IDataReader's and Connections, so it's pretty unlikely that we're doing
something outwardly wrong that would excuse this sort of behavior. The
stack is:

TARGETSITE: System.Data.SqlClient.SqlDataReader
ExecuteReader(System.Data.CommandBehavior,
System.Data.SqlClient.RunBehavior, Boolean)

STACKTRACE: at
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior
cmdBehavior,
RunBehavior runBehavior, Boolean returnStream)
at System.Data.SqlClient.SqlCommand.ExecuteReader()
at Web.MyPage.Page_Load(Object sender, EventArgs e)
<removed by me>

Perhaps related, we also see random errors like this about columns we
KNOW
exist in recordsets that are being returned:

MESSAGE: EventId
SOURCE: System.Data
TARGETSITE: Int32 GetOrdinal(System.String)
STACKTRACE: at System.Data.Common.FieldNameLookup.GetOrdinal(String
fieldName)
at System.Data.SqlClient.SqlDataReader.GetOrdinal(String name)
at MyObject..ctor(IDataReader eventReader, EventData eventData)
<removed by me>

This is definitely unacceptable behavior from ADO, SQL Server, or
both...
No
code that an end user writes in a managed language should be able to
create
this kind of untraceable havoc.

Any thoughts?

 

[Max Metral]

The system is *definitely* loaded. At that time we were doing about 50
pages per second, all hitting the db. It's just the nature of the app...
We tuned things down another notch, and it's gotten a bit better, but I
still see errors, and this isn't the way the system should fail... (i'll
try the QFE Angel suggests, maybe that will move the problem one step
further).

Another strange thing... The db is CPU bound it seems. The disk queue
length stays very low, even under heavy load, but the dual hyperthreaded
procs are maxed out. I've never seen the db be the bottleneck in a db
before, it seems it's almost always disk or mem or something. What sorts of
things might be big culprits here? UDFs? The memory bus bandwidth? The
Quake server? (just kidding)

--Max

 

[Eric Stoll]

Angel,
Any luck finding that KB number? I am interested in it also.

Thanks,
Eric Stoll

Bill,
I apologize, I know next to nothing about how KBs are handled. Like you I
cannot find any information on this error. I will try to hunt down what
happened to the KB article for this.

--
Angel Saenz-Badillos [MS] Managed Providers
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging about ADO.NET: http://weblogs.asp.net/angelsb/

Angel, I don't get any hits on Q814114.

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________

Max,
I believe from your description that you are running into Q814114. Please
contact PSS directly for the fix

http://support.microsoft.com/default.aspx?scid=/support/contact/default.asp

(this is a free call).

The bug addressed by this QFE is almost exactly what you describe, a dirty
connection gets in the pool. This only repros when the network you are using
is high traffic or high latency, this forces the network libraries to
include multiple TDS Packets per read. Normally on connection close we clean
the data waiting on the connection, there is a bug where we fail to

clean
up

multiple packet TDS under certain conditions.

I apologize for the obvious pain this is causing, let me know if this does
not fix your problem.
--
Angel Saenz-Badillos [MS] Managed Providers
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging about ADO.NET: http://weblogs.asp.net/angelsb/




LAN connection, which isn't especially loaded (.27% according to perfmon).
The pool stabilizes at about 80 connections under heavy load, although I
just got one of these errors with 10 in the pool. I also wonder whether
it's some sort of "dirty" connection in the pool, is this possible? i.e.
something like:

1) Expensive query in web page
2) user hits stop, recordset still "in" the connection
3) connection goes back in pool
4) another thread gets conn
5) conn is in weird state, all bets off (also would explain errors like
missing columns, sort of)

I realize this kind of diagnosis is indistinguishable from guessing,

but
you

never know

Turn on Perfmon and watch the connection pool. Is it constantly

growing
over

time or does it flatten out after a few minutes under load? Is the network
stable? Are you connecting over a LAN, WAN or web? If you lose a connection
due to IO problems on the wire, you can get this type of exception.

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________

I've seen other posts about such errors, but never a solution. It

seems
to

appear randomly under load. We are pretty vigilant about "using"
IDataReader's and Connections, so it's pretty unlikely that we're doing
something outwardly wrong that would excuse this sort of behavior. The
stack is:

TARGETSITE: System.Data.SqlClient.SqlDataReader
ExecuteReader(System.Data.CommandBehavior,
System.Data.SqlClient.RunBehavior, Boolean)

STACKTRACE: at
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior cmdBehavior,
RunBehavior runBehavior, Boolean returnStream)
at System.Data.SqlClient.SqlCommand.ExecuteReader()
at Web.MyPage.Page_Load(Object sender, EventArgs e)
<removed by me>

Perhaps related, we also see random errors like this about columns

we
KNOW

exist in recordsets that are being returned:

MESSAGE: EventId
SOURCE: System.Data
TARGETSITE: Int32 GetOrdinal(System.String)
STACKTRACE: at System.Data.Common.FieldNameLookup.GetOrdinal(String
fieldName)
at System.Data.SqlClient.SqlDataReader.GetOrdinal(String name)
at MyObject..ctor(IDataReader eventReader, EventData eventData)
<removed by me>

This is definitely unacceptable behavior from ADO, SQL Server, or

both...
No

code that an end user writes in a managed language should be able

to
create

this kind of untraceable havoc.

Any thoughts?

 

[Angel Saenz-Badillos[MS]]

Eric,
As far as I have been able to determine so far this article is no longer
available. I am trying to find out why and what (if anything) has replaced
it, but I am really not the best person to look into this and the one person
that I know could get this done is currently in vacation.

The QFE should still be valid, I am particularly interested in hearing
wither it does fix Max's problem.
--
Angel Saenz-Badillos [MS] Managed Providers
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging about ADO.NET: http://weblogs.asp.net/angelsb/

Angel,
Any luck finding that KB number? I am interested in it also.

Thanks,
Eric Stoll

"Angel Saenz-Badillos[MS]" <[email protected]> wrote in message

Bill,
I apologize, I know next to nothing about how KBs are handled. Like you I
cannot find any information on this error. I will try to hunt down what
happened to the KB article for this.

--
Angel Saenz-Badillos [MS] Managed Providers
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging about ADO.NET: http://weblogs.asp.net/angelsb/

Angel, I don't get any hits on Q814114.

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________

Max,
I believe from your description that you are running into Q814114. Please
contact PSS directly for the fix

http://support.microsoft.com/default.aspx?scid=/support/contact/default.asp

(this is a free call).

The bug addressed by this QFE is almost exactly what you describe, a dirty
connection gets in the pool. This only repros when the network you

are
using

is high traffic or high latency, this forces the network libraries to
include multiple TDS Packets per read. Normally on connection close

we
clean

the data waiting on the connection, there is a bug where we fail to clean
up
multiple packet TDS under certain conditions.

I apologize for the obvious pain this is causing, let me know if

this
does

not fix your problem.
--
Angel Saenz-Badillos [MS] Managed Providers
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging about ADO.NET: http://weblogs.asp.net/angelsb/




LAN connection, which isn't especially loaded (.27% according to perfmon).
The pool stabilizes at about 80 connections under heavy load,

although
I

just got one of these errors with 10 in the pool. I also wonder whether
it's some sort of "dirty" connection in the pool, is this

possible?
i.e.

something like:

1) Expensive query in web page
2) user hits stop, recordset still "in" the connection
3) connection goes back in pool
4) another thread gets conn
5) conn is in weird state, all bets off (also would explain errors like
missing columns, sort of)

I realize this kind of diagnosis is indistinguishable from

guessing,
but
you

never know

Turn on Perfmon and watch the connection pool. Is it constantly growing
over
time or does it flatten out after a few minutes under load? Is

the
network

stable? Are you connecting over a LAN, WAN or web? If you lose a connection
due to IO problems on the wire, you can get this type of exception.

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers

no
rights.

__________________________________

I've seen other posts about such errors, but never a solution.

It
seems
to

appear randomly under load. We are pretty vigilant about "using"
IDataReader's and Connections, so it's pretty unlikely that

we're
doing

something outwardly wrong that would excuse this sort of

behavior.
The

stack is:

TARGETSITE: System.Data.SqlClient.SqlDataReader
ExecuteReader(System.Data.CommandBehavior,
System.Data.SqlClient.RunBehavior, Boolean)

STACKTRACE: at
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior cmdBehavior,
RunBehavior runBehavior, Boolean returnStream)
at System.Data.SqlClient.SqlCommand.ExecuteReader()
at Web.MyPage.Page_Load(Object sender, EventArgs e)
<removed by me>

Perhaps related, we also see random errors like this about

columns
we
KNOW

exist in recordsets that are being returned:

MESSAGE: EventId
SOURCE: System.Data
TARGETSITE: Int32 GetOrdinal(System.String)
STACKTRACE: at System.Data.Common.FieldNameLookup.GetOrdinal(String
fieldName)
at System.Data.SqlClient.SqlDataReader.GetOrdinal(String name)
at MyObject..ctor(IDataReader eventReader, EventData eventData)
<removed by me>

This is definitely unacceptable behavior from ADO, SQL Server,

or
both...
No

code that an end user writes in a managed language should be

able
to
create

this kind of untraceable havoc.

Any thoughts?

 

[William \(Bill\) Vaughn]

When I visited the power shack at a firebase and smelled a hot generator, I
knew it was time to buy another generator to spread the load. I think you
could spend quite a bit of time working on streamlining your server, but you
might find it best to add another server and build a farm--especially if it
really is CPU bound. If you have Extended Stored procedures, you might get
some help with Yukon... but by then it might be too late.
That said, there are lots of other things to look at. First, I would hire
Kimberly Tripp and get her to take a look at your Indexes, procedure cache
hits and the other parameters that affect performance. More often than not,
it's the questions you ask that bring the server to its knees--the queries.
We'll be talking about all of these issues in October at our Immersion
conference in Chicago.
You might also try adding RAM to the systems. With more RAM, the chance that
a row or procedure is in cache increases.
I also do a perf talk at the Connections conference that discusses how to
tell if a SP is building a good reusable plan. The real key here is
_reusable_. If the plan does not fit the parameters, the QP won't change it.
There are a lot of factors to consider. Of course, you could ask some of
your customers to go away... but they might do that on their own if they
haven't already.

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________

 

[Max Metral]

I like this error even less. Anything interesting on this:

MESSAGE: A severe error occurred on the current command. The results, if
any, should be discarded.

SOURCE: .Net SqlClient Data Provider

STACKTRACE: at
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior cmdBehavior,
RunBehavior runBehavior, Boolean returnStream)

at System.Data.SqlClient.SqlCommand.ExecuteReader()

Max,
I believe from your description that you are running into Q814114. Please
contact PSS directly for the fix
http://support.microsoft.com/default.aspx?scid=/support/contact/default.asp
(this is a free call).

The bug addressed by this QFE is almost exactly what you describe, a dirty
connection gets in the pool. This only repros when the network you are using
is high traffic or high latency, this forces the network libraries to
include multiple TDS Packets per read. Normally on connection close we clean
the data waiting on the connection, there is a bug where we fail to clean up
multiple packet TDS under certain conditions.

I apologize for the obvious pain this is causing, let me know if this does
not fix your problem.
--
Angel Saenz-Badillos [MS] Managed Providers
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging about ADO.NET: http://weblogs.asp.net/angelsb/

LAN connection, which isn't especially loaded (.27% according to perfmon).
The pool stabilizes at about 80 connections under heavy load, although I
just got one of these errors with 10 in the pool. I also wonder whether
it's some sort of "dirty" connection in the pool, is this possible? i.e.
something like:

1) Expensive query in web page
2) user hits stop, recordset still "in" the connection
3) connection goes back in pool
4) another thread gets conn
5) conn is in weird state, all bets off (also would explain errors like
missing columns, sort of)

I realize this kind of diagnosis is indistinguishable from guessing, but you
never know

growing
over we
KNOW

 

[Angel Saenz-Badillos[MS]]

I take it that this is _after_ you apply the QFE patch?



--
Angel Saenz-Badillos [MS] Managed Providers
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging about ADO.NET: http://weblogs.asp.net/angelsb/

I like this error even less. Anything interesting on this:

MESSAGE: A severe error occurred on the current command. The results, if
any, should be discarded.

SOURCE: .Net SqlClient Data Provider

STACKTRACE: at
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior cmdBehavior,
RunBehavior runBehavior, Boolean returnStream)

at System.Data.SqlClient.SqlCommand.ExecuteReader()

Max,
I believe from your description that you are running into Q814114. Please
contact PSS directly for the fix

http://support.microsoft.com/default.aspx?scid=/support/contact/default.asp

(this is a free call).

The bug addressed by this QFE is almost exactly what you describe, a dirty
connection gets in the pool. This only repros when the network you are using
is high traffic or high latency, this forces the network libraries to
include multiple TDS Packets per read. Normally on connection close we clean
the data waiting on the connection, there is a bug where we fail to

clean

up
multiple packet TDS under certain conditions.

I apologize for the obvious pain this is causing, let me know if this does
not fix your problem.
--
Angel Saenz-Badillos [MS] Managed Providers
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging about ADO.NET: http://weblogs.asp.net/angelsb/

LAN connection, which isn't especially loaded (.27% according to perfmon).
The pool stabilizes at about 80 connections under heavy load, although I
just got one of these errors with 10 in the pool. I also wonder whether
it's some sort of "dirty" connection in the pool, is this possible? i.e.
something like:

1) Expensive query in web page
2) user hits stop, recordset still "in" the connection
3) connection goes back in pool
4) another thread gets conn
5) conn is in weird state, all bets off (also would explain errors like
missing columns, sort of)

I realize this kind of diagnosis is indistinguishable from guessing,

but
you

never know

Turn on Perfmon and watch the connection pool. Is it constantly growing
over
time or does it flatten out after a few minutes under load? Is the network
stable? Are you connecting over a LAN, WAN or web? If you lose a
connection
due to IO problems on the wire, you can get this type of exception.

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no
rights.
__________________________________

I've seen other posts about such errors, but never a solution. It seems
to
appear randomly under load. We are pretty vigilant about "using"
IDataReader's and Connections, so it's pretty unlikely that we're doing
something outwardly wrong that would excuse this sort of behavior. The
stack is:

TARGETSITE: System.Data.SqlClient.SqlDataReader
ExecuteReader(System.Data.CommandBehavior,
System.Data.SqlClient.RunBehavior, Boolean)

STACKTRACE: at
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior
cmdBehavior,
RunBehavior runBehavior, Boolean returnStream)
at System.Data.SqlClient.SqlCommand.ExecuteReader()
at Web.MyPage.Page_Load(Object sender, EventArgs e)
<removed by me>

Perhaps related, we also see random errors like this about columns we
KNOW
exist in recordsets that are being returned:

MESSAGE: EventId
SOURCE: System.Data
TARGETSITE: Int32 GetOrdinal(System.String)
STACKTRACE: at System.Data.Common.FieldNameLookup.GetOrdinal(String
fieldName)
at System.Data.SqlClient.SqlDataReader.GetOrdinal(String name)
at MyObject..ctor(IDataReader eventReader, EventData eventData)
<removed by me>

This is definitely unacceptable behavior from ADO, SQL Server, or
both...
No
code that an end user writes in a managed language should be able to
create
this kind of untraceable havoc.

Any thoughts?

 

[Max Metral]

Another note... I was monitoring perf when this error occurred. The DB CPU
went through the roof, and other connections failed. So here's my newest
flaky theory:

1) A "fatal error" happens, because of some bug somewhere, probably MSFTs in
ADO.Net, but who knows.
2) The connection pool gets tossed out and rebuilt
3) Everybody and their mother tries to connect to the db
4) The db CPU gets pegged
5) Goto step 1.

One thing bolstering this theory: The perf counter "SqlClient: Current #
pooled and nonpooled connections" on one of the web servers reads 415, even
though there are only 30 conns to the db overall.

I like this error even less. Anything interesting on this:

MESSAGE: A severe error occurred on the current command. The results, if
any, should be discarded.

SOURCE: .Net SqlClient Data Provider

STACKTRACE: at
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior cmdBehavior,
RunBehavior runBehavior, Boolean returnStream)

at System.Data.SqlClient.SqlCommand.ExecuteReader()

Max,
I believe from your description that you are running into Q814114. Please
contact PSS directly for the fix

http://support.microsoft.com/default.aspx?scid=/support/contact/default.asp

(this is a free call).

The bug addressed by this QFE is almost exactly what you describe, a dirty
connection gets in the pool. This only repros when the network you are using
is high traffic or high latency, this forces the network libraries to
include multiple TDS Packets per read. Normally on connection close we clean
the data waiting on the connection, there is a bug where we fail to

clean

up
multiple packet TDS under certain conditions.

I apologize for the obvious pain this is causing, let me know if this does
not fix your problem.
--
Angel Saenz-Badillos [MS] Managed Providers
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging about ADO.NET: http://weblogs.asp.net/angelsb/

LAN connection, which isn't especially loaded (.27% according to perfmon).
The pool stabilizes at about 80 connections under heavy load, although I
just got one of these errors with 10 in the pool. I also wonder whether
it's some sort of "dirty" connection in the pool, is this possible? i.e.
something like:

1) Expensive query in web page
2) user hits stop, recordset still "in" the connection
3) connection goes back in pool
4) another thread gets conn
5) conn is in weird state, all bets off (also would explain errors like
missing columns, sort of)

I realize this kind of diagnosis is indistinguishable from guessing,

but
you

never know

Turn on Perfmon and watch the connection pool. Is it constantly growing
over
time or does it flatten out after a few minutes under load? Is the network
stable? Are you connecting over a LAN, WAN or web? If you lose a
connection
due to IO problems on the wire, you can get this type of exception.

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no
rights.
__________________________________

I've seen other posts about such errors, but never a solution. It seems
to
appear randomly under load. We are pretty vigilant about "using"
IDataReader's and Connections, so it's pretty unlikely that we're doing
something outwardly wrong that would excuse this sort of behavior. The
stack is:

TARGETSITE: System.Data.SqlClient.SqlDataReader
ExecuteReader(System.Data.CommandBehavior,
System.Data.SqlClient.RunBehavior, Boolean)

STACKTRACE: at
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior
cmdBehavior,
RunBehavior runBehavior, Boolean returnStream)
at System.Data.SqlClient.SqlCommand.ExecuteReader()
at Web.MyPage.Page_Load(Object sender, EventArgs e)
<removed by me>

Perhaps related, we also see random errors like this about columns we
KNOW
exist in recordsets that are being returned:

MESSAGE: EventId
SOURCE: System.Data
TARGETSITE: Int32 GetOrdinal(System.String)
STACKTRACE: at System.Data.Common.FieldNameLookup.GetOrdinal(String
fieldName)
at System.Data.SqlClient.SqlDataReader.GetOrdinal(String name)
at MyObject..ctor(IDataReader eventReader, EventData eventData)
<removed by me>

This is definitely unacceptable behavior from ADO, SQL Server, or
both...
No
code that an end user writes in a managed language should be able to
create
this kind of untraceable havoc.

Any thoughts?

 

[Angel Saenz-Badillos[MS]]

Max,
Can you verify the connection number with the server side performance
counters? The client side counters are unfortunately next to useless because
of this: http://support.microsoft.com/default.aspx?scid=kb;en-us;Q314429

--
Angel Saenz-Badillos [MS] Managed Providers
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging about ADO.NET: http://weblogs.asp.net/angelsb/

Another note... I was monitoring perf when this error occurred. The DB CPU
went through the roof, and other connections failed. So here's my newest
flaky theory:

1) A "fatal error" happens, because of some bug somewhere, probably MSFTs in
ADO.Net, but who knows.
2) The connection pool gets tossed out and rebuilt
3) Everybody and their mother tries to connect to the db
4) The db CPU gets pegged
5) Goto step 1.

One thing bolstering this theory: The perf counter "SqlClient: Current #
pooled and nonpooled connections" on one of the web servers reads 415, even
though there are only 30 conns to the db overall.

I like this error even less. Anything interesting on this:

MESSAGE: A severe error occurred on the current command. The results, if
any, should be discarded.

SOURCE: .Net SqlClient Data Provider

STACKTRACE: at
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior cmdBehavior,
RunBehavior runBehavior, Boolean returnStream)

at System.Data.SqlClient.SqlCommand.ExecuteReader()

http://support.microsoft.com/default.aspx?scid=/support/contact/default.asp

(this is a free call).

The bug addressed by this QFE is almost exactly what you describe, a dirty
connection gets in the pool. This only repros when the network you are using
is high traffic or high latency, this forces the network libraries to
include multiple TDS Packets per read. Normally on connection close we clean
the data waiting on the connection, there is a bug where we fail to

clean

up
multiple packet TDS under certain conditions.

I apologize for the obvious pain this is causing, let me know if this does
not fix your problem.
--
Angel Saenz-Badillos [MS] Managed Providers
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging about ADO.NET: http://weblogs.asp.net/angelsb/




LAN connection, which isn't especially loaded (.27% according to perfmon).
The pool stabilizes at about 80 connections under heavy load,

although

I columns
we

able

to

 

[A Zmajus]

Experienced exactly the same behavior though in our case it does not
take that much load to reproduce. Very erratic. Very unnerving. Very,
very, scarry!!!

 

[Ananth]

Hi All

I am also facing the same problem " Internal connection fatal error" . Angel Saenz-Badillos do you have any update. how to resolve this.

Thanks
Ananth.

From http://www.developmentnow.com/g/7_2004_7_0_0_132599/Internal-connection-fatal-error.ht

Posted via DevelopmentNow.com Group
http://www.developmentnow.com