Exception Logging Strategy

[Cramer]

I plan to implement an exception logging feature in an ASP.NET Web
application that writes encountered exceptions to disk. The exception data
will be stored as XML.

I am planning on having each exception written to its own XML file. I plan
to NOT append exceptions to a single XML file because I don't want for the
exception logging component to have to deal with contention for access to
the file. This is possible when, for example, multiple sessions of the Web
application are encountering the exception at the same time... each would
want access to the file. So rather than having "the file" (one file), I'm
thinking that it might be good to have the exception logging component keep
track of which exceptions have already been logged (within a sliding window)
and log only the exceptions that have not yet been logged. This would
prevent excessive logging of the exact same exception.

Thoughts?

I can't be the first person to come up with a robust exception logging
capability for ASP.NET Web applications - in which thousands of sessions
could encounter the exact same exception simultaneously (or within seconds
of each other).

Any suggestions to help me not "re-invent the wheel" on this would be much
appreciated.

-Cramer

 

[OHM \( One Handed Man \)]

Well my first question would be why on earth store exceptions in an xml data
file when you can use SQL ( or other ) to store it in, that way you wont
have any contentions to deal with and you can use reporting services or
crystal reports to report errors etc.

You can categorize the errors under things like, Security, Data, Validation
etc and have severitiy codes, etc. If you use reflection you can also store
the class and member function which contains the line which generated the
exception.

I also dont understand why you would not want to log exceptions which had
already been logged, they would be seperate events and could have been
caused for different reasons. When you are trying to diagnose problems in
your applications, you need as much information as possible to help
establish a route to a rapid remedy.

Thats my two euros worth anyway.

HTH

 

[Cramer]

Well my first question would be why on earth store exceptions in an xml
data file when you can use SQL ( or other ) to store it in, that way you
wont have any contentions to deal with and you can use reporting services
or crystal reports to report errors etc.

You assume that SQL Server is available. What would you do if connectivity
to the SQL Server was broken? Oh, let me guess - you'd write the exception
data to the local disk. At this point you can store it in a plain text file,
system log, or XML file. I'm going with XML for reasons that should be
obvious compared to the alternatives.

It's not relevant to the OP, but my plan is to have a separate process
periodically transfer the exception XML data to the SQL Server. So I agree
with the benefits you point out, but I completely disagree that the
exceptions should be logged directly to the SQL Server.

You can categorize the errors under things like, Security, Data,
Validation etc and have severitiy codes, etc. If you use reflection you
can also store the class and member function which contains the line
which generated the exception.

What? You seem to think that if I were to store exception data in an XML
file that it couldn't include all the detail you talk about.

The storage medium has nothing to do with whether I can use Reflection, log
security, have severity codes, etc.

I also dont understand why you would not want to log exceptions which had
already been logged,

Consider that this is an ASP.NET Web application that could have thousands
of concurrent users. Suppose it connects to a SQL Server database to get
data to display on pages. The connectivity to the SQL Server gets broken. I
don't need or want thousands of duplicate errors getting logged - regardless
of the storage medium.

Note that in my IOP I referred to a sliding window. That could be minutes to
hours. Tell me about a broken SQL Server connection once. I don't need to
know about the many different features that consequently broke and reported
an exception. The sliding window lets me know about the first that happens
and not the subsequent thousands. BTW, this sliding window only impacts
what's written to the XML files. The component that manages the sliding
window could keep track of the total number of exceptions NOT logged, so I
would still know the overall number of times the exception was encountered.

they would be seperate events and could have been caused for different
reasons.

Then, by definition, and my exception handling model, those are different
exceptions - each of which gets logged (once per sliding window).

When you are trying to diagnose problems in your applications, you need as
much information as possible to help establish a route to a rapid remedy.

Yes - but you also want a good signal-to-noise ratio. It's simply not
helpful to have redundant information redundant information redundant
information redundant information redundant information redundant
information redundant information redundant information redundant
information redundant information redundant information redundant
information redundant information redundant information redundant
information redundant information redundant information redundant
information redundant information redundant information redundant
information redundant information.

right?

 

[Arne Vajhøj]

I plan to implement an exception logging feature in an ASP.NET Web
application that writes encountered exceptions to disk. The exception data
will be stored as XML.

I am planning on having each exception written to its own XML file. I plan
to NOT append exceptions to a single XML file because I don't want for the
exception logging component to have to deal with contention for access to
the file. This is possible when, for example, multiple sessions of the Web
application are encountering the exception at the same time... each would
want access to the file. So rather than having "the file" (one file), I'm
thinking that it might be good to have the exception logging component keep
track of which exceptions have already been logged (within a sliding window)
and log only the exceptions that have not yet been logged. This would
prevent excessive logging of the exact same exception.

Thoughts?

I can't be the first person to come up with a robust exception logging
capability for ASP.NET Web applications - in which thousands of sessions
could encounter the exact same exception simultaneously (or within seconds
of each other).

Any suggestions to help me not "re-invent the wheel" on this would be much
appreciated.

log4net has a robust logging capability.

I don't like the multiple file approach.

Single file, database, syslog whatever would be better.

The synchronization overhead should not be a problem. Actually I would
expect the creating many files to have more overhead than synchronizing
writing to a single file.

I would log all exceptions. In case you want to figure out what happen,
then it is nice to have everything and not try to avoid redundancy.

Arne

 

[Arne Vajhøj]

You assume that SQL Server is available. What would you do if connectivity
to the SQL Server was broken? Oh, let me guess - you'd write the exception
data to the local disk. At this point you can store it in a plain text file,
system log, or XML file. I'm going with XML for reasons that should be
obvious compared to the alternatives.

For many applications if the database is down the entire app is
completely down until the database comes up again.

And usually it is not really that useful to have logged the
exceptions telling that the database was down.

It's not relevant to the OP, but my plan is to have a separate process
periodically transfer the exception XML data to the SQL Server. So I agree
with the benefits you point out, but I completely disagree that the
exceptions should be logged directly to the SQL Server.

It has been done many times before.

It can be done again.

Consider that this is an ASP.NET Web application that could have thousands
of concurrent users. Suppose it connects to a SQL Server database to get
data to display on pages. The connectivity to the SQL Server gets broken. I
don't need or want thousands of duplicate errors getting logged - regardless
of the storage medium.

Note that in my IOP I referred to a sliding window. That could be minutes to
hours. Tell me about a broken SQL Server connection once. I don't need to
know about the many different features that consequently broke and reported
an exception. The sliding window lets me know about the first that happens
and not the subsequent thousands. BTW, this sliding window only impacts
what's written to the XML files. The component that manages the sliding
window could keep track of the total number of exceptions NOT logged, so I
would still know the overall number of times the exception was encountered.


Then, by definition, and my exception handling model, those are different
exceptions - each of which gets logged (once per sliding window).


Yes - but you also want a good signal-to-noise ratio. It's simply not
helpful to have redundant information redundant information redundant
information redundant information redundant information redundant

I am still a bit skeptical about whether you will be able to code
logic that correctly will be able to guess whether an exception is
sufficient closely related to another exception, that the troubleshooter
will not be interested.

And disk space is rather cheap.

I would rather log MB's of stuff that is never needed than one day
sit there and "I wish I had ...".

Arne

 

[OHM \( One Handed Man \)]

You assume that SQL Server is available. What would you do if connectivity
to the SQL Server was broken? Oh, let me guess - you'd write the exception
data to the local disk.

What is it with you and broken SQL Servers, are they allways broken in your
part of the galaxy, and why cant you have an sql server instance on your
host machine ?

At this point you can store it in a plain text file, system log, or XML
file. I'm going with XML for reasons that should be obvious compared to the
alternatives.

1.) If you read my text, it says SQL ( or other ).
2.) What reasons should be obvious to me that you would choose XML over the
alternatives,
I would like to hear them provided you are going to be contructive rather
than attacking.

What? You seem to think that if I were to store exception data in an XML
file that it couldn't include all the detail you talk about.

No, I didnt say that at all. It was simply pointing out that you could.

The storage medium has nothing to do with whether I can use Reflection,
log security, have severity codes, etc.

Again, I never said it was.

Consider that this is an ASP.NET Web application that could have thousands
of concurrent users. Suppose it connects to a SQL Server database to get
data to display on pages. The connectivity to the SQL Server gets broken.
I don't need or want thousands of duplicate errors getting logged -
regardless of the storage medium.

There is a difference between duplicate errors and errors which relate to
the
same origin of failure. The challenge you seem to have here is how to
discern
between the two.

Note that in my IOP I referred to a sliding window. That could be minutes
to hours. Tell me about a broken SQL Server connection once. I don't need
to know about the many different features that consequently broke and
reported an exception. The sliding window lets me know about the first
that happens and not the subsequent thousands. BTW, this sliding window
only impacts what's written to the XML files. The component that manages
the sliding window could keep track of the total number of exceptions NOT
logged, so I would still know the overall number of times the exception
was encountered.



Then, by definition, and my exception handling model, those are different
exceptions - each of which gets logged (once per sliding window).

Again, I ask how will you know the difference, if its an sql error for
example,
you will need to know the number of the error message and perhaps more
explicit detail in order to know if the exception was caused by the same
root
failure or something else.

Yes - but you also want a good signal-to-noise ratio. It's simply not
information redundant information redundant information redundant

I see your point, and again it comes down to how to determine the difference
between
an error caused by a common failure and one which is not.

If we assume that you have some method of being able to seperate a single
causal failure
which results in mutiple exceptions, then what method would you use to
determine how that set
of commonly caused exceptions were different from another set. Same hour,
Same Day, Set resolved etc.

I do see your predicament, but linking exceptions with common causes may be
very difficult to do in
many cases expecially in a complex system. This is a little bit akin to a
helpdesk system where the help desk assistants
take calls and when a main server goes down, 500 calls for the same subject
get "Linked", the difference here is
that this is humans making a judgment for that linkage, but with computers,
you need to be more specific and that is
a lot harder in this scenario.

 

[Cramer]

You assume that SQL Server is available. What would you do if

What is it with you and broken SQL Servers, are they allways broken in
your
part of the galaxy, and why cant you have an sql server instance on your
host machine ?

Just because we *can* install SQL Server on a Web server doesn't mean we
*should*. In my case I can, but I don't want to have a busy Web server doing
any important tasks that have absolutely nothing to do with serving Web
pages.

And, in my part of the galaxy I don't always have broken or unreliable
connectivity to SQL Server. But, like for many systems, when the
connectivity is gone, that's a major problem that I want to know about as
soon as it happens.

BTW: I'm not attacking you - but I will point out bad ideas when I see
them... ideas like installing SQL Server on a Web server just for the sake
of logging exceptions (or even worse, doing more than logging exceptions). I
understand that this is a forum for freely exchanging ideas. It's also a
forum for critiquing ideas and pointing out both the good and the bad. You
don't need to be offended if one of your ideas is found to be less than
useful (like your most recent implication to install SQL Server on a Web
server. Nobody who knows what they are doing would ever recommend doing
that).

 

[Arne Vajhøj]

BTW: I'm not attacking you - but I will point out bad ideas when I see
them... ideas like installing SQL Server on a Web server just for the sake
of logging exceptions (or even worse, doing more than logging exceptions).

That is fair.

Nobody who knows what they are doing would ever recommend doing
that).

But this is more than pointing out it is a bad idea. This *is* a
personal attack.

Arne

 

[Alvin Bruney [ASP.NET MVP]]

the functionality is already implemented in detail in the Enterprise
Library. All the scenarios described in this thread are covered by design
time policy. I note here that this has to be coded into the logic if you
aren't using EL. Why re-invent the wheel? For EL, code does not decide where
to log, policy determines that and policy is set by the maintenance folk not
the programmer. Have a look at EL, I see it as your best option with very
little coding.

 

[Peter Bromberg [C# MVP]]

I really have to agree with some of the other posters here. In almost all
cases, if you are unable to write an exception log record to your database,
the rest of your application is already hosed anyway. I think what you're
suggesting represents a bit of "overkill", and is unlikely to buy you much
more than some extra headaches due to unneccessary complexity in your app.
-- Peter
To be a success, arm yourself with the tools you need and learn how to use
them.

Site: http://www.eggheadcafe.com
http://petesbloggerama.blogspot.com
http://ittyurl.net

 

[Cramer]

I really have to agree with some of the other posters here. In almost all
cases, if you are unable to write an exception log record to your
database,
the rest of your application is already hosed anyway.

Yes - but the implication then becomes that we don't log the fact that the
exception occurred at all ("we can't connect to the SQL Server, so don't log
the exception").

So then all we have to go on is knowledge that the application is hosed,
with no error log to look at as to why it's hosed, or what went wrong just
prior to connectivity going awry.

In my case I have a Windows server that looks for the XML exceptions. When
connectivity to the database is lost (and my app is hosed as you accurately
summed), then the Windows service tells me right away (sends a text message
to my cell phone). This happened only once in the last 3 years - but I had
the Web server up and running in a matter of minutes - with zero calls to
tech support. I doubt any of my customers even knew their sites were
offline.

I think what you're
suggesting represents a bit of "overkill", and is unlikely to buy you much
more than some extra headaches due to unneccessary complexity in your app.

Given the above production support scenario (which actually happened) I
don't consider it to be overkill. Maybe it would be if this were a hobby Web
site or I could afford to let angry customers tell me when their Web sites
are down. In my case I'm trying to be proactive.

I'm still wondering if anyone else who supports ASP.NET Web applications
does anything even remotely similar to what I'm doing for logging - or if
you all just write directly to SQL Server and therefore knowingly losing
knowledge of all exceptions that occur when the SQL Server is unavailable.

 

[Daniel Ruehle]

Simple answer, yes, there are others that do what your end goal seems to be.
Just maybe not the way you are proposing.

I typically don't use logging to determine if a system is down. Logs are
useful for figuring out what happened, not what is happening. I always use
monitoring to ensure that applications are running. Simply build test case
handlers that exercise key functions of your system. Connect to the
database, log a test user in, perform a query, run a report, etc. Assuming
all goes well, provide an appropriate response. There are many monitoring
tools out there that can do this easily. This method has the added benefit
that you may know about an outage before any clients receive an error where
logging only informs you when a problem occurs.

That said, log4net is an easy way to consolidate logs. You can configure it
live to write logs to files, database tables, web services or emails if you
like to get spammed. It takes care of all of the contention and can even be
configured to work correctly to write to files in a web garden. Of course
EL can as well, but it takes more setup and configuration. What I usually
do is make database connection exceptions cause Fatal level log entries
which I configure to send an email (to me or hopefully a support person) so
if the monitor doesn't catch it, someone will be notified immediately, and
no additional Windows service is required to monitor log files.

 

[Alexey Smirnov]

Any suggestions to help me not "re-invent the wheel" on this would be much
appreciated.

Look at the ELMAH project, it does logging to the SQL Server database
http://www.google.com/search?hl=en&q=elmah

 

[Microsoft Newsserver]

Exactly.

But for the record, I disagree that this is necessarily a bad idea, it
really depends on the circumstances. In my opinion, there are seldom bad
ideas, just ones which form a better solution better than others depending
on ones criteria.

You quote it as a "Busy Server" but where is you impirical evidence to
support the fact that adding an SQL instance would have any noticable impact
on performance. 'error' logging by its nature tends not to be an intensive
operation, if it did, then you have major issues, and in such, the
performance of the application is likely to be secondary to the faults which
precipitated any onslaught of the same.

If I revisit the OP, then it is clear that most of the detail related to
this has come out as a result of discussion rather than having been
presented as a cogent argument for your present strategy in the first
instance; although no doubt you will counterpoint that assertion.

You (OP) need in my opinion to be more objective and less emotional about
solving the issue. That way, having removed the emotion and having had
reasoned discussions with members of this forum, some concensus may drop out
and lead you to the most favourable solution.

 

[Arne Vajhøj]

Yes - but the implication then becomes that we don't log the fact that the
exception occurred at all ("we can't connect to the SQL Server, so don't log
the exception").

So then all we have to go on is knowledge that the application is hosed,
with no error log to look at as to why it's hosed, or what went wrong just
prior to connectivity going awry.

In my case I have a Windows server that looks for the XML exceptions. When
connectivity to the database is lost (and my app is hosed as you accurately
summed), then the Windows service tells me right away (sends a text message
to my cell phone). This happened only once in the last 3 years - but I had
the Web server up and running in a matter of minutes - with zero calls to
tech support. I doubt any of my customers even knew their sites were
offline.


Given the above production support scenario (which actually happened) I
don't consider it to be overkill. Maybe it would be if this were a hobby Web
site or I could afford to let angry customers tell me when their Web sites
are down. In my case I'm trying to be proactive.

I'm still wondering if anyone else who supports ASP.NET Web applications
does anything even remotely similar to what I'm doing for logging - or if
you all just write directly to SQL Server and therefore knowingly losing
knowledge of all exceptions that occur when the SQL Server is unavailable.

Application logging is for finding problems in the application.

To detect whether your SQLServer is running or not you should not use
your applications log file.

You should monitor the SQLServer directly for that.

Arne

 

[adda]

My suggestion would be to have a look at the Microsoft Enterprise
logging and exception block, more than powerful enough for your needs.

Chris

 

[Alexey Smirnov]

My suggestion would be to have a look at the Microsoft Enterprise
logging and exception block, more than powerful enough for your needs.

Chris







- Show quoted text -

Here's how to use it
http://www.codeproject.com/KB/aspnet/ExceptionHandling.aspx
http://msdn.microsoft.com/en-us/library/cc512464.aspx

 

[Alvin Bruney [ASP.NET MVP]]

capability for ASP.NET Web applications - in which thousands of sessionsyou have to be careful here, I believe EL simply renames the log file when a
clash occurs meaning you have to review log files and date stamps when
troubleshooting.

--

Regards,
Alvin Bruney [MVP ASP.NET]

[Shameless Author plug]
The O.W.C. Black Book, 2nd Edition
Exclusively on www.lulu.com/owc $19.99