Help needed with an issue related to ASP.Net and WCF Service

[Nadeem Ashraf]

Hi,

We are developing a web based application "UltraLearn.com" with a mix of
junior/senior Microsoft technologies. That includes Microsoft Silverlight,
ASP.Net Ajax and WCF/WF. Recently, we have hit the wall with some of
tested/proven technologies. The problem statement follows:

We are calling a WCF Service from our site pages to track user navigation.
The service in turn makes call to a gateway/server (using HttpWebRequest,
HttpWebResponse) which just logs entries into the database. A call to the
service is initiated from the Page_Init method of Master Page so that it is
ensured that page will be tracked even if there is some problem while
rendering the page. The call is asynchronous/non-blocking due to obvious
performance reason (Quick Load Test proved that asynchronous version affects
page's response
time significantly)

Here is the piece of code that is making the call from Page_Init method of
master page:

SiteTrackingServiceRef.SiteTrackingServiceClient trackingClient = new
SiteTrackingServiceRef.SiteTrackingServiceClient trackingClient();
trackingClient.StartTrackingCompleted += new
EventHandler<SiteTrackingServiceRef.StartTrackingCompletedEventArgs>(trackingClient_StartTrackingCompleted);
trackingClient.StartTrackingAsync(pageVisit);

The site works perfectly fine until you put some load onto it after which it
slows down extremely. At first, we thought that it was some networking issue
but different tests related to congestion, DNS entries, Dos attacks etc.
suggested against it. Then we thought that it was issue with IIS 6.0 running
on Win2k3 (our production environment), but IIS logs and performance
counters didn't suggest the same. Also, other sites on the same
IIS/environment run perfectly fine. Then we unit/load tested our WCF Service
individually to find out that
it wasn't causing bottleneck either leaving us clueless.

It may be worthwhile to mention that when the site starts slowing down after
some time with variable load on the site, our WCF Service
(SiteTrackingService) still does great job with logging the navigation. It's
just that something happens to the page loading (or server response time)

The following suggested solutions didn't impact the outcome in any ways.

1) WCF Service Throttling (Making the service Singleton, PerCall, PerSession
and/or adjusting MaxConcurrentCalls, MaxConcurrentSessions etc.)
2) Calling the WCF Service synchronously as opposed to asynchronously

Somebody also suggested to remove the WCF Service altogether and initiating
navigation logging directly from ASP.Net page. But we really need to
understand this strange behavior as our site is heavily dependent upon the
happy marriage of ASP.Net and WCF Services. We can avoid this issue here by
removing WCF Service altogether but not in other places. Really need some
expert advice/help from you guys soon.

Please note that most obvious performance counters related to CPU, Memory
and ASP.Net processing on the server shows system to be in healthy state
when the site slows down.

Thanks,
Nadeem

 

[bruce barker]

you are incorrectly using async processing on your page. an async call runs
on the current thread and does a callback when complete on the same thread.
unless you page request does a wait until the async completes and turns off
thread agility (aspcompat=true), during load the callback will happen on a
different request. you will also stack up async calls on the same thread.

if you use async processing, you need to switch to the
AddOnPreRenderCompleteAsync which is the only supported way to do async
processing on an asp.net page. this will run the aync request (and wait for
completion just before prerender).

you probably want to switch from async to a background thread pool to do
your logging.


-- bruce (sqlwork.com)