EPiWiki.se  - EPiServer notes shared with others
 

Troubleshooting a low CPU hang in community 3.2

[Edit]
Investigation of a low CPUhang in community 3.2 can be easy when using stackdump.exe to investigate it.

Problem description:


Date: 2010-04-14
EPiServer product: EPiServer community 3.2
OS: Windows 2008 .NET 3.5
Database: Microsoft SQL 2008
The web site stops responding, low CPU on both database and application servers.

Investigate what the application hangs on:


I was lazy and use the application stack dump to see what the application was doing when it dosent deliver web pages and my request to it only get time out.
In the task manager I added the pid column in show column, the w3wp process to investigate was easy to find sense I absolutely consume the most of all memory and has the pid set to 4711.
I execute stackdump.exe four times so it would be possible to se differences in the .NET stacks over time.

Stackdump.exe 4711 > log1.txt
Stackdump.exe 4711 > log2.txt
Stackdump.exe 4711 > log3.txt
Stackdump.exe 4711 > log4.txt

After looking in to the .NET stacks either was hanging on mostly three different functions in the code
1) Calls to an external web service

OS Thread Id:11552
  System.Net.Sockets.Socket.Receive(N/AN/AN/AN/AN/AN/A)
  ...
  HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(...

2) And the other one Getting images

OS Thread Id:5840
  SNINativeMethodWrapper.SNIReadSync(N/A)
  ...
  EPiServer.Common.Data.DatabaseHandler.GetReader(N/AN/A)
  MyCode.GetImagesForUser(...
  ...

3) And a third thread (most frequent) trying to read images from the cache

OS Thread Id:7844
  EPiServer.Common.Cache.CacheHandler.GetCachedObjectImplementation...
  EPiServer.Common.Cache.CacheHandler.GetCachedObject(N/AN/AN/A)
  EPiServer.Community.ImageGallery.ImageGalleryHandler.GetImage(N/A)
  MyCode.GetImage(System.Int32)

So it seems that something is locking the database from reading pictures, and that lock stops the cache from deliver it from the cache.
After some harder investigation a web service call that looks very strange was found:

OS Thread Id:7956
  System.Net.Sockets.Socket.Receive(N/AN/AN/AN/AN/AN/A)
  ...
  MyCode.AddToIndexImage(N/A)
  ...
  EPiServer.Community.ImageGallery.ImageGalleryHandler.OnNewImage(N/A)
  EPiServer.Community.ImageGallery.Data.ImageGalleryFactory.AddImage(N/A)
  EPiServer.Community.ImageGallery.ImageGalleryHandler.AddImage(N/AN/A)
  MyCode.AddImage(N/A)
  ...
  MyCode.HttpHandlers.ImageUpload.ProcessRequest(N/A)
  ...
  System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(N/AN/AN/AN/A)

This means that:


A web request that uploads an image to the community connect to a event that ends up in a web service call.
When looking with reflector on the code EPiServer.Community.ImageGallery.ImageGalleryHandler I find out that the event OnNewImage is handled inside the transaction. And locks down almost the whole image handling until the call to the web service are finished.

Solution:


Make a in memory map of added image and update the index with the web service outside the transaction, for example in the end request event.

Remarks:


It probably should be a good idea for EPiServer to make events that are handled inside a database transaction more concrete for the developer and name it something like OnNewImageInTransaction or something similar.
Version author:
Mattias Lövström

EPiServer version

'EPiServer CMS 5'