EPiWiki.se  - EPiServer notes shared with others
 

Troubleshoot a high CPU usage of an .NET application server

[Edit]
Troubleshoot a high CPU usage of an .NET web server.

References


Troubleshoot high cpu usage when the cache is purged (Importent for EPiServer CMS sites).

Symptoms:


Over 80% CPU utilization on the application server.
The PerfMon counter "ASP.NET => cache total entries" is stable with a high value.

Salvation


Test if it’s possible to reproduce the issue in a test environment, if it is - troubleshoot the application (the next step can help even in a developing environment).
In the easy cases there is only one functionality that causes this issue then the easiest way to troubleshoot the application is by using a profiler product. I think the easiest tool for this is JetBrains dotTrace it cost but I think it worse it. Visual Studio has one built in but I haven’t figure out how I can get it working.

With WinDbg


The hard way to do this is by taking several hang dumps with WinDbg when the utilization of the processor is over 80%

List threads CPU time



!runaway

List the stack trace to see what the application is doing



!clrstack

Using StackDump


A middle way is to use StackDump to list all active .Net threads on a running application. This program has a low impact on the web server so it possible to run it in a production environment.
1) Download StackDump and save it to a folder on the server you want to investigate.
2) Get process identity with process explorer or task manager.
3) List all managed threads for the process

Example



StackDump 4711

Generates the output:
This output says that fifteen threads are running for the process. Four of the threads are running managed code.
The first managed thread is rewriting HTML for the friendly URL rewriter.
The second one sends a remote event.
The third is a thread waiting for EPiServer scheduler.
The forth tread is waiting for EPiServer workflow.

OS Thread Id:7936
OS Thread Id:4480
OS Thread Id:3548
OS Thread Id:6648
OS Thread Id:8728
OS Thread Id:4388
OS Thread Id:9048
  System.Management.ManagementObjectCollection.ManagementObjectEnumerator.MoveNext(N/A)
  ElektroPost.Licensing.RestrictionTypes.MACRestriction.ValidateManagementCollection(ElektroPost.Licensing.RestrictionTypes.MACRestrictionN/A)
  ElektroPost.Licensing.RestrictionTypes.MACRestriction.Setup(ElektroPost.Licensing.RestrictionTypes.MACRestriction)
  ElektroPost.Licensing.LicenseData.Validate(ElektroPost.Licensing.LicenseData)
  EPiServer.Core.Licensing.Test()
  EPiServer.LocalPageProvider.CheckLicense(N/A)
  EPiServer.LocalPageProvider.GetLocalPage(N/AN/AN/A)
  EPiServer.Core.PageProviderBase.<>c__DisplayClass4.<GetPageInternal>b__3(N/A)
  EPiServer.Core.OptimisticCache`1.Read(System.StringN/A)
  EPiServer.Core.PageProviderBase.GetPageInternal(N/AN/AN/A)
  EPiServer.Core.PageProviderBase.GetPage(N/AN/AN/A)
  EPiServer.DataFactory.GetPage(EPiServer.DataFactoryN/AN/A)
  EPiServer.Web.FriendlyUrlRewriteProvider.GetFriendlyPagePath(N/AN/A)
  EPiServer.Web.FriendlyUrlRewriteProvider.ConvertToExternalInternal(N/AEPiServer.UrlBuilderN/AN/A)
  EPiServer.Web.FriendlyUrlRewriteProvider.ConvertToExternal(N/AN/AN/AN/A)
  EPiServer.Web.FriendlyHtmlRewriteToExternal.HtmlRewriteUrl(N/AN/AN/AN/AN/ASystem.Object&)
  EPiServer.Web.FriendlyHtmlRewriteToExternal.rewritePipe_HtmlRewriteUrl(N/AN/AEPiServer.Web.HtmlRewritePipe.MyHtmlRewriteEventArgs)
  EPiServer.Web.RewritePipe.OnHtmlRewriteUrl(N/AN/A)
  EPiServer.Web.HtmlRewritePipe.WriteElement(N/AN/AN/AN/A)
  EPiServer.Web.HtmlRewritePipe.RewriteToEnd(N/AN/ASystem.IO.StreamWriter)
  EPiServer.Web.HtmlRewriteStream.Close(EPiServer.Web.HtmlRewriteStream)
  System.Web.HttpWriter.FilterIntegrated(N/ASystem.BooleanN/A)
  System.Web.HttpResponse.FilterOutput(System.Web.HttpResponse)
  System.Web.HttpApplication.CallFilterExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute(System.Web.HttpApplication.CallFilterExecutionStep)
  System.Web.HttpApplication.ExecuteStep(ASP.global_asaxSystem.Web.HttpApplication.CallFilterExecutionStepSystem.Boolean&)
  System.Web.HttpApplication.PipelineStepManager.ResumeSteps(System.Web.HttpApplication.PipelineStepManagerN/A)
  System.Web.HttpApplication.BeginProcessRequestNotification(N/AN/AN/A)
  System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.HttpRuntimeSystem.Web.Hosting.IIS7WorkerRequestSystem.Web.HttpContext)
  System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(N/ASystem.IntPtrN/AN/A)
  System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(N/AN/AN/AN/A)
  System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(N/AN/AN/AN/A)
  System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(N/AN/AN/AN/A)
OS Thread Id:6584
OS Thread Id:9740
  System.Net.Sockets.Socket.DoConnect(N/AN/AN/A)
  System.Net.Sockets.Socket.Connect(N/AN/A)
  System.ServiceModel.Channels.SocketConnectionInitiator.Connect(System.ServiceModel.Channels.SocketConnectionInitiatorSystem.UriN/A)
  System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(N/AN/AN/A)
  System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.DuplexConnectionPoolHelperN/A)
  System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(System.ServiceModel.Channels.ClientFramingDuplexSessionChannelN/A)
  System.ServiceModel.Channels.CommunicationObject.Open(System.ServiceModel.Channels.ClientFramingDuplexSessionChannelN/A)
  System.ServiceModel.Channels.ServiceChannel.OnOpen(N/AN/A)
  System.ServiceModel.Channels.CommunicationObject.Open(System.ServiceModel.Channels.ServiceChannelN/A)
  System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(System.ServiceModel.Channels.ServiceChannel.CallOnceManagerN/AN/A)
  System.ServiceModel.Channels.ServiceChannel.EnsureOpened(N/AN/A)
  System.ServiceModel.Channels.ServiceChannel.Call(System.ServiceModel.Channels.ServiceChannelN/AN/ASystem.ServiceModel.Dispatcher.ProxyOperationRuntimeN/ASystem.Object[]N/A)
  System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(N/AN/AN/A)
  System.ServiceModel.Channels.ServiceChannelProxy.Invoke(System.ServiceModel.Channels.ServiceChannelProxySystem.Runtime.Remoting.Messaging.Message)
  System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(N/AN/AN/A)
  EPiServer.Events.Proxies.EventReplicationProxy.RaiseEvent(N/AN/AN/AN/AN/AN/AN/A)
  EPiServer.Events.Remote.EventReplicationWCFProviders.RaiseEvent(N/ASystem.GuidSystem.StringSystem.Int32System.Byte[]System.GuidSystem.String)
  EPiServer.Events.Remote.RemoteEventsManager.RaiseEventsThreadMain(N/A)
  System.Threading.ExecutionContext.runTryCode(N/A)
  System.Threading.ExecutionContext.Run(N/AN/AN/A)
  System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(N/A)
  System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(N/A)
OS Thread Id:8704
OS Thread Id:9868
OS Thread Id:1364
OS Thread Id:2364
OS Thread Id:7184
  Microsoft.Samples.Runtime.Remoting.Channels.Pipe.PipeConnection.WaitForConnect(N/A)
  Microsoft.Samples.Runtime.Remoting.Channels.Pipe.PipeServerChannel.ListenerMain(Microsoft.Samples.Runtime.Remoting.Channels.Pipe.PipeServerChannel)
  System.Threading.ExecutionContext.runTryCode(N/A)
  System.Threading.ExecutionContext.Run(N/AN/AN/A)
  System.Threading.ThreadHelper.ThreadStart(N/A)
OS Thread Id:8724
  System.Threading.WaitHandle.WaitOne(N/AN/AN/A)
  EPiServer.WorkflowFoundation.Services.CommunicationService.SendEventsOnThread()
  System.Threading.ExecutionContext.runTryCode(N/A)
  System.Threading.ExecutionContext.Run(N/AN/AN/A)
  System.Threading.ThreadHelper.ThreadStart(N/A)

C:\Temp\StackDump>
Version author:
Mattias Lövström

EPiServer version

All