Troubleshoot a high CPU usage of an .NET web server.
Symptoms:
Over 80% CPU utilization on the application server.
Salvation
Test if it’s possible to reproduce the issue in a test environment, if it is - troubleshoot the application (the next step can help even in a developing environment).
In the easy cases there is only one functionality that causes this issue then the easiest way to troubleshoot the application is by using a profiler product. I think the easiest tool for this is
JetBrains dotTrace it cost but I think it worse it. Visual Studio has one built in but I haven’t figure out how I can get it working.
With WinDbg
The hard way to do this is by taking several hang dumps with WinDbg when the utilization of the processor is over 80%
List threads CPU time
!runaway
List the stack trace to see what the application is doing
!clrstack
Using StackDump
A middle way is to use
StackDump to list all active .Net threads on a running application. This program has a low impact on the web server so it possible to run it in a production environment.
1) Download StackDump and save it to a folder on the server you want to investigate.
2) Get process identity with process explorer or task manager.
3) List all managed threads for the process
Example
StackDump 4711
Generates the output:
This output says that fifteen threads are running for the process. Four of the threads are running managed code.
The first managed thread is rewriting HTML for the friendly URL rewriter.
The second one sends a remote event.
The third is a thread waiting for EPiServer scheduler.
The forth tread is waiting for EPiServer workflow.
OS Thread Id:7936
OS Thread Id:4480
OS Thread Id:3548
OS Thread Id:6648
OS Thread Id:8728
OS Thread Id:4388
OS Thread Id:9048
System.Management.ManagementObjectCollection.ManagementObjectEnumerator.MoveNext(N/A)
ElektroPost.Licensing.RestrictionTypes.MACRestriction.ValidateManagementCollection(ElektroPost.Licensing.RestrictionTypes.MACRestrictionN/A)
ElektroPost.Licensing.RestrictionTypes.MACRestriction.Setup(ElektroPost.Licensing.RestrictionTypes.MACRestriction)
ElektroPost.Licensing.LicenseData.Validate(ElektroPost.Licensing.LicenseData)
EPiServer.Core.Licensing.Test()
EPiServer.LocalPageProvider.CheckLicense(N/A)
EPiServer.LocalPageProvider.GetLocalPage(N/AN/AN/A)
EPiServer.Core.PageProviderBase.<>c__DisplayClass4.<GetPageInternal>b__3(N/A)
EPiServer.Core.OptimisticCache`1.Read(System.StringN/A)
EPiServer.Core.PageProviderBase.GetPageInternal(N/AN/AN/A)
EPiServer.Core.PageProviderBase.GetPage(N/AN/AN/A)
EPiServer.DataFactory.GetPage(EPiServer.DataFactoryN/AN/A)
EPiServer.Web.FriendlyUrlRewriteProvider.GetFriendlyPagePath(N/AN/A)
EPiServer.Web.FriendlyUrlRewriteProvider.ConvertToExternalInternal(N/AEPiServer.UrlBuilderN/AN/A)
EPiServer.Web.FriendlyUrlRewriteProvider.ConvertToExternal(N/AN/AN/AN/A)
EPiServer.Web.FriendlyHtmlRewriteToExternal.HtmlRewriteUrl(N/AN/AN/AN/AN/ASystem.Object&)
EPiServer.Web.FriendlyHtmlRewriteToExternal.rewritePipe_HtmlRewriteUrl(N/AN/AEPiServer.Web.HtmlRewritePipe.MyHtmlRewriteEventArgs)
EPiServer.Web.RewritePipe.OnHtmlRewriteUrl(N/AN/A)
EPiServer.Web.HtmlRewritePipe.WriteElement(N/AN/AN/AN/A)
EPiServer.Web.HtmlRewritePipe.RewriteToEnd(N/AN/ASystem.IO.StreamWriter)
EPiServer.Web.HtmlRewriteStream.Close(EPiServer.Web.HtmlRewriteStream)
System.Web.HttpWriter.FilterIntegrated(N/ASystem.BooleanN/A)
System.Web.HttpResponse.FilterOutput(System.Web.HttpResponse)
System.Web.HttpApplication.CallFilterExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute(System.Web.HttpApplication.CallFilterExecutionStep)
System.Web.HttpApplication.ExecuteStep(ASP.global_asaxSystem.Web.HttpApplication.CallFilterExecutionStepSystem.Boolean&)
System.Web.HttpApplication.PipelineStepManager.ResumeSteps(System.Web.HttpApplication.PipelineStepManagerN/A)
System.Web.HttpApplication.BeginProcessRequestNotification(N/AN/AN/A)
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.HttpRuntimeSystem.Web.Hosting.IIS7WorkerRequestSystem.Web.HttpContext)
System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(N/ASystem.IntPtrN/AN/A)
System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(N/AN/AN/AN/A)
System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(N/AN/AN/AN/A)
System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(N/AN/AN/AN/A)
OS Thread Id:6584
OS Thread Id:9740
System.Net.Sockets.Socket.DoConnect(N/AN/AN/A)
System.Net.Sockets.Socket.Connect(N/AN/A)
System.ServiceModel.Channels.SocketConnectionInitiator.Connect(System.ServiceModel.Channels.SocketConnectionInitiatorSystem.UriN/A)
System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(N/AN/AN/A)
System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.DuplexConnectionPoolHelperN/A)
System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(System.ServiceModel.Channels.ClientFramingDuplexSessionChannelN/A)
System.ServiceModel.Channels.CommunicationObject.Open(System.ServiceModel.Channels.ClientFramingDuplexSessionChannelN/A)
System.ServiceModel.Channels.ServiceChannel.OnOpen(N/AN/A)
System.ServiceModel.Channels.CommunicationObject.Open(System.ServiceModel.Channels.ServiceChannelN/A)
System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(System.ServiceModel.Channels.ServiceChannel.CallOnceManagerN/AN/A)
System.ServiceModel.Channels.ServiceChannel.EnsureOpened(N/AN/A)
System.ServiceModel.Channels.ServiceChannel.Call(System.ServiceModel.Channels.ServiceChannelN/AN/ASystem.ServiceModel.Dispatcher.ProxyOperationRuntimeN/ASystem.Object[]N/A)
System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(N/AN/AN/A)
System.ServiceModel.Channels.ServiceChannelProxy.Invoke(System.ServiceModel.Channels.ServiceChannelProxySystem.Runtime.Remoting.Messaging.Message)
System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(N/AN/AN/A)
EPiServer.Events.Proxies.EventReplicationProxy.RaiseEvent(N/AN/AN/AN/AN/AN/AN/A)
EPiServer.Events.Remote.EventReplicationWCFProviders.RaiseEvent(N/ASystem.GuidSystem.StringSystem.Int32System.Byte[]System.GuidSystem.String)
EPiServer.Events.Remote.RemoteEventsManager.RaiseEventsThreadMain(N/A)
System.Threading.ExecutionContext.runTryCode(N/A)
System.Threading.ExecutionContext.Run(N/AN/AN/A)
System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(N/A)
System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(N/A)
OS Thread Id:8704
OS Thread Id:9868
OS Thread Id:1364
OS Thread Id:2364
OS Thread Id:7184
Microsoft.Samples.Runtime.Remoting.Channels.Pipe.PipeConnection.WaitForConnect(N/A)
Microsoft.Samples.Runtime.Remoting.Channels.Pipe.PipeServerChannel.ListenerMain(Microsoft.Samples.Runtime.Remoting.Channels.Pipe.PipeServerChannel)
System.Threading.ExecutionContext.runTryCode(N/A)
System.Threading.ExecutionContext.Run(N/AN/AN/A)
System.Threading.ThreadHelper.ThreadStart(N/A)
OS Thread Id:8724
System.Threading.WaitHandle.WaitOne(N/AN/AN/A)
EPiServer.WorkflowFoundation.Services.CommunicationService.SendEventsOnThread()
System.Threading.ExecutionContext.runTryCode(N/A)
System.Threading.ExecutionContext.Run(N/AN/AN/A)
System.Threading.ThreadHelper.ThreadStart(N/A)
C:\Temp\StackDump>