Add dotnet-trace to Debug ThreadPool Starvation #46251
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dotnet-policy-service
bot
added
dotnet-fundamentals/svc
community-contribution
noahfalk
reviewed
May 19, 2025
| @@ -7,7 +7,7 @@ ms.date: 04/19/2022 | |||
|
|
|||
| # Debug ThreadPool Starvation | |||
|
|
|||
| **This article applies to: ✔️** .NET Core 3.1 and later versions | |||
| **This article applies to: ✔️** .NET 9.0 and later versions | |||
There was a problem hiding this comment.
The metrics shown by default in dotnet-counters changed in .NET 9.0, but the article still refers to pre-9.0 metrics. I know it wasn't the focus of your change, but if we are changing this to a 9.0 guide we should change that part to make it consistent.
| - [Sample web app](/samples/dotnet/samples/diagnostic-scenarios) to demonstrate ThreadPool starvation behavior | ||
| - [Bombardier](https://github.com/codesenberg/bombardier/releases) to generate load for the sample web app | ||
| - [dotnet-counters](dotnet-counters.md) to observe performance counters | ||
| - [dotnet-stack](dotnet-stack.md) to examine thread stacks | ||
| - [dotnet-trace](dotnet-trace.md) to collect wait events | ||
| - Optional: [PerfView](https://github.com/microsoft/perfview/releases) to analyze .nettrace files |
There was a problem hiding this comment.
Suggested change
| - Optional: [PerfView](https://github.com/microsoft/perfview/releases) to analyze .nettrace files | |
| - Optional: [PerfView](https://github.com/microsoft/perfview/releases) to analyze the wait events |
its true that it analyzes .nettrace files of course, but this way its clearer that the output of dotnet-trace will be the input of PerfView.
| @@ -192,7 +177,9 @@ On .NET 6 you should observe the pool increase the thread count more quickly tha | |||
|
|
|||
| ## Resolving ThreadPool starvation | |||
|
|
|||
| To eliminate ThreadPool starvation, ThreadPool threads need to remain unblocked so that they're available to handle incoming work items. There are two ways to determine what each thread was doing, either using the [dotnet-stack](dotnet-stack.md) tool or capturing a dump with [dotnet-dump](dotnet-dump.md) that can be viewed in [Visual Studio](/visualstudio/debugger/using-dump-files). dotnet-stack can be faster because it shows the thread stacks immediately on the console, but Visual Studio dump debugging offers better visualizations that map frames to source, Just My Code can filter out runtime implementation frames, and the Parallel Stacks feature can help group large numbers of threads with similar stacks. This tutorial shows the dotnet-stack option. See the [diagnosing ThreadPool starvation tutorial video](/shows/on-net/diagnosing-thread-pool-exhaustion-issues-in-net-core-apps) for an example of investigating the thread stacks using Visual Studio. | |||
| To eliminate ThreadPool starvation, ThreadPool threads need to remain unblocked so that they're available to handle incoming work items. There are multiple ways to determine what each thread was doing: using the [dotnet-stack](dotnet-stack.md) tool, capturing a dump with [dotnet-dump](dotnet-dump.md) that can be viewed in [Visual Studio](/visualstudio/debugger/using-dump-files), or collecting wait events with [dotnet-trace](dotnet-trace.md). dotnet-stack can be faster because it shows the thread stacks immediately on the console, but Visual Studio dump debugging offers better visualizations that map frames to source, Just My Code can filter out runtime implementation frames, and the Parallel Stacks feature can help group large numbers of threads with similar stacks. This tutorial first shows the dotnet-stack option, then demonstrates how to use dotnet-trace. See the [diagnosing ThreadPool starvation tutorial video](/shows/on-net/diagnosing-thread-pool-exhaustion-issues-in-net-core-apps) for an example of investigating the thread stacks using Visual Studio. | |||
There was a problem hiding this comment.
Suggested change
| To eliminate ThreadPool starvation, ThreadPool threads need to remain unblocked so that they're available to handle incoming work items. There are multiple ways to determine what each thread was doing: using the [dotnet-stack](dotnet-stack.md) tool, capturing a dump with [dotnet-dump](dotnet-dump.md) that can be viewed in [Visual Studio](/visualstudio/debugger/using-dump-files), or collecting wait events with [dotnet-trace](dotnet-trace.md). dotnet-stack can be faster because it shows the thread stacks immediately on the console, but Visual Studio dump debugging offers better visualizations that map frames to source, Just My Code can filter out runtime implementation frames, and the Parallel Stacks feature can help group large numbers of threads with similar stacks. This tutorial first shows the dotnet-stack option, then demonstrates how to use dotnet-trace. See the [diagnosing ThreadPool starvation tutorial video](/shows/on-net/diagnosing-thread-pool-exhaustion-issues-in-net-core-apps) for an example of investigating the thread stacks using Visual Studio. | |
| To eliminate ThreadPool starvation, ThreadPool threads need to remain unblocked so that they're available to handle incoming work items. There are multiple ways to determine what each thread was doing. If the issue occurs only occasionally then collecting a trace with [dotnet-trace](dotnet-trace.md) is best to record application behavior over a period of time. If the issue is constantly occurring then you can use the [dotnet-stack](dotnet-stack.md) tool or capture a dump with [dotnet-dump](dotnet-dump.md) that can be viewed in [Visual Studio](/visualstudio/debugger/using-dump-files). dotnet-stack can be faster because it shows the thread stacks immediately on the console, but Visual Studio dump debugging offers better visualizations that map frames to source, Just My Code can filter out runtime implementation frames, and the Parallel Stacks feature can help group large numbers of threads with similar stacks. This tutorial shows the dotnet-stack and dotnet-trace options. See the [diagnosing ThreadPool starvation tutorial video](/shows/on-net/diagnosing-thread-pool-exhaustion-issues-in-net-core-apps) for an example of investigating the thread stacks using Visual Studio. |
| To eliminate ThreadPool starvation, ThreadPool threads need to remain unblocked so that they're available to handle incoming work items. There are two ways to determine what each thread was doing, either using the [dotnet-stack](dotnet-stack.md) tool or capturing a dump with [dotnet-dump](dotnet-dump.md) that can be viewed in [Visual Studio](/visualstudio/debugger/using-dump-files). dotnet-stack can be faster because it shows the thread stacks immediately on the console, but Visual Studio dump debugging offers better visualizations that map frames to source, Just My Code can filter out runtime implementation frames, and the Parallel Stacks feature can help group large numbers of threads with similar stacks. This tutorial shows the dotnet-stack option. See the [diagnosing ThreadPool starvation tutorial video](/shows/on-net/diagnosing-thread-pool-exhaustion-issues-in-net-core-apps) for an example of investigating the thread stacks using Visual Studio. | ||
| To eliminate ThreadPool starvation, ThreadPool threads need to remain unblocked so that they're available to handle incoming work items. There are multiple ways to determine what each thread was doing: using the [dotnet-stack](dotnet-stack.md) tool, capturing a dump with [dotnet-dump](dotnet-dump.md) that can be viewed in [Visual Studio](/visualstudio/debugger/using-dump-files), or collecting wait events with [dotnet-trace](dotnet-trace.md). dotnet-stack can be faster because it shows the thread stacks immediately on the console, but Visual Studio dump debugging offers better visualizations that map frames to source, Just My Code can filter out runtime implementation frames, and the Parallel Stacks feature can help group large numbers of threads with similar stacks. This tutorial first shows the dotnet-stack option, then demonstrates how to use dotnet-trace. See the [diagnosing ThreadPool starvation tutorial video](/shows/on-net/diagnosing-thread-pool-exhaustion-issues-in-net-core-apps) for an example of investigating the thread stacks using Visual Studio. | ||
|
|
||
| ### dotnet-stack |
There was a problem hiding this comment.
Suggested change
| ### dotnet-stack | |
| ### Diagnose a continuous issue with dotnet-stack |
| @@ -275,6 +262,67 @@ Thread (0x25968): | |||
| DiagnosticScenarios!testwebapi.Controllers.DiagScenarioController.TaskWait() | |||
| ``` | |||
|
|
|||
| ### dotnet-trace | |||
There was a problem hiding this comment.
Suggested change
| ### dotnet-trace | |
| ### Diagnose an intermittent issue with dotnet-trace |
|
|
||
| The dotnet-stack approach is effective only for obvious, consistent blocking operations that occur in every request. In some scenarios, the blocking could happen sporadically only every few minutes, making dotnet-stack less useful for diagnosing the issue. dotnet-trace can be used to collect events over a period of time and save them in a nettrace file that can be analyzed later. | ||
|
|
||
| There is one particular event that helps diagnosing thread pool starvation that was introduced in .NET 9, it's the WaitHandleWait event. It's emitted when a thread pool thread becomes blocked by operations such as sync-over-async calls (e.g. `Task.Result`, `Task.Wait`, `Task.GetAwaiter().GetResult()`) or by other locking operations like `lock`, `Monitor.Enter`, `ManualResetEventSlim.Wait`, or `SemaphoreSlim.Wait`. |
There was a problem hiding this comment.
Suggested change
| There is one particular event that helps diagnosing thread pool starvation that was introduced in .NET 9, it's the WaitHandleWait event. It's emitted when a thread pool thread becomes blocked by operations such as sync-over-async calls (e.g. `Task.Result`, `Task.Wait`, `Task.GetAwaiter().GetResult()`) or by other locking operations like `lock`, `Monitor.Enter`, `ManualResetEventSlim.Wait`, or `SemaphoreSlim.Wait`. | |
| There is one particular event that helps diagnosing thread pool starvation that was introduced in .NET 9, the WaitHandleWait event. It's emitted when a thread becomes blocked by operations such as sync-over-async calls (e.g. `Task.Result`, `Task.Wait`, `Task.GetAwaiter().GetResult()`) or by other locking operations like `lock`, `Monitor.Enter`, `ManualResetEventSlim.Wait`, or `SemaphoreSlim.Wait`. |
|
|
||
| Let's see how to use each tool to read the nettrace file. | ||
|
|
||
| ### Analyze a nettrace with Perfview |
There was a problem hiding this comment.
Suggested change
| ### Analyze a nettrace with Perfview | |
| #### Analyze a nettrace with Perfview |
|  | ||
| 7. This stack trace can be read as: `Task<T>.Result` emitted a WaitHandleWait event with a WaitSource MonitorWait (`Task<T>.Result` uses `Monitor.Wait` to perform a wait). It was called by `DiagScenarioController.TaskWait`, which was called by some lambda, which was called by some ASP.NET code | ||
|
|
||
| ### Analyze a nettrace with .NET Events Viewer |
There was a problem hiding this comment.
Suggested change
| ### Analyze a nettrace with .NET Events Viewer | |
| #### Analyze a nettrace with .NET Events Viewer |
|
|
||
| ### Analyze a nettrace with .NET Events Viewer | ||
|
|
||
| 1. Go to [verdie-g.github.io/dotnet-events-viewer](https://verdie-g.github.io/dotnet-events-viewer) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
In .NET 9, a new event was added (dotnet/runtime#94737) to help diagnose thread pool starvation. It is documented in dotnet-trace #40852.
In this PR, I'm adding a section to Debug ThreadPool Starvation to show how to collect this event, and analyze it using both PerfView and a cross-platform tool that I made in Blazor.
cc @kouvel @noahfalk who helped implementing this event
Internal previews