How to Add Performance Counters to your .NET Core Application

Ever had the problem where your .NET Core application was slow and you just weren’t sure why? Or maybe you just wanted to monitor your application that’s running in production so you could see how it’s performing? What you’re looking for is data on how your application is running. If you want to get that, you’re going to want to know how to add performance counters to your .NET Core application.

To put this in a “DevOps nerd” kind of a way, this is all about enabling observability.

TL;DR – here’s the source code

How Does Application Insights Fit In?

Now before I go too far into this, I want to mention Application Insights in Azure. If you’re writing a web application or a Web API application, you can get a lot of great performance info without having to do much of anything just by configuring Application Insights. For a lot of use cases, that’s good enough.

But there are limits to what Application Insights can do. Or maybe you don’t want to use Application Insights or can’t use it for some reason. Or you’ve chosen Application Insights and you need to publish a custom metric that can’t automatically get captured. Or maybe you’re doing local performance tuning and the reporting delay for Application Insights makes it just too hard to debug your performance issues.

If any of those things are true for you, you’ll want to start instrumenting your application with metrics.

PerfMon Metrics for .NET Core

Back in the olden days before .NET Core — you know — back when we were writing .NET Framework apps for Windows, we’d instrument our app so that our application would emit PerfMon events. PerfMon is a tool in Windows that captures performance data about a process and displays the info for the user. It’s super helpful for tracking down performance issues and for monitoring the performance of live applications. In fact, I used to write a lot about how to implement Perfmon counters and how to load test your applications.

But all of that was focused on Windows. And .NET Core is cross-platform. So what’s the answer? How do we do performance metrics in .NET Core applications?

The answer starts with System.Diagnostics.Metrics and a tool called dotnet-counters.

Meters & Metrics in .NET Core

This stuff in .NET Core has gone through some iterations and changes. (In fact, there are going to be a bunch of additional changes to Metrics coming in .NET 8.) The latest and greatest uses System.Diagnostics.Metrics and emits data that’s usable by OpenTelemetry. It’s always good when .NET Core works with popular open-source projects and standard. So while it can be a little confusing right now to figure out, I think the solution is pretty-well baked.

NOTE: What you DON’T want to do is start implementing performance tracking using EventCounters. While this isn’t officially deprecated, it seems like the EventCounters options are on the path to a dead end.

When you start implementing using System.Diagnostics.Metrics, everything starts with an instance of a Meter. Meters have a name. The code below creates an instance of a Meter called “myapp.performanceinfo”. The meter is the container off of which you’ll report the actual performance data.

_meter = new Meter("myapp.performanceinfo");

Report Your Performance Data using Instruments

The Meter is the container but doesn’t have any data of its own. When you want to report data from your application, you’ll attach Instruments.

When I’m doing performance analysis for an application, I typically like to know for each controller and/or method:

• Number of successful calls per second
• Number of errors per second
• Average execution duration in milliseconds

For the number of successful calls and number of errors, I’ll typically use a Counter<T>. For the execution duration in milliseconds, I’ll use a Histogram<T>. The instruments also get names that are based on their parent Meter and the Meter has utility methods for creating the instruments.

Avoiding Metric Code Sprawl & Staying Organized

If your application is simple, then you create some of these meters and instruments and make some method calls. It’s pretty easy. But if your application is even minimally complex, this code ends up getting spread all over the place in the application. And it’s super repetitive, too so the risk of suddenly having a zillion lines of duplicate instrumentation code is very high.

When I’m instrumenting an application, I usually am focusing on instrumenting the Controllers in my app and the methods within those controllers. In order to avoid this sprawl and avoid having tons of duplicate code, I created two classes: ControllerPerformanceMetrics.cs and MethodMetrics.cs.

ControllerPerformanceMetrics<T> handles the setup and instancing of the Meter for the Controller class. MethodMetrics encapsulates the initialization logic for the Instruments that hang off of the parent Meter. In .NET 7 and below, the instancing of these Meters and Instruments can be a little tricky because access to the Meter is always done via a static (singleton) instance. To avoid having to create 9 billion static instances, ControllerPerformanceMetric<T> allows me to record performance data dynamically by method name.

Instrumenting the Controller

Instrumenting the Controller methods is really easy. The code below is a sample action for an ASP.NET Core MVC application. I’ve put the relevant parts in bold. To get the execution duration for the method, I use an instance Stopwatch. After I am done with the relevant operation for the controller, I call Stop() on the stopwatch and then report the information the ControllerPerformanceMetrics<T> instance for this controller method using the RecordRequest() method. That call to RecordRequest() takes the name of the method that we’re reporting for and the number of milliseconds it took to run.

If an error happens during execution, I call RecordError() to record the fact that we encountered an error. I’ll point out that I’m not capturing the details of the error in this code sample for simplicity. For this kind of metrics gathering code, I usually don’t care about the errors themselves, I just want to see trends in the number of errors reported. If I’ve got a bunch of errors, I’ll use logging to figure out what’s actually going on.

public IActionResult Index(bool throwException = false)
{
    try
    {
        if (throwException == true)
        {
            throw new Exception("This is a test exception.");
        }

        var stopwatch = new Stopwatch();
        stopwatch.Start();


        var model = new RandomDelayViewModel();

        model.DelayDuration = WaitForRandomAmountOfTime();

        stopwatch.Stop();


        ControllerPerformanceMetrics<HomeController>.Log.RecordRequest(
            nameof(Index), stopwatch.ElapsedMilliseconds);


        return View(model);
    }
    catch (Exception ex)
    {
        _logger.LogError(ex, "Unhandled exception in HomeController.Index().");

        ControllerPerformanceMetrics<HomeController>.Log.RecordError(
            nameof(Index));

        
        return View(new RandomDelayViewModel() { Message = ex.ToString() });
    }
}

Capturing & Viewing the Metrics Data in Real-Time with dotnet-counters

The dotnet-counters application is a dotnet tool that’s downloadable/installable via NuGet. When you run your application and run dotnet-counters, you’ll get an output that looks like the following.

To start the dotnet-counters app, you’ll go to the command line and run dotnet-counters monitor for your application’s process. NOTE: your application should already be running!

Once you’ve started dotnet-counters, it will display the data for any of the Meters you told it to capture. One thing to note is that the list of counters isn’t hardcoded at startup so if your application hasn’t published data for a meter or instrument, it probably won’t show up in the dotnet-counters display. As soon as your app publishes instrument data for that instrument name and/or meter name, it’ll appear.

Sample Application

I’ve created a simple proof-of-concept application so that you can understand the code and see it in action in dotnet-counters. The source code is available on GitHub.

The first step is to install the dotnet-counters application. To do that, open a command line terminal and run the following command:

dotnet tool install --global dotnet-counters

After you’ve installed that, you can run the sample application.

To run it on Windows, open a Powershell window and run the run-and-collect-counters.ps1 script. This will start the web application and then start the data collection in dotnet-counters. Open a browser for the port show in the terminal window and start clicking around on the actions.

To run it on Mac or Linux, first open a terminal window and run run-web-app.sh. Then open a second terminal window and run collect-counters.sh.

Summary

Monitoring your .NET Core application — also known as observability — probably requires you to create Meters and Instruments for your application. The sample code has classes that help make that simple and maintainable. Once you’ve instrumented your code, you’ll use dotnet-counters to view the live data for your app.

I hope this helps.

-Ben

Performance problems in your .NET Core applications got you down? Not sure how to instrument your applications for observability? Not sure how to get going with Application Insights? Need some help tracking down and solving performance problems in your .NET Core applications? We can help. Drop us a line at info@benday.com.