Application Insights extensibility : tricks worth knowing #1

monitoring, diagnostics, azure, aspnetcore

By Cyprien ,

Application Insights is a great web application monitoring and analytics tool. If you are not familiar with it, you should definitely have a look. You can get it running, in a matter of minutes and you'll get invaluable troubleshooting data and metrics for just a couple of coins per month.

But, that's not all! It is a highly configurable tool and you can tune it to match you exact needs. I will share with you what we've learn across a blog serie demonstrating how to use different AppInsights extensibility endpoints with .Net core. Wi will start with telemetry filtering which can be very interesting to reduce noise, data volume and related costs.

Filtering out irrelevant telemetry events

On some environments, we are using Hangfire with the SQL Server Provider. Hangfire is a background job processing and scheduler for .net (and .net core). Now when you see Queues and SQL in the same sentence you are right to think about polling...

The thing is Application Insights will monitor your dependencies (and this is something we love about it). It will for instance log the sql queries. In our case it will monitor all the polling requests. Let's say it is configured to poll for new jobs every 15 seconds you are looking at roughly 650k requests per month per node. With that kind of volume, sampling will start kicking in and you might exceed the free data allowance, hence waste money on additional data. Given the fact they are not part of your code base and are 99.9% success, are you really interested in logging them? Probably not. Let's see how you can drop them quite easily.

One of the extensibility point is made of ITelemetryProcessor (docs).

public interface ITelemetryProcessor
{
    void Process(ITelemetry item);
}

It will allow you to hook into the telemetry processing pipeline to alter/enrich or drop telemetry. It works in a very similar way to the middlewares in aspnet core. As part of the AppInsights pipeline, your processor will be instantiated with a continuation ITelemetryProcessor. Your implementation of Process will run the custom logic and pass on the execution to the next processor. If you wish to drop the telemetry you just have to not call the next processor.

In our case we will need to identify Hangfire successful SQL traces and drop them entirely. Those telemetries will be DependencyTelemetry with its type set to SQL type and the Name set to database server | database name. Our Hangfire database always contain hangfire as part of their name.

So our TelemetryProcessor will look like this:

    public class FilterHangfireDependencyTracking : ITelemetryProcessor
    {
        public FilterHangfireDependencyTracking(ITelemetryProcessor next)
        {
            Next = next;
        }

        private ITelemetryProcessor Next { get; set; }

        public void Process(ITelemetry item)
        {
            var dependency = item as DependencyTelemetry;
            if (dependency != null)
            {
                // Drop successful SQL hangfire dependency calls
                if (dependency.Type == "SQL"
                    && dependency.Name.Contains("hangfire")
                    && dependency.Success.GetValueOrDefault(false))
                {
                    return;
                }
            }

            Next.Process(item);
        }
    }

You can now add this to the pipeline by adding the following to the Configure method in you Startup:

var configuration = app.ApplicationServices.GetService<TelemetryConfiguration>();

configuration.TelemetryProcessorChainBuilder
                .Use(next => new Utilities.FilterHangfireDependencyTracking(next))
                .Build();

Conclusion

As you can see, filtering logs is pretty straighforward. Stay tuned for our next post which will be about logging http headers. Things will start to get more... Let's say "interesting"!