How to benchmark Umbraco

Hello fellow Umbracians and welcome back to my Garage!
As I was playing around with the Benchmark.net Nuget package a lot lately I was thinking if it wasn’t possible to benchmark actual Umbraco code aswell.
Turns out it is possible!
And as it is rather straight forward I am going to show you how to set up everything yourself!

Prerequisites

To make the setup and later testing easier I would recommend you to create a normal Umbraco project first, unless you already have one you can use, as we will need to copy some files over to the benchmark project. Also since the Benchmark.net tests are only possible to run as a Console Application, you will not have access to the backoffice otherwise.

Project

Create a basic Console Application and add references to the BenchmarkDotNet and Umbraco.Cms nuget packages.

  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.13.1" />
    <PackageReference Include="Umbraco.Cms" Version="10.0.0" />
  </ItemGroup>

Program.cs

In the Program.cs file we need to tell Benchmark.net where it should look for benchmarks tests.

using BenchmarkDotNet.Running;

public class Program
{
    public static void Main(string[] args)
    {
        BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run();
    }
}

Startup.cs

From your existing Umbraco project copy over the Startup.cs file into the Benchmark project.

Since Benchmark.Net runs its tests in a subdirectory of the bin folder it can’t find the default Media folder location as this points to a directory some levels above the directory it gets executed.
Thats why we have to change the MediaFileSystem and point it to a folder within the executing directory, it's important here to make sure that this folder really exists thats why I do a Directory.CreateDirectory just to be safe.

All you have to change in your copied Startup.cs is the ConfigureServices method by extending it with the .SetMediaFileSystem extension.

public void ConfigureServices(IServiceCollection services)
        {
#pragma warning disable IDE0022 // Use expression body for methods
            services.AddUmbraco(_env, _config)
                .AddBackOffice()
                .AddWebsite()
                .AddComposers()
                .SetMediaFileSystem((factory) =>
                {
                    Umbraco.Cms.Core.Hosting.IHostingEnvironment hostingEnvironment = factory.GetRequiredService<Umbraco.Cms.Core.Hosting.IHostingEnvironment>();
                    ILogger<PhysicalFileSystem> logger = factory.GetRequiredService<ILogger<PhysicalFileSystem>>();
                    GlobalSettings globalSettings = factory.GetRequiredService<IOptions<GlobalSettings>>().Value;

                    var rootPath = Path.IsPathRooted(globalSettings.UmbracoMediaPhysicalRootPath) ? globalSettings.UmbracoMediaPhysicalRootPath : hostingEnvironment.MapPathWebRoot(globalSettings.UmbracoMediaPhysicalRootPath);
                    Directory.CreateDirectory(rootPath);

                    var environment = factory.GetRequiredService<Umbraco.Cms.Core.Hosting.IHostingEnvironment>();
                    return new PhysicalFileSystem(factory.GetRequiredService<IIOHelper>(),
                        environment,
                        factory.GetRequiredService<ILogger<PhysicalFileSystem>>(),
                        rootPath,
                        environment.ToAbsolute("/Media"));
                })
                .Build();
#pragma warning restore IDE0022 // Use expression body for methods

        }

Appsettings.json

Copy over the appsettings.json file from your Umbraco project and make sure that the correct connectionstring is configured aswell as that the Build Action for the file is set to Content and Copy to Output Directory is set to Copy if newer or Copy always

UmbracoSetup.cs

Thats a static helper class in which the “magic” happens and we initialize Umbraco.
At first glance it will look like a Program.cs file you would find in any Umbraco project.
But if you look a bit closer there are actually some differences.
First off there is a CreateWebHostBuilder method instead of CreateHostBuilder.
Thats because we are in a Console Application and thus need to tell the application to create a WebHostBuilder. This WebHostBuilder we have to give a url to use, you can use any url here as it is not important, and also we need to tell it to use our modified Startup.

In the Startup method we then use this WebHostBuilder use Build() and then Start() on it. It is very important to use Start and not Run here as Run would listen for logs on the url we gave it and therefore never run any test.

using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Hosting;

namespace Benchmarking
{
    internal static class UmbracoSetup
    {
        public static void Startup()
        {
            CreateHostBuilder()
                .Build()
                .Start();
        }

        public static IWebHostBuilder CreateHostBuilder() =>
            Host.CreateDefaultBuilder()
                .ConfigureUmbracoDefaults()
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseUrls("http://localhost:5002");
                    webBuilder.UseStaticWebAssets();
                    webBuilder.UseStartup<Startup>();
                });
    }
}

Benchmark

Now that everything is setup we can actually start and write our benchmark test.
As an example I will show you how you can benchmark the serialization of a ContentType.
For this we will need to get the an instance of the IContentTypeService aswell as of the IEntityXmlSerializer, since we can’t use dependency injection here we have to use the StaticServiceProvider Umbraco provides us.

Next we need to start up Umbraco and then get the ContentType we want to serialize. This is done in the Setup method which is decorated with the [GlobalSetup] attribute which indicates that this method will only be executed once before any benchmarking is done.

Now we only have to write our benchmark test to actually have something to execute.

And then we are done!

using BenchmarkDotNet.Attributes;
using Microsoft.Extensions.DependencyInjection;
using Umbraco.Cms.Core.Models;
using Umbraco.Cms.Core.Services;
using Umbraco.Cms.Web.Common.DependencyInjection;

namespace Benchmarking.Benchmarks
{
    [MemoryDiagnoser]
    public class XmlSerializerBenchmark
    {
        private static IContentTypeService _service = StaticServiceProvider.Instance.GetRequiredService<IContentTypeService>();
        private static IEntityXmlSerializer _xmlSerializer = StaticServiceProvider.Instance.GetRequiredService<IEntityXmlSerializer>();

        private IContentType type;

        [GlobalSetup]
        public void Setup()
        {
            UmbracoSetup.Startup();

            type = _service.GetAll().FirstOrDefault() ?? throw new Exception();
        }

        [Benchmark]
        public void Serialize()
        {
            _xmlSerializer.Serialize(type);
        }
    }
}

Now you can run your application in Release mode and see how it performs!

Conclusion

You should always keep in mind that it does not make sense to benchmark methods that make a call to the database as such calls are effected by more than just the quality of the code.
If you have to get something from the database always do it in the Setup method, like shown in my example.
While this means that a lot of the code still can’t be performance tested, there are definitly a lot of new opportunities for it, like the XmlSerializer or a lot of extension methods.

With that beeing said it was a lot of fun getting all of this to run and then play around with it.
I hope I will not be the only one who will be enjoying this!