When benchmarking against multiple TFMs including a Native AOT one, BenchmarkDotNet gets confused about framework identifiers

[akamsteeg]

(I'm using BenchmarkDotNet v0.14. The issue occurs on Windows, I haven't tested it on Linux or Mac OS.)

I have a Program.cs inside a benchmark project that looks like this:

public static class Program
{
  public static void Main(string[] args)
  {
    var config = GetConfig();
    BenchmarkSwitcher
      .FromAssembly(Assembly.GetExecutingAssembly())
      .Run(args, config);
  }

  private static ManualConfig GetConfig()
  {
    var job = Job.Default;

    var config = ManualConfig.Create(DefaultConfig.Instance)
      .AddJob(
        job.WithToolchain(CsProjCoreToolchain.NetCoreApp80).AsBaseline(),
        Job.Default.WithRuntime(NativeAotRuntime.Net80),
        job.WithToolchain(CsProjCoreToolchain.NetCoreApp60)
        )
      .AddDiagnoser(MemoryDiagnoser.Default);

    if (Environment.OSVersion.Platform == PlatformID.Win32NT)
    {
      config.AddJob(job.WithToolchain(CsProjClassicNetToolchain.Net481));
    }

    config.SummaryStyle = SummaryStyle.Default
      .WithRatioStyle(RatioStyle.Percentage);

    config.AddValidator(JitOptimizationsValidator.FailOnError); // Fail when any of the referenced assemblies are not optimized

    config.WithOrderer(new DefaultOrderer(SummaryOrderPolicy.FastestToSlowest));

    return config;
  }
}

So I'm running benchmarks against .NET 8 as my baseline, .NET 8 with Native AOT, .NET 6 and .NET 4.8.1 when on Windows. When running this using dotnet build -c Release && dotnet run -c Release -f net8.0 the final report in the console shows this:

// * Summary *

BenchmarkDotNet v0.14.0, Windows 11 (10.0.22631.3958/23H2/2023Update/SunValley3)
AMD Ryzen 7 7800X3D, 1 CPU, 16 logical and 8 physical cores
.NET SDK 9.0.100-preview.6.24328.19
  [Host]     : .NET 8.0.7 (8.0.724.31311), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-IRRDUC : .NET 8.0.7 (8.0.724.31311), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-BCZWNO : .NET 8.0.5, X64 NativeAOT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-EMODDW : .NET 6.0.32 (6.0.3224.31407), X64 RyuJIT AVX2
  Job-LCVGYL : .NET Framework 4.8.1 (4.8.9256.0), X64 RyuJIT VectorSize=256


| Method                        | Runtime       | password             | Mean       | Error    | StdDev   | Ratio    | RatioSD | Gen0   | Gen1   | Allocated | Alloc Ratio |
|------------------------------ |-------------- |--------------------- |-----------:|---------:|---------:|---------:|--------:|-------:|-------:|----------:|------------:|
| GetKAnonimityPartsForPassword | .NET 8.0      | -&HxcB_d             |   277.4 ns |  4.09 ns |  3.83 ns | baseline |         | 0.0043 |      - |     232 B |             |
| GetKAnonimityPartsForPassword | NativeAOT 8.0 | -&HxcB_d             |   284.4 ns |  5.69 ns |  7.20 ns |      +3% |    2.8% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | -&HxcB_d             |   288.1 ns |  4.49 ns |  4.20 ns |      +4% |    1.9% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | -&HxcB_d             | 3,249.2 ns | 54.70 ns | 51.16 ns |  +1,072% |    2.0% | 0.7477 | 0.0038 |    4710 B |     +1,930% |
|                               |               |                      |            |          |          |          |         |        |        |           |             |
| GetKAnonimityPartsForPassword | .NET 8.0      | -&Hxc(...)QbuAz [64] |   361.9 ns |  7.22 ns |  6.75 ns | baseline |         | 0.0043 |      - |     232 B |             |
| GetKAnonimityPartsForPassword | NativeAOT 8.0 | -&Hxc(...)QbuAz [64] |   362.2 ns |  4.28 ns |  4.00 ns |      +0% |    2.1% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | -&Hxc(...)QbuAz [64] |   369.0 ns |  3.93 ns |  3.28 ns |      +2% |    2.0% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | -&Hxc(...)QbuAz [64] | 3,274.8 ns | 62.43 ns | 61.32 ns |    +805% |    2.6% | 0.7553 | 0.0038 |    4766 B |     +1,954% |
|                               |               |                      |            |          |          |          |         |        |        |           |             |
| GetKAnonimityPartsForPassword | .NET 8.0      | これはパスワードです           |   305.4 ns |  3.88 ns |  3.44 ns | baseline |         | 0.0043 |      - |     232 B |             |
| GetKAnonimityPartsForPassword | NativeAOT 8.0 | これはパスワードです           |   329.3 ns |  6.50 ns |  7.49 ns |      +8% |    2.5% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | これはパスワードです           |   330.8 ns |  3.53 ns |  3.30 ns |      +8% |    1.4% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | これはパスワードです           | 3,248.5 ns | 42.48 ns | 39.73 ns |    +964% |    1.6% | 0.7515 | 0.0038 |    4734 B |     +1,941% |

In the first part of the summary, where the jobs are listed, it correctly identifies the runtimes used. However, in the overview of the metrics per benchmark/runtime combination, it gets confused about it and only lists .NET 8.0 or NativeAOT 8.0 and not .NET 6 and .NET 4.8.1. The metrics are okay though, so it looks like just a display issue for the runtime names.

⚠️ Removing the AOT job (Job.Default.WithRuntime(NativeAotRuntime.Net80)) solves this issue so it seems to be related to having Native AOT benchmarks mixed with 'normal' ones.

For reproduction the code I ran is on Github: akamsteeg/AtleX.HaveIBeenPwned@c6c6cd4. Running it with dotnet build -c Release && dotnet run -c Release -f net8.0 and picking any benchmark should reproduce this. It's consistent for me on two Windows 11 machines.