Unable to Retrieve Benchmarks Using Coreruns

[AvishaiDotan]

Environment Setup

I attempted to set up the environment for running benchmarks to complete this issue by building two CoreRun executables:

windows.x64.Debug
windows.x64.Release
The builds were successful, and no errors were encountered during the process. I followed the instructions provided in the documentation for runtime repository.

Benchmark Example

Here is the benchmark code I used:

> [DisassemblyDiagnoser]
> public class IntroDisassembly
> {
>     private const int Iterations = 10;
> 
>     [Benchmark]
>     public string StringConcatenationWithPlus()
>     {
>         string result = "";
>         for (int i = 0; i < Iterations; i++)
>         {
>             result += i.ToString();
>         }
>         return result;
>     }
> 
>     [Benchmark]
>     public string StringConcatenationWithStringBuilder()
>     {
>         var sb = new System.Text.StringBuilder();
>         for (int i = 0; i < Iterations; i++)
>         {
>             sb.Append(i);
>         }
>         return sb.ToString();
>     }
> }
> 
> class Program
> {
>     static void Main(string[] args)
>     {
>         var summary = BenchmarkRunner.Run<IntroDisassembly>(null, args);
>     }
> }

Command Used

I ran the benchmark using the following command:
dotnet run -c Release --corerun some-path\windows.x64.Release\corerun.exe some-path\windows.x64.Debug\corerun.exe

Issue Observed

Despite the setup being seemingly correct, the benchmarks did not execute successfully. Here is the output:

// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Found 4 benchmark(s) in total *****
// ***** Building 2 exe(s) in Parallel: Start   *****
// ***** Done, took 00:01:26 (86.92 sec)   *****
// Found 4 benchmarks:
//   IntroDisassembly.StringConcatenationWithPlus: Job-YCGGKX(Toolchain=\windows.x64.Debug\corerun.exe)
//   IntroDisassembly.StringConcatenationWithStringBuilder: Job-YCGGKX(Toolchain=\windows.x64.Debug\corerun.exe)
//   IntroDisassembly.StringConcatenationWithPlus: Job-GKFKQX(Toolchain=\windows.x64.Release\corerun.exe)
//   IntroDisassembly.StringConcatenationWithStringBuilder: Job-GKFKQX(Toolchain=\windows.x64.Release\corerun.exe)

Setup power plan (GUID: 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c FriendlyName: High performance)
// **************************
// Benchmark: IntroDisassembly.StringConcatenationWithPlus: Job-YCGGKX(Toolchain=\windows.x64.Debug\corerun.exe)
// *** Execute ***
// Launch: 1 / 1
// Execute: F:\OpenSource\runtime\artifacts\bin\coreclr\b53bf288-4775-4a8f-a902-6d613279c5a9\corerun.exe 5ed8e74d-4c51-43a3-b724-0dbe9f39ca6d.dll --anonymousPipes 1496 1756 --benchmarkName BenchmarkDemo.IntroDisassembly.StringConcatenationWithPlus --job Toolchain=\windows.x64.Debug\corerun.exe --benchmarkId 0 in F:\OpenSource\Tests\BenchmarkDemo\bin\Release\net8.0\5ed8e74d-4c51-43a3-b724-0dbe9f39ca6d\bin\Release\net8.0\publish
ExitCode != 0 and no results reported
No Workload Results were obtained from the run.
// Benchmark Process 35920 has exited with code -532462766.

// ** Remained 3 (75.0%) benchmark(s) to run. Estimated finish 2024-12-21 20:34 (0h 0m from now) **
Setup power plan (GUID: 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c FriendlyName: High performance)
// **************************
// Benchmark: IntroDisassembly.StringConcatenationWithStringBuilder: Job-YCGGKX(Toolchain=\windows.x64.Debug\corerun.exe)
// *** Execute ***
// Launch: 1 / 1
// Execute: F:\OpenSource\runtime\artifacts\bin\coreclr\b53bf288-4775-4a8f-a902-6d613279c5a9\corerun.exe 5ed8e74d-4c51-43a3-b724-0dbe9f39ca6d.dll --anonymousPipes 2000 1996 --benchmarkName BenchmarkDemo.IntroDisassembly.StringConcatenationWithStringBuilder --job Toolchain=\windows.x64.Debug\corerun.exe --benchmarkId 1 in F:\OpenSource\Tests\BenchmarkDemo\bin\Release\net8.0\5ed8e74d-4c51-43a3-b724-0dbe9f39ca6d\bin\Release\net8.0\publish
ExitCode != 0 and no results reported
No Workload Results were obtained from the run.
// Benchmark Process 1604 has exited with code -532462766.

// ** Remained 2 (50.0%) benchmark(s) to run. Estimated finish 2024-12-21 20:34 (0h 0m from now) **
Setup power plan (GUID: 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c FriendlyName: High performance)
// **************************
// Benchmark: IntroDisassembly.StringConcatenationWithPlus: Job-GKFKQX(Toolchain=\windows.x64.Release\corerun.exe)
// *** Execute ***
// Launch: 1 / 1
// Execute: F:\OpenSource\runtime\artifacts\bin\coreclr\6f8b4d58-36dd-4e1f-b34a-3b9edee9857a\corerun.exe 571de9ef-b65b-4415-8329-ba2a6c33caec.dll --anonymousPipes 2028 1784 --benchmarkName BenchmarkDemo.IntroDisassembly.StringConcatenationWithPlus --job Toolchain=\windows.x64.Release\corerun.exe --benchmarkId 0 in F:\OpenSource\Tests\BenchmarkDemo\bin\Release\net8.0\571de9ef-b65b-4415-8329-ba2a6c33caec\bin\Release\net8.0\publish
ExitCode != 0 and no results reported
No Workload Results were obtained from the run.
// Benchmark Process 36464 has exited with code -532462766.

// ** Remained 1 (25.0%) benchmark(s) to run. Estimated finish 2024-12-21 20:34 (0h 0m from now) **
Setup power plan (GUID: 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c FriendlyName: High performance)
// **************************
// Benchmark: IntroDisassembly.StringConcatenationWithStringBuilder: Job-GKFKQX(Toolchain=\windows.x64.Release\corerun.exe)
// *** Execute ***
// Launch: 1 / 1
// Execute: F:\OpenSource\runtime\artifacts\bin\coreclr\6f8b4d58-36dd-4e1f-b34a-3b9edee9857a\corerun.exe 571de9ef-b65b-4415-8329-ba2a6c33caec.dll --anonymousPipes 1496 2004 --benchmarkName BenchmarkDemo.IntroDisassembly.StringConcatenationWithStringBuilder --job Toolchain=\windows.x64.Release\corerun.exe --benchmarkId 1 in F:\OpenSource\Tests\BenchmarkDemo\bin\Release\net8.0\571de9ef-b65b-4415-8329-ba2a6c33caec\bin\Release\net8.0\publish
ExitCode != 0 and no results reported
No Workload Results were obtained from the run.
// Benchmark Process 35880 has exited with code -532462766.

// ** Remained 0 (0.0%) benchmark(s) to run. Estimated finish 2024-12-21 20:34 (0h 0m from now) **
Successfully reverted power plan (GUID: 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c FriendlyName: High performance)
// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts\results\BenchmarkDemo.IntroDisassembly-report.csv
  BenchmarkDotNet.Artifacts\results\BenchmarkDemo.IntroDisassembly-report-github.md
  BenchmarkDotNet.Artifacts\results\BenchmarkDemo.IntroDisassembly-report.html
  BenchmarkDotNet.Artifacts\results\BenchmarkDemo.IntroDisassembly-asm.md

// * Detailed results *
IntroDisassembly.StringConcatenationWithPlus: Job-YCGGKX(Toolchain=\windows.x64.Debug\corerun.exe)
Runtime = ; GC =
There are not any results runs

IntroDisassembly.StringConcatenationWithPlus: Job-GKFKQX(Toolchain=\windows.x64.Release\corerun.exe)
Runtime = ; GC =
There are not any results runs

IntroDisassembly.StringConcatenationWithStringBuilder: Job-YCGGKX(Toolchain=\windows.x64.Debug\corerun.exe)
Runtime = ; GC =
There are not any results runs

IntroDisassembly.StringConcatenationWithStringBuilder: Job-GKFKQX(Toolchain=\windows.x64.Release\corerun.exe)
Runtime = ; GC =
There are not any results runs

// * Summary *

BenchmarkDotNet v0.14.0, Windows 10 (10.0.19045.5247/22H2/2022Update)
AMD Ryzen 5 3600, 1 CPU, 12 logical and 6 physical cores
.NET SDK 9.0.101
  [Host] : .NET 8.0.11 (8.0.1124.51707), X64 RyuJIT AVX2


| Method                               | Job        | Toolchain                        | Mean | Error | Ratio | RatioSD |
|------------------------------------- |----------- |--------------------------------- |-----:|------:|------:|--------:|
| StringConcatenationWithPlus          | Job-YCGGKX | \windows.x64.Debug\corerun.exe   |   NA |    NA |     ? |       ? |
| StringConcatenationWithPlus          | Job-GKFKQX | \windows.x64.Release\corerun.exe |   NA |    NA |     ? |       ? |
|                                      |            |                                  |      |       |       |         |
| StringConcatenationWithStringBuilder | Job-YCGGKX | \windows.x64.Debug\corerun.exe   |   NA |    NA |     ? |       ? |
| StringConcatenationWithStringBuilder | Job-GKFKQX | \windows.x64.Release\corerun.exe |   NA |    NA |     ? |       ? |

Benchmarks with issues:
  IntroDisassembly.StringConcatenationWithPlus: Job-YCGGKX(Toolchain=\windows.x64.Debug\corerun.exe)
  IntroDisassembly.StringConcatenationWithPlus: Job-GKFKQX(Toolchain=\windows.x64.Release\corerun.exe)
  IntroDisassembly.StringConcatenationWithStringBuilder: Job-YCGGKX(Toolchain=\windows.x64.Debug\corerun.exe)
  IntroDisassembly.StringConcatenationWithStringBuilder: Job-GKFKQX(Toolchain=\windows.x64.Release\corerun.exe)

// * Warnings *
BaselineCustomAnalyzer
  Summary -> A question mark '?' symbol indicates that it was not possible to compute the (Ratio, RatioSD) column(s) because the baseline value is too close to zero.
Environment
  Summary -> Detected error exit code from one of the benchmarks. It might be caused by following antivirus software:
        - Windows Defender (windowsdefender://)
        - Kaspersky Total Security (C:\Program Files (x86)\Kaspersky Lab\Kaspersky Total Security 21.3\wmiav.exe)
Use InProcessEmitToolchain or InProcessNoEmitToolchain to avoid new process creation.


// * Legends *
  Mean    : Arithmetic mean of all measurements
  Error   : Half of 99.9% confidence interval
  Ratio   : Mean of the ratio distribution ([Current]/[Baseline])
  RatioSD : Standard deviation of the ratio distribution ([Current]/[Baseline])
  1 ns    : 1 Nanosecond (0.000000001 sec)

// * Diagnostic Output - DisassemblyDiagnoser *
No benchmarks were disassembled

// ***** BenchmarkRunner: End *****
Run time: 00:00:17 (17.43 sec), executed benchmarks: 4

Global total time: 00:01:44 (104.67 sec), executed benchmarks: 4
// * Artifacts cleanup *
Artifacts cleanup is finished