Today I wanted to show you something that I hope could be a very useful addition to your .NET Core development toolbox, and that is an ultra-lightweight of doing performance benchmarking for your code using BenchmarkDotNet and dotnet-script.
We just released 0.19.0 of dotnet-script, that supports benchmarking, yesterday.
So what is it all about? π
Now, this is not really a post about how to get started with BenchmarkDotNet - it’s a pretty well known tool, and it has excellent documentation that can help you get started. It is a terrific benchmarking tool.
On a similar note, I don’t think I need to introduce dotnet-script again, chances are if you visit this blog sometimes, you know of it already - it’s a .NET Core C# script runner.
A C# script (CSX file) seems like a perfect vessel to use for benchmarking, because it is so simple - literally just a single file, where you dump your benchmarking code, and off you go. No ceremony, no project files, no Nuget set up, not even a Program class or static void Main. And of course dotnet-script is supported in OmniSharp, meaning you can get very good language service experience in VS Code for example.
So it all sounds like a perfect match, but until the last release, dotnet-script didn’t support running C# scripts under OptimizationLevel.Release, which is prerequisite for benchmarking.
The reason for this is that the ScriptCompiler from Roslyn runs in debug mode by default (it is actually hardcoded in there at the moment). While not ideal, there are some understandable reasons behind it, like for example the simple fact that probably in scripts you normally don’t care about maximum performance, but instead you want to have maximum stacktrace and diagnostic information.
However, in latest version of dotnet-script we allow you to override the Roslyn defaults and opt into running in Release mode, by using the -c or -configuration option.
dotnet script main.csx -c Release
How do I do it? π
So imagine you want to test the performance of Span.Slice() vs Array.Skip(). In order to do this with dotnet-script, you need a CSX file. You could just create one by hand, but probably the easiest is to create some local folder for your little “project” and call:
dotnet script init
This creates the default dotnet-script main.csx file as well as a hidden OmniSharp/VS Code config so that you could enjoy fully fledged language services and debugging capabilities in VS Code right at your fingertips.
Next, you need to reference BenchmarkDotNet and any packages you need to write your benchmark, from Nuget. In our case, it’s System.Memory, where Span can be found. dotnet-script supports inline Nuget references so it’s very easy to add Nuget packages.
Note: at the moment, due to limitation in Roslyn, the OmniSharp tooling doesn’t resolve Nuget packages in real-time. You will need to restart OmniSharp (ctrl+shift+P/cmd+shift+P and select “Restart OmniSharp”) to have them picked up by intellisense. This will be improved in the future.
#! "netcoreapp2.0"
#r "nuget:System.Memory, 4.5.0-preview1-26216-02"
#r "nuget:BenchmarkDotNet, 0.10.12"
// code will go here
What’s left is to just write our benchmark. We will use BenchmarkDotNet’s inprocess toolchain, as we will not emit a separate DLL. In order to run the test, we will use BenchmarkRunner, and since we are in a C# script, we can just invoke it straight away (global expressions and statements are allowed in CSX).
The full code sample is shown below.
#! "netcoreapp2.0"
#r "nuget:System.Memory, 4.5.0-preview1-26216-02"
#r "nuget:BenchmarkDotNet, 0.10.12"
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Attributes.Jobs;
using BenchmarkDotNet.Running;
BenchmarkRunner.Run<Test>();
[InProcess]
public class Test
{
int[] values = new[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };
[Benchmark]
public void Take()
{
var a = values.Skip(5);
}
[Benchmark]
public void Slice()
{
var span = new Span<int>(values, 0, values.Length);
var a = span.Slice(5, span.Length - 5);
}
}
To run the benchmark, execute:
dotnet script main.csx -c Release
You should see the usual BenchmarkDotNet output, for example:
BenchmarkDotNet=v0.10.12, OS=macOS 10.12.6 (16G29) [Darwin 16.7.0]
Intel Core i7-4870HQ CPU 2.80GHz (Haswell), 1 CPU, 8 logical cores and 4 physical cores
.NET Core SDK=2.0.0
[Host] : .NET Core 2.0.0 (Framework 4.6.00001.0), 64bit RyuJIT
Job=InProcess Toolchain=InProcessToolchain
Method | Mean | Error | StdDev |
------- |---------:|----------:|----------:|
Take | 57.54 ns | 1.3227 ns | 1.4701 ns |
Slice | 12.26 ns | 0.2668 ns | 0.2855 ns |
// * Legends *
Mean : Arithmetic mean of all measurements
Error : Half of 99.9% confidence interval
StdDev : Standard deviation of all measurements
1 ns : 1 Nanosecond (0.000000001 sec)
As you can see from the output, I actually ran this on a macOS, because, of course, it’s .NET Core and it runs everywhere. And that’s it - hopefully something that is something that you will find useful in the future.
