Benchmark Harness Usage

This directory stores reproducible benchmark notes and reports for Mercury tools. The mhash harness lives at scripts/benchmark-mhash.ps1 and writes generated artifacts under target\mhash-benchmark\results by default.

Default Digest Benchmark

pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash.ps1

The default matrix keeps runtime reasonable: 1MiB, 64MiB, and 256MiB inputs; MD5, SHA-1, SHA-256, and SHA-512; 1 warmup and 3 measured runs. It compares Mercury mhash auto/read modes against available non-admin external tools.

Fast Smoke Check

pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash-smoke.ps1 -SkipBuild

The smoke check runs a 1KiB SHA-256 Mercury-only matrix, validates required telemetry fields, checks JSONL export counts, and verifies -SizeOnly emits artifact telemetry without process records.

Useful Modes

pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash.ps1 -SizeOnly -SkipBuild

Use -SizeOnly when you only need binary artifact telemetry. It skips fixture creation and process benchmarks, but still writes raw JSON, artifact JSON/CSV, empty summary files, and a Markdown report.

To inspect a prebuilt size-focused binary, point the harness at it:

cargo build --profile release-size -p mercury-mhash --bin mhash
pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash.ps1 -SizeOnly -SkipBuild -DigestPath .\target\release-size\mhash.exe

pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash.ps1 -Extended -Hyperfine

Use -Extended for a deeper matrix. Unless explicitly overridden, it raises the run count to 5, warmups to 2, adds a 1GiB fixture, and benchmarks Mercury sum, jsonl, and json output formats. Use -Hyperfine to add optional hyperfine timing exports when hyperfine is available on PATH.

pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash.ps1 -Sizes 1KiB -Algorithms sha256 -DigestFormats sum,jsonl -Repeat 1 -Warmup 0 -NoExternal -SkipBuild

Use explicit small matrices for local harness checks or quick regression triage.

Emitted Telemetry

Each run now records wall time, CPU time, CPU as percent of one core and of the machine, peak working set, mhash output format, binary path, binary size, and a short stdout/stderr sample. Summaries include mean wall time, min wall time, mean CPU milliseconds, throughput, peak working set, CPU percent of one core, and binary size.

The raw report includes reproducibility metadata: invocation, script path, workspace root, OS and process architecture, PowerShell version, logical processor count, git branch/commit/status entries, fixture SHA-256 hashes, tool paths/versions, artifact sizes, artifact SHA-256 hashes, and optional hyperfine suites.

Generated files include:

*.raw.json: full environment, records, summary, artifact, and hyperfine data.
*.records.jsonl: one process measurement per line.
*.summary.json and *.summary.jsonl: grouped summary rows.
*.summary.csv: spreadsheet-friendly grouped summary rows.
*.artifacts.json and *.artifacts.csv: binary size and identity metadata.
*.hyperfine.json and *.hyperfine.csv: aggregate optional hyperfine output.
*.md: human-readable report.

Measurement Caveats

Windows does not expose a safe non-admin cache-drop primitive, so results are warm-cache CLI timings. Very small fixtures mostly measure process startup and output overhead. Peak working set comes from process telemetry and can miss very short-lived allocation spikes. Hyperfine is opt-in and complements the built-in wall/CPU/memory telemetry; it does not replace the harness process metrics.