3.7 KiB
Benchmark Harness Usage
This directory stores reproducible benchmark notes and reports for Mercury tools.
The mhash harness lives at scripts/benchmark-mhash.ps1 and writes generated
artifacts under target\mhash-benchmark\results by default.
Default Digest Benchmark
pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash.ps1
The default matrix keeps runtime reasonable: 1MiB, 64MiB, and 256MiB inputs;
MD5, SHA-1, SHA-256, and SHA-512; 1 warmup and 3 measured runs. It compares
Mercury mhash auto/read modes against available non-admin external tools.
Fast Smoke Check
pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash-smoke.ps1 -SkipBuild
The smoke check runs a 1KiB SHA-256 Mercury-only matrix, validates required
telemetry fields, checks JSONL export counts, and verifies -SizeOnly emits
artifact telemetry without process records.
Useful Modes
pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash.ps1 -SizeOnly -SkipBuild
Use -SizeOnly when you only need binary artifact telemetry. It skips fixture
creation and process benchmarks, but still writes raw JSON, artifact JSON/CSV,
empty summary files, and a Markdown report.
To inspect a prebuilt size-focused binary, point the harness at it:
cargo build --profile release-size -p mercury-mhash --bin mhash
pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash.ps1 -SizeOnly -SkipBuild -DigestPath .\target\release-size\mhash.exe
pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash.ps1 -Extended -Hyperfine
Use -Extended for a deeper matrix. Unless explicitly overridden, it raises
the run count to 5, warmups to 2, adds a 1GiB fixture, and benchmarks Mercury
sum, jsonl, and json output formats. Use -Hyperfine to add optional
hyperfine timing exports when hyperfine is available on PATH.
pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash.ps1 -Sizes 1KiB -Algorithms sha256 -DigestFormats sum,jsonl -Repeat 1 -Warmup 0 -NoExternal -SkipBuild
Use explicit small matrices for local harness checks or quick regression triage.
Emitted Telemetry
Each run now records wall time, CPU time, CPU as percent of one core and of the machine, peak working set, mhash output format, binary path, binary size, and a short stdout/stderr sample. Summaries include mean wall time, min wall time, mean CPU milliseconds, throughput, peak working set, CPU percent of one core, and binary size.
The raw report includes reproducibility metadata: invocation, script path, workspace root, OS and process architecture, PowerShell version, logical processor count, git branch/commit/status entries, fixture SHA-256 hashes, tool paths/versions, artifact sizes, artifact SHA-256 hashes, and optional hyperfine suites.
Generated files include:
*.raw.json: full environment, records, summary, artifact, and hyperfine data.*.records.jsonl: one process measurement per line.*.summary.jsonand*.summary.jsonl: grouped summary rows.*.summary.csv: spreadsheet-friendly grouped summary rows.*.artifacts.jsonand*.artifacts.csv: binary size and identity metadata.*.hyperfine.jsonand*.hyperfine.csv: aggregate optional hyperfine output.*.md: human-readable report.
Measurement Caveats
Windows does not expose a safe non-admin cache-drop primitive, so results are warm-cache CLI timings. Very small fixtures mostly measure process startup and output overhead. Peak working set comes from process telemetry and can miss very short-lived allocation spikes. Hyperfine is opt-in and complements the built-in wall/CPU/memory telemetry; it does not replace the harness process metrics.