91 lines
3.7 KiB
Markdown
91 lines
3.7 KiB
Markdown
# Benchmark Harness Usage
|
|
|
|
This directory stores reproducible benchmark notes and reports for Mercury tools.
|
|
The mhash harness lives at `scripts/benchmark-mhash.ps1` and writes generated
|
|
artifacts under `target\mhash-benchmark\results` by default.
|
|
|
|
## Default Digest Benchmark
|
|
|
|
```powershell
|
|
pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash.ps1
|
|
```
|
|
|
|
The default matrix keeps runtime reasonable: 1MiB, 64MiB, and 256MiB inputs;
|
|
MD5, SHA-1, SHA-256, and SHA-512; 1 warmup and 3 measured runs. It compares
|
|
Mercury `mhash` auto/read modes against available non-admin external tools.
|
|
|
|
## Fast Smoke Check
|
|
|
|
```powershell
|
|
pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash-smoke.ps1 -SkipBuild
|
|
```
|
|
|
|
The smoke check runs a 1KiB SHA-256 Mercury-only matrix, validates required
|
|
telemetry fields, checks JSONL export counts, and verifies `-SizeOnly` emits
|
|
artifact telemetry without process records.
|
|
|
|
## Useful Modes
|
|
|
|
```powershell
|
|
pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash.ps1 -SizeOnly -SkipBuild
|
|
```
|
|
|
|
Use `-SizeOnly` when you only need binary artifact telemetry. It skips fixture
|
|
creation and process benchmarks, but still writes raw JSON, artifact JSON/CSV,
|
|
empty summary files, and a Markdown report.
|
|
|
|
To inspect a prebuilt size-focused binary, point the harness at it:
|
|
|
|
```powershell
|
|
cargo build --profile release-size -p mercury-mhash --bin mhash
|
|
pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash.ps1 -SizeOnly -SkipBuild -DigestPath .\target\release-size\mhash.exe
|
|
```
|
|
|
|
```powershell
|
|
pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash.ps1 -Extended -Hyperfine
|
|
```
|
|
|
|
Use `-Extended` for a deeper matrix. Unless explicitly overridden, it raises
|
|
the run count to 5, warmups to 2, adds a 1GiB fixture, and benchmarks Mercury
|
|
`sum`, `jsonl`, and `json` output formats. Use `-Hyperfine` to add optional
|
|
`hyperfine` timing exports when `hyperfine` is available on `PATH`.
|
|
|
|
```powershell
|
|
pwsh -NoProfile -ExecutionPolicy Bypass -File .\scripts\benchmark-mhash.ps1 -Sizes 1KiB -Algorithms sha256 -DigestFormats sum,jsonl -Repeat 1 -Warmup 0 -NoExternal -SkipBuild
|
|
```
|
|
|
|
Use explicit small matrices for local harness checks or quick regression
|
|
triage.
|
|
|
|
## Emitted Telemetry
|
|
|
|
Each run now records wall time, CPU time, CPU as percent of one core and of the
|
|
machine, peak working set, mhash output format, binary path, binary size, and a
|
|
short stdout/stderr sample. Summaries include mean wall time, min wall time,
|
|
mean CPU milliseconds, throughput, peak working set, CPU percent of one core,
|
|
and binary size.
|
|
|
|
The raw report includes reproducibility metadata: invocation, script path,
|
|
workspace root, OS and process architecture, PowerShell version, logical
|
|
processor count, git branch/commit/status entries, fixture SHA-256 hashes,
|
|
tool paths/versions, artifact sizes, artifact SHA-256 hashes, and optional
|
|
hyperfine suites.
|
|
|
|
Generated files include:
|
|
|
|
- `*.raw.json`: full environment, records, summary, artifact, and hyperfine data.
|
|
- `*.records.jsonl`: one process measurement per line.
|
|
- `*.summary.json` and `*.summary.jsonl`: grouped summary rows.
|
|
- `*.summary.csv`: spreadsheet-friendly grouped summary rows.
|
|
- `*.artifacts.json` and `*.artifacts.csv`: binary size and identity metadata.
|
|
- `*.hyperfine.json` and `*.hyperfine.csv`: aggregate optional hyperfine output.
|
|
- `*.md`: human-readable report.
|
|
|
|
## Measurement Caveats
|
|
|
|
Windows does not expose a safe non-admin cache-drop primitive, so results are
|
|
warm-cache CLI timings. Very small fixtures mostly measure process startup and
|
|
output overhead. Peak working set comes from process telemetry and can miss very
|
|
short-lived allocation spikes. Hyperfine is opt-in and complements the built-in
|
|
wall/CPU/memory telemetry; it does not replace the harness process metrics.
|