chore(release): prepare public source release
This commit is contained in:
MercuryToolbox Release
@@ -0,0 +1,329 @@
|
||||
# Mercury Toolbox V2 Design
|
||||
|
||||
## Goal
|
||||
|
||||
Promote the repository from a generic AI-friendly CLI toolbox into the formally named `Mercury Toolbox` / `水星工具箱`, add four new high-leverage commands for search-to-context, diagnostics, log reduction, and binary string triage, and ship a first-class Windows + PowerShell installation path that automatically places the toolbox on the user PATH.
|
||||
|
||||
## Scope
|
||||
|
||||
- Rename the toolbox brand in repository-level docs, workspace metadata, install-facing text, and user help from the generic placeholder wording to `Mercury Toolbox` / `水星工具箱`.
|
||||
- Add four new commands:
|
||||
- `hitsnip`
|
||||
- `diagpick`
|
||||
- `logshape`
|
||||
- `stringscan`
|
||||
- Keep natural binary names such as `snip`, `outline`, `hitsnip`, and `diagpick`; do not introduce a global binary prefix.
|
||||
- Add a user-facing PowerShell installer that:
|
||||
- builds the workspace release binaries
|
||||
- installs all toolbox binaries into `$env:LOCALAPPDATA\MercuryToolbox\bin`
|
||||
- automatically adds that directory to the user PATH if missing
|
||||
- reports what changed and how to verify the install
|
||||
- Add a complementary uninstall path and installation documentation.
|
||||
- Preserve the shared CLI contract:
|
||||
- compact ASCII text output by default
|
||||
- `--json` for machine consumption
|
||||
- PowerShell-friendly stdin behavior
|
||||
- stable exit code mapping
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- No GUI, TUI, or long-running background service.
|
||||
- No package publishing, Scoop bucket, installer EXE, or code signing in this iteration.
|
||||
- No heavyweight parser stack such as tree-sitter, Roslyn, or full symbol servers.
|
||||
- No single “master” control binary; the toolbox remains a workspace of small independent commands.
|
||||
|
||||
## Product Direction
|
||||
|
||||
The existing toolbox already covers “find a file”, “probe the environment”, and “read a known target”. The new batch fills the remaining gaps that are especially painful for AI-assisted terminal workflows:
|
||||
|
||||
- `hitsnip` answers “I already have search hits; now give me the smallest useful context.”
|
||||
- `diagpick` answers “this compiler or runtime log is noisy; show me the actionable diagnostics.”
|
||||
- `logshape` answers “this log is too repetitive; summarize the patterns before I read it.”
|
||||
- `stringscan` answers “this binary or generated blob is opaque; show me the strings that reveal identity, runtime, and ecosystem.”
|
||||
|
||||
The installation work is part of the feature set, not a side quest. A toolbox that is annoying to install has near-zero real-world value even if the commands are good.
|
||||
|
||||
## Branding And Naming
|
||||
|
||||
### Repository Name
|
||||
|
||||
- Primary English name: `Mercury Toolbox`
|
||||
- Primary Chinese name: `水星工具箱`
|
||||
- Documentation should treat these as the formal product name, not as aliases for the old placeholder wording.
|
||||
|
||||
### Binary Names
|
||||
|
||||
- Keep natural command names:
|
||||
- `jsonlgrep`
|
||||
- `recent`
|
||||
- `pathshadow`
|
||||
- `portping`
|
||||
- `binmeta`
|
||||
- `fileprobe`
|
||||
- `outline`
|
||||
- `snip`
|
||||
- `chunkcat`
|
||||
- `hitsnip`
|
||||
- `diagpick`
|
||||
- `logshape`
|
||||
- `stringscan`
|
||||
- Do not add `mercury-` prefixes.
|
||||
- The installer owns product-level grouping; the binaries stay small and task-specific.
|
||||
|
||||
## Installation Design
|
||||
|
||||
### Main Installation Path
|
||||
|
||||
- Add `scripts/install-toolbox.ps1`
|
||||
- Default behavior:
|
||||
- verify `cargo` exists
|
||||
- run `cargo build --release --workspace`
|
||||
- collect all known toolbox executables from `target\release`
|
||||
- create `$env:LOCALAPPDATA\MercuryToolbox\bin` if needed
|
||||
- copy all toolbox executables into that directory
|
||||
- inspect the user PATH
|
||||
- append the install directory if missing
|
||||
- print a compact summary of installed commands and PATH status
|
||||
|
||||
### PATH Policy
|
||||
|
||||
- PATH updates target the user-level environment variable, not machine-wide PATH.
|
||||
- The script should avoid duplicate path entries by normalizing path comparison.
|
||||
- Current-session PATH may also be patched so the commands are usable immediately after install without opening a new shell.
|
||||
|
||||
### Safety And UX
|
||||
|
||||
- Support `-NoPathUpdate` for explicit opt-out.
|
||||
- Support `-InstallRoot <PATH>` for advanced/manual installs, while defaulting to `$env:LOCALAPPDATA\MercuryToolbox`.
|
||||
- Support `-Configuration Debug|Release`, defaulting to `Release`.
|
||||
- Text output should clearly distinguish:
|
||||
- install root
|
||||
- bin directory
|
||||
- copied binaries
|
||||
- whether PATH was already configured, updated, or skipped
|
||||
|
||||
### Uninstall Path
|
||||
|
||||
- Add `scripts/uninstall-toolbox.ps1`
|
||||
- Default behavior:
|
||||
- remove installed binaries from the Mercury Toolbox bin directory
|
||||
- remove the Mercury Toolbox bin directory from the user PATH when present
|
||||
- leave unrelated files untouched
|
||||
|
||||
### Fallback Installation Path
|
||||
|
||||
- README also documents manual per-command install via `cargo install --path crates/<command>`.
|
||||
- This is a fallback path for Rust-native users, not the primary recommendation.
|
||||
|
||||
## Command Designs
|
||||
|
||||
### `hitsnip`
|
||||
|
||||
#### Purpose
|
||||
|
||||
Convert search hits into deduplicated, compact context windows so neither humans nor AI have to manually reopen each hit.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Explicit files plus line references in one of these forms:
|
||||
- `path:line`
|
||||
- `path:line:column`
|
||||
- stdin text from tools such as `rg -n`
|
||||
- JSONL hit records with at least `path` and `line`
|
||||
|
||||
#### Output
|
||||
|
||||
- Text mode:
|
||||
- one header per merged snippet
|
||||
- numbered lines under each snippet
|
||||
- compact reason metadata such as hit count and merged line span
|
||||
- JSON mode:
|
||||
- one JSON array of snippet objects
|
||||
- fields:
|
||||
- `path`
|
||||
- `start_line`
|
||||
- `end_line`
|
||||
- `hit_lines`
|
||||
- `hit_count`
|
||||
- `lines`
|
||||
|
||||
#### Behavior
|
||||
|
||||
- Merge nearby hits in the same file when the distance between hit windows is at most `--max-gap`.
|
||||
- Expand each hit by `--context` lines.
|
||||
- Deduplicate repeated identical hit lines.
|
||||
- Ignore malformed lines with clear usage/runtime errors rather than panicking.
|
||||
|
||||
#### Key Flags
|
||||
|
||||
- `--context <N>`
|
||||
- `--max-gap <N>`
|
||||
- `--limit <N>`
|
||||
- `--input-format auto|lines|jsonl`
|
||||
- shared `--json`
|
||||
|
||||
### `diagpick`
|
||||
|
||||
#### Purpose
|
||||
|
||||
Extract actionable diagnostics from compiler, build, and runtime logs and optionally attach source context.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Plain text logs from stdin or files
|
||||
- JSONL records with fields such as `path`, `line`, `column`, `severity`, `message`
|
||||
|
||||
#### Recognized Text Patterns
|
||||
|
||||
- Rust diagnostics:
|
||||
- `error[E0425]: ...`
|
||||
- `--> path:line:column`
|
||||
- MSVC/C#/Unity-style:
|
||||
- `path(line,column): error CSxxxx: ...`
|
||||
- `path:line:column: error: ...`
|
||||
- Generic runtime stack/log references where a path and line are present
|
||||
|
||||
#### Output
|
||||
|
||||
- Text mode:
|
||||
- one compact record per diagnostic
|
||||
- optional source snippet below when `--with-source` is enabled
|
||||
- JSON mode:
|
||||
- array of diagnostics with stable fields:
|
||||
- `path`
|
||||
- `line`
|
||||
- `column`
|
||||
- `severity`
|
||||
- `code`
|
||||
- `message`
|
||||
- `source`
|
||||
- `tool_hint`
|
||||
|
||||
#### Key Flags
|
||||
|
||||
- `--with-source`
|
||||
- `--context <N>`
|
||||
- `--limit <N>`
|
||||
- `--severity error|warning|note|all`
|
||||
|
||||
### `logshape`
|
||||
|
||||
#### Purpose
|
||||
|
||||
Collapse repetitive logs into pattern groups before anyone spends tokens reading the raw stream.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- Plain log lines from stdin or files
|
||||
|
||||
#### Heuristic Normalization
|
||||
|
||||
- Replace volatile fragments with placeholders:
|
||||
- timestamps
|
||||
- decimal and hex numbers
|
||||
- UUID-like tokens
|
||||
- long paths and addresses
|
||||
- Preserve level-like prefixes such as `INFO`, `WARN`, `ERROR` when possible
|
||||
|
||||
#### Output
|
||||
|
||||
- Text mode:
|
||||
- one compact line per template
|
||||
- includes count and a representative sample
|
||||
- JSON mode:
|
||||
- array of groups with fields:
|
||||
- `pattern`
|
||||
- `count`
|
||||
- `first_line`
|
||||
- `last_line`
|
||||
- `sample`
|
||||
|
||||
#### Key Flags
|
||||
|
||||
- `--top <N>`
|
||||
- `--min-count <N>`
|
||||
- `--show-samples`
|
||||
- `--keep-level`
|
||||
|
||||
### `stringscan`
|
||||
|
||||
#### Purpose
|
||||
|
||||
Expose the high-signal strings inside binaries, generated files, and opaque artifacts for reverse engineering and quick triage.
|
||||
|
||||
#### Inputs
|
||||
|
||||
- One or more file paths from argv or stdin
|
||||
|
||||
#### Output
|
||||
|
||||
- Text mode:
|
||||
- one compact summary line per file in summary mode
|
||||
- optional per-string output when filtering by category
|
||||
- JSON mode:
|
||||
- per-file reports with:
|
||||
- `path`
|
||||
- `is_binary`
|
||||
- `string_count`
|
||||
- `categories`
|
||||
- `matches`
|
||||
|
||||
#### Heuristic Categories
|
||||
|
||||
- `url`
|
||||
- `path`
|
||||
- `dll`
|
||||
- `namespace`
|
||||
- `unity`
|
||||
- `dotnet`
|
||||
- `il2cpp`
|
||||
- `bepinex`
|
||||
- `generic`
|
||||
|
||||
#### Key Flags
|
||||
|
||||
- `--min-len <N>`
|
||||
- `--kind all|url|path|dll|namespace|unity|dotnet|il2cpp|bepinex|generic`
|
||||
- `--unique`
|
||||
- `--limit <N>`
|
||||
- `--details`
|
||||
|
||||
## Shared Fixtures
|
||||
|
||||
Add fixtures that support these commands without introducing heavyweight dependencies:
|
||||
|
||||
- `fixtures/hits/rg-output.txt`
|
||||
- `fixtures/diag/rust-errors.txt`
|
||||
- `fixtures/diag/unity-errors.txt`
|
||||
- `fixtures/logs/repetitive.log`
|
||||
- `fixtures/binaries/stringscan-sample.bin`
|
||||
|
||||
The binary fixture should embed obvious markers such as URLs, DLL names, Unity namespace fragments, and BepInEx-like strings so category detection can be tested reliably.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- Unit tests:
|
||||
- hit parsing and merge planning for `hitsnip`
|
||||
- diagnostic parsing helpers for `diagpick`
|
||||
- normalization and grouping for `logshape`
|
||||
- string extraction and category heuristics for `stringscan`
|
||||
- installer path normalization and PATH edit helpers
|
||||
- Integration tests:
|
||||
- `--help` examples for all new binaries
|
||||
- PowerShell pipeline flows
|
||||
- JSON output shape
|
||||
- install/uninstall script dry-run or temp-root behavior
|
||||
- Verification gates stay under the existing Jade standard, including `check-jade.ps1`.
|
||||
|
||||
## Risks
|
||||
|
||||
- Search-hit parsing can get messy when input mixes drive-letter paths and colon-separated line syntax.
|
||||
- Prefer a path-aware parser that handles Windows drive prefixes before splitting on the final line/column segments.
|
||||
- Diagnostic parsers can drift when tools change exact wording.
|
||||
- Keep the patterns heuristic and additive rather than pretending to be a full parser for each toolchain.
|
||||
- Over-normalizing logs can merge meaningfully different failures.
|
||||
- Keep the normalization rule set conservative in v1.
|
||||
- Binary string scanning can explode in output volume.
|
||||
- Default to concise summaries and require explicit detail flags for large per-string dumps.
|
||||
- PATH editing is user-hostile if it duplicates entries or stomps unrelated content.
|
||||
- Use minimal, append-only edits with idempotent detection.
|
||||
@@ -0,0 +1,137 @@
|
||||
# Toolbox Reading V2 Design
|
||||
|
||||
## Goal
|
||||
|
||||
Add four new PowerShell-friendly, AI-friendly reading commands to the toolbox so the workflow can move from "find a file" to "read the right part" without falling back to `Get-Content` or wasting tokens on full-file dumps.
|
||||
|
||||
## Scope
|
||||
|
||||
- Add `snip` for precise snippet extraction from files or stdin.
|
||||
- Add `outline` for heuristic structure summaries of common source and config files.
|
||||
- Add `fileprobe` for fast file-type and usefulness heuristics before opening a file.
|
||||
- Add `chunkcat` for deterministic chunk listing and chunk extraction from large text files.
|
||||
- Add help examples and README usage for all four commands.
|
||||
- Reuse the existing workspace CLI contract: shared `--json`, `--input-format`, exit codes, and PowerShell pipe support.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- No TUI, pager, fuzzy picker, or interactive mode.
|
||||
- No tree-sitter, Roslyn, or heavyweight language parser in this iteration.
|
||||
- No write or patch workflow in this batch.
|
||||
- No token estimation command, preview patch command, or replacement engine in this batch.
|
||||
|
||||
## Design Direction
|
||||
|
||||
- Prefer heuristic parsing over heavyweight parsing.
|
||||
- This keeps the binaries small, predictable, and fast enough for ad hoc shell use.
|
||||
- False positives are acceptable when they are clearly labeled as heuristic output.
|
||||
- Prefer compact text output that is useful to both humans and AI.
|
||||
- Text mode should avoid banners and avoid dumping redundant metadata.
|
||||
- JSON mode should expose stable machine fields so scripts can select the next action.
|
||||
- Treat these commands as a chain rather than isolated tools.
|
||||
- `fileprobe` answers "what is this file and should I read it?"
|
||||
- `outline` answers "where is the interesting structure?"
|
||||
- `snip` answers "show me the exact region."
|
||||
- `chunkcat` answers "how do I traverse this large file safely?"
|
||||
|
||||
## Command Designs
|
||||
|
||||
### `snip`
|
||||
|
||||
- Input:
|
||||
- one or more file paths from argv
|
||||
- or stdin content when piped
|
||||
- Selectors:
|
||||
- `--lines <START[:END]>`
|
||||
- `--around <REGEX>`
|
||||
- `--symbol <NAME>`
|
||||
- exactly one selector must be present
|
||||
- Supporting flags:
|
||||
- `--context <N>` for `--around` and `--symbol`
|
||||
- `--max-matches <N>` for `--around`
|
||||
- Output:
|
||||
- text mode emits `path:start-end reason=...` followed by numbered lines
|
||||
- JSON mode emits an array of snippets with `path`, `start_line`, `end_line`, `reason`, and `lines`
|
||||
- Heuristic behavior:
|
||||
- `--symbol` uses language-aware regexes for Rust and C# plus generic fallbacks for other text files
|
||||
- when possible, symbol extraction expands to a balanced block instead of a single line
|
||||
|
||||
### `outline`
|
||||
|
||||
- Input:
|
||||
- one or more file paths from argv
|
||||
- stdin paths in line mode
|
||||
- Supported heuristic families:
|
||||
- Rust: `mod`, `struct`, `enum`, `trait`, `impl`, `fn`, `const`, `static`, `type`
|
||||
- C#: `namespace`, `class`, `struct`, `enum`, `interface`, `record`, method-like members
|
||||
- JSON: object keys traversed by depth
|
||||
- TOML: tables and keys
|
||||
- YAML: indentation-based key outline
|
||||
- Flags:
|
||||
- `--depth <N>` to cap nested output
|
||||
- `--kind all|code|config`
|
||||
- Output:
|
||||
- text mode emits one compact line per item: `line depth kind name`
|
||||
- JSON mode emits per-file objects with stable `items`
|
||||
- Heuristic behavior:
|
||||
- items are marked by file-relative line number and depth, not claimed as exact AST nodes
|
||||
|
||||
### `fileprobe`
|
||||
|
||||
- Input:
|
||||
- one or more file paths from argv or stdin
|
||||
- Output fields:
|
||||
- path, extension, size, modified time
|
||||
- `exists`, `is_dir`, `is_binary`, `encoding_hint`
|
||||
- `family` such as `source`, `config`, `data`, `binary`, `archive`, `unknown`
|
||||
- `language_hint`
|
||||
- `line_count`, `blank_lines`, `longest_line`
|
||||
- heuristic flags such as `likely_generated`, `likely_minified`, `likely_test`, `likely_lockfile`, `likely_vendor`
|
||||
- optional `container_hint` such as `pe`, `zip`, `sqlite`, `pdf`
|
||||
- Heuristic behavior:
|
||||
- detect binary vs text from bytes and UTF-8 validity
|
||||
- infer family and language from extension plus lightweight content checks
|
||||
- do not parse PE deeply here; `binmeta` remains the dedicated PE inspector
|
||||
|
||||
### `chunkcat`
|
||||
|
||||
- Input:
|
||||
- one text file path at a time in v1
|
||||
- Flags:
|
||||
- `--max-lines <N>` defaulting to a budget-friendly size
|
||||
- `--overlap <N>` for deterministic overlap between chunks
|
||||
- `--chunk <INDEX>` to emit a specific chunk
|
||||
- Output:
|
||||
- without `--chunk`, emit chunk inventory only
|
||||
- with `--chunk`, emit the selected chunk with numbered lines
|
||||
- JSON mode emits chunk metadata, and chunk content only when `--chunk` is set
|
||||
- Behavior:
|
||||
- chunking is line-based and deterministic
|
||||
- line ranges are stable for the same file contents and options
|
||||
- overlapping chunks use a fixed stride of `max_lines - overlap`
|
||||
|
||||
## Shared Testing Strategy
|
||||
|
||||
- Add unit tests for:
|
||||
- selector parsing
|
||||
- heuristic file classification
|
||||
- chunk calculation
|
||||
- outline extraction helpers
|
||||
- Add integration tests for:
|
||||
- `--help` examples
|
||||
- PowerShell pipeline scenarios
|
||||
- JSON output shape for automation
|
||||
- Add fixtures for:
|
||||
- Rust source with multiple symbols
|
||||
- C# source with Unity-style class structure
|
||||
- JSON, TOML, and YAML config files
|
||||
- minified/generated-ish text and simple binary-like samples
|
||||
|
||||
## Risks
|
||||
|
||||
- Heuristic structure detection can misclassify edge-case syntax.
|
||||
- This is acceptable if outputs are useful and clearly heuristic.
|
||||
- `snip --symbol` block expansion can drift on malformed files.
|
||||
- Prefer a safe fallback to line-only snippets over panics or empty output.
|
||||
- `chunkcat` can become noisy if default chunk size is too small.
|
||||
- Pick a conservative default tuned for AI reading, not log streaming.
|
||||
@@ -0,0 +1,80 @@
|
||||
# Toolbox V1.5 Design
|
||||
|
||||
## Goal
|
||||
|
||||
Promote the existing four commands from "usable prototypes" to daily-driver tools by pairing feature upgrades with first-class help and documentation, and add a production-grade `binmeta` command for fast binary triage in AI-heavy terminal workflows.
|
||||
|
||||
## Scope
|
||||
|
||||
- Upgrade `jsonlgrep` with nested field paths and negative field predicates.
|
||||
- Upgrade `recent` with regex-based name filtering.
|
||||
- Upgrade `pathshadow` with clearer shadowing explanations in text and JSON output.
|
||||
- Upgrade `portping` with selectable HTTP method and expected status checks.
|
||||
- Add concrete examples to `--help` output for all commands.
|
||||
- Update `README.md` with scenario-driven examples and PowerShell pipeline usage.
|
||||
- Add `binmeta` as a new CLI command for binary metadata inspection on Windows PE files, with graceful handling for non-PE files.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- No GUI or TUI work.
|
||||
- No installer, release automation, or Scoop packaging in this iteration.
|
||||
- No multi-format binary analysis beyond production-grade PE support plus safe fallback summaries for unknown formats.
|
||||
|
||||
## Command Changes
|
||||
|
||||
### `jsonlgrep`
|
||||
|
||||
- Extend query syntax to support `field!=value` and `field!~=regex`.
|
||||
- Resolve dotted field paths such as `event.user.name`.
|
||||
- Treat scalar JSON values consistently so exact matches and text projections also work for numbers, booleans, and null.
|
||||
- Improve `--help` with copy-pasteable file and pipeline examples.
|
||||
|
||||
### `recent`
|
||||
|
||||
- Add `--name <REGEX>` to filter by file or directory basename.
|
||||
- Preserve `.gitignore`-aware traversal and current sorting behavior.
|
||||
- Improve `--help` with examples for recent Rust work, recent directories, and JSON pipeline usage.
|
||||
|
||||
### `pathshadow`
|
||||
|
||||
- Append stable explanation fields to JSON output so scripts and AI can tell why an entry won or lost.
|
||||
- Include PATH rank and winner path context where applicable.
|
||||
- Keep text output compact but clearer about origin and shadowing reason.
|
||||
- Improve `--help` with explicit PATH diagnosis examples.
|
||||
|
||||
### `portping`
|
||||
|
||||
- Add `--method GET|HEAD` for HTTP(S) requests.
|
||||
- Add `--expect-status` to turn mismatched HTTP status codes into probe failure while keeping actual status visible.
|
||||
- Preserve TCP behavior unchanged.
|
||||
- Improve `--help` with health-check and HEAD examples.
|
||||
|
||||
### `binmeta`
|
||||
|
||||
- Input: one or more file paths from argv or stdin lines.
|
||||
- Text output: one concise summary line per file for human scanning.
|
||||
- JSON output: one object per file with stable fields for automation and AI use.
|
||||
- PE analysis:
|
||||
- file kind, machine/architecture, subsystem, executable vs DLL hint, PE timestamp
|
||||
- section summaries
|
||||
- imported DLLs and imported symbol count summary
|
||||
- exported symbol count summary when available
|
||||
- SHA-256, file size, modified time
|
||||
- heuristic hints for `.NET`, Unity, IL2CPP, Mono, and BepInEx-related artifacts
|
||||
- Non-PE fallback:
|
||||
- still emit hash, size, modified time, file extension, and "not_pe" kind without failing the whole command
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- Add focused unit tests for new parsing helpers and negative query logic.
|
||||
- Expand integration tests to cover help examples and new CLI flags.
|
||||
- Add `binmeta` integration tests against:
|
||||
- a known text fixture for non-PE fallback
|
||||
- compiled workspace binaries for PE parsing
|
||||
- Keep the existing Jade verification gate unchanged.
|
||||
|
||||
## Risks
|
||||
|
||||
- PE parsing APIs can be noisy; the implementation should wrap parser details behind small helpers.
|
||||
- `binmeta` hints must remain heuristic and clearly labeled as hints, not definitive claims.
|
||||
- Help text examples must stay in sync with actual behavior, so tests should assert representative help fragments.
|
||||
Reference in New Issue
Block a user