Benchmark methodology

Every number this project publishes comes from a run you can open, on a machine whose specification is recorded with the result. Three tiers:

Runs live in the Benchmarks workflow; the synthetic tier is charted per commit at dev/bench by github-action-benchmark, which fails the run on a regression beyond 2×.

How queries are timed

• The index is built once; queries run against it warm, 50 results per page, after a warm-up iteration. p50 and p95 are over repeated invocations of the same query.
• “Common” is a word present in most files (return); “identifier” a name present in a few; “no match” a string in none (which measures candidate rejection alone).
• Regex cases: a pattern with a required literal, a case-insensitive one, and one with no literal (a full scan).
• The incremental case rewrites one file and applies the change through the same path the watcher uses.

What the machine is

GitHub-hosted ubuntu-latest runners: 4 vCPU, 16 GB RAM, SSD, shared tenancy. Absolute numbers on your hardware will differ; the value of the CI numbers is that they are comparable commit to commit. Local runs are labelled as such wherever they appear.

Reproduce

cargo run --release --example corpus_bench -- path/to/repo [more paths] --label mine
cargo bench                                 # synthetic tier
cargo bench --bench search_benchmark        # search only

corpus_bench prints the same table CI does (--markdown) and can emit bencher lines for tracking (--bencher).

Reading the trend charts

The trend page charts every measurement per commit to main. Each chart is a cost, so lower is better; the vertical axis is time per operation in the unit that fits the chart (or bytes for memory). The shaded band is that run’s spread: median to 95th percentile for the real-corpus rows, Criterion’s reported deviation for the synthetic rows. A step on every chart at the same commit is a change of runner, not of code. Each chart carries a sentence saying what it measures and what would move it.

Ranking quality

Latency says how fast the engine is; this says whether what it puts first is what a reader wanted. benches/ranking_quality.toml holds labelled queries over the pinned corpora — text, symbols, regex and references — each naming the file and line that should come first (struct Runtime → the definition in runtime.rs, not a doc comment; Field in symbols mode → the model field, not FieldOverridePost in a test). The suite reports precision@1 (the expected result is the first hit) and precision@5, runs on every push, and fails the run below 0.90 / 0.95, so a change that makes searches faster but worse cannot land. Its first run scored 0.30 and surfaced six ranking defects; see the changelog.

cargo run --release --example ranking_quality -- bench-corpus/tokio bench-corpus/django
cargo run --release --example ranking_quality -- … --explain 'struct Runtime'   # why a query ranks as it does

Correctness against ripgrep

Fast and well ranked is worthless if a match is missing. A backtest runs 124 searches through fcs and through ripgrep on the same trees and fails on any line one finds and the other does not; see Correctness against ripgrep for how it works and how to extend it.

Comparison with scan tools

scripts/bench/compare.sh <dir> times the same queries through ripgrep, ugrep (if installed) and fcs against a server on the same tree, warm cache, median of seven runs, and records the machine. It reports both the fcs client’s wall time (process start plus HTTP) and the engine’s own time. The point is not that a scan tool is slow — it is the right tool for a one-off search — but that an index answers in the same time however large the tree is, and ranks. Results are on the latest results page when the comparison job has run.