Add benchmark pipeline with Rust-native A/B validation#912

Open

YuanyuanTian-hh wants to merge 39 commits intomainfrom

user/tianyuanyuan/benchmark-regression

Commits on Apr 9, 2026

Migrate DiskANN benchmark pipeline from ADO to GitHub Actions
Yuanyuan Tian (from Dev Box)
committed
Fix openai-100K distance metric and add gitignore patterns
Yuanyuan Tian (from Dev Box)
committed
Add push trigger to benchmarks workflow for pre-merge testing
Yuanyuan Tian (from Dev Box)
committed
Fix baseline run: use input config from current branch checkout
Yuanyuan Tian (from Dev Box)
committed
Fix markdown conversion: replace broken csvtomd with inline Python
Yuanyuan Tian (from Dev Box)
committed
Update benchmark configs: align build/search params to fix low recall
Yuanyuan Tian (from Dev Box)
committed
Fix recall_at: set to 100 to match groundtruth file K=100
Yuanyuan Tian (from Dev Box)
committed
Remove stale absolute contracts (qps/recall/total_time): calibrated for ADO mimir-enron, not applicable to public datasets on GitHub runners. Threshold calibration tracked in PBI.
Yuanyuan Tian (from Dev Box)
committed
remove comments
Yuanyuan Tian (from Dev Box)
committed
Fix build_ram_limit_gb: reduce 10->4 to fit GitHub runner RAM (7GB limit)
Yuanyuan Tian (from Dev Box)
committed
Fix wikipedia distance: cosine_normalized->cosine (vectors are not L2-normalized, metric is inner product)
Yuanyuan Tian (from Dev Box)
committed
Fix wikipedia distance: cosine->inner_product (groundtruth uses raw ip, not cosine similarity)
Yuanyuan Tian (from Dev Box)
committed
Align build threads
Yuanyuan Tian (from Dev Box)
committed
Speed up benchmarks: build threads 1->4, openai pq_chunks 384->192
Yuanyuan Tian (from Dev Box)
committed
Temp: disable concurrency cancellation for A/A batch testing
Yuanyuan Tian (from Dev Box)
committed
revert A/A test settings, update OpenAI config to SQ_1_2.0
Yuanyuan Tian (from Dev Box)
committed
Remove micro-benchmark-iai comments
Yuanyuan Tian (from Dev Box)
committed
use GitHub Release assets for benchmark datasets
Yuanyuan Tian (from Dev Box)
committed
extract csv-to-markdown into reusable script
Yuanyuan Tian (from Dev Box)
committed
calibrate contract thresholds from GitHub runner data
Yuanyuan Tian (from Dev Box)
committed
add daily A/A benchmark stability test with failure notification
Yuanyuan Tian (from Dev Box)
committed
widen mean_cpus threshold to 15% for shared-runner CPU noise
Yuanyuan Tian (from Dev Box)
committed
move benchmark datasets to separate repo (YuanyuanTian-hh/diskann-benchmark-data)
Yuanyuan Tian (from Dev Box)
committed
consolidate 3 benchmark scripts into single benchmark_validate.py
Yuanyuan Tian (from Dev Box)
committed
switch benchmark jobs to self-hosted 1ES runner pool (diskann-github)
Yuanyuan Tian (from Dev Box)
committed
replace gh CLI with curl for dataset downloads (gh not available on 1ES runners)
Yuanyuan Tian (from Dev Box)
committed
fix latency_95: read p95_latency instead of p999_latency from benchmark JSON
Yuanyuan Tian (from Dev Box)
committed
revert to ubuntu-latest runners, switch dataset source to BAB v0.4.0
Yuanyuan Tian (from Dev Box)
committed
address PR review: reduce search_list to 200, remove hardcoded Rust version, fix missing-field handling, clean up orphaned thresholds, switch data source to BAB v0.4.0
Yuanyuan Tian (from Dev Box)
committed
widen latency_95 threshold to 15% for shared-runner noise
Yuanyuan Tian (from Dev Box)
committed
replace push trigger with pull_request trigger targeting main
Yuanyuan Tian (from Dev Box)
committed
implement Regression trait for disk-index benchmarks
Yuanyuan Tian (from Dev Box)
committed
test: set strict tolerances (0.1%) to verify pipeline failure detection
Yuanyuan Tian (from Dev Box)
committed
revert tolerances to production values
Yuanyuan Tian (from Dev Box)
committed
test for failure
Yuanyuan Tian (from Dev Box)
committed
revert test code
Yuanyuan Tian (from Dev Box)
committed
address review: refactor workflows to matrix strategy, update build params
Yuanyuan Tian (from Dev Box)
committed

Commits on Apr 10, 2026

Address hildebrandmw review: Direction enum, Table display, composite action, workflow renames
Yuanyuan Tian (from Dev Box)
committed
Fix E0521: clone strings for Table::Row::insert ('static bound)
Yuanyuan Tian (from Dev Box)
committed