The Gauntlet runs six families in one pass and checks exact correctness in each:
| Family | What you learn |
|---|---|
| OLTP | Mixed point writes/reads: durable rows/s and point-lookup p99 |
| OLAP | Scans and aggregations over the packed copy: GB/s and rows/s |
| HTAP | Writers plus analytical readers at once: interference penalty with split writer/reader latencies |
| Freshness | Per-row commit-to-visible latency |
| Resource | RSS, file descriptors, and data/WAL footprint before and after churn |
| Scale | Load plus scan behavior at each measured row count |
How to run#
In-engine qualification (fast CI defaults; writes target/gauntlet_report.md):
cargo test -p plomid-gauntletWire-level harness (through a real server; writes benchmarks/harness/results/gauntlet.json):
python3 benchmarks/harness/workloads/gauntlet.pyFull 1M/2M scale points only run when opted in:
PLOMID_SCALE_ROWS=full python3 benchmarks/harness/workloads/gauntlet.pyOr pass an explicit comma list: PLOMID_SCALE_ROWS=1000000,2000000.
Pipeline#
Every qualification follows the same stages in order:
INSPECT → MEASURE → DIAGNOSE → DESIGN → IMPLEMENT → BENCHMARK → COMPARE → TEST → FIX → REGRESSION → REPORT → STOP
The markdown report carries this checklist. STOP completes when the report has been reviewed.
Thresholds and targets#
All targets are internal to PloMID — the Gauntlet never compares against other databases:
- Row engine: minimize point-lookup p99; maximize durable rows/s.
- Column engine: maximize scan GB/s and rows/s.
- HTAP penalty: p99 latency under combined load divided by p99 alone. Near
1.0means writes and analytics barely disturb each other. (An earlier revision of this page called it a throughput ratio — the harness's headline penalty is the p99 ratio.) - Freshness: per-row commit-to-visible latency stays bounded while writes continue.
The seven questions#
The report answers each with YES or NO:
- Did throughput improve?
- Did p99 improve?
- Did CPU efficiency improve?
- Did memory improve?
- Did storage footprint improve?
- Did correctness remain exact?
- Did OLTP and OLAP and HTAP interference improve?
A single run records healthy measurement; cross-run improvement is judged by comparing reports over time. NO means "not demonstrated by this run", never "proven worse".
Scale honesty rule#
Only measured row counts are reported — unmeasured scales are never extrapolated. Smoke runs use small points; the 1M/2M points appear only when PLOMID_SCALE_ROWS opts in.