docs: honest scoreboard — startup note, us methodology, current hardware
This commit is contained in:
+22
-15
@@ -10,18 +10,25 @@ These aren't guidelines. They're the deal.
|
||||
### 1. The benchmark is the contract
|
||||
|
||||
`./benchmarks/test-all.sh` must pass. Not "mostly pass." Not "pass on
|
||||
your machine." Pass. Every suite — words, lines (monsters included),
|
||||
and stdin — races fastwc against GNU wc.
|
||||
your machine." Pass. Every suite — words (growing to 10M lines), lines
|
||||
(monsters included), and stdin — races fastwc against GNU coreutils
|
||||
wc, busybox wc, and toybox wc at microsecond resolution.
|
||||
|
||||
The moment fastwc is slower than GNU wc — or disagrees with it on a
|
||||
single count — your change does not ship. We did not spend this much
|
||||
effort being faster than a forty-year-old dependency museum just so you
|
||||
could add a `strlen()` in the hot loop.
|
||||
The moment fastwc is slower than any of them on a real race — or
|
||||
disagrees with any of them on a single count — your change does not
|
||||
ship. We did not spend this much effort being faster than a
|
||||
forty-year-old dependency museum just so you could add a `strlen()` in
|
||||
the hot loop.
|
||||
|
||||
The old standing exception — the 1M-line case, where GNU's hand-tuned
|
||||
AVX-512 assembly squeaked out a one-millisecond win — is closed. We
|
||||
beat them there too now (see docs/PERFORMANCE.md). There are no
|
||||
exceptions left, and there is no excuse to be slower anywhere else.
|
||||
The benchmark is honest about what it measures. Both sides are timed
|
||||
by a C timer, and a case the reference finishes in under 5 ms is
|
||||
startup, not throughput: fastwc must still count it right, but the
|
||||
case is reported as startup-bound and excluded from the averages.
|
||||
Raced cases allow a 2% dead-heat margin so a genuine tie can't flake
|
||||
on scheduler jitter. That margin exists because GNU's threaded counter
|
||||
occasionally dead-heats with us on mid-size files — it has never won a
|
||||
raced case, and no exception is carved out for it
|
||||
(see docs/PERFORMANCE.md).
|
||||
|
||||
### 2. C99, or don't bother
|
||||
|
||||
@@ -68,11 +75,11 @@ feature needs a new file, your feature is over-engineered.
|
||||
### 7. Tests are the benchmarks
|
||||
|
||||
There is no separate test suite, because the benchmark *is* the test
|
||||
suite — it checks correctness against GNU wc (the only oracle around,
|
||||
and it's wrong often enough to keep us humble) and speed in the same
|
||||
breath. Add a flag? It gets benchmarked. Change the counting? It gets
|
||||
benchmarked. The fail-fast scripts in `benchmarks/` are your tests, and
|
||||
all of them must pass before you open a PR.
|
||||
suite — it checks correctness against GNU coreutils wc, busybox wc,
|
||||
and toybox wc (all three are wrong often enough to keep us humble) and
|
||||
speed in the same breath. Add a flag? It gets benchmarked. Change the
|
||||
counting? It gets benchmarked. The fail-fast scripts in `benchmarks/`
|
||||
are your tests, and all of them must pass before you open a PR.
|
||||
|
||||
### 8. No AI slop
|
||||
|
||||
|
||||
Reference in New Issue
Block a user