bench: race busybox again, verify the wc oracle, average the speedup
Busybox wc is back in every suite, for shits and giggles. checkwc now walks PATH and identifies each wc by its --version answer (coreutils and fastwc respond; busybox names itself in the error it prints), so a wc symlinked to fastwc is detected and skipped instead of silently racing us against ourselves, and a missing coreutils is a clear configure error. Every run now ends with the average speedup of fastwc against coreutils and busybox, computed from the per-case ratios. The fail-fast verdict now requires a measurable (>0ms) reference time, so sub-millisecond cases stop flaking on startup noise, and the 100M line monster is pinned back to the coreutils oracle instead of whatever oracle was raced last.
This commit is contained in:
@@ -41,8 +41,10 @@ GNU wc's lone win — 1M lines, by a single millisecond, on hand-tuned
|
||||
AVX-512 assembly — is a historical footnote now. We closed it, then the
|
||||
10M case, then the 100M monster, and then we added a 1B-line race with
|
||||
no opponent, because beating GNU wc at this point is just cruelty to
|
||||
software. Busybox was removed from the suite for the same reason: it
|
||||
stopped being a challenge and started being a participation trophy.
|
||||
software. Busybox is back in the suite too — for shits and giggles. It
|
||||
stopped being a challenge years ago, but watching it count is the
|
||||
closest thing this project has to a hobby. Every run ends with the
|
||||
average speedup against each oracle, so the cruelty is quantified.
|
||||
|
||||
## Why
|
||||
|
||||
@@ -99,13 +101,16 @@ make bench # build release + run every suite
|
||||
|
||||
The suites live under `benchmarks/files/{lines,words}` (file input) and
|
||||
`benchmarks/stdin/piping` (standard input), and fail fast the moment
|
||||
fastwc loses a single case. The lines suite also carries the monsters:
|
||||
100M lines raced against coreutils, and 1B lines timed solo — there is
|
||||
no reference for that one; we are the reference now. The suites
|
||||
interleave runs so both commands see identical cache warmth, keep the
|
||||
minimum, and fail the moment fastwc loses a single case. GNU `wc` is
|
||||
used as an oracle the same way you'd use a broken clock: occasionally
|
||||
it's right, and it's the only one around.
|
||||
fastwc loses a single case. Every suite races against GNU coreutils wc
|
||||
and, when it is installed, busybox wc. The benchmark locates the real
|
||||
oracles by their `--version` answer — a `wc` that answers as fastwc is a
|
||||
symlink somebody made, and is skipped rather than raced against itself.
|
||||
The lines suite also carries the monsters: 100M lines raced against
|
||||
coreutils, and 1B lines timed solo — there is no reference for that
|
||||
one; we are the reference now. The suites interleave runs so both
|
||||
commands see identical cache warmth, keep the minimum, and fail the
|
||||
moment fastwc loses a single case. When the run ends, the average
|
||||
speedup of fastwc against each oracle is printed.
|
||||
|
||||
## Development
|
||||
|
||||
|
||||
Reference in New Issue
Block a user