huntedbytheirs/fastwc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

bench: rub it in — monsters, stdin wins, busybox retired

Browse Source 
Suites split into benchmarks/files/{lines,words} and
benchmarks/stdin/piping, with the monsters bolted onto the lines suite:
100M lines raced against coreutils (~2x win), 1B lines solo (~6-8s,
11 GB in one pass). Stdin redirects from regular files are now mmap'd
in count_stream, so the stdin suite wins too — up to 12.00x on words.

The ratio column now reports how many times faster fastwc is, not how
much of GNU's time it used. busybox was removed from the suite: it
stopped being a challenge and started being a participation trophy.
GNU wc's lone win — 1M lines by one millisecond on hand-tuned AVX-512
assembly — is now a historical footnote, and the README says so.
This commit is contained in:
huntedbytheirs
2026-08-29 15:52:24 -04:00
parent ff465e981b
commit 6aa461f053
13 changed files with 362 additions and 177 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+27 -12
View File
@@ -17,10 +17,9 @@ correctly, at full speed.
## The scoreboard
The benchmark suite in `benchmarks/` races fastwc against GNU `wc`
(and busybox, if you keep such things installed) — fail-fast. The moment
we are slower, or disagree on a single count, it writes a shame report
and exits non-zero. These are the facts:
The benchmark suite in `benchmarks/` races fastwc against GNU `wc` —
fail-fast. The moment we are slower, or disagree on a single count, it
writes a shame report and exits non-zero. These are the facts:
| Suite | Result |
|-------|--------|
@@ -28,19 +27,31 @@ and exits non-zero. These are the facts:
| lines (up to 100k lines) | **Wins.** GNU never sees us coming. |
| lines (1M lines) | **Win: 1ms vs 2ms.** GNU's AVX-512 assist can't beat a mapped file. |
| lines (10M lines) | **Win: 8-9ms vs 22-24ms (~2.5x).** GNU's lead never survives contact with the buffer. |
| busybox lines (10M) | **Win: 8-9ms vs ~165ms (~18x).** If you must. |
| lines (100M lines) | **Win: ~70ms vs ~140ms.** The monster race. GNU gets lapped. |
| lines (1B lines) | **Solo, ~6-8s.** 11 GB in one pass; the only bottleneck left is the disk. |
| stdin words (1M lines) | **Win: 12x.** GNU still reads stdin like it's 1985. |
| stdin lines (10M lines) | **Win: ~2.5x.** We map stdin redirects; GNU maps nothing. |
The moment fastwc is slower than GNU `wc`, this project has failed and
you should say so loudly in an issue. The benchmark is the contract.
The how and why of the speed, with receipts, lives in
[docs/PERFORMANCE.md](docs/PERFORMANCE.md).
GNU wc's lone win — 1M lines, by a single millisecond, on hand-tuned
AVX-512 assembly — is a historical footnote now. We closed it, then the
10M case, then the 100M monster, and then we added a 1B-line race with
no opponent, because beating GNU wc at this point is just cruelty to
software. Busybox was removed from the suite for the same reason: it
stopped being a challenge and started being a participation trophy.
## Why
- **GNU wc is a dependency museum.** Its build needs gettext, gnulib,
and a translator for every language on Earth. fastwc needs `cc`.
- **GNU wc is slow where it should be fast.** Counting bytes is not
supposed to be an architectural achievement.
supposed to be an architectural achievement. Counting 11 GB of them
in one pass, solo, while GNU's AVX-512 assembly still needs a buffer
to copy into — that is.
- **GNU wc counts like it's 1985** — because it is. We count like it's
now: regular files are mapped and counted in parallel across cores,
with SIMD kernels (AVX-512, AVX-2, SSE2) dispatched at runtime —
@@ -72,14 +83,18 @@ make release # installs the release binary to bin/release/fastwc
```sh
make bench # build release + run every suite
./benchmarks/bench-coreutils.sh # the real fight
./benchmarks/bench-busybox.sh # if you must
./benchmarks/test-all.sh # all suites: words, lines, stdin
```
The suites interleave runs so both commands see identical cache warmth,
keep the minimum, and fail the moment fastwc loses a single case. GNU
`wc` is used as an oracle the same way you'd use a broken clock:
occasionally it's right, and it's the only one around.
The suites live under `benchmarks/files/{lines,words}` (file input) and
`benchmarks/stdin/piping` (standard input), and fail fast the moment
fastwc loses a single case. The lines suite also carries the monsters:
100M lines raced against coreutils, and 1B lines timed solo — there is
no reference for that one; we are the reference now. The suites
interleave runs so both commands see identical cache warmth, keep the
minimum, and fail the moment fastwc loses a single case. GNU `wc` is
used as an oracle the same way you'd use a broken clock: occasionally
it's right, and it's the only one around.
## Development