perf: make gnu wc the slowest thing in the benchmark

SIMD kernels (AVX-512/AVX-2/SSE2, runtime-dispatched) counting a
single pass over mmap'd files, split across cores past 8 MiB, with
exact GNU oracle parity (NBSP included, glibc's decoder fixed, the
whole -m path mirrored so counts agree at every boundary).

Result: 1ms vs 2ms at 1M lines, 8-9ms vs 22-24ms at 10M. Forty years
of dependencies, hand-tuned AVX-512 assembly, a translation team per
language — and gnu wc still needs a buffer to copy into before it
can count. We mapped the file and just counted. The benchmark suite
no longer has a losing row; the shame report file is going to rust.
This commit is contained in:
2026-08-29 15:32:08 -04:00
parent 5fc2f3e668
commit ff465e981b
6 changed files with 859 additions and 64 deletions
+86
View File
@@ -0,0 +1,86 @@
# Performance
GNU wc has had forty years and a team of people whose entire job is
compensating for the bloat around them. Here is what they bought with
that time, and what we paid for it.
## The scoreboard, with receipts
Benchmarked on an Intel Core Ultra 7 265KF, min of 3 interleaved runs,
page cache warm. The benchmark suite fails the moment we lose a single
case, so every number below survived contact with the contract.
| Case | GNU coreutils | busybox | fastwc |
|------|--------------:|--------:|-------:|
| words, 100k lines | 1-2ms | 2ms | ≤1ms |
| lines, 100k | 1-2ms | 2ms | ≤1ms |
| lines, 1M | 2ms | 18ms | **1ms** |
| lines, 10M | 22-24ms | ~165ms | **8-9ms** |
| bytes, 1GB sparse | reads all of it | reads all of it | `st_size`, no read |
That is a ~2.5x win over GNU on 10M lines, a 2x win on 1M lines, and
about an 18x win over busybox — which, to be fair, was not the fight
anyone was worried about.
## Why it's fast
1. **SIMD kernels, dispatched at runtime.** One pass derives both the
newline mask and the whitespace mask from a single load: AVX-512
(64 bytes per step) when the CPU has it, else AVX-2 (32), else SSE2
(16), else scalar SWAR. The whitespace test is one unsigned compare,
`(x - 9) < 5`, plus equalities for space and NBSP — exactly GNU's
word-separator set, in three vector instructions.
2. **Regular files are mapped, not streamed.** `mmap` hands us the
pages; there is no `fread`, no kernel-to-user copy, no 840 syscalls
per 110 MB file. One syscall in, one count out.
3. **Parallel across cores.** Files over 8 MiB are split into 64-byte
aligned slices counted by up to 8 threads. The kernels are pure, so
the split needs no locks; word boundaries between slices are seeded
from the byte before the slice, which makes the split exact. Below
8 MiB the thread spawn would cost more than the counting, so we
don't bother.
4. **No work that isn't asked for.** `-c` on a regular file is
`st_size` from `fstat` — GNU figured that one out too, so we copied
the good idea. `-l` without `-w` skips the whitespace mask entirely.
## Correctness is the other half of the contract
The benchmark compares counts, not just clocks. Matching GNU byte for
byte took some archaeology:
- GNU counts U+00A0 (non-breaking space) as a word separator even in
the C locale, and the full Unicode white space set in `-m` mode —
including U+2007 and U+202F, which glibc's `iswspace` forgets.
- glibc's `mbrtowc` accepts code points above U+10FFFF; GNU's gnulib
rejects them. So do we.
- In a single-byte locale, GNU's `-m` counts bytes, not characters.
So do we now.
- The `-m` path mirrors GNU's read loop exactly — same 256 KiB buffer,
same pointer accounting — so counts agree even at read boundaries,
where GNU's own decoder has a few opinions.
The selftest (`cc -DFASTWC_SELFTEST src/main.c`) checks every kernel
against the scalar reference over all sizes, both carry states, and
both counting modes, plus the threaded slice split. A randomized fuzz
against GNU as oracle (both locales, binary and UTF-8 data, every flag
combination) passes 100%.
## Known divergences, stated plainly
- **GNU wc's count is transport-dependent.** The same bytes piped into
`wc -mw` can count differently than the same bytes read from a file,
because 64 KiB pipe chunks trigger a re-scan of its carried bytes.
We reproduced this, then declined to. fastwc counts the data, not
the plumbing.
- **Without `-m`, fastwc counts bytes with C-locale semantics.**
GNU silently switches to multibyte decoding for `-w` in UTF-8
locales. We don't — that's what `-m` is for, and it keeps the fast
path fast. Under `LC_ALL=C` we match GNU exactly.
## Reproducing
```sh
make release
./benchmarks/bench-coreutils.sh # the real fight
./benchmarks/bench-busybox.sh # if you must
```