# Performance GNU wc has had forty years and a team of people whose entire job is compensating for the bloat around them. Here is what they bought with that time, and what we paid for it. ## The scoreboard, with receipts Benchmarked on an AMD Ryzen AI 7 PRO 350 — 8 cores / 16 threads, boost up to 5.09 GHz, 384 KiB L1d / 256 KiB L1i / 8 MiB L2 / 16 MiB L3 — with 64 GiB of DDR5-4800 dual-channel. Opponents: GNU coreutils 9.11, busybox 1.36.1, toybox 0.8.13. Page-cache-warm files, min of 3 interleaved runs, both sides timed at microsecond resolution by a C timer. Only *raced* cases appear below — anything the reference finishes in under 5 ms is startup, not throughput, and gets reported but excluded (see "On startup" below). Every number survived contact with the contract. | Case | GNU coreutils | busybox | toybox | fastwc | |------|--------------:|--------:|-------:|-------:| | words, 1M lines (11 MB) | 7.9ms | 16.2ms | 15.9ms | **1.6ms** | | words, 10M lines (110 MB) | 66ms | 160ms | 157ms | **5.1ms** | | lines, 1M (11 MB) | startup | 16.9ms | 17.6ms | **1.6ms** | | lines, 10M (110 MB) | 5.8ms | 160ms | 154ms | **4.6ms** | | lines, 100M (monster) | 47.5ms | — | — | **29.2ms** | | lines, 1B (solo) | — | — | — | **266ms** | | stdin lines, 10M | 5.9ms | 161ms | 154ms | **4.6ms** | | stdin words, 1M | 7.1ms | 16.0ms | 16.5ms | **1.5ms** | That is a 13x win over GNU on 10M lines of words, 4.9x on 1M, and 1.6x on the 100M monster. GNU's threaded counter manages a genuine dead heat at 110 MB of plain lines (5.8ms vs our 4.6ms — it never wins; we allow it a 2% tie margin and it still can't take it). The moment the file stops fitting in a polite buffer, the dead heat stops being polite. At 1B lines — 11 GB, which this machine's 64 GiB actually fits — the solo run lands at 266ms: 41.4 GB/s, 3.76 billion lines per second. The reference is us now. Busybox and toybox, meanwhile, are here for the cruelty: 10-31x slower depending on the case, and their word counting has *bugs*. ## The locale tax, gone Every number above is C locale — the setting that flatters the opponents most: GNU `wc -w` under a UTF-8 locale stops counting bytes and decodes every one of them through `mbrtowc`, even when the file is pure ASCII and decoding changes nothing. We used to pay that same tax: the multibyte gate looked only at `MB_CUR_MAX`, so a UTF-8 locale silently traded the SIMD kernels for the decoder, and the 11 MB words race above flipped from a 4.9x win to a 25% loss against GNU. The kernels now double as a probe — one vector move-mask per load flags the first byte ≥ 0x80, free when unused — so only files that actually contain a high byte fall back to the decoder. Receipts, min-of-N interleaved, `en_US.UTF-8`, the same 11 MB ASCII words file: | Case | GNU coreutils | fastwc | gap | |------|--------------:|-------:|----:| | words | 10.43ms | **1.38ms** | 7.6x | | default (`-lwc`) | 10.55ms | **1.50ms** | 7.0x | | characters (`-m`) | 10.58ms | **2.06ms** | 5.1x | | longest line (`-L`) | 10.56ms | **6.83ms** | 1.5x | GNU's decoder bill for that file is unchanged: 10.4ms, for bytes that were never multibyte. Files that genuinely are multibyte still decode at parity — 10.5 MB of mixed CJK+latin, 56.7ms against GNU's 56.9ms — because there both sides decode. The one case GNU keeps is *lightly* multibyte files: sparse UTF-8 costs us one wasted fast pass before the fallback (793 KB, 1.30ms vs GNU's 1.18ms). We judged the tax worth it; ASCII is the rule, multibyte is the exception. ## On startup A word counter that loses one-line races to a slower counter is not slower — it's slower to *start*, and the benchmark used to punish that with integer-millisecond `date`-fork rounding. We fixed the honest half and then the honest reporting: - **What startup used to cost.** Staged-exit probes with a min-of-N exec timer showed ~105µs of avoidable work above the glibc exec floor: `setlocale` (~44µs), a `nl_langinfo` quote-style probe used only by one error message (~26µs), two 256-entry `isspace`/`isprint` table fills (~39µs), and runtime CPU dispatch — all of it paid even for `--help` and `-c`. - **What we did about it.** The locale chain (`LC_ALL` → `LC_CTYPE` → `LANG`) is resolved first and `setlocale` is skipped entirely when it resolves to C/POSIX or is unset — glibc's initial locale *is* C, so the semantics are identical. The quote-style probe went lazy (only the `--total=` error diagnostics ever print curly quotes). The whitespace/print tables are constant-filled under the C locale and built only when the flags need them (`-w`, `-L`). Option parsing moved ahead of everything, so `--help`, `--version`, and usage errors exit with zero startup cost. - **The receipts.** Min-of-400 interleaved on an 11-byte file: fastwc ~0.25ms, GNU ~0.33ms, toybox ~0.37ms, busybox ~0.14ms. Before the work, fastwc `-w` on a tiny file measured ~562µs; after, ~425µs. Nobody will ever notice a difference that small. Those were the dynamic-link numbers; the static musl default below starts ~3x sooner than even those. - **The honest half.** Because those microseconds don't matter, the benchmark no longer pretends they do. Any case the reference finishes in under 5ms is filed under `startup-bound`: fastwc must still match the count, but the case is excluded from the averages and the throughput scoreboard. Busybox used to win these outright — it is a static musl binary, and skipping the dynamic loader bought it the better part of a hundred microseconds on every exec. That excuse retired itself when the default build went static musl too (see the README): on a 12-byte file, min-of-400, fastwc `-l` now lands at 78µs against busybox's 83µs, GNU's 253µs, and toybox's 254µs. The cases in the table above are the ones where counting takes longer than starting. ## Why it's fast 1. **SIMD kernels, dispatched at runtime.** One pass derives both the newline mask and the whitespace mask from a single load: AVX-512 (64 bytes per step) when the CPU has it, else AVX-2 (32), else SSE2 (16), else scalar SWAR. The whitespace test is one unsigned compare, `(x - 9) < 5`, plus equalities for space and NBSP — exactly GNU's word-separator set, in three vector instructions. Dispatch itself is lazy: the CPUID probe runs only when a count will actually use it. 2. **Regular files are mapped, not streamed.** `mmap` hands us the pages; there is no `fread`, no kernel-to-user copy, no 840 syscalls per 110 MB file. One syscall in, one count out. A stdin redirect from a regular file (`fastwc -l < file`) gets the same treatment — the data comes through stdin, but how we read it is our business. The stdin suite is why this shows up in the scoreboard too. 3. **Parallel across cores.** Files over 8 MiB are split into 64-byte aligned slices counted by up to one thread per core (capped at 24) past 256 MiB, 12 past 32 MiB, 4 past 8 MiB. The kernels are pure, so the split needs no locks; word boundaries between slices are seeded from the byte before the slice, which makes the split exact. Below 8 MiB the thread spawn would cost more than the counting, so we don't bother. 4. **No work that isn't asked for.** `-c` on a regular file is `st_size` from `fstat` — GNU figured that one out too, so we copied the good idea. `-l` without `-w` skips the whitespace mask entirely; `-w` without `-L` never builds the print table. 5. **ASCII pays nothing, even in a UTF-8 locale.** Multibyte decoding is expensive, so we don't volunteer for it. The SIMD kernels double as a probe: when asked, they flag the first byte ≥ 0x80 with a vector move-mask — no extra pass, no cost on pure-ASCII input. A file that stays pure ASCII keeps the full-speed byte path, and its counts are identical to what the decoder would produce, because ASCII decodes to itself. Only files that actually contain a high byte pay for the multibyte decoder, and then only from the first high byte on. ## Correctness is the other half of the contract The benchmark compares counts, not just clocks — against all three oracles. Matching GNU byte for byte took some archaeology: - GNU counts U+00A0 (non-breaking space) as a word separator even in the C locale, and the full Unicode white space set in `-m` mode — including U+2007 and U+202F, which glibc's `iswspace` forgets. - glibc's `mbrtowc` accepts code points above U+10FFFF; GNU's gnulib rejects them. So do we. - In a single-byte locale, GNU's `-m` counts bytes, not characters. So do we now. - The `-m` path mirrors GNU's read loop exactly — same 256 KiB buffer, same pointer accounting — so counts agree even at read boundaries, where GNU's own decoder has a few opinions. The selftest (`cc -DFASTWC_SELFTEST src/main.c`) checks every kernel against the scalar reference over all sizes, both carry states, and both counting modes, plus the threaded slice split. A randomized fuzz against GNU as oracle (both locales, binary and UTF-8 data, every flag combination) passes 100%. ## Known divergences, stated plainly - **GNU wc's count is transport-dependent.** The same bytes piped into `wc -mw` can count differently than the same bytes read from a file, because 64 KiB pipe chunks trigger a re-scan of its carried bytes. We reproduced this, then declined to. fastwc counts the data, not the plumbing. ## Reproducing ```sh make bench # build release + run every suite ./benchmarks/test-all.sh # words, lines (monsters included), stdin ``` `test-all.sh` compiles its two helper tools on the spot (`tools/genfile` for the test data, `tools/timeit` for microsecond timing) — no separate setup. The suites live in `benchmarks/files/{lines,words}` and `benchmarks/stdin/piping`, and race all three oracles in one run. The lines suite ends with the monsters: 100M lines raced against coreutils, and 1B lines timed solo (no reference to beat — the reference is us now).