Performance

GNU wc has had forty years and a team of people whose entire job is compensating for the bloat around them. Here is what they bought with that time, and what we paid for it.

The scoreboard, with receipts

Benchmarked on an AMD Ryzen AI 7 PRO 350 — 8 cores / 16 threads, boost up to 5.09 GHz, 384 KiB L1d / 256 KiB L1i / 8 MiB L2 / 16 MiB L3 — with 64 GiB of DDR5-4800 dual-channel. Opponents: GNU coreutils 9.11, busybox 1.36.1, toybox 0.8.13. Page-cache-warm files, min of 3 interleaved runs, both sides timed at microsecond resolution by a C timer. Only raced cases appear below — anything the reference finishes in under 5 ms is startup, not throughput, and gets reported but excluded (see "On startup" below). Every number survived contact with the contract.

Case	GNU coreutils	busybox	toybox	fastwc
words, 1M lines (11 MB)	7.9ms	16.2ms	15.9ms	1.6ms
words, 10M lines (110 MB)	66ms	160ms	157ms	5.1ms
lines, 1M (11 MB)	startup	16.9ms	17.6ms	1.6ms
lines, 10M (110 MB)	5.8ms	160ms	154ms	4.6ms
lines, 100M (monster)	47.5ms	—	—	29.2ms
lines, 1B (solo)	—	—	—	266ms
stdin lines, 10M	5.9ms	161ms	154ms	4.6ms
stdin words, 1M	7.1ms	16.0ms	16.5ms	1.5ms

That is a 13x win over GNU on 10M lines of words, 4.9x on 1M, and 1.6x on the 100M monster. GNU's threaded counter manages a genuine dead heat at 110 MB of plain lines (5.8ms vs our 4.6ms — it never wins; we allow it a 2% tie margin and it still can't take it). The moment the file stops fitting in a polite buffer, the dead heat stops being polite. At 1B lines — 11 GB, which this machine's 64 GiB actually fits — the solo run lands at 266ms: 41.4 GB/s, 3.76 billion lines per second. The reference is us now. Busybox and toybox, meanwhile, are here for the cruelty: 10-31x slower depending on the case, and their word counting has bugs.

The locale tax, gone

Every number above is C locale — the setting that flatters the opponents most: GNU wc -w under a UTF-8 locale stops counting bytes and decodes every one of them through mbrtowc, even when the file is pure ASCII and decoding changes nothing. We used to pay that same tax: the multibyte gate looked only at MB_CUR_MAX, so a UTF-8 locale silently traded the SIMD kernels for the decoder, and the 11 MB words race above flipped from a 4.9x win to a 25% loss against GNU.

The kernels now double as a probe — one vector move-mask per load flags the first byte ≥ 0x80, free when unused — so only files that actually contain a high byte fall back to the decoder. Receipts, min-of-N interleaved, en_US.UTF-8, the same 11 MB ASCII words file:

Case	GNU coreutils	fastwc	gap
words	10.43ms	1.38ms	7.6x
default (`-lwc`)	10.55ms	1.50ms	7.0x
characters (`-m`)	10.58ms	2.06ms	5.1x
longest line (`-L`)	10.56ms	6.83ms	1.5x

GNU's decoder bill for that file is unchanged: 10.4ms, for bytes that were never multibyte. Files that genuinely are multibyte still decode at parity — 10.5 MB of mixed CJK+latin, 56.7ms against GNU's 56.9ms — because there both sides decode. The one case GNU keeps is lightly multibyte files: sparse UTF-8 costs us one wasted fast pass before the fallback (793 KB, 1.30ms vs GNU's 1.18ms). We judged the tax worth it; ASCII is the rule, multibyte is the exception.

On startup

A word counter that loses one-line races to a slower counter is not slower — it's slower to start, and the benchmark used to punish that with integer-millisecond date-fork rounding. We fixed the honest half and then the honest reporting:

What startup used to cost. Staged-exit probes with a min-of-N exec timer showed ~105µs of avoidable work above the glibc exec floor: setlocale (~44µs), a nl_langinfo quote-style probe used only by one error message (~26µs), two 256-entry isspace/isprint table fills (~39µs), and runtime CPU dispatch — all of it paid even for --help and -c.
What we did about it. The locale chain (LC_ALL → LC_CTYPE → LANG) is resolved first and setlocale is skipped entirely when it resolves to C/POSIX or is unset — glibc's initial locale is C, so the semantics are identical. The quote-style probe went lazy (only the --total= error diagnostics ever print curly quotes). The whitespace/print tables are constant-filled under the C locale and built only when the flags need them (-w, -L). Option parsing moved ahead of everything, so --help, --version, and usage errors exit with zero startup cost.
The receipts. Min-of-400 interleaved on an 11-byte file: fastwc ~0.25ms, GNU ~0.33ms, toybox ~0.37ms, busybox ~0.14ms. Before the work, fastwc -w on a tiny file measured ~562µs; after, ~425µs. Nobody will ever notice a difference that small. Those were the dynamic-link numbers; the static musl default below starts ~3x sooner than even those.
The honest half. Because those microseconds don't matter, the benchmark no longer pretends they do. Any case the reference finishes in under 5ms is filed under startup-bound: fastwc must still match the count, but the case is excluded from the averages and the throughput scoreboard. Busybox used to win these outright — it is a static musl binary, and skipping the dynamic loader bought it the better part of a hundred microseconds on every exec. That excuse retired itself when the default build went static musl too (see the README): on a 12-byte file, min-of-400, fastwc -l now lands at 78µs against busybox's 83µs, GNU's 253µs, and toybox's 254µs. The cases in the table above are the ones where counting takes longer than starting.

Why it's fast

SIMD kernels, dispatched at runtime. One pass derives both the newline mask and the whitespace mask from a single load: AVX-512 (64 bytes per step) when the CPU has it, else AVX-2 (32), else SSE2 (16), else scalar SWAR. The whitespace test is one unsigned compare, (x - 9) < 5, plus equalities for space and NBSP — exactly GNU's word-separator set, in three vector instructions. Dispatch itself is lazy: the CPUID probe runs only when a count will actually use it.
Regular files are mapped, not streamed. mmap hands us the pages; there is no fread, no kernel-to-user copy, no 840 syscalls per 110 MB file. One syscall in, one count out. A stdin redirect from a regular file (fastwc -l < file) gets the same treatment — the data comes through stdin, but how we read it is our business. The stdin suite is why this shows up in the scoreboard too.
Parallel across cores. Files over 8 MiB are split into 64-byte aligned slices counted by up to one thread per core (capped at 24) past 256 MiB, 12 past 32 MiB, 4 past 8 MiB. The kernels are pure, so the split needs no locks; word boundaries between slices are seeded from the byte before the slice, which makes the split exact. Below 8 MiB the thread spawn would cost more than the counting, so we don't bother.
No work that isn't asked for. -c on a regular file is st_size from fstat — GNU figured that one out too, so we copied the good idea. -l without -w skips the whitespace mask entirely; -w without -L never builds the print table.
ASCII pays nothing, even in a UTF-8 locale. Multibyte decoding is expensive, so we don't volunteer for it. The SIMD kernels double as a probe: when asked, they flag the first byte ≥ 0x80 with a vector move-mask — no extra pass, no cost on pure-ASCII input. A file that stays pure ASCII keeps the full-speed byte path, and its counts are identical to what the decoder would produce, because ASCII decodes to itself. Only files that actually contain a high byte pay for the multibyte decoder, and then only from the first high byte on.

Correctness is the other half of the contract

The benchmark compares counts, not just clocks — against all three oracles. Matching GNU byte for byte took some archaeology:

GNU counts U+00A0 (non-breaking space) as a word separator even in the C locale, and the full Unicode white space set in -m mode — including U+2007 and U+202F, which glibc's iswspace forgets.
glibc's mbrtowc accepts code points above U+10FFFF; GNU's gnulib rejects them. So do we.
In a single-byte locale, GNU's -m counts bytes, not characters. So do we now.
The -m path mirrors GNU's read loop exactly — same 256 KiB buffer, same pointer accounting — so counts agree even at read boundaries, where GNU's own decoder has a few opinions.

The selftest (cc -DFASTWC_SELFTEST src/main.c) checks every kernel against the scalar reference over all sizes, both carry states, and both counting modes, plus the threaded slice split. A randomized fuzz against GNU as oracle (both locales, binary and UTF-8 data, every flag combination) passes 100%.

Known divergences, stated plainly

GNU wc's count is transport-dependent. The same bytes piped into wc -mw can count differently than the same bytes read from a file, because 64 KiB pipe chunks trigger a re-scan of its carried bytes. We reproduced this, then declined to. fastwc counts the data, not the plumbing.

Reproducing

make bench                            # build release + run every suite
./benchmarks/test-all.sh              # words, lines (monsters included), stdin

test-all.sh compiles its two helper tools on the spot (tools/genfile for the test data, tools/timeit for microsecond timing) — no separate setup. The suites live in benchmarks/files/{lines,words} and benchmarks/stdin/piping, and race all three oracles in one run. The lines suite ends with the monsters: 100M lines raced against coreutils, and 1B lines timed solo (no reference to beat — the reference is us now).

9.9 KiB Raw Permalink Blame History