docs: the locale tax, gone — utf-8 receipts and the startup race won

This commit is contained in:
2026-09-09 16:10:19 -04:00
parent e52e97a825
commit f8361c31a3
2 changed files with 101 additions and 26 deletions
+50 -9
View File
@@ -38,6 +38,36 @@ second. The reference is us now. Busybox and toybox, meanwhile, are
here for the cruelty: 10-31x slower depending on the case, and their
word counting has *bugs*.
## The locale tax, gone
Every number above is C locale — the setting that flatters the
opponents most: GNU `wc -w` under a UTF-8 locale stops counting bytes
and decodes every one of them through `mbrtowc`, even when the file is
pure ASCII and decoding changes nothing. We used to pay that same tax:
the multibyte gate looked only at `MB_CUR_MAX`, so a UTF-8 locale
silently traded the SIMD kernels for the decoder, and the 11 MB words
race above flipped from a 4.9x win to a 25% loss against GNU.
The kernels now double as a probe — one vector move-mask per load
flags the first byte ≥ 0x80, free when unused — so only files that
actually contain a high byte fall back to the decoder. Receipts,
min-of-N interleaved, `en_US.UTF-8`, the same 11 MB ASCII words file:
| Case | GNU coreutils | fastwc | gap |
|------|--------------:|-------:|----:|
| words | 10.43ms | **1.38ms** | 7.6x |
| default (`-lwc`) | 10.55ms | **1.50ms** | 7.0x |
| characters (`-m`) | 10.58ms | **2.06ms** | 5.1x |
| longest line (`-L`) | 10.56ms | **6.83ms** | 1.5x |
GNU's decoder bill for that file is unchanged: 10.4ms, for bytes that
were never multibyte. Files that genuinely are multibyte still decode
at parity — 10.5 MB of mixed CJK+latin, 56.7ms against GNU's 56.9ms —
because there both sides decode. The one case GNU keeps is *lightly*
multibyte files: sparse UTF-8 costs us one wasted fast pass before the
fallback (793 KB, 1.30ms vs GNU's 1.18ms). We judged the tax worth
it; ASCII is the rule, multibyte is the exception.
## On startup
A word counter that loses one-line races to a slower counter is not
@@ -63,15 +93,21 @@ half and then the honest reporting:
- **The receipts.** Min-of-400 interleaved on an 11-byte file: fastwc
~0.25ms, GNU ~0.33ms, toybox ~0.37ms, busybox ~0.14ms. Before the
work, fastwc `-w` on a tiny file measured ~562µs; after, ~425µs.
Nobody will ever notice a difference that small.
Nobody will ever notice a difference that small. Those were the
dynamic-link numbers; the static musl default below starts ~3x
sooner than even those.
- **The honest half.** Because those microseconds don't matter, the
benchmark no longer pretends they do. Any case the reference
finishes in under 5ms is filed under `startup-bound`: fastwc must
still match the count, but the case is excluded from the averages
and the throughput scoreboard. Busybox's genuinely faster startup
(242µs vs our 534µs on one line) is reported exactly that way. The
cases in the table above are the ones where counting takes longer
than starting.
and the throughput scoreboard. Busybox used to win these outright —
it is a static musl binary, and skipping the dynamic loader bought
it the better part of a hundred microseconds on every exec. That
excuse retired itself when the default build went static musl too
(see the README): on a 12-byte file, min-of-400, fastwc `-l` now
lands at 78µs against busybox's 83µs, GNU's 253µs, and toybox's
254µs. The cases in the table above are the ones where counting
takes longer than starting.
## Why it's fast
@@ -99,6 +135,15 @@ half and then the honest reporting:
`st_size` from `fstat` — GNU figured that one out too, so we copied
the good idea. `-l` without `-w` skips the whitespace mask entirely;
`-w` without `-L` never builds the print table.
5. **ASCII pays nothing, even in a UTF-8 locale.** Multibyte decoding
is expensive, so we don't volunteer for it. The SIMD kernels
double as a probe: when asked, they flag the first byte ≥ 0x80
with a vector move-mask — no extra pass, no cost on pure-ASCII
input. A file that stays pure ASCII keeps the full-speed byte
path, and its counts are identical to what the decoder would
produce, because ASCII decodes to itself. Only files that actually
contain a high byte pay for the multibyte decoder, and then only
from the first high byte on.
## Correctness is the other half of the contract
@@ -129,10 +174,6 @@ combination) passes 100%.
because 64 KiB pipe chunks trigger a re-scan of its carried bytes.
We reproduced this, then declined to. fastwc counts the data, not
the plumbing.
- **Without `-m`, fastwc counts bytes with C-locale semantics.**
GNU silently switches to multibyte decoding for `-w` in UTF-8
locales. We don't — that's what `-m` is for, and it keeps the fast
path fast. Under `LC_ALL=C` we match GNU exactly.
## Reproducing