docs: the locale tax, gone — utf-8 receipts and the startup race won
This commit is contained in:
+50
-9
@@ -38,6 +38,36 @@ second. The reference is us now. Busybox and toybox, meanwhile, are
|
||||
here for the cruelty: 10-31x slower depending on the case, and their
|
||||
word counting has *bugs*.
|
||||
|
||||
## The locale tax, gone
|
||||
|
||||
Every number above is C locale — the setting that flatters the
|
||||
opponents most: GNU `wc -w` under a UTF-8 locale stops counting bytes
|
||||
and decodes every one of them through `mbrtowc`, even when the file is
|
||||
pure ASCII and decoding changes nothing. We used to pay that same tax:
|
||||
the multibyte gate looked only at `MB_CUR_MAX`, so a UTF-8 locale
|
||||
silently traded the SIMD kernels for the decoder, and the 11 MB words
|
||||
race above flipped from a 4.9x win to a 25% loss against GNU.
|
||||
|
||||
The kernels now double as a probe — one vector move-mask per load
|
||||
flags the first byte ≥ 0x80, free when unused — so only files that
|
||||
actually contain a high byte fall back to the decoder. Receipts,
|
||||
min-of-N interleaved, `en_US.UTF-8`, the same 11 MB ASCII words file:
|
||||
|
||||
| Case | GNU coreutils | fastwc | gap |
|
||||
|------|--------------:|-------:|----:|
|
||||
| words | 10.43ms | **1.38ms** | 7.6x |
|
||||
| default (`-lwc`) | 10.55ms | **1.50ms** | 7.0x |
|
||||
| characters (`-m`) | 10.58ms | **2.06ms** | 5.1x |
|
||||
| longest line (`-L`) | 10.56ms | **6.83ms** | 1.5x |
|
||||
|
||||
GNU's decoder bill for that file is unchanged: 10.4ms, for bytes that
|
||||
were never multibyte. Files that genuinely are multibyte still decode
|
||||
at parity — 10.5 MB of mixed CJK+latin, 56.7ms against GNU's 56.9ms —
|
||||
because there both sides decode. The one case GNU keeps is *lightly*
|
||||
multibyte files: sparse UTF-8 costs us one wasted fast pass before the
|
||||
fallback (793 KB, 1.30ms vs GNU's 1.18ms). We judged the tax worth
|
||||
it; ASCII is the rule, multibyte is the exception.
|
||||
|
||||
## On startup
|
||||
|
||||
A word counter that loses one-line races to a slower counter is not
|
||||
@@ -63,15 +93,21 @@ half and then the honest reporting:
|
||||
- **The receipts.** Min-of-400 interleaved on an 11-byte file: fastwc
|
||||
~0.25ms, GNU ~0.33ms, toybox ~0.37ms, busybox ~0.14ms. Before the
|
||||
work, fastwc `-w` on a tiny file measured ~562µs; after, ~425µs.
|
||||
Nobody will ever notice a difference that small.
|
||||
Nobody will ever notice a difference that small. Those were the
|
||||
dynamic-link numbers; the static musl default below starts ~3x
|
||||
sooner than even those.
|
||||
- **The honest half.** Because those microseconds don't matter, the
|
||||
benchmark no longer pretends they do. Any case the reference
|
||||
finishes in under 5ms is filed under `startup-bound`: fastwc must
|
||||
still match the count, but the case is excluded from the averages
|
||||
and the throughput scoreboard. Busybox's genuinely faster startup
|
||||
(242µs vs our 534µs on one line) is reported exactly that way. The
|
||||
cases in the table above are the ones where counting takes longer
|
||||
than starting.
|
||||
and the throughput scoreboard. Busybox used to win these outright —
|
||||
it is a static musl binary, and skipping the dynamic loader bought
|
||||
it the better part of a hundred microseconds on every exec. That
|
||||
excuse retired itself when the default build went static musl too
|
||||
(see the README): on a 12-byte file, min-of-400, fastwc `-l` now
|
||||
lands at 78µs against busybox's 83µs, GNU's 253µs, and toybox's
|
||||
254µs. The cases in the table above are the ones where counting
|
||||
takes longer than starting.
|
||||
|
||||
## Why it's fast
|
||||
|
||||
@@ -99,6 +135,15 @@ half and then the honest reporting:
|
||||
`st_size` from `fstat` — GNU figured that one out too, so we copied
|
||||
the good idea. `-l` without `-w` skips the whitespace mask entirely;
|
||||
`-w` without `-L` never builds the print table.
|
||||
5. **ASCII pays nothing, even in a UTF-8 locale.** Multibyte decoding
|
||||
is expensive, so we don't volunteer for it. The SIMD kernels
|
||||
double as a probe: when asked, they flag the first byte ≥ 0x80
|
||||
with a vector move-mask — no extra pass, no cost on pure-ASCII
|
||||
input. A file that stays pure ASCII keeps the full-speed byte
|
||||
path, and its counts are identical to what the decoder would
|
||||
produce, because ASCII decodes to itself. Only files that actually
|
||||
contain a high byte pay for the multibyte decoder, and then only
|
||||
from the first high byte on.
|
||||
|
||||
## Correctness is the other half of the contract
|
||||
|
||||
@@ -129,10 +174,6 @@ combination) passes 100%.
|
||||
because 64 KiB pipe chunks trigger a re-scan of its carried bytes.
|
||||
We reproduced this, then declined to. fastwc counts the data, not
|
||||
the plumbing.
|
||||
- **Without `-m`, fastwc counts bytes with C-locale semantics.**
|
||||
GNU silently switches to multibyte decoding for `-w` in UTF-8
|
||||
locales. We don't — that's what `-m` is for, and it keeps the fast
|
||||
path fast. Under `LC_ALL=C` we match GNU exactly.
|
||||
|
||||
## Reproducing
|
||||
|
||||
|
||||
Reference in New Issue
Block a user