docs: the locale tax, gone — utf-8 receipts and the startup race won

README.md

@@ -48,6 +48,20 @@ count, and the case is reported, but it is excluded from the averages.
 Raced cases allow a 2% dead-heat margin so a genuine tie can't flake
 on scheduler jitter. All of it, exactly as `benchmarks/` prescribes.
 
+And every number above was measured in the C locale — the setting that
+flatters the opponents most. Under `en_US.UTF-8`, GNU `wc` stops
+counting bytes and starts decoding them, one `mbrtowc` at a time, even
+when the file is pure ASCII and decoding changes nothing. We used to
+make that exact mistake: a UTF-8 locale silently swapped our SIMD
+kernels for the same decoder, and the 11 MB words race flipped from a
+4.9x win to a GNU win. The kernels now probe for non-ASCII bytes while
+they count — a vector move-mask per load, free when unused — so ASCII
+files never see the decoder. Same 11 MB words file, the locale you
+actually run: **1.4 ms vs GNU 10.4 ms (7.6x).** GNU still pays that
+10.4 ms for bytes that were never multibyte; we pay for the decoder
+only when a file genuinely needs it. Receipts in
+[docs/PERFORMANCE.md](docs/PERFORMANCE.md).
+
 The moment fastwc is slower than any of them, this project has failed
 and you should say so loudly in an issue. The benchmark is the
 contract. The how and why of the speed, with receipts, lives in
@@ -61,21 +75,26 @@ opponent that keeps score, and even its threaded counter only manages
 a dead heat on mid-size files — never a win, and the moment the file
 stops fitting in a polite buffer, the dead heat stops being polite.
 
-A note on startup, in the interest of honesty: on a one-line file the
+A note on startup, in the interest of honesty — and of gloating: on a
-whole race happens in the low hundreds of microseconds — fastwc
+one-line file the whole race happens in the low hundreds of
-~0.25 ms, GNU ~0.33 ms, toybox ~0.37 ms, busybox ~0.14 ms. Microseconds
+microseconds, and fastwc now wins it outright. The default build is
-either way. Nobody will ever notice a difference that small, and it
+static musl, so there is no dynamic loader to pay: min-of-400 on a
-does not matter in the bigger picture: the tiny cases in the table are
+12-byte file puts fastwc `-l` at 78µs against busybox's 83µs, GNU's
-here to prove fastwc is never *wrong*, not to brag about a head start
+253µs, and toybox's 254µs. Busybox's one structural advantage — a
-that evaporates the moment the page cache warms up. That is why the
+loader it never had to start — is no longer an advantage; we don't
-suite now times both sides at microsecond resolution and files anything
+start one either. None of this matters in the bigger picture, and we
-the reference finishes in under 5 ms under "startup-bound": correct,
+will not pretend otherwise: nobody will ever notice a difference that
-reported, and excluded from the averages — because nobody should be
+small, and the tiny cases in the table are here to prove fastwc is
-racing startup, least of all a word counter. The cases that matter are
+never *wrong*, not to brag about a head start that evaporates the
-the ones where counting takes longer than starting — and those are the
+moment the page cache warms up. That is why the suite now times both
-ones in this table. Every run ends with the average speedup against
+sides at microsecond resolution and files anything the reference
-each oracle — coreutils ~4.5x, busybox ~13-15x, toybox ~15-18x — so
+finishes in under 5 ms under "startup-bound": correct, reported, and
-the cruelty is quantified.
+excluded from the averages — because nobody should be racing startup,
+least of all a word counter. The cases that matter are the ones where
+counting takes longer than starting — and those are the ones in this
+table. Every run ends with the average speedup against each oracle —
+coreutils ~4.5x, busybox ~13-15x, toybox ~15-18x — so the cruelty is
+quantified.
 
 ## Why
 
@@ -89,6 +108,12 @@ the cruelty is quantified.
   now: regular files are mapped and counted in parallel across cores,
   with SIMD kernels (AVX-512, AVX-2, SSE2) dispatched at runtime —
   zero function calls in the hot path.
+- **GNU wc slows down in the locale you actually run.** Under a UTF-8
+  locale it decodes every byte it counts — pure ASCII included, which
+  decoding cannot change — so the 11 MB words file that takes it
+  7.9 ms in the C locale takes 10.4 ms there, against our 1.4 ms.
+  fastwc's kernels probe for non-ASCII bytes as they count and only
+  decode files that need it. The locale that taxes GNU is free for us.
 
 ## What it does
 
@@ -117,10 +142,19 @@ fastwc [OPTION]... --files0-from=F
 Requires a C compiler and autotools. That's it. No gettext. No gnulib.
 No translators.
 
+The default build links statically against musl (via `musl-gcc`),
+which is why fastwc now wins the startup cases above outright —
+there is no dynamic loader to pay, and the ~80µs exec floor is the
+same one busybox pays. If `musl-gcc` isn't installed the configure
+script warns and falls back to the system compiler; the glibc build
+is one flag away:
+
 ```sh
-./autogen.sh     # autoreconf -fi && ./configure
+./autogen.sh            # autoreconf -fi && ./configure (static musl)
 make
 make release            # installs the release binary to bin/release/fastwc
+
+./configure --enable-glibc && make   # or: dynamic glibc build
 ```
 
 ## Benchmark

docs/PERFORMANCE.md

@@ -38,6 +38,36 @@ second. The reference is us now. Busybox and toybox, meanwhile, are
 here for the cruelty: 10-31x slower depending on the case, and their
 word counting has *bugs*.
 
+## The locale tax, gone
+
+Every number above is C locale — the setting that flatters the
+opponents most: GNU `wc -w` under a UTF-8 locale stops counting bytes
+and decodes every one of them through `mbrtowc`, even when the file is
+pure ASCII and decoding changes nothing. We used to pay that same tax:
+the multibyte gate looked only at `MB_CUR_MAX`, so a UTF-8 locale
+silently traded the SIMD kernels for the decoder, and the 11 MB words
+race above flipped from a 4.9x win to a 25% loss against GNU.
+
+The kernels now double as a probe — one vector move-mask per load
+flags the first byte ≥ 0x80, free when unused — so only files that
+actually contain a high byte fall back to the decoder. Receipts,
+min-of-N interleaved, `en_US.UTF-8`, the same 11 MB ASCII words file:
+
+| Case | GNU coreutils | fastwc | gap |
+|------|--------------:|-------:|----:|
+| words | 10.43ms | **1.38ms** | 7.6x |
+| default (`-lwc`) | 10.55ms | **1.50ms** | 7.0x |
+| characters (`-m`) | 10.58ms | **2.06ms** | 5.1x |
+| longest line (`-L`) | 10.56ms | **6.83ms** | 1.5x |
+
+GNU's decoder bill for that file is unchanged: 10.4ms, for bytes that
+were never multibyte. Files that genuinely are multibyte still decode
+at parity — 10.5 MB of mixed CJK+latin, 56.7ms against GNU's 56.9ms —
+because there both sides decode. The one case GNU keeps is *lightly*
+multibyte files: sparse UTF-8 costs us one wasted fast pass before the
+fallback (793 KB, 1.30ms vs GNU's 1.18ms). We judged the tax worth
+it; ASCII is the rule, multibyte is the exception.
+
 ## On startup
 
 A word counter that loses one-line races to a slower counter is not
@@ -63,15 +93,21 @@ half and then the honest reporting:
 - **The receipts.** Min-of-400 interleaved on an 11-byte file: fastwc
   ~0.25ms, GNU ~0.33ms, toybox ~0.37ms, busybox ~0.14ms. Before the
   work, fastwc `-w` on a tiny file measured ~562µs; after, ~425µs.
-  Nobody will ever notice a difference that small.
+  Nobody will ever notice a difference that small. Those were the
+  dynamic-link numbers; the static musl default below starts ~3x
+  sooner than even those.
 - **The honest half.** Because those microseconds don't matter, the
   benchmark no longer pretends they do. Any case the reference
   finishes in under 5ms is filed under `startup-bound`: fastwc must
   still match the count, but the case is excluded from the averages
-  and the throughput scoreboard. Busybox's genuinely faster startup
+  and the throughput scoreboard. Busybox used to win these outright —
-  (242µs vs our 534µs on one line) is reported exactly that way. The
+  it is a static musl binary, and skipping the dynamic loader bought
-  cases in the table above are the ones where counting takes longer
+  it the better part of a hundred microseconds on every exec. That
-  than starting.
+  excuse retired itself when the default build went static musl too
+  (see the README): on a 12-byte file, min-of-400, fastwc `-l` now
+  lands at 78µs against busybox's 83µs, GNU's 253µs, and toybox's
+  254µs. The cases in the table above are the ones where counting
+  takes longer than starting.
 
 ## Why it's fast
 
@@ -99,6 +135,15 @@ half and then the honest reporting:
    `st_size` from `fstat` — GNU figured that one out too, so we copied
    the good idea. `-l` without `-w` skips the whitespace mask entirely;
    `-w` without `-L` never builds the print table.
+5. **ASCII pays nothing, even in a UTF-8 locale.** Multibyte decoding
+   is expensive, so we don't volunteer for it. The SIMD kernels
+   double as a probe: when asked, they flag the first byte ≥ 0x80
+   with a vector move-mask — no extra pass, no cost on pure-ASCII
+   input. A file that stays pure ASCII keeps the full-speed byte
+   path, and its counts are identical to what the decoder would
+   produce, because ASCII decodes to itself. Only files that actually
+   contain a high byte pay for the multibyte decoder, and then only
+   from the first high byte on.
 
 ## Correctness is the other half of the contract
 
@@ -129,10 +174,6 @@ combination) passes 100%.
   because 64 KiB pipe chunks trigger a re-scan of its carried bytes.
   We reproduced this, then declined to. fastwc counts the data, not
   the plumbing.
-- **Without `-m`, fastwc counts bytes with C-locale semantics.**
-  GNU silently switches to multibyte decoding for `-w` in UTF-8
-  locales. We don't — that's what `-m` is for, and it keeps the fast
-  path fast. Under `LC_ALL=C` we match GNU exactly.
 
 ## Reproducing