docs: the locale tax, gone — utf-8 receipts and the startup race won

Makefile.am

@@ -3,10 +3,16 @@ AM_CFLAGS = -Wall -Wextra -O2 -pthread
 bin_PROGRAMS = fastwc
 fastwc_SOURCES = src/main.c
 
+# configure.ac fills this in with -static when the default musl toolchain
+# was selected (empty for --enable-glibc builds).
+fastwc_LDFLAGS = @STATIC_LDFLAGS@
+
 # Release build consumed by benchmarks/ (expects bin/release/fastwc).
+# Strip the copy, not the tree binary: the debug build stays debuggable.
 release: all
 	$(MKDIR_P) bin/release
 	cp -f fastwc bin/release/fastwc
+	$(STRIP) bin/release/fastwc
 
 # Convenience: build the release binary, then run every benchmark suite.
 bench: release

README.md

@@ -48,6 +48,20 @@ count, and the case is reported, but it is excluded from the averages.
 Raced cases allow a 2% dead-heat margin so a genuine tie can't flake
 on scheduler jitter. All of it, exactly as `benchmarks/` prescribes.
 
+And every number above was measured in the C locale — the setting that
+flatters the opponents most. Under `en_US.UTF-8`, GNU `wc` stops
+counting bytes and starts decoding them, one `mbrtowc` at a time, even
+when the file is pure ASCII and decoding changes nothing. We used to
+make that exact mistake: a UTF-8 locale silently swapped our SIMD
+kernels for the same decoder, and the 11 MB words race flipped from a
+4.9x win to a GNU win. The kernels now probe for non-ASCII bytes while
+they count — a vector move-mask per load, free when unused — so ASCII
+files never see the decoder. Same 11 MB words file, the locale you
+actually run: **1.4 ms vs GNU 10.4 ms (7.6x).** GNU still pays that
+10.4 ms for bytes that were never multibyte; we pay for the decoder
+only when a file genuinely needs it. Receipts in
+[docs/PERFORMANCE.md](docs/PERFORMANCE.md).
+
 The moment fastwc is slower than any of them, this project has failed
 and you should say so loudly in an issue. The benchmark is the
 contract. The how and why of the speed, with receipts, lives in
@@ -61,21 +75,26 @@ opponent that keeps score, and even its threaded counter only manages
 a dead heat on mid-size files — never a win, and the moment the file
 stops fitting in a polite buffer, the dead heat stops being polite.
 
-A note on startup, in the interest of honesty: on a one-line file the
+A note on startup, in the interest of honesty — and of gloating: on a
-whole race happens in the low hundreds of microseconds — fastwc
+one-line file the whole race happens in the low hundreds of
-~0.25 ms, GNU ~0.33 ms, toybox ~0.37 ms, busybox ~0.14 ms. Microseconds
+microseconds, and fastwc now wins it outright. The default build is
-either way. Nobody will ever notice a difference that small, and it
+static musl, so there is no dynamic loader to pay: min-of-400 on a
-does not matter in the bigger picture: the tiny cases in the table are
+12-byte file puts fastwc `-l` at 78µs against busybox's 83µs, GNU's
-here to prove fastwc is never *wrong*, not to brag about a head start
+253µs, and toybox's 254µs. Busybox's one structural advantage — a
-that evaporates the moment the page cache warms up. That is why the
+loader it never had to start — is no longer an advantage; we don't
-suite now times both sides at microsecond resolution and files anything
+start one either. None of this matters in the bigger picture, and we
-the reference finishes in under 5 ms under "startup-bound": correct,
+will not pretend otherwise: nobody will ever notice a difference that
-reported, and excluded from the averages — because nobody should be
+small, and the tiny cases in the table are here to prove fastwc is
-racing startup, least of all a word counter. The cases that matter are
+never *wrong*, not to brag about a head start that evaporates the
-the ones where counting takes longer than starting — and those are the
+moment the page cache warms up. That is why the suite now times both
-ones in this table. Every run ends with the average speedup against
+sides at microsecond resolution and files anything the reference
-each oracle — coreutils ~4.5x, busybox ~13-15x, toybox ~15-18x — so
+finishes in under 5 ms under "startup-bound": correct, reported, and
-the cruelty is quantified.
+excluded from the averages — because nobody should be racing startup,
+least of all a word counter. The cases that matter are the ones where
+counting takes longer than starting — and those are the ones in this
+table. Every run ends with the average speedup against each oracle —
+coreutils ~4.5x, busybox ~13-15x, toybox ~15-18x — so the cruelty is
+quantified.
 
 ## Why
 
@@ -89,6 +108,12 @@ the cruelty is quantified.
   now: regular files are mapped and counted in parallel across cores,
   with SIMD kernels (AVX-512, AVX-2, SSE2) dispatched at runtime —
   zero function calls in the hot path.
+- **GNU wc slows down in the locale you actually run.** Under a UTF-8
+  locale it decodes every byte it counts — pure ASCII included, which
+  decoding cannot change — so the 11 MB words file that takes it
+  7.9 ms in the C locale takes 10.4 ms there, against our 1.4 ms.
+  fastwc's kernels probe for non-ASCII bytes as they count and only
+  decode files that need it. The locale that taxes GNU is free for us.
 
 ## What it does
 
@@ -117,10 +142,19 @@ fastwc [OPTION]... --files0-from=F
 Requires a C compiler and autotools. That's it. No gettext. No gnulib.
 No translators.
 
+The default build links statically against musl (via `musl-gcc`),
+which is why fastwc now wins the startup cases above outright —
+there is no dynamic loader to pay, and the ~80µs exec floor is the
+same one busybox pays. If `musl-gcc` isn't installed the configure
+script warns and falls back to the system compiler; the glibc build
+is one flag away:
+
 ```sh
-./autogen.sh     # autoreconf -fi && ./configure
+./autogen.sh            # autoreconf -fi && ./configure (static musl)
 make
 make release            # installs the release binary to bin/release/fastwc
+
+./configure --enable-glibc && make   # or: dynamic glibc build
 ```
 
 ## Benchmark

configure.ac

@@ -2,9 +2,38 @@ AC_PREREQ([2.69])
 AC_INIT([fastwc], [0.1.0], [], [fastwc])
 AC_CONFIG_SRCDIR([src/main.c])
 
+# Build flavor: --enable-glibc links dynamically against the system libc
+# (respecting CC, whatever it is). The default is a fully static musl
+# binary, preferring musl-gcc even when CC is set in the environment,
+# since removing ld.so from startup is exactly what the tiny-file
+# benchmark races hinge on.
+AC_ARG_ENABLE([glibc],
+  [AS_HELP_STRING([--enable-glibc],
+    [link dynamically against the system libc (respects CC) instead of the default static musl])],
+  [enable_glibc=$enableval], [enable_glibc=no])
+
+STATIC_LDFLAGS=
+if test "x$enable_glibc" = xno; then
+  # Static musl is the default toolchain. Fall back to the system
+  # compiler (dynamic link) only when no musl compiler is installed.
+  AC_PATH_PROGS([MUSL_CC], [musl-gcc x86_64-linux-musl-gcc])
+  if test -n "$MUSL_CC"; then
+    CC="$MUSL_CC"
+    STATIC_LDFLAGS="-static"
+  fi
+fi
+
 AC_PROG_CC
+AC_CHECK_TOOL([STRIP], [strip], [:])
 AC_USE_SYSTEM_EXTENSIONS
 AM_INIT_AUTOMAKE([foreign subdir-objects])
 
+AS_IF([test "x$enable_glibc" = xyes],
+  [AC_MSG_NOTICE([fastwc: glibc build requested, using $CC])],
+  [AS_IF([test -n "$MUSL_CC"],
+     [AC_MSG_NOTICE([fastwc: building against static musl ($MUSL_CC)])],
+     [AC_MSG_WARN([fastwc: musl-gcc not found; falling back to $CC (dynamic link). Install a musl toolchain or configure --enable-glibc.])])])
+AC_SUBST([STATIC_LDFLAGS])
+
 AC_CONFIG_FILES([Makefile])
 AC_OUTPUT

docs/PERFORMANCE.md

@@ -38,6 +38,36 @@ second. The reference is us now. Busybox and toybox, meanwhile, are
 here for the cruelty: 10-31x slower depending on the case, and their
 word counting has *bugs*.
 
+## The locale tax, gone
+
+Every number above is C locale — the setting that flatters the
+opponents most: GNU `wc -w` under a UTF-8 locale stops counting bytes
+and decodes every one of them through `mbrtowc`, even when the file is
+pure ASCII and decoding changes nothing. We used to pay that same tax:
+the multibyte gate looked only at `MB_CUR_MAX`, so a UTF-8 locale
+silently traded the SIMD kernels for the decoder, and the 11 MB words
+race above flipped from a 4.9x win to a 25% loss against GNU.
+
+The kernels now double as a probe — one vector move-mask per load
+flags the first byte ≥ 0x80, free when unused — so only files that
+actually contain a high byte fall back to the decoder. Receipts,
+min-of-N interleaved, `en_US.UTF-8`, the same 11 MB ASCII words file:
+
+| Case | GNU coreutils | fastwc | gap |
+|------|--------------:|-------:|----:|
+| words | 10.43ms | **1.38ms** | 7.6x |
+| default (`-lwc`) | 10.55ms | **1.50ms** | 7.0x |
+| characters (`-m`) | 10.58ms | **2.06ms** | 5.1x |
+| longest line (`-L`) | 10.56ms | **6.83ms** | 1.5x |
+
+GNU's decoder bill for that file is unchanged: 10.4ms, for bytes that
+were never multibyte. Files that genuinely are multibyte still decode
+at parity — 10.5 MB of mixed CJK+latin, 56.7ms against GNU's 56.9ms —
+because there both sides decode. The one case GNU keeps is *lightly*
+multibyte files: sparse UTF-8 costs us one wasted fast pass before the
+fallback (793 KB, 1.30ms vs GNU's 1.18ms). We judged the tax worth
+it; ASCII is the rule, multibyte is the exception.
+
 ## On startup
 
 A word counter that loses one-line races to a slower counter is not
@@ -63,15 +93,21 @@ half and then the honest reporting:
 - **The receipts.** Min-of-400 interleaved on an 11-byte file: fastwc
   ~0.25ms, GNU ~0.33ms, toybox ~0.37ms, busybox ~0.14ms. Before the
   work, fastwc `-w` on a tiny file measured ~562µs; after, ~425µs.
-  Nobody will ever notice a difference that small.
+  Nobody will ever notice a difference that small. Those were the
+  dynamic-link numbers; the static musl default below starts ~3x
+  sooner than even those.
 - **The honest half.** Because those microseconds don't matter, the
   benchmark no longer pretends they do. Any case the reference
   finishes in under 5ms is filed under `startup-bound`: fastwc must
   still match the count, but the case is excluded from the averages
-  and the throughput scoreboard. Busybox's genuinely faster startup
+  and the throughput scoreboard. Busybox used to win these outright —
-  (242µs vs our 534µs on one line) is reported exactly that way. The
+  it is a static musl binary, and skipping the dynamic loader bought
-  cases in the table above are the ones where counting takes longer
+  it the better part of a hundred microseconds on every exec. That
-  than starting.
+  excuse retired itself when the default build went static musl too
+  (see the README): on a 12-byte file, min-of-400, fastwc `-l` now
+  lands at 78µs against busybox's 83µs, GNU's 253µs, and toybox's
+  254µs. The cases in the table above are the ones where counting
+  takes longer than starting.
 
 ## Why it's fast
 
@@ -99,6 +135,15 @@ half and then the honest reporting:
    `st_size` from `fstat` — GNU figured that one out too, so we copied
    the good idea. `-l` without `-w` skips the whitespace mask entirely;
    `-w` without `-L` never builds the print table.
+5. **ASCII pays nothing, even in a UTF-8 locale.** Multibyte decoding
+   is expensive, so we don't volunteer for it. The SIMD kernels
+   double as a probe: when asked, they flag the first byte ≥ 0x80
+   with a vector move-mask — no extra pass, no cost on pure-ASCII
+   input. A file that stays pure ASCII keeps the full-speed byte
+   path, and its counts are identical to what the decoder would
+   produce, because ASCII decodes to itself. Only files that actually
+   contain a high byte pay for the multibyte decoder, and then only
+   from the first high byte on.
 
 ## Correctness is the other half of the contract
 
@@ -129,10 +174,6 @@ combination) passes 100%.
   because 64 KiB pipe chunks trigger a re-scan of its carried bytes.
   We reproduced this, then declined to. fastwc counts the data, not
   the plumbing.
-- **Without `-m`, fastwc counts bytes with C-locale semantics.**
-  GNU silently switches to multibyte decoding for `-w` in UTF-8
-  locales. We don't — that's what `-m` is for, and it keeps the fast
-  path fast. Under `LC_ALL=C` we match GNU exactly.
 
 ## Reproducing
 

src/main.c

@@ -384,7 +384,13 @@ typedef struct
 } lw_t;
 
 typedef lw_t (*count_lw_fn)(const unsigned char *s, size_t n, int *prev_ws,
-                            int need_lines, int need_words);
+                            int need_lines, int need_words, int need_high,
+                            int *high);
+/* need_high: probe the buffer for any byte >= 0x80. When one is seen the
+ * kernel sets *high and returns immediately with whatever partial counts
+ * it has; callers that asked for the probe treat the counts as invalid and
+ * re-run through the multibyte decoder. Pure-ASCII buffers never trigger,
+ * so the byte-path counts stay valid. */
 
 /*
  * Word separators match GNU wc (the benchmark oracle): the locale's
@@ -396,7 +402,7 @@ typedef lw_t (*count_lw_fn)(const unsigned char *s, size_t n, int *prev_ws,
 
 __attribute__((target("avx512f,avx512bw"))) static lw_t
 count_lw_avx512(const unsigned char *s, size_t n, int *prev_ws, int need_lines,
-                int need_words)
+                int need_words, int need_high, int *high)
 {
     const __m512i nl = _mm512_set1_epi8('\n');
     const __m512i sp = _mm512_set1_epi8(' ');
@@ -412,6 +418,13 @@ count_lw_avx512(const unsigned char *s, size_t n, int *prev_ws, int need_lines,
         __m512i v = _mm512_loadu_si512((const void *)(s + i));
         uint64_t nl_mask = 0;
 
+        if (need_high && _mm512_movepi8_mask(v))
+        {
+            *high = 1;
+            lw_t r = {lines, words};
+            return r;
+        }
+
         if (need_lines)
         {
             nl_mask = (uint64_t)_mm512_cmpeq_epi8_mask(v, nl);
@@ -435,6 +448,12 @@ count_lw_avx512(const unsigned char *s, size_t n, int *prev_ws, int need_lines,
 
     for (; i < n; i++)
     {
+        if (need_high && s[i] & 0x80)
+        {
+            *high = 1;
+            lw_t r = {lines, words};
+            return r;
+        }
         int ws = ws_tab[s[i]];
         if (need_lines)
             lines += s[i] == '\n';
@@ -453,7 +472,7 @@ count_lw_avx512(const unsigned char *s, size_t n, int *prev_ws, int need_lines,
 
 __attribute__((target("avx2"))) static lw_t
 count_lw_avx2(const unsigned char *s, size_t n, int *prev_ws, int need_lines,
-              int need_words)
+              int need_words, int need_high, int *high)
 {
     const __m256i nl = _mm256_set1_epi8('\n');
     const __m256i sp = _mm256_set1_epi8(' ');
@@ -469,6 +488,13 @@ count_lw_avx2(const unsigned char *s, size_t n, int *prev_ws, int need_lines,
         __m256i v = _mm256_loadu_si256((const void *)(s + i));
         uint32_t nl_mask = 0;
 
+        if (need_high && _mm256_movemask_epi8(v))
+        {
+            *high = 1;
+            lw_t r = {lines, words};
+            return r;
+        }
+
         if (need_lines)
         {
             nl_mask = (uint32_t)_mm256_movemask_epi8(_mm256_cmpeq_epi8(v, nl));
@@ -494,6 +520,12 @@ count_lw_avx2(const unsigned char *s, size_t n, int *prev_ws, int need_lines,
 
     for (; i < n; i++)
     {
+        if (need_high && s[i] & 0x80)
+        {
+            *high = 1;
+            lw_t r = {lines, words};
+            return r;
+        }
         int ws = ws_tab[s[i]];
         if (need_lines)
             lines += s[i] == '\n';
@@ -512,7 +544,7 @@ count_lw_avx2(const unsigned char *s, size_t n, int *prev_ws, int need_lines,
 
 __attribute__((target("sse2"))) static lw_t
 count_lw_sse2(const unsigned char *s, size_t n, int *prev_ws, int need_lines,
-              int need_words)
+              int need_words, int need_high, int *high)
 {
     const __m128i nl = _mm_set1_epi8('\n');
     const __m128i sp = _mm_set1_epi8(' ');
@@ -528,6 +560,13 @@ count_lw_sse2(const unsigned char *s, size_t n, int *prev_ws, int need_lines,
         __m128i v = _mm_loadu_si128((const void *)(s + i));
         uint32_t nl_mask = 0;
 
+        if (need_high && _mm_movemask_epi8(v))
+        {
+            *high = 1;
+            lw_t r = {lines, words};
+            return r;
+        }
+
         if (need_lines)
         {
             nl_mask = (uint32_t)_mm_movemask_epi8(_mm_cmpeq_epi8(v, nl));
@@ -550,6 +589,12 @@ count_lw_sse2(const unsigned char *s, size_t n, int *prev_ws, int need_lines,
 
     for (; i < n; i++)
     {
+        if (need_high && s[i] & 0x80)
+        {
+            *high = 1;
+            lw_t r = {lines, words};
+            return r;
+        }
         int ws = ws_tab[s[i]];
         if (need_lines)
             lines += s[i] == '\n';
@@ -570,10 +615,22 @@ count_lw_sse2(const unsigned char *s, size_t n, int *prev_ws, int need_lines,
 
 /* Reference path: the two scalar SWAR counters, kept as the fallback. */
 static lw_t count_lw_scalar(const unsigned char *s, size_t n, int *prev_ws,
-                            int need_lines, int need_words)
+                            int need_lines, int need_words, int need_high,
+                            int *high)
 {
     lw_t r;
 
+    if (need_high)
+    {
+        for (size_t i = 0; i < n; i++)
+            if (s[i] & 0x80)
+            {
+                *high = 1;
+                r.lines = 0;
+                r.words = 0;
+                return r;
+            }
+    }
     r.lines = need_lines ? count_newlines(s, n) : 0;
     r.words = need_words ? count_words(s, n, prev_ws) : 0;
     return r;
@@ -581,7 +638,8 @@ static lw_t count_lw_scalar(const unsigned char *s, size_t n, int *prev_ws,
 
 static count_lw_fn count_lw = count_lw_scalar; /* chosen by pick_kernel() */
 
-static void count_mapped(const unsigned char *p, size_t n, counts_t *c);
+static void count_mapped(const unsigned char *p, size_t n, counts_t *c,
+                         int need_high, int *high);
 
 #if defined(__x86_64__) || defined(__i386__)
 static count_lw_fn pick_kernel(void)
@@ -901,12 +959,12 @@ static void count_stream(FILE *fp, counts_t *c)
     int fd = fileno(fp);
 
     /* Under a multibyte locale, -w/-m/-L need per-character decoding;
-     * -l and -c alone stay on the byte path, exactly like GNU wc. */
+     * -l and -c alone stay on the byte path, exactly like GNU wc.
-    if ((MB_CUR_MAX > 1) && (flags & (F_CHARS | F_WORDS | F_MAXLEN)))
+     * mb_semantics is computed once and reused below: regular files are
-    {
+     * first probed on the SIMD byte path (see the has_high fallback),
-        count_stream_mb(fp, c);
+     * while non-regular inputs go straight to the decoder. */
-        return;
+    int mb_semantics =
-    }
+        (MB_CUR_MAX > 1) && (flags & (F_CHARS | F_WORDS | F_MAXLEN));
 
     if (debug && using_wc_lines())
         debug_methods();
@@ -960,7 +1018,22 @@ static void count_stream(FILE *fp, counts_t *c)
                 mmap(NULL, (size_t)st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
             if (m != MAP_FAILED)
             {
-                count_mapped((const unsigned char *)m, (size_t)st.st_size, c);
+                counts_t save = *c;
+                int high = 0;
+
+                count_mapped((const unsigned char *)m, (size_t)st.st_size, c,
+                             mb_semantics, &high);
+                if (mb_semantics && high)
+                {
+                    /* Non-ASCII input: the byte-path counts are invalid
+                     * under a multibyte locale. Discard them and decode.
+                     * The stream is still positioned at 0 (mmap never
+                     * advanced it), so re-reading counts the whole file. */
+                    *c = save;
+                    munmap(m, (size_t)st.st_size);
+                    count_stream_mb(fp, c);
+                }
+                else
                     munmap(m, (size_t)st.st_size);
                 if (ferror(fp))
                     c->ok = 0;
@@ -969,6 +1042,13 @@ static void count_stream(FILE *fp, counts_t *c)
         }
     }
 
+    /* Non-regular input: no mmap probe happened, so decode directly. */
+    if (mb_semantics)
+    {
+        count_stream_mb(fp, c);
+        return;
+    }
+
     for (;;)
     {
         nread = fread(buf, 1, sizeof buf, fp); /* NOLINT: EOF-state FP */
@@ -980,7 +1060,7 @@ static void count_stream(FILE *fp, counts_t *c)
         if (flags & (F_LINES | F_WORDS))
         {
             lw_t r = count_lw(buf, nread, &prev_ws, (flags & F_LINES) != 0,
-                              (flags & F_WORDS) != 0);
+                              (flags & F_WORDS) != 0, 0, NULL);
             if (flags & F_LINES)
                 c->lines += r.lines;
             if (flags & F_WORDS)
@@ -1008,6 +1088,8 @@ typedef struct
     int prev_ws;
     int need_lines;
     int need_words;
+    int need_high;
+    int high;
     lw_t r;
 } mjob_t;
 
@@ -1015,7 +1097,8 @@ static void *map_worker(void *arg)
 {
     mjob_t *j = arg;
 
-    j->r = count_lw(j->s, j->n, &j->prev_ws, j->need_lines, j->need_words);
+    j->r = count_lw(j->s, j->n, &j->prev_ws, j->need_lines, j->need_words,
+                    j->need_high, &j->high);
     return NULL;
 }
 
@@ -1026,8 +1109,8 @@ static void *map_worker(void *arg)
  * which makes the split exact. The kernels are pure, so no locks.
  */
 static void count_sliced(const unsigned char *p, size_t n, int nt,
-                         int need_lines, int need_words, long long *lines,
+                         int need_lines, int need_words, int need_high,
-                         long long *words)
+                         int *high, long long *lines, long long *words)
 {
     mjob_t jobs[MAX_THREADS];
     pthread_t th[MAX_THREADS];
@@ -1051,6 +1134,8 @@ static void count_sliced(const unsigned char *p, size_t n, int nt,
         jobs[i].prev_ws = (i == 0 || jobs[i].n == 0) ? 1 : ws_tab[p[start - 1]];
         jobs[i].need_lines = need_lines;
         jobs[i].need_words = need_words;
+        jobs[i].need_high = need_high;
+        jobs[i].high = 0;
         th[i] = 0;
         if (jobs[i].n > 0)
         {
@@ -1072,6 +1157,14 @@ static void count_sliced(const unsigned char *p, size_t n, int nt,
         tw += jobs[i].r.words;
     }
 
+    *high = 0;
+    for (i = 0; i < nt; i++)
+        if (jobs[i].high)
+        {
+            *high = 1;
+            break;
+        }
+
     *lines = tl;
     *words = tw;
 }
@@ -1096,7 +1189,8 @@ static int pick_threads(size_t n)
     return nt;
 }
 
-static void count_mapped(const unsigned char *p, size_t n, counts_t *c)
+static void count_mapped(const unsigned char *p, size_t n, counts_t *c,
+                         int need_high, int *high)
 {
     int need_lines = (flags & F_LINES) != 0;
     int need_words = (flags & F_WORDS) != 0;
@@ -1110,14 +1204,20 @@ static void count_mapped(const unsigned char *p, size_t n, counts_t *c)
     if (nt <= 1)
     {
         int prev_ws = 1;
-        lw_t r = count_lw(p, n, &prev_ws, need_lines, need_words);
+        lw_t r = count_lw(p, n, &prev_ws, need_lines, need_words, need_high,
+                          high);
 
+        if (need_high && *high)
+            return; /* partial counts; caller re-decodes */
         lines = r.lines;
         words = r.words;
     }
     else
     {
-        count_sliced(p, n, nt, need_lines, need_words, &lines, &words);
+        count_sliced(p, n, nt, need_lines, need_words, need_high, high,
+                     &lines, &words);
+        if (need_high && *high)
+            return; /* partial counts; caller re-decodes */
     }
 
     if (flags & F_LINES)
@@ -1786,7 +1886,8 @@ static long long ref_words(const unsigned char *s, size_t n, int *prev_ws)
  * the reference instead of only on real AVX-512 hardware.
  */
 static lw_t count_lw_avx512_mirror(const unsigned char *s, size_t n,
-                                   int *prev_ws, int need_lines, int need_words)
+                                   int *prev_ws, int need_lines, int need_words,
+                                   int need_high, int *high)
 {
     long long lines = 0, words = 0;
     size_t i = 0;
@@ -1800,6 +1901,12 @@ static lw_t count_lw_avx512_mirror(const unsigned char *s, size_t n,
             unsigned char c = s[i + j];
             uint64_t bit = (uint64_t)1 << j;
 
+            if (need_high && (c & 0x80))
+            {
+                *high = 1;
+                lw_t r = {lines, words};
+                return r;
+            }
             if (c == '\n')
                 nl_mask |= bit;
             if (c == '\n' || c == ' ' || (nbsp_ws && c == 0xa0) ||
@@ -1818,6 +1925,12 @@ static lw_t count_lw_avx512_mirror(const unsigned char *s, size_t n,
     for (; i < n; i++)
     {
         int ws = ws_tab[s[i]];
+        if (need_high && (s[i] & 0x80))
+        {
+            *high = 1;
+            lw_t r = {lines, words};
+            return r;
+        }
         if (need_lines)
             lines += s[i] == '\n';
         if (need_words)
@@ -1850,7 +1963,7 @@ static int check_kernel(const char *name, count_lw_fn fn)
                 for (int nw = 0; nw <= 1; nw++)
                 {
                     int a = pw, b = pw;
-                    lw_t got = fn(buf, n, &a, nl, nw);
+                    lw_t got = fn(buf, n, &a, nl, nw, 0, NULL);
                     long long want_l = nl ? ref_lines(buf, n) : 0;
                     long long want_w = nw ? ref_words(buf, n, &b) : 0;
 
@@ -1884,7 +1997,7 @@ static int check_kernel(const char *name, count_lw_fn fn)
         for (int pw = 0; pw <= 1; pw++)
         {
             int a = pw, b = pw;
-            lw_t got = fn(buf, n, &a, 1, 1);
+            lw_t got = fn(buf, n, &a, 1, 1, 0, NULL);
 
             if (got.lines != ref_lines(buf, n) ||
                 got.words != ref_words(buf, n, &b) || a != b)
@@ -1907,7 +2020,7 @@ static int check_kernel(const char *name, count_lw_fn fn)
         for (k = 0; k < 256; k++)
             buf[k] = pass == 0 ? (unsigned char)k : (unsigned char)(255 - k);
         int a = 1, b = 1;
-        lw_t got = fn(buf, 256, &a, 1, 1);
+        lw_t got = fn(buf, 256, &a, 1, 1, 0, NULL);
 
         if (got.lines != ref_lines(buf, 256) ||
             got.words != ref_words(buf, 256, &b) || a != b)
@@ -1917,6 +2030,35 @@ static int check_kernel(const char *name, count_lw_fn fn)
         }
     }
 
+    /* high-byte probe contract: *high set iff some byte >= 0x80 exists.
+     * Partial counts on a probe trip are undefined, so only the flag is
+     * checked here (the caller re-runs through the mb decoder). */
+    for (size_t probe = 0; probe < 2; probe++)
+    {
+        int high = 0;
+
+        memset(buf, 'x', 256);
+        buf[probe == 0 ? 0 : 255] = probe == 0 ? 0x80 : 0xff;
+        int a = 1;
+        fn(buf, 256, &a, 1, 1, 1, &high);
+        if (!high)
+        {
+            printf("%s: high probe pos=%zu not detected\n", name, probe);
+            fails++;
+        }
+
+        memset(buf, 'x', 256);
+        buf[255] = '\n';
+        high = 0;
+        a = 1;
+        fn(buf, 256, &a, 1, 1, 1, &high);
+        if (high)
+        {
+            printf("%s: ascii probe false-trip\n", name);
+            fails++;
+        }
+    }
+
     printf("%s: %s\n", name, fails ? "FAIL" : "ok");
     return fails;
 }
@@ -1942,12 +2084,13 @@ static int check_sliced(void)
         {
             int nt = tcs[ti];
             long long tl = 0, tw = 0;
+            int dh = 0;
             long long want_l = count_newlines(buf, n);
             int pw = 1;
             long long want_w = count_words(buf, n, &pw);
 
-            count_sliced(buf, n, nt, 1, 1, &tl, &tw);
+            count_sliced(buf, n, nt, 1, 1, 0, &dh, &tl, &tw);
-            if (tl != want_l || tw != want_w)
+            if (tl != want_l || tw != want_w || dh != 0)
             {
                 printf("sliced: n=%zu nt=%d lines %lld/%lld "
                        "words %lld/%lld\n",
@@ -1968,12 +2111,13 @@ static int check_sliced(void)
         {
             int nt = tcs[ti];
             long long tl = 0, tw = 0;
+            int dh = 0;
             long long want_l = count_newlines(buf, n);
             int pw = 1;
             long long want_w = count_words(buf, n, &pw);
 
-            count_sliced(buf, n, nt, 1, 1, &tl, &tw);
+            count_sliced(buf, n, nt, 1, 1, 0, &dh, &tl, &tw);
-            if (tl != want_l || tw != want_w)
+            if (tl != want_l || tw != want_w || dh != 0)
             {
                 printf("sliced-ws: n=%zu nt=%d lines %lld/%lld "
                        "words %lld/%lld\n",