count_sliced computed the slice size from the requested thread count and
only then clamped nt to MAX_THREADS, so the last slice ended before the
file tail: raising the cap (as pick_threads now does) silently dropped
the tail from every count. Cap nt first, then derive per.
Monsters (>=256 MiB) now get one thread per core (capped at 24) instead
of a hard 16, which measures ~1.17x on the 1B-line case and is flat on
warm files. check_sliced now sweeps the cap boundary (16, 24, 25).
Five distro recipes in packaging/, all following the same recipe:
unpack the Gitea release archive (which extracts to fastwc/), run
autoreconf -fi (the generated configure/Makefile.in are not committed),
configure, make, make install - the binary needs only glibc.
Each recipe notes the release prerequisite: the archive URL requires a
v0.1.0 tag on the remote, and Arch/Nix/Void carry an integrity
placeholder to fill in once the tag exists. The PKGBUILD was validated
with makepkg --printsrcinfo, and the full build recipe was run
end-to-end from a Gitea-layout tarball (autoreconf, configure, make,
make install DESTDIR) producing a working /usr/bin/fastwc.
The scalar AVX-512 mirror built its whitespace mask from ws_tab lookups,
so a bug in the kernel's actual predicate - the (x - 9) < 5 unsigned
range trick plus newline/space/NBSP compares - would pass the selftest
on hosts without AVX-512 and only surface on real hardware. The mirror
now computes the mask with the kernel's exact expression, and
check_kernel gains an exhaustive 256-byte ramp (forward and reversed)
so any divergence from the reference table changes a count. Verified by
mutation: shifting the range constant to (x - 8) < 5 makes the selftest
fail.
Review (5-lane) found one MAJOR: count_sliced ignored pthread_create's
return value - on EAGAIN it joined an indeterminate pthread_t and summed
an uninitialized slice, UB plus a silently wrong count. Threads are now
initialized to 0, a failed create counts its slice inline, and nt is
hard-capped at a named MAX_THREADS (the fixed jobs[8]/th[8] arrays
smash the stack past 8 threads - reproduced by QA's thread sweep).
Shipped from the measured optimization hunt:
- -w-only mode skips the newline compare/popcount entirely (the
(x-9)<5 range already covers '\n' in the whitespace mask) - a
need_lines gate threads through every kernel, the scalar reference,
the avx512 mirror, and the sliced workers.
- Thread retune: 12 threads past 32 MiB, 16 past 256 MiB (was 8) -
up to 18% on 110 MB, ~3% on 1.1 GB in QA's interleaved sweep.
- Benchmark suites pin LC_ALL=C so GNU wc -w can't silently switch to
multibyte decoding and inflate the win.
- popcount16 is guarded to x86 builds (zero-warnings on other arches);
checkwc's stray argument dropped.
Verified: selftest (kernels + sliced, now with need_lines coverage),
120-trial fuzz vs GNU (file + stdin, both locales), full test-all suite
three consecutive times, format-check and clang-tidy clean. The 1B
solo monster now lands at 3.7-6s (up to 268 Mlines/s) depending on how
warm the page cache is feeling.
Suites split into benchmarks/files/{lines,words} and
benchmarks/stdin/piping, with the monsters bolted onto the lines suite:
100M lines raced against coreutils (~2x win), 1B lines solo (~6-8s,
11 GB in one pass). Stdin redirects from regular files are now mmap'd
in count_stream, so the stdin suite wins too — up to 12.00x on words.
The ratio column now reports how many times faster fastwc is, not how
much of GNU's time it used. busybox was removed from the suite: it
stopped being a challenge and started being a participation trophy.
GNU wc's lone win — 1M lines by one millisecond on hand-tuned AVX-512
assembly — is now a historical footnote, and the README says so.
SIMD kernels (AVX-512/AVX-2/SSE2, runtime-dispatched) counting a
single pass over mmap'd files, split across cores past 8 MiB, with
exact GNU oracle parity (NBSP included, glibc's decoder fixed, the
whole -m path mirrored so counts agree at every boundary).
Result: 1ms vs 2ms at 1M lines, 8-9ms vs 22-24ms at 10M. Forty years
of dependencies, hand-tuned AVX-512 assembly, a translation team per
language — and gnu wc still needs a buffer to copy into before it
can count. We mapped the file and just counted. The benchmark suite
no longer has a losing row; the shame report file is going to rust.
huntedbytheirs