Adds -L/--max-line-length (display width, tab stops every 8), the long
options --lines/--words/--chars/--bytes/--max-line-length, --files0-from
(NUL-delimited name lists), --total=auto|always|only|never, --debug
(acceleration diagnostics), and POSIXLY_CORRECT handling. Ports GNU's
column width rule (digits of the sum of regular file sizes, floor 7 for
non-regular inputs), its multibyte word/line-length semantics (-w/-m/-L
decode per character under a multibyte locale), the -c sizing heuristic
(page-aligned st_size reads a tail block), and its error texts, quoting
and exit codes.
Verified byte-identical stdout, exit codes and (normalized) stderr
against coreutils wc 9.11: 125-case parity suite and 552-case fuzz
across C, C.UTF-8, en_US.UTF-8 and POSIX locales. The benchmark contract
holds: words, lines (incl. the 100M monster) and stdin all pass with
fastwc never slower than GNU wc.
Review (5-lane) found one MAJOR: count_sliced ignored pthread_create's
return value - on EAGAIN it joined an indeterminate pthread_t and summed
an uninitialized slice, UB plus a silently wrong count. Threads are now
initialized to 0, a failed create counts its slice inline, and nt is
hard-capped at a named MAX_THREADS (the fixed jobs[8]/th[8] arrays
smash the stack past 8 threads - reproduced by QA's thread sweep).
Shipped from the measured optimization hunt:
- -w-only mode skips the newline compare/popcount entirely (the
(x-9)<5 range already covers '\n' in the whitespace mask) - a
need_lines gate threads through every kernel, the scalar reference,
the avx512 mirror, and the sliced workers.
- Thread retune: 12 threads past 32 MiB, 16 past 256 MiB (was 8) -
up to 18% on 110 MB, ~3% on 1.1 GB in QA's interleaved sweep.
- Benchmark suites pin LC_ALL=C so GNU wc -w can't silently switch to
multibyte decoding and inflate the win.
- popcount16 is guarded to x86 builds (zero-warnings on other arches);
checkwc's stray argument dropped.
Verified: selftest (kernels + sliced, now with need_lines coverage),
120-trial fuzz vs GNU (file + stdin, both locales), full test-all suite
three consecutive times, format-check and clang-tidy clean. The 1B
solo monster now lands at 3.7-6s (up to 268 Mlines/s) depending on how
warm the page cache is feeling.
Suites split into benchmarks/files/{lines,words} and
benchmarks/stdin/piping, with the monsters bolted onto the lines suite:
100M lines raced against coreutils (~2x win), 1B lines solo (~6-8s,
11 GB in one pass). Stdin redirects from regular files are now mmap'd
in count_stream, so the stdin suite wins too — up to 12.00x on words.
The ratio column now reports how many times faster fastwc is, not how
much of GNU's time it used. busybox was removed from the suite: it
stopped being a challenge and started being a participation trophy.
GNU wc's lone win — 1M lines by one millisecond on hand-tuned AVX-512
assembly — is now a historical footnote, and the README says so.
SIMD kernels (AVX-512/AVX-2/SSE2, runtime-dispatched) counting a
single pass over mmap'd files, split across cores past 8 MiB, with
exact GNU oracle parity (NBSP included, glibc's decoder fixed, the
whole -m path mirrored so counts agree at every boundary).
Result: 1ms vs 2ms at 1M lines, 8-9ms vs 22-24ms at 10M. Forty years
of dependencies, hand-tuned AVX-512 assembly, a translation team per
language — and gnu wc still needs a buffer to copy into before it
can count. We mapped the file and just counted. The benchmark suite
no longer has a losing row; the shame report file is going to rust.
huntedbytheirs