Busybox wc is back in every suite, for shits and giggles. checkwc now
walks PATH and identifies each wc by its --version answer (coreutils and
fastwc respond; busybox names itself in the error it prints), so a wc
symlinked to fastwc is detected and skipped instead of silently racing
us against ourselves, and a missing coreutils is a clear configure
error. Every run now ends with the average speedup of fastwc against
coreutils and busybox, computed from the per-case ratios.
The fail-fast verdict now requires a measurable (>0ms) reference time,
so sub-millisecond cases stop flaking on startup noise, and the 100M
line monster is pinned back to the coreutils oracle instead of whatever
oracle was raced last.
Adds -L/--max-line-length (display width, tab stops every 8), the long
options --lines/--words/--chars/--bytes/--max-line-length, --files0-from
(NUL-delimited name lists), --total=auto|always|only|never, --debug
(acceleration diagnostics), and POSIXLY_CORRECT handling. Ports GNU's
column width rule (digits of the sum of regular file sizes, floor 7 for
non-regular inputs), its multibyte word/line-length semantics (-w/-m/-L
decode per character under a multibyte locale), the -c sizing heuristic
(page-aligned st_size reads a tail block), and its error texts, quoting
and exit codes.
Verified byte-identical stdout, exit codes and (normalized) stderr
against coreutils wc 9.11: 125-case parity suite and 552-case fuzz
across C, C.UTF-8, en_US.UTF-8 and POSIX locales. The benchmark contract
holds: words, lines (incl. the 100M monster) and stdin all pass with
fastwc never slower than GNU wc.
Sub-millisecond runs showed as a flat 0ms, hiding real differences. The
suite now times fastwc with a time_us helper, keeps the minimum reading
in µs, derives ms from it, and prints both (fastwc: 0ms (767µs)). GNU wc
stays at ms precision; solo case shows µs too.
count_sliced computed the slice size from the requested thread count and
only then clamped nt to MAX_THREADS, so the last slice ended before the
file tail: raising the cap (as pick_threads now does) silently dropped
the tail from every count. Cap nt first, then derive per.
Monsters (>=256 MiB) now get one thread per core (capped at 24) instead
of a hard 16, which measures ~1.17x on the 1B-line case and is flat on
warm files. check_sliced now sweeps the cap boundary (16, 24, 25).
Five distro recipes in packaging/, all following the same recipe:
unpack the Gitea release archive (which extracts to fastwc/), run
autoreconf -fi (the generated configure/Makefile.in are not committed),
configure, make, make install - the binary needs only glibc.
Each recipe notes the release prerequisite: the archive URL requires a
v0.1.0 tag on the remote, and Arch/Nix/Void carry an integrity
placeholder to fill in once the tag exists. The PKGBUILD was validated
with makepkg --printsrcinfo, and the full build recipe was run
end-to-end from a Gitea-layout tarball (autoreconf, configure, make,
make install DESTDIR) producing a working /usr/bin/fastwc.
The scalar AVX-512 mirror built its whitespace mask from ws_tab lookups,
so a bug in the kernel's actual predicate - the (x - 9) < 5 unsigned
range trick plus newline/space/NBSP compares - would pass the selftest
on hosts without AVX-512 and only surface on real hardware. The mirror
now computes the mask with the kernel's exact expression, and
check_kernel gains an exhaustive 256-byte ramp (forward and reversed)
so any divergence from the reference table changes a count. Verified by
mutation: shifting the range constant to (x - 8) < 5 makes the selftest
fail.
Review (5-lane) found one MAJOR: count_sliced ignored pthread_create's
return value - on EAGAIN it joined an indeterminate pthread_t and summed
an uninitialized slice, UB plus a silently wrong count. Threads are now
initialized to 0, a failed create counts its slice inline, and nt is
hard-capped at a named MAX_THREADS (the fixed jobs[8]/th[8] arrays
smash the stack past 8 threads - reproduced by QA's thread sweep).
Shipped from the measured optimization hunt:
- -w-only mode skips the newline compare/popcount entirely (the
(x-9)<5 range already covers '\n' in the whitespace mask) - a
need_lines gate threads through every kernel, the scalar reference,
the avx512 mirror, and the sliced workers.
- Thread retune: 12 threads past 32 MiB, 16 past 256 MiB (was 8) -
up to 18% on 110 MB, ~3% on 1.1 GB in QA's interleaved sweep.
- Benchmark suites pin LC_ALL=C so GNU wc -w can't silently switch to
multibyte decoding and inflate the win.
- popcount16 is guarded to x86 builds (zero-warnings on other arches);
checkwc's stray argument dropped.
Verified: selftest (kernels + sliced, now with need_lines coverage),
120-trial fuzz vs GNU (file + stdin, both locales), full test-all suite
three consecutive times, format-check and clang-tidy clean. The 1B
solo monster now lands at 3.7-6s (up to 268 Mlines/s) depending on how
warm the page cache is feeling.
Suites split into benchmarks/files/{lines,words} and
benchmarks/stdin/piping, with the monsters bolted onto the lines suite:
100M lines raced against coreutils (~2x win), 1B lines solo (~6-8s,
11 GB in one pass). Stdin redirects from regular files are now mmap'd
in count_stream, so the stdin suite wins too — up to 12.00x on words.
The ratio column now reports how many times faster fastwc is, not how
much of GNU's time it used. busybox was removed from the suite: it
stopped being a challenge and started being a participation trophy.
GNU wc's lone win — 1M lines by one millisecond on hand-tuned AVX-512
assembly — is now a historical footnote, and the README says so.
SIMD kernels (AVX-512/AVX-2/SSE2, runtime-dispatched) counting a
single pass over mmap'd files, split across cores past 8 MiB, with
exact GNU oracle parity (NBSP included, glibc's decoder fixed, the
whole -m path mirrored so counts agree at every boundary).
Result: 1ms vs 2ms at 1M lines, 8-9ms vs 22-24ms at 10M. Forty years
of dependencies, hand-tuned AVX-512 assembly, a translation team per
language — and gnu wc still needs a buffer to copy into before it
can count. We mapped the file and just counted. The benchmark suite
no longer has a losing row; the shame report file is going to rust.
huntedbytheirs