Files

T

huntedbytheirs ae5068d4e7 fix: thread-safety in count_sliced, ship measured wins

Review (5-lane) found one MAJOR: count_sliced ignored pthread_create's
return value - on EAGAIN it joined an indeterminate pthread_t and summed
an uninitialized slice, UB plus a silently wrong count. Threads are now
initialized to 0, a failed create counts its slice inline, and nt is
hard-capped at a named MAX_THREADS (the fixed jobs[8]/th[8] arrays
smash the stack past 8 threads - reproduced by QA's thread sweep).

Shipped from the measured optimization hunt:
- -w-only mode skips the newline compare/popcount entirely (the
  (x-9)<5 range already covers '\n' in the whitespace mask) - a
  need_lines gate threads through every kernel, the scalar reference,
  the avx512 mirror, and the sliced workers.
- Thread retune: 12 threads past 32 MiB, 16 past 256 MiB (was 8) -
  up to 18% on 110 MB, ~3% on 1.1 GB in QA's interleaved sweep.
- Benchmark suites pin LC_ALL=C so GNU wc -w can't silently switch to
  multibyte decoding and inflate the win.
- popcount16 is guarded to x86 builds (zero-warnings on other arches);
  checkwc's stray argument dropped.

Verified: selftest (kernels + sliced, now with need_lines coverage),
120-trial fuzz vs GNU (file + stdin, both locales), full test-all suite
three consecutive times, format-check and clang-tidy clean. The 1B
solo monster now lands at 3.7-6s (up to 268 Mlines/s) depending on how
warm the page cache is feeling.

2026-08-29 17:03:31 -04:00

4.5 KiB

Raw Blame History

Performance

GNU wc has had forty years and a team of people whose entire job is compensating for the bloat around them. Here is what they bought with that time, and what we paid for it.

The scoreboard, with receipts

Benchmarked on an Intel Core Ultra 7 265KF, min of 3 interleaved runs, page cache warm. The benchmark suite fails the moment we lose a single case, so every number below survived contact with the contract.

Case	GNU coreutils	fastwc
words, 100k lines	1-2ms	≤1ms
lines, 100k	1-2ms	≤1ms
lines, 1M	2-3ms	1ms
lines, 10M	21-24ms	8-9ms
lines, 100M (monster)	~140ms	~70ms
lines, 1B (solo)	—	~4-6s
bytes, 1GB sparse	reads all of it	`st_size`, no read

That is a ~2.5x win over GNU on 10M lines, a 2x win on 1M lines, and a 2x win on the 100M monster. At 1B lines — 11 GB of data — the solo run lands around 4-6 seconds (200-270 Mlines/s, warm cache), and the bottleneck is honest to admit: an 11 GB file does not fit in the 15 GB of RAM this machine has, so the last monster is racing the disk. The 100M case, which fits, runs at ~17 GB/s, and that number is the counting.

Why it's fast

SIMD kernels, dispatched at runtime. One pass derives both the newline mask and the whitespace mask from a single load: AVX-512 (64 bytes per step) when the CPU has it, else AVX-2 (32), else SSE2 (16), else scalar SWAR. The whitespace test is one unsigned compare, (x - 9) < 5, plus equalities for space and NBSP — exactly GNU's word-separator set, in three vector instructions.
Regular files are mapped, not streamed. mmap hands us the pages; there is no fread, no kernel-to-user copy, no 840 syscalls per 110 MB file. One syscall in, one count out. A stdin redirect from a regular file (fastwc -l < file) gets the same treatment — the data comes through stdin, but how we read it is our business. The stdin suite is why this shows up in the scoreboard too.
Parallel across cores. Files over 8 MiB are split into 64-byte aligned slices counted by up to 16 threads (12 past 32 MiB, 16 past 256 MiB). The kernels are pure, so the split needs no locks; word boundaries between slices are seeded from the byte before the slice, which makes the split exact. Below 8 MiB the thread spawn would cost more than the counting, so we don't bother.
No work that isn't asked for. -c on a regular file is st_size from fstat — GNU figured that one out too, so we copied the good idea. -l without -w skips the whitespace mask entirely.

Correctness is the other half of the contract

The benchmark compares counts, not just clocks. Matching GNU byte for byte took some archaeology:

GNU counts U+00A0 (non-breaking space) as a word separator even in the C locale, and the full Unicode white space set in -m mode — including U+2007 and U+202F, which glibc's iswspace forgets.
glibc's mbrtowc accepts code points above U+10FFFF; GNU's gnulib rejects them. So do we.
In a single-byte locale, GNU's -m counts bytes, not characters. So do we now.
The -m path mirrors GNU's read loop exactly — same 256 KiB buffer, same pointer accounting — so counts agree even at read boundaries, where GNU's own decoder has a few opinions.

The selftest (cc -DFASTWC_SELFTEST src/main.c) checks every kernel against the scalar reference over all sizes, both carry states, and both counting modes, plus the threaded slice split. A randomized fuzz against GNU as oracle (both locales, binary and UTF-8 data, every flag combination) passes 100%.

Known divergences, stated plainly

GNU wc's count is transport-dependent. The same bytes piped into wc -mw can count differently than the same bytes read from a file, because 64 KiB pipe chunks trigger a re-scan of its carried bytes. We reproduced this, then declined to. fastwc counts the data, not the plumbing.
Without -m, fastwc counts bytes with C-locale semantics. GNU silently switches to multibyte decoding for -w in UTF-8 locales. We don't — that's what -m is for, and it keeps the fast path fast. Under LC_ALL=C we match GNU exactly.

Reproducing

make release
./benchmarks/test-all.sh          # words, lines (monsters included), stdin

The suites live in benchmarks/files/{lines,words} and benchmarks/stdin/piping. The lines suite ends with the monsters: 100M lines raced against coreutils, and 1B lines timed solo (no reference to beat — the reference is us now).

4.5 KiB Raw Blame History