huntedbytheirs
af61660c42
count_sliced computed the slice size from the requested thread count and only then clamped nt to MAX_THREADS, so the last slice ended before the file tail: raising the cap (as pick_threads now does) silently dropped the tail from every count. Cap nt first, then derive per. Monsters (>=256 MiB) now get one thread per core (capped at 24) instead of a hard 16, which measures ~1.17x on the 1B-line case and is flat on warm files. check_sliced now sweeps the cap boundary (16, 24, 25).
fastwc
The word counter GNU wishes it could be.
GNU coreutils wc is the Apple of the FOSS world. Forty years of
accumulated bureaucracy wrapped in a binary. Translation teams. gettext.
--help output in fourteen languages. An autotools contraption the size
of a small city, all so you can count newlines. And when it can't keep
up, it doesn't get faster — it gets more dependencies.
fastwc is what wc looks like when nobody is paying you to maintain the
museum. One file. One purpose. No translators. No gnulib. No AVX-512
kernels hand-tuned by people whose entire job is compensating for the
bloat around them — just our own: AVX-512, AVX-2, and SSE2 intrinsics
with runtime dispatch, and a scalar SWAR fallback. Just counting,
correctly, at full speed.
The scoreboard
The benchmark suite in benchmarks/ races fastwc against GNU wc —
fail-fast. The moment we are slower, or disagree on a single count, it
writes a shame report and exits non-zero. These are the facts:
| Suite | Result |
|---|---|
| words (6 cases) | 6/6 wins. Never slower, never wrong. |
| lines (up to 100k lines) | Wins. GNU never sees us coming. |
| lines (1M lines) | Win: 1ms vs 2ms. GNU's AVX-512 assist can't beat a mapped file. |
| lines (10M lines) | Win: 8-9ms vs 22-24ms (~2.5x). GNU's lead never survives contact with the buffer. |
| lines (100M lines) | Win: ~70ms vs ~140ms. The monster race. GNU gets lapped. |
| lines (1B lines) | Solo, ~4-6s. 11 GB in one pass; the only bottleneck left is the disk. |
| stdin words (1M lines) | Win: 12x. GNU still reads stdin like it's 1985. |
| stdin lines (10M lines) | Win: ~2.5x. We map stdin redirects; GNU maps nothing. |
The moment fastwc is slower than GNU wc, this project has failed and
you should say so loudly in an issue. The benchmark is the contract.
The how and why of the speed, with receipts, lives in
docs/PERFORMANCE.md.
GNU wc's lone win — 1M lines, by a single millisecond, on hand-tuned AVX-512 assembly — is a historical footnote now. We closed it, then the 10M case, then the 100M monster, and then we added a 1B-line race with no opponent, because beating GNU wc at this point is just cruelty to software. Busybox was removed from the suite for the same reason: it stopped being a challenge and started being a participation trophy.
Why
- GNU wc is a dependency museum. Its build needs gettext, gnulib,
and a translator for every language on Earth. fastwc needs
cc. - GNU wc is slow where it should be fast. Counting bytes is not supposed to be an architectural achievement. Counting 11 GB of them in one pass, solo, while GNU's AVX-512 assembly still needs a buffer to copy into — that is.
- GNU wc counts like it's 1985 — because it is. We count like it's now: regular files are mapped and counted in parallel across cores, with SIMD kernels (AVX-512, AVX-2, SSE2) dispatched at runtime — zero function calls in the hot path.
What it does
fastwc [-lwc] [-m] [file...]
-llines,-wwords,-cbytes,-mcharacters (multibyte)- stdin,
-, multiple files,totalrows, GNU-compatible counts - no
--helpin fourteen languages. One--help, in English, the language of people who ship software
Build
Requires a C compiler and autotools. That's it. No gettext. No gnulib. No translators.
./autogen.sh # autoreconf -fi && ./configure
make
make release # installs the release binary to bin/release/fastwc
Benchmark
make bench # build release + run every suite
./benchmarks/test-all.sh # all suites: words, lines, stdin
The suites live under benchmarks/files/{lines,words} (file input) and
benchmarks/stdin/piping (standard input), and fail fast the moment
fastwc loses a single case. The lines suite also carries the monsters:
100M lines raced against coreutils, and 1B lines timed solo — there is
no reference for that one; we are the reference now. The suites
interleave runs so both commands see identical cache warmth, keep the
minimum, and fail the moment fastwc loses a single case. GNU wc is
used as an oracle the same way you'd use a broken clock: occasionally
it's right, and it's the only one around.
Development
The editor setup is one command:
make compile-commands # compile_commands.json for clangd
clangd reads .clangd, .clang-tidy, and .clang-format — the style
guide, enforced by robots. We use bear when it's installed; the
fallback hand-rolls the single entry from the Makefile, because one
source file doesn't need a database.
make format— make the code confess to the style guidemake format-check— verify without touchingmake lint— clang-tidy, static analysis included
.editorconfig and .gitattributes keep every editor honest. Your
editor has opinions. So do we. Ours are in the repo.
License
MIT. Do whatever you want. We're not GNU, we won't sue you — we'll just be faster.
Contributing
See CONTRIBUTING.md — the rules are the deal. See STYLEGUIDE.md — the style is the law.