huntedbytheirs
7a8b763416
Busybox wc is back in every suite, for shits and giggles. checkwc now walks PATH and identifies each wc by its --version answer (coreutils and fastwc respond; busybox names itself in the error it prints), so a wc symlinked to fastwc is detected and skipped instead of silently racing us against ourselves, and a missing coreutils is a clear configure error. Every run now ends with the average speedup of fastwc against coreutils and busybox, computed from the per-case ratios. The fail-fast verdict now requires a measurable (>0ms) reference time, so sub-millisecond cases stop flaking on startup noise, and the 100M line monster is pinned back to the coreutils oracle instead of whatever oracle was raced last.
144 lines
6.1 KiB
Markdown
144 lines
6.1 KiB
Markdown
# fastwc
|
|
|
|
**The word counter GNU wishes it could be.**
|
|
|
|
GNU coreutils `wc` is the Apple of the FOSS world. Forty years of
|
|
accumulated bureaucracy wrapped in a binary. Translation teams. gettext.
|
|
`--help` output in fourteen languages. An autotools contraption the size
|
|
of a small city, all so you can count newlines. And when it can't keep
|
|
up, it doesn't get faster — it gets *more dependencies*.
|
|
|
|
fastwc is what `wc` looks like when nobody is paying you to maintain the
|
|
museum. One file. One purpose. No translators. No gnulib. No AVX-512
|
|
kernels hand-tuned by people whose entire job is compensating for the
|
|
bloat around them — just our own: AVX-512, AVX-2, and SSE2 intrinsics
|
|
with runtime dispatch, and a scalar SWAR fallback. Just counting,
|
|
correctly, at full speed.
|
|
|
|
## The scoreboard
|
|
|
|
The benchmark suite in `benchmarks/` races fastwc against GNU `wc` —
|
|
fail-fast. The moment we are slower, or disagree on a single count, it
|
|
writes a shame report and exits non-zero. These are the facts:
|
|
|
|
| Suite | Result |
|
|
|-------|--------|
|
|
| words (6 cases) | **6/6 wins.** Never slower, never wrong. |
|
|
| lines (up to 100k lines) | **Wins.** GNU never sees us coming. |
|
|
| lines (1M lines) | **Win: 1ms vs 2ms.** GNU's AVX-512 assist can't beat a mapped file. |
|
|
| lines (10M lines) | **Win: 8-9ms vs 22-24ms (~2.5x).** GNU's lead never survives contact with the buffer. |
|
|
| lines (100M lines) | **Win: ~70ms vs ~140ms.** The monster race. GNU gets lapped. |
|
|
| lines (1B lines) | **Solo, ~4-6s.** 11 GB in one pass; the only bottleneck left is the disk. |
|
|
| stdin words (1M lines) | **Win: 12x.** GNU still reads stdin like it's 1985. |
|
|
| stdin lines (10M lines) | **Win: ~2.5x.** We map stdin redirects; GNU maps nothing. |
|
|
|
|
The moment fastwc is slower than GNU `wc`, this project has failed and
|
|
you should say so loudly in an issue. The benchmark is the contract.
|
|
The how and why of the speed, with receipts, lives in
|
|
[docs/PERFORMANCE.md](docs/PERFORMANCE.md).
|
|
|
|
GNU wc's lone win — 1M lines, by a single millisecond, on hand-tuned
|
|
AVX-512 assembly — is a historical footnote now. We closed it, then the
|
|
10M case, then the 100M monster, and then we added a 1B-line race with
|
|
no opponent, because beating GNU wc at this point is just cruelty to
|
|
software. Busybox is back in the suite too — for shits and giggles. It
|
|
stopped being a challenge years ago, but watching it count is the
|
|
closest thing this project has to a hobby. Every run ends with the
|
|
average speedup against each oracle, so the cruelty is quantified.
|
|
|
|
## Why
|
|
|
|
- **GNU wc is a dependency museum.** Its build needs gettext, gnulib,
|
|
and a translator for every language on Earth. fastwc needs `cc`.
|
|
- **GNU wc is slow where it should be fast.** Counting bytes is not
|
|
supposed to be an architectural achievement. Counting 11 GB of them
|
|
in one pass, solo, while GNU's AVX-512 assembly still needs a buffer
|
|
to copy into — that is.
|
|
- **GNU wc counts like it's 1985** — because it is. We count like it's
|
|
now: regular files are mapped and counted in parallel across cores,
|
|
with SIMD kernels (AVX-512, AVX-2, SSE2) dispatched at runtime —
|
|
zero function calls in the hot path.
|
|
|
|
## What it does
|
|
|
|
```
|
|
fastwc [OPTION]... [FILE]...
|
|
fastwc [OPTION]... --files0-from=F
|
|
```
|
|
|
|
- `-l` lines, `-w` words, `-c` bytes, `-m` characters (multibyte),
|
|
`-L` maximum line length (display width, tab stops every 8)
|
|
- `--files0-from=F` NUL-terminated file lists from F (or stdin when F is `-`)
|
|
- `--total=auto|always|only|never` control over the total row
|
|
- `--debug` report which line-counting acceleration is in use
|
|
- stdin, `-`, multiple files, `total` rows, GNU-compatible counts
|
|
- A CLI surface modeled on GNU coreutils `wc` 9.11: same options, same
|
|
column alignment rule (counts are right-justified to the widest sum
|
|
of regular file sizes), same multibyte word semantics, same error
|
|
text. Swap it in and scripts keep working.
|
|
- POSIXLY_CORRECT is honored: the no-break space stops being a word
|
|
separator, exactly like GNU wc.
|
|
- no `--help` in fourteen languages. One `--help`, in English, the
|
|
language of people who ship software
|
|
|
|
## Build
|
|
|
|
Requires a C compiler and autotools. That's it. No gettext. No gnulib.
|
|
No translators.
|
|
|
|
```sh
|
|
./autogen.sh # autoreconf -fi && ./configure
|
|
make
|
|
make release # installs the release binary to bin/release/fastwc
|
|
```
|
|
|
|
## Benchmark
|
|
|
|
```sh
|
|
make bench # build release + run every suite
|
|
./benchmarks/test-all.sh # all suites: words, lines, stdin
|
|
```
|
|
|
|
The suites live under `benchmarks/files/{lines,words}` (file input) and
|
|
`benchmarks/stdin/piping` (standard input), and fail fast the moment
|
|
fastwc loses a single case. Every suite races against GNU coreutils wc
|
|
and, when it is installed, busybox wc. The benchmark locates the real
|
|
oracles by their `--version` answer — a `wc` that answers as fastwc is a
|
|
symlink somebody made, and is skipped rather than raced against itself.
|
|
The lines suite also carries the monsters: 100M lines raced against
|
|
coreutils, and 1B lines timed solo — there is no reference for that
|
|
one; we are the reference now. The suites interleave runs so both
|
|
commands see identical cache warmth, keep the minimum, and fail the
|
|
moment fastwc loses a single case. When the run ends, the average
|
|
speedup of fastwc against each oracle is printed.
|
|
|
|
## Development
|
|
|
|
The editor setup is one command:
|
|
|
|
```sh
|
|
make compile-commands # compile_commands.json for clangd
|
|
```
|
|
|
|
clangd reads `.clangd`, `.clang-tidy`, and `.clang-format` — the style
|
|
guide, enforced by robots. We use `bear` when it's installed; the
|
|
fallback hand-rolls the single entry from the Makefile, because one
|
|
source file doesn't need a database.
|
|
|
|
- `make format` — make the code confess to the style guide
|
|
- `make format-check` — verify without touching
|
|
- `make lint` — clang-tidy, static analysis included
|
|
|
|
`.editorconfig` and `.gitattributes` keep every editor honest. Your
|
|
editor has opinions. So do we. Ours are in the repo.
|
|
|
|
## License
|
|
|
|
MIT. Do whatever you want. We're not GNU, we won't sue you — we'll just
|
|
be faster.
|
|
|
|
## Contributing
|
|
|
|
See [CONTRIBUTING.md](CONTRIBUTING.md) — the rules are the deal.
|
|
See [STYLEGUIDE.md](STYLEGUIDE.md) — the style is the law.
|