huntedbytheirs
6aa461f053
Suites split into benchmarks/files/{lines,words} and
benchmarks/stdin/piping, with the monsters bolted onto the lines suite:
100M lines raced against coreutils (~2x win), 1B lines solo (~6-8s,
11 GB in one pass). Stdin redirects from regular files are now mmap'd
in count_stream, so the stdin suite wins too — up to 12.00x on words.
The ratio column now reports how many times faster fastwc is, not how
much of GNU's time it used. busybox was removed from the suite: it
stopped being a challenge and started being a participation trophy.
GNU wc's lone win — 1M lines by one millisecond on hand-tuned AVX-512
assembly — is now a historical footnote, and the README says so.
128 lines
5.1 KiB
Markdown
128 lines
5.1 KiB
Markdown
# fastwc
|
|
|
|
**The word counter GNU wishes it could be.**
|
|
|
|
GNU coreutils `wc` is the Apple of the FOSS world. Forty years of
|
|
accumulated bureaucracy wrapped in a binary. Translation teams. gettext.
|
|
`--help` output in fourteen languages. An autotools contraption the size
|
|
of a small city, all so you can count newlines. And when it can't keep
|
|
up, it doesn't get faster — it gets *more dependencies*.
|
|
|
|
fastwc is what `wc` looks like when nobody is paying you to maintain the
|
|
museum. One file. One purpose. No translators. No gnulib. No AVX-512
|
|
kernels hand-tuned by people whose entire job is compensating for the
|
|
bloat around them — just our own: AVX-512, AVX-2, and SSE2 intrinsics
|
|
with runtime dispatch, and a scalar SWAR fallback. Just counting,
|
|
correctly, at full speed.
|
|
|
|
## The scoreboard
|
|
|
|
The benchmark suite in `benchmarks/` races fastwc against GNU `wc` —
|
|
fail-fast. The moment we are slower, or disagree on a single count, it
|
|
writes a shame report and exits non-zero. These are the facts:
|
|
|
|
| Suite | Result |
|
|
|-------|--------|
|
|
| words (6 cases) | **6/6 wins.** Never slower, never wrong. |
|
|
| lines (up to 100k lines) | **Wins.** GNU never sees us coming. |
|
|
| lines (1M lines) | **Win: 1ms vs 2ms.** GNU's AVX-512 assist can't beat a mapped file. |
|
|
| lines (10M lines) | **Win: 8-9ms vs 22-24ms (~2.5x).** GNU's lead never survives contact with the buffer. |
|
|
| lines (100M lines) | **Win: ~70ms vs ~140ms.** The monster race. GNU gets lapped. |
|
|
| lines (1B lines) | **Solo, ~6-8s.** 11 GB in one pass; the only bottleneck left is the disk. |
|
|
| stdin words (1M lines) | **Win: 12x.** GNU still reads stdin like it's 1985. |
|
|
| stdin lines (10M lines) | **Win: ~2.5x.** We map stdin redirects; GNU maps nothing. |
|
|
|
|
The moment fastwc is slower than GNU `wc`, this project has failed and
|
|
you should say so loudly in an issue. The benchmark is the contract.
|
|
The how and why of the speed, with receipts, lives in
|
|
[docs/PERFORMANCE.md](docs/PERFORMANCE.md).
|
|
|
|
GNU wc's lone win — 1M lines, by a single millisecond, on hand-tuned
|
|
AVX-512 assembly — is a historical footnote now. We closed it, then the
|
|
10M case, then the 100M monster, and then we added a 1B-line race with
|
|
no opponent, because beating GNU wc at this point is just cruelty to
|
|
software. Busybox was removed from the suite for the same reason: it
|
|
stopped being a challenge and started being a participation trophy.
|
|
|
|
## Why
|
|
|
|
- **GNU wc is a dependency museum.** Its build needs gettext, gnulib,
|
|
and a translator for every language on Earth. fastwc needs `cc`.
|
|
- **GNU wc is slow where it should be fast.** Counting bytes is not
|
|
supposed to be an architectural achievement. Counting 11 GB of them
|
|
in one pass, solo, while GNU's AVX-512 assembly still needs a buffer
|
|
to copy into — that is.
|
|
- **GNU wc counts like it's 1985** — because it is. We count like it's
|
|
now: regular files are mapped and counted in parallel across cores,
|
|
with SIMD kernels (AVX-512, AVX-2, SSE2) dispatched at runtime —
|
|
zero function calls in the hot path.
|
|
|
|
## What it does
|
|
|
|
```
|
|
fastwc [-lwc] [-m] [file...]
|
|
```
|
|
|
|
- `-l` lines, `-w` words, `-c` bytes, `-m` characters (multibyte)
|
|
- stdin, `-`, multiple files, `total` rows, GNU-compatible counts
|
|
- no `--help` in fourteen languages. One `--help`, in English, the
|
|
language of people who ship software
|
|
|
|
## Build
|
|
|
|
Requires a C compiler and autotools. That's it. No gettext. No gnulib.
|
|
No translators.
|
|
|
|
```sh
|
|
./autogen.sh # autoreconf -fi && ./configure
|
|
make
|
|
make release # installs the release binary to bin/release/fastwc
|
|
```
|
|
|
|
## Benchmark
|
|
|
|
```sh
|
|
make bench # build release + run every suite
|
|
./benchmarks/test-all.sh # all suites: words, lines, stdin
|
|
```
|
|
|
|
The suites live under `benchmarks/files/{lines,words}` (file input) and
|
|
`benchmarks/stdin/piping` (standard input), and fail fast the moment
|
|
fastwc loses a single case. The lines suite also carries the monsters:
|
|
100M lines raced against coreutils, and 1B lines timed solo — there is
|
|
no reference for that one; we are the reference now. The suites
|
|
interleave runs so both commands see identical cache warmth, keep the
|
|
minimum, and fail the moment fastwc loses a single case. GNU `wc` is
|
|
used as an oracle the same way you'd use a broken clock: occasionally
|
|
it's right, and it's the only one around.
|
|
|
|
## Development
|
|
|
|
The editor setup is one command:
|
|
|
|
```sh
|
|
make compile-commands # compile_commands.json for clangd
|
|
```
|
|
|
|
clangd reads `.clangd`, `.clang-tidy`, and `.clang-format` — the style
|
|
guide, enforced by robots. We use `bear` when it's installed; the
|
|
fallback hand-rolls the single entry from the Makefile, because one
|
|
source file doesn't need a database.
|
|
|
|
- `make format` — make the code confess to the style guide
|
|
- `make format-check` — verify without touching
|
|
- `make lint` — clang-tidy, static analysis included
|
|
|
|
`.editorconfig` and `.gitattributes` keep every editor honest. Your
|
|
editor has opinions. So do we. Ours are in the repo.
|
|
|
|
## License
|
|
|
|
MIT. Do whatever you want. We're not GNU, we won't sue you — we'll just
|
|
be faster.
|
|
|
|
## Contributing
|
|
|
|
See [CONTRIBUTING.md](CONTRIBUTING.md) — the rules are the deal.
|
|
See [STYLEGUIDE.md](STYLEGUIDE.md) — the style is the law.
|