huntedbytheirs/fastwc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

bench: rub it in — monsters, stdin wins, busybox retired

Browse Source 
Suites split into benchmarks/files/{lines,words} and
benchmarks/stdin/piping, with the monsters bolted onto the lines suite:
100M lines raced against coreutils (~2x win), 1B lines solo (~6-8s,
11 GB in one pass). Stdin redirects from regular files are now mmap'd
in count_stream, so the stdin suite wins too — up to 12.00x on words.

The ratio column now reports how many times faster fastwc is, not how
much of GNU's time it used. busybox was removed from the suite: it
stopped being a challenge and started being a participation trophy.
GNU wc's lone win — 1M lines by one millisecond on hand-tuned AVX-512
assembly — is now a historical footnote, and the README says so.
This commit is contained in:
huntedbytheirs
2026-08-29 15:52:24 -04:00
parent ff465e981b
commit 6aa461f053
13 changed files with 362 additions and 177 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/PERFORMANCE.md
+24 -12
View File
@@ -10,17 +10,22 @@ Benchmarked on an Intel Core Ultra 7 265KF, min of 3 interleaved runs,
page cache warm. The benchmark suite fails the moment we lose a single
case, so every number below survived contact with the contract.
| Case | GNU coreutils | busybox | fastwc |
|------|--------------:|--------:|-------:|
| words, 100k lines | 1-2ms | 2ms | ≤1ms |
| lines, 100k | 1-2ms | 2ms | ≤1ms |
| lines, 1M | 2ms | 18ms | **1ms** |
| lines, 10M | 22-24ms | ~165ms | **8-9ms** |
| bytes, 1GB sparse | reads all of it | reads all of it | `st_size`, no read |
| Case | GNU coreutils | fastwc |
|------|--------------:|-------:|
| words, 100k lines | 1-2ms | ≤1ms |
| lines, 100k | 1-2ms | ≤1ms |
| lines, 1M | 2-3ms | **1ms** |
| lines, 10M | 21-24ms | **8-9ms** |
| lines, 100M (monster) | ~140ms | **~70ms** |
| lines, 1B (solo) | — | **~6s** |
| bytes, 1GB sparse | reads all of it | `st_size`, no read |
That is a ~2.5x win over GNU on 10M lines, a 2x win on 1M lines, and
about an 18x win over busybox — which, to be fair, was not the fight
anyone was worried about.
a 2x win on the 100M monster. At 1B lines — 11 GB of data — the solo
run lands around 6-8 seconds (125-170 Mlines/s), and the bottleneck is
honest to admit: an 11 GB file does not fit in the 15 GB of RAM this
machine has, so the last monster is racing the disk. The 100M case,
which fits, runs at ~17 GB/s, and that number is the counting.
## Why it's fast
@@ -32,7 +37,10 @@ anyone was worried about.
word-separator set, in three vector instructions.
2. **Regular files are mapped, not streamed.** `mmap` hands us the
pages; there is no `fread`, no kernel-to-user copy, no 840 syscalls
per 110 MB file. One syscall in, one count out.
per 110 MB file. One syscall in, one count out. A stdin redirect
from a regular file (`fastwc -l < file`) gets the same treatment —
the data comes through stdin, but how we read it is our business.
The stdin suite is why this shows up in the scoreboard too.
3. **Parallel across cores.** Files over 8 MiB are split into 64-byte
aligned slices counted by up to 8 threads. The kernels are pure, so
the split needs no locks; word boundaries between slices are seeded
@@ -81,6 +89,10 @@ combination) passes 100%.
```sh
make release
./benchmarks/bench-coreutils.sh # the real fight
./benchmarks/bench-busybox.sh # if you must
./benchmarks/test-all.sh # words, lines (monsters included), stdin
```
The suites live in `benchmarks/files/{lines,words}` and
`benchmarks/stdin/piping`. The lines suite ends with the monsters:
100M lines raced against coreutils, and 1B lines timed solo (no
reference to beat — the reference is us now).