first: setup

setting up repo.

.gitignore

@@ -52,3 +52,26 @@ Module.symvers
 Mkfile.old
 dkms.conf
 
+# ---> fastwc benchmarks
+benchmarks/tools/genfile
+benchmarks/.data/
+benchmarks/FAILED-benchmark.txt
+
+# ---> fastwc build artifacts
+bin/
+fastwc
+
+# ---> autotools generated
+Makefile
+Makefile.in
+aclocal.m4
+autom4te.cache/
+compile
+config.log
+config.status
+configure
+depcomp
+.deps/
+install-sh
+missing
+

CONTRIBUTING.md

@@ -0,0 +1,139 @@
+# Contributing to fastwc
+
+We're building the word counter GNU wc is still trying to be after forty
+years. If that sounds like your kind of project, keep reading.
+
+## The rules
+
+These aren't guidelines. They're the deal.
+
+### 1. The benchmark is the contract
+
+`./benchmarks/bench-coreutils.sh` must pass. Not "mostly pass." Not
+"pass on your machine." Pass.
+
+The moment fastwc is slower than GNU wc — or disagrees with it on a
+single count — your change does not ship. We did not spend this much
+effort being faster than a forty-year-old dependency museum just so you
+could add a `strlen()` in the hot loop.
+
+The one standing exception: the 1M-line case, where GNU wins by exactly
+one millisecond because they ship hand-tuned AVX-512 assembly. Closing
+that gap is the project's open goal, not your excuse to be slower
+anywhere else.
+
+### 2. C99, or don't bother
+
+We compile with `-Wall -Wextra` and zero warnings. Zero. If the compiler
+is complaining, the compiler is right and you are wrong. We target C99
+with POSIX extensions, not whatever your compiler vendor's imagination
+happens to ship this quarter.
+
+### 3. No new dependencies. Ever.
+
+The C standard library is the only thing we trust. No gettext. No gnulib.
+No `libutf8pro` with its own translation team. GNU wc needs a build
+system the size of a small country to count newlines. We need `cc`.
+
+If your feature needs a library, the feature is too big. Make the
+feature smaller.
+
+### 4. The hot path is sacred
+
+`count_stream()`, `count_newlines()`, and `count_words()` in
+`src/main.c` are the entire point of this project. You may touch them
+only if `./benchmarks/bench-coreutils.sh` still passes afterward.
+Rules for the counting loops:
+
+- fixed stride, no per-token function calls
+- no allocation, no locking, no syscalls — the read is the only syscall
+- branchless where it costs nothing; lookup tables are the foundation,
+  not a hack
+- if you make it "clearer" but 2x slower, it's not clearer, it's worse
+
+### 5. Never suppress the truth
+
+No casts to silence warnings. No `-Wno-*` flags to make the noise go
+away. No `void*` where a real type exists. If the type system is
+fighting you, you're fighting the design. Fix the design.
+
+### 6. The codebase is one file. Keep it that way.
+
+`src/main.c` is one file and it will stay one file. GNU wc needs
+forty-two translation units and a build system with its own mailing
+list. We have one file and we can read all of it in one sitting. If your
+feature needs a new file, your feature is over-engineered.
+
+### 7. Tests are the benchmarks
+
+There is no separate test suite, because the benchmark *is* the test
+suite — it checks correctness against GNU wc (the only oracle around,
+and it's wrong often enough to keep us humble) and speed in the same
+breath. Add a flag? It gets benchmarked. Change the counting? It gets
+benchmarked. The fail-fast scripts in `benchmarks/` are your tests, and
+all of them must pass before you open a PR.
+
+### 8. No AI slop
+
+If it looks like ChatGPT wrote it, it gets rejected. We can tell. Write
+code like a human who's been doing this for a decade and is tired of
+GNU's excuses.
+
+## How to contribute
+
+### Pick something
+
+Good first issues:
+
+- Closing the 1M-line AVX-512 gap (the only loss on the board)
+- Benchmark case coverage for `-m` and multiple files
+- `bench-busybox.sh` CI on a machine that actually has busybox
+
+Ambitious issues:
+
+- A word counter that doesn't just tie GNU — it embarrasses it
+- Bigger SWAR chunks, wider strides, less patience
+
+### Send a PR
+
+1. Fork the repo
+2. Create a branch: `feat/my-thing` or `fix/my-bug`
+3. Write code that follows the rules above
+4. Run `./benchmarks/test-all.sh` — everything must pass
+5. Open a PR against `main`
+
+### PR requirements
+
+- Build must pass with zero warnings: `make && make release`
+- Benchmarks must pass: `./benchmarks/test-all.sh`
+- Follow [STYLEGUIDE.md](STYLEGUIDE.md). The style is the law.
+- No commented-out code. No dead code. No TODO without a date.
+- Commit messages in imperative: `Count newlines in 8-byte strides` not
+  `Added newline counting optimization`
+
+## What we won't merge
+
+- **Anything slower than GNU wc.** We didn't come this far to tie.
+- **Dependencies.** If it needs a library, it needs to not need a library.
+- **GNU-style bloat.** Long options with help strings in fourteen
+  languages. Translation infrastructure. "Enterprise" anything. This is
+  a word counter, not a CRM.
+- **Abstract nonsense.** Three layers of indirection to add a feature
+  means the feature is too complicated. GNU wc is what happens when you
+  let abstractions win.
+- **AI slop.** See rule 8.
+
+## Communication
+
+We don't have a Discord. We don't have a forum. We don't have a mailing
+list with a code of conduct and a weekly digest. Open an issue. Write a
+clear title, a reproduction case, and what you expected. We'll respond
+when we respond.
+
+If you want to propose a major feature, open an issue first. Surprise
+PRs that rewrite the counting loops get closed without review — the
+benchmark is the referee, and it doesn't know you.
+
+---
+
+fastwc is 0.1.0. Everything is subject to change except the rules above.

Makefile.am

@@ -0,0 +1,18 @@
+AM_CFLAGS = -Wall -Wextra -O2
+
+bin_PROGRAMS = fastwc
+fastwc_SOURCES = src/main.c
+
+# Release build consumed by benchmarks/ (expects bin/release/fastwc).
+release: all
+	$(MKDIR_P) bin/release
+	cp -f fastwc bin/release/fastwc
+
+# Convenience: build the release binary, then run every benchmark suite.
+bench: release
+	./benchmarks/test-all.sh
+
+clean-local:
+	rm -rf bin
+
+.PHONY: release bench

README.md

@@ -1,3 +1,86 @@
 # fastwc
 
-A extremely fast wc replacement.
+**The word counter GNU wishes it could be.**
+
+GNU coreutils `wc` is the Apple of the FOSS world. Forty years of
+accumulated bureaucracy wrapped in a binary. Translation teams. gettext.
+`--help` output in fourteen languages. An autotools contraption the size
+of a small city, all so you can count newlines. And when it can't keep
+up, it doesn't get faster — it gets *more dependencies*.
+
+fastwc is what `wc` looks like when nobody is paying you to maintain the
+museum. One file. One purpose. No translators. No gnulib. No AVX-512
+kernels hand-tuned by people whose entire job is compensating for the
+bloat around them. Just counting, correctly, at full speed.
+
+## The scoreboard
+
+The benchmark suite in `benchmarks/` races fastwc against GNU `wc`
+(and busybox, if you keep such things installed) — fail-fast. The moment
+we are slower, or disagree on a single count, it writes a shame report
+and exits non-zero. These are the facts:
+
+| Suite | Result |
+|-------|--------|
+| words (6 cases) | **6/6 wins.** Never slower, never wrong. |
+| lines (up to 100k lines) | **Wins.** GNU never sees us coming. |
+| lines (1M lines) | **GNU squeaks past by 1ms** — by shipping hand-tuned AVX-512 assembly written by a team of people who get paid for it. We call that cheating. Our SIMD pass is coming, and it will not be subtle. |
+| lines (10M lines) | Not yet run. The benchmark aborts at the first loss. Coward. |
+
+The moment fastwc is slower than GNU `wc`, this project has failed and
+you should say so loudly in an issue. The benchmark is the contract.
+
+## Why
+
+- **GNU wc is a dependency museum.** Its build needs gettext, gnulib,
+  and a translator for every language on Earth. fastwc needs `cc`.
+- **GNU wc is slow where it should be fast.** Counting bytes is not
+  supposed to be an architectural achievement.
+- **GNU wc counts like it's 1985** — because it is. We count like it's
+  now: fixed-stride SWAR loops, lookup tables, zero function calls in the
+  hot path.
+
+## What it does
+
+```
+fastwc [-lwc] [-m] [file...]
+```
+
+- `-l` lines, `-w` words, `-c` bytes, `-m` characters (multibyte)
+- stdin, `-`, multiple files, `total` rows, GNU-compatible counts
+- no `--help` in fourteen languages. One `--help`, in English, the
+  language of people who ship software
+
+## Build
+
+Requires a C compiler and autotools. That's it. No gettext. No gnulib.
+No translators.
+
+```sh
+./autogen.sh     # autoreconf -fi && ./configure
+make
+make release     # installs the release binary to bin/release/fastwc
+```
+
+## Benchmark
+
+```sh
+make bench                          # build release + run every suite
+./benchmarks/bench-coreutils.sh     # the real fight
+./benchmarks/bench-busybox.sh       # if you must
+```
+
+The suites interleave runs so both commands see identical cache warmth,
+keep the minimum, and fail the moment fastwc loses a single case. GNU
+`wc` is used as an oracle the same way you'd use a broken clock:
+occasionally it's right, and it's the only one around.
+
+## License
+
+MIT. Do whatever you want. We're not GNU, we won't sue you — we'll just
+be faster.
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) — the rules are the deal.
+See [STYLEGUIDE.md](STYLEGUIDE.md) — the style is the law.

STYLEGUIDE.md

@@ -0,0 +1,92 @@
+# fastwc Style Guide
+
+The style is the law. GNU wc can afford to look like a committee
+designed it — it was. We're one file, one purpose, one opinion.
+
+## The language
+
+- **C99 with POSIX.** `_POSIX_C_SOURCE 200809L`. We target the
+  standard, not a compiler vendor's mood.
+- **Zero warnings** with `-Wall -Wextra`. A warning is a bug you
+  haven't fixed yet. Fix it.
+- **No compiler-specific extensions** unless they're guarded and the
+  benchmark gets faster because of it. `__builtin_popcountll` is the
+  one exception — it's the price of admission on every compiler that
+  matters, and the benchmark proves it.
+
+## Format
+
+- **4 spaces.** Not tabs, not 8 spaces. GNU uses tabs because GNU was
+  written in 1985 by people who had strong opinions about typewriters.
+- **80 columns.** If your line is longer, your idea is longer than it
+  needs to be.
+- **Braces on the next line** (Allman). Everywhere. Functions, blocks,
+  switch cases. Consistency is the only religion here, and we're devout.
+- **Snake case for everything.** `count_stream`, `ws_tab`, `nread`.
+  No CamelCase — that's for languages that need help knowing what a
+  word is.
+
+## Structure
+
+- **One file: `src/main.c`.** See CONTRIBUTING.md rule 6. This is not a
+  suggestion.
+- **Everything is `static`.** The binary exports nothing. We're not a
+  library, we're a verdict.
+- **Types end in `_t`.** `counts_t`, and not much else. GNU has a type
+  for everything and a committee to name it. We have counts, and we
+  count.
+- **No globals except what's truly process-wide.** `flags` and `ws_tab`
+  live at file scope because they're the program's identity. Everything
+  else gets passed around like it's hot.
+
+## The hot path
+
+`count_stream()`, `count_newlines()`, `count_words()` are the product.
+The rest of the file is just the packaging.
+
+- **Fixed stride.** No per-token function calls. No `isspace()` in a
+  loop — that's what the lookup table is for.
+- **No allocation, no locks, no syscalls in the counting loop.** The
+  `fread` is the only syscall, and it's not yours to add to.
+- **Branchless where it costs nothing.** A predictable branch is fine;
+  a mispredicted one is a lie you told the CPU.
+- **The buffer is `static`, 128 KiB, and never grows.** GNU's wc reads
+  in chunks too — ours just doesn't make a ceremony of it.
+
+## Counts
+
+- **`long long` for everything that counts.** Files can be bigger than
+  your opinions. Signed, because overflow should scream, not wrap.
+- **Lines are `'\n'`s.** Not "lines of text," not "records." A file
+  without a trailing newline has fewer newlines than it has lines, and
+  that's correct. GNU agrees with us on this one, so it must be right.
+- **Words are whitespace-delimited runs.** The lookup table decides
+  what whitespace is; the popcount decides how many words; you don't
+  get an opinion.
+
+## Errors
+
+- **`fastwc: <file>: <message>` on stderr**, and keep going. GNU wc
+  fails gracefully across multiple files, and we'll credit them that
+  one thing. Then we'll be faster at it.
+- **Exit 1 if anything failed**, 0 otherwise. No drama, no
+  `EXIT_FAILURE` poetry.
+
+## Comments
+
+- **Only when the code can't speak for itself.** The SWAR newline
+  counter in `count_newlines()` gets a comment because
+  `(x ^ nl) & cl` is a sentence in a language you don't speak yet.
+  `i++` does not get a comment.
+- **No commented-out code.** The dead don't get to live in the file.
+- **No TODO without a date.** A TODO without a date is a promise you
+  plan to break.
+
+## What the style is not
+
+- It is not GNU's style. GNU's style was written by a committee,
+  ratified by a foundation, and translated into fourteen languages.
+- It is not "whatever your editor defaults to." Your editor has
+  opinions. So do we. Ours are right.
+- It is not negotiable in review. If a PR violates the style, the PR
+  is a rewrite request with extra steps.

autogen.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env sh
+# Generate the autotools build system and configure an in-tree build.
+set -e
+
+autoreconf -fi
+./configure "$@"

benchmarks/bench-busybox.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# bench-busybox.sh — benchmark the busybox wc applet against the release
+# build of fastwc. Fails fast: the moment fastwc is slower than (or
+# disagrees with) busybox wc, a human readable report is written to
+# FAILED-benchmark.txt and this script exits non-zero.
+#
+# usage: ./bench-busybox.sh
+set -u
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=std.sh
+source "$SCRIPT_DIR/std.sh"
+
+checkfastwc
+checkwc busybox
+BENCH_NAME="busybox"
+
+run_benchmark_suite

benchmarks/bench-coreutils.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# bench-coreutils.sh — benchmark the GNU Coreutils wc implementation against
+# the release build of fastwc. Fails fast: the moment fastwc is slower than
+# (or disagrees with) coreutils wc, a human readable report is written to
+# FAILED-benchmark.txt and this script exits non-zero.
+#
+# usage: ./bench-coreutils.sh
+set -u
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=std.sh
+source "$SCRIPT_DIR/std.sh"
+
+checkfastwc
+checkwc coreutils
+BENCH_NAME="coreutils"
+
+run_benchmark_suite

benchmarks/std.sh

@@ -0,0 +1,257 @@
+#!/usr/bin/env bash
+#
+# std.sh — shared "standard library" for the fastwc benchmark scripts.
+#
+# Provides the helpers every benchmark script needs:
+#   checkfastwc()      verify a release build of fastwc exists
+#   checkwc()          locate the coreutils or busybox wc implementation
+#   createrandstr()    print one random 10-character alphanumeric string
+#   createtxt()        create (or reuse) a text file with N such lines
+#   run_benchmark_suite()  run every word/line case for the selected wc
+#
+# Source this file from a benchmark script, then:
+#   checkfastwc
+#   checkwc coreutils          # or: checkwc busybox
+#   BENCH_NAME="coreutils"
+#   run_benchmark_suite
+#
+# The suite fails fast: the moment fastwc is slower than (or disagrees
+# with) the selected wc implementation, it writes a human readable report
+# to FAILED-benchmark.txt and returns non-zero.
+
+set -u
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+FASTWC="$SCRIPT_DIR/../bin/release/fastwc"
+DATA_DIR="$SCRIPT_DIR/.data"
+GENFILE="$SCRIPT_DIR/tools/genfile"   # optional C helper, built by test-all.sh
+
+BENCH_NAME="${BENCH_NAME:-wc}"        # set by the caller: coreutils | busybox
+BENCH_REPS="${BENCH_REPS:-3}"         # interleaved runs per case; minimum is kept
+RESULT_ROWS=""                        # accumulated results table
+
+WC_CMD=()                             # filled by checkwc(), e.g. (wc) or (busybox wc)
+TEXT_FILE=""                          # filled by createtxt()
+
+if [[ -z "$BENCH_REPS" || "$BENCH_REPS" -lt 1 ]]; then
+    BENCH_REPS=1
+fi
+
+# checkfastwc — make sure ../bin/release/fastwc exists and is executable.
+# Mimics autotools configure: prints "checking for ... yes/no" and bails
+# out with a helpful message when the release build is missing.
+checkfastwc() {
+    printf 'checking for release build fastwc... '
+    if [[ -x "$FASTWC" ]]; then
+        printf 'yes\n'
+        return 0
+    fi
+    printf 'no\n'
+    printf 'configure: error: no release build of fastwc found at %s\n' "$FASTWC" >&2
+    printf 'configure: error: run "make release" first to generate one\n' >&2
+    exit 1
+}
+
+# checkwc <coreutils|busybox> — locate the requested wc implementation and
+# store its invocation in $WC_CMD. Mimics autotools configure output and
+# exits on failure.
+checkwc() {
+    local impl="$1"
+    case "$impl" in
+        coreutils)
+            printf 'checking for coreutils wc... '
+            if command -v wc >/dev/null 2>&1 \
+               && wc --version 2>/dev/null | head -n1 | grep -qi 'GNU coreutils'; then
+                printf 'yes\n'
+                WC_CMD=(wc)
+            else
+                printf 'no\n'
+                printf 'configure: error: GNU Coreutils wc not found in PATH\n' >&2
+                exit 1
+            fi
+            ;;
+        busybox)
+            printf 'checking for busybox wc... '
+            if command -v busybox >/dev/null 2>&1 \
+               && busybox --list 2>/dev/null | grep -qx 'wc'; then
+                printf 'yes\n'
+                WC_CMD=(busybox wc)
+            else
+                printf 'no\n'
+                printf 'configure: error: busybox (with the wc applet) not found in PATH\n' >&2
+                exit 1
+            fi
+            ;;
+        *)
+            printf 'checkwc: error: unknown implementation "%s" (expected coreutils or busybox)\n' "$impl" >&2
+            exit 1
+            ;;
+    esac
+}
+
+# createrandstr — print one 10-character random alphanumeric combination.
+createrandstr() {
+    local chars='abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789'
+    local out='' i
+    for ((i = 0; i < 10; i++)); do
+        out+="${chars:$((RANDOM % ${#chars})):1}"
+    done
+    printf '%s\n' "$out"
+}
+
+# createtxt <lines> — make sure a text file with <lines> rows of random
+# 10-character alphanumeric strings exists. A copy generated by a previous
+# run is reused (checked by exact byte size: 10 chars + '\n' per line), so
+# repeated benchmark runs are cheap. Prints the path and sets $TEXT_FILE.
+createtxt() {
+    local lines="$1"
+    local expect=$((lines * 11))
+    local have=0
+
+    TEXT_FILE="$DATA_DIR/words-$lines.txt"
+
+    if [[ -f "$TEXT_FILE" ]]; then
+        have=$(stat -c '%s' "$TEXT_FILE" 2>/dev/null || printf '0')
+    fi
+
+    if [[ "$have" -ne "$expect" ]]; then
+        mkdir -p "$DATA_DIR"
+        if [[ -x "$GENFILE" ]]; then
+            "$GENFILE" "$lines" > "$TEXT_FILE" || {
+                printf 'createtxt: error: failed to generate %s\n' "$TEXT_FILE" >&2
+                return 1
+            }
+        else
+            printf 'createtxt: warning: %s not built, using slow shell fallback\n' "$GENFILE" >&2
+            printf 'createtxt: warning: run ./test-all.sh to build the helper tools\n' >&2
+            : > "$TEXT_FILE"
+            for ((i = 0; i < lines; i++)); do
+                createrandstr >> "$TEXT_FILE"
+            done
+        fi
+    fi
+
+    printf '%s\n' "$TEXT_FILE"
+}
+
+# time_ms <cmd...> — run a command once and print elapsed wall time in ms.
+time_ms() {
+    local s e
+    s=$(date +%s%N)
+    "$@" >/dev/null 2>&1
+    e=$(date +%s%N)
+    printf '%s\n' "$(( (e - s) / 1000000 ))"
+}
+
+# capture_count <cmd...> — print the first whitespace-separated field of a
+# command's output, i.e. the count reported by `wc -w/-l` or `fastwc -w/-l`.
+capture_count() {
+    "$@" 2>/dev/null | awk 'NR == 1 { print $1 }'
+}
+
+# write_failed_report <label> <reason> <wc_ms> <fast_ms> — write the
+# human readable failure report to FAILED-benchmark.txt.
+write_failed_report() {
+    local label="$1" reason="$2" wc_ms="$3" fast_ms="$4"
+    local report="$SCRIPT_DIR/FAILED-benchmark.txt"
+
+    {
+        printf 'fastwc benchmark FAILED\n'
+        printf '=======================\n'
+        printf 'implementation : %s wc\n' "$BENCH_NAME"
+        printf 'failed test    : %s\n' "$label"
+        printf 'failure        : %s\n' "$reason"
+        printf '\nresults\n'
+        printf '%s\n' '-------'
+        printf '%-28s %10s %10s %8s %s\n' 'test' 'wc' 'fastwc' 'ratio' 'status'
+        printf '%s' "$RESULT_ROWS"
+        printf '\nfastwc must never be slower than %s wc — benchmark aborted.\n' "$BENCH_NAME"
+    } > "$report"
+
+    printf '\nbenchmark FAILED (%s): %s\n' "$BENCH_NAME" "$reason" >&2
+    printf 'full results written to %s\n' "$report" >&2
+}
+
+# run_case <words|lines> <n-lines> <-w|-l> — create (or reuse) the text file,
+# then race the selected wc against fastwc. Fails the benchmark the moment
+# fastwc is slower or reports a different count.
+run_case() {
+    local mode="$1" lines="$2" flag="$3"
+    local noun='lines'; [[ "$lines" -eq 1 ]] && noun='line'
+    local label="${mode} (${lines} ${noun})"
+    local file wc_count fast_count wc_ms fast_ms ratio verdict reason row
+    local i d
+
+    file=$(createtxt "$lines") || return 1
+
+    # correctness: fastwc must report the same count as the reference wc
+    wc_count=$(capture_count "${WC_CMD[@]}" "$flag" "$file")
+    fast_count=$(capture_count "$FASTWC" "$flag" "$file")
+
+    # speed: interleaved timing so both commands see identical cache warmth;
+    # keep the minimum of $BENCH_REPS runs each to reduce noise
+    wc_ms=''
+    fast_ms=''
+    for ((i = 0; i < BENCH_REPS; i++)); do
+        d=$(time_ms "${WC_CMD[@]}" "$flag" "$file")
+        [[ -z "$wc_ms" || "$d" -lt "$wc_ms" ]] && wc_ms="$d"
+
+        d=$(time_ms "$FASTWC" "$flag" "$file")
+        [[ -z "$fast_ms" || "$d" -lt "$fast_ms" ]] && fast_ms="$d"
+    done
+
+    if [[ -n "$wc_ms" && "$wc_ms" -gt 0 ]]; then
+        ratio=$(awk -v f="$fast_ms" -v w="$wc_ms" 'BEGIN { printf "%.2fx", f / w }')
+    else
+        ratio='-'
+    fi
+
+    verdict='PASS'
+    reason=''
+    if [[ "$fast_count" != "$wc_count" ]]; then
+        verdict='FAIL'
+        reason="output mismatch (fastwc: ${fast_count}, ${BENCH_NAME} wc: ${wc_count})"
+    elif (( fast_ms > wc_ms )); then
+        verdict='FAIL'
+        reason="fastwc was slower (fastwc: ${fast_ms}ms vs ${BENCH_NAME} wc: ${wc_ms}ms)"
+    fi
+
+    row=$(printf '%-28s %10s %10s %8s %s\n' \
+        "$label" "wc: ${wc_ms}ms" "fastwc: ${fast_ms}ms" "$ratio" "$verdict")
+    RESULT_ROWS+="${row}"$'\n'
+    printf '%s\n' "$row"
+
+    if [[ "$verdict" == 'FAIL' ]]; then
+        write_failed_report "$label" "$reason" "$wc_ms" "$fast_ms"
+        return 1
+    fi
+    return 0
+}
+
+# run_benchmark_suite — run every word and line case for the wc selected by
+# checkwc(). Returns non-zero the first time fastwc loses.
+run_benchmark_suite() {
+    local rc=0 size
+
+    printf 'benchmarking %s wc vs fastwc (%s interleaved runs each, minimum kept)\n' \
+        "$BENCH_NAME" "$BENCH_REPS"
+    printf '%-28s %10s %10s %8s %s\n' 'test' 'wc' 'fastwc' 'ratio' 'status'
+
+    printf '%s\n' '--- words ---'
+    for size in 1 10 100 1000 10000 100000; do
+        run_case words "$size" -w || { rc=1; break; }
+    done
+
+    if [[ $rc -eq 0 ]]; then
+        printf '%s\n' '--- lines ---'
+        for size in 10000 100000 1000000 10000000; do
+            run_case lines "$size" -l || { rc=1; break; }
+        done
+    fi
+
+    if [[ $rc -eq 0 ]]; then
+        printf '\nall %s benchmarks passed — fastwc was never slower than %s wc\n' \
+            "$BENCH_NAME" "$BENCH_NAME"
+    fi
+    return $rc
+}

benchmarks/test-all.sh

@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+# test-all.sh — compile the benchmark helper tools first, then run every
+# benchmark (coreutils and busybox). Exits non-zero if any of them fails.
+#
+# usage: ./test-all.sh
+set -u
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# 1. build the helper tools before running any benchmark so createtxt()
+#    can use the fast C generator instead of the slow shell fallback
+mkdir -p "$SCRIPT_DIR/tools"
+if ! cc -O2 -Wall -o "$SCRIPT_DIR/tools/genfile" "$SCRIPT_DIR/tools/genfile.c"; then
+    printf 'test-all: error: failed to compile %s\n' "$SCRIPT_DIR/tools/genfile.c" >&2
+    exit 1
+fi
+printf 'built %s\n' "$SCRIPT_DIR/tools/genfile"
+
+# 2. run each benchmark (both run regardless, so every result is reported)
+"$SCRIPT_DIR/bench-coreutils.sh"
+rc_coreutils=$?
+"$SCRIPT_DIR/bench-busybox.sh"
+rc_busybox=$?
+
+# 3. summarize
+if [[ $rc_coreutils -ne 0 || $rc_busybox -ne 0 ]]; then
+    printf '\ntest-all: FAILED (coreutils=%s, busybox=%s)\n' "$rc_coreutils" "$rc_busybox" >&2
+    exit 1
+fi
+printf '\ntest-all: all benchmarks passed\n'

benchmarks/tools/genfile.c

@@ -0,0 +1,70 @@
+/*
+ * genfile — generate a text file of N lines, each a random 10-character
+ * alphanumeric string followed by a newline. Used by the fastwc benchmarks
+ * (createtxt in std.sh) to build test data quickly; std.sh falls back to a
+ * slow shell loop when this binary has not been built.
+ *
+ * usage: genfile <lines>
+ */
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <time.h>
+#include <unistd.h>
+
+#define LINE_LEN 10
+#define CHUNK (64 * 1024)
+
+static const char alpha[] =
+    "abcdefghijklmnopqrstuvwxyz"
+    "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+    "0123456789";
+#define ALPHA_LEN (sizeof(alpha) - 1)
+
+static uint64_t state;
+
+static uint64_t next(void)
+{
+    state ^= state << 13;
+    state ^= state >> 7;
+    state ^= state << 17;
+    return state;
+}
+
+int main(int argc, char **argv)
+{
+    long n;
+    long i;
+    int j;
+    char buf[CHUNK];
+    size_t used = 0;
+
+    if (argc != 2) {
+        fprintf(stderr, "usage: %s <lines>\n", argv[0]);
+        return 2;
+    }
+    n = atol(argv[1]);
+    if (n < 0) {
+        fprintf(stderr, "genfile: invalid line count: %s\n", argv[1]);
+        return 2;
+    }
+
+    /* seed from time + pid so each run produces fresh data */
+    state = (uint64_t)time(NULL) ^ ((uint64_t)getpid() << 32) ^ 0x9E3779B97F4A7C15ULL;
+
+    for (i = 0; i < n; i++) {
+        for (j = 0; j < LINE_LEN; j++)
+            buf[used++] = alpha[next() % ALPHA_LEN];
+        buf[used++] = '\n';
+        if (used + LINE_LEN + 1 > CHUNK) {
+            if (fwrite(buf, 1, used, stdout) != used)
+                return 1;
+            used = 0;
+        }
+    }
+    if (used && fwrite(buf, 1, used, stdout) != used)
+        return 1;
+    if (fclose(stdout) != 0)
+        return 1;
+    return 0;
+}

configure.ac

@@ -0,0 +1,10 @@
+AC_PREREQ([2.69])
+AC_INIT([fastwc], [0.1.0], [], [fastwc])
+AC_CONFIG_SRCDIR([src/main.c])
+
+AC_PROG_CC
+AC_USE_SYSTEM_EXTENSIONS
+AM_INIT_AUTOMAKE([foreign subdir-objects])
+
+AC_CONFIG_FILES([Makefile])
+AC_OUTPUT

src/main.c

@@ -0,0 +1,358 @@
+/*
+ * fastwc - a fast wc replacement.
+ *
+ * Kickstart stub: functionally correct, with the standard fast-counting
+ * tricks already in place (memchr for newlines, a whitespace lookup table
+ * plus popcount for words). The next level of speed (SIMD / SWAR bulk
+ * scanning) plugs into count_stream() below.
+ */
+
+#define _POSIX_C_SOURCE 200809L
+
+#include <ctype.h>
+#include <errno.h>
+#include <locale.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <wchar.h>
+#include <wctype.h>
+
+enum {
+    F_LINES = 1 << 0,   /* -l: count '\n' */
+    F_WORDS = 1 << 1,   /* -w: whitespace-separated tokens */
+    F_CHARS = 1 << 2,   /* -m: multibyte characters */
+    F_BYTES = 1 << 3,   /* -c: bytes */
+};
+
+static int flags = 0;
+
+typedef struct {
+    long long lines;
+    long long words;
+    long long chars;
+    long long bytes;
+    int       ok;       /* read succeeded */
+} counts_t;
+
+static unsigned char ws_tab[256];   /* ws_tab[c] = 1 if c is whitespace */
+
+static void init_ws_tab(void)
+{
+    for (int i = 0; i < 256; i++)
+        ws_tab[i] = isspace((unsigned char)i) ? 1 : 0;
+}
+
+static void usage(FILE *out)
+{
+    fprintf(out,
+        "usage: fastwc [-lwc] [-m] [file...]\n"
+        "\n"
+        "Count lines, words, and bytes (default) or selected counts.\n"
+        "With no file, or when file is -, read standard input.\n"
+        "\n"
+        "  -l  count lines\n"
+        "  -w  count words\n"
+        "  -c  count bytes\n"
+        "  -m  count characters\n"
+        "  --help     display this help and exit\n"
+        "  --version  output version information and exit\n");
+}
+
+/*
+ * Count '\n' in fixed 8-byte SWAR chunks. XOR turns '\n' bytes into zero
+ * bytes, the classic has-zero-byte trick flags them, popcount sums them.
+ * Constant stride (no per-newline memchr calls), so it stays fast even on
+ * files with dense newlines.
+ */
+static long long count_newlines(const unsigned char *s, size_t n)
+{
+    const uint64_t nl = 0x0a0a0a0a0a0a0a0aULL;
+    const uint64_t lo = 0x0101010101010101ULL;
+    const uint64_t hi = 0x8080808080808080ULL;
+    const uint64_t cl = 0x7f7f7f7f7f7f7f7fULL;
+    long long k = 0;
+    size_t i = 0;
+
+    for (; i + 8 <= n; i += 8) {
+        uint64_t x;
+        memcpy(&x, s + i, 8);
+        x = (x ^ nl) & cl;
+        k += (long long)__builtin_popcountll((x - lo) & ~x & hi);
+    }
+    for (; i < n; i++)
+        k += s[i] == '\n';
+    return k;
+}
+
+/*
+ * Count word starts (whitespace -> non-whitespace transitions) 8 bytes at
+ * a time. For a chunk, build a bitmask where bit j = 1 if byte j is
+ * whitespace; word starts inside the chunk are the 1->0 transitions of
+ * that mask, plus one for the left edge if the previous byte was
+ * whitespace. *prev_ws carries the boundary across chunks.
+ */
+static long long count_words(const unsigned char *s, size_t n, int *prev_ws)
+{
+    long long w = 0;
+    size_t i = 0;
+    int prev = *prev_ws;
+
+    for (; i + 8 <= n; i += 8) {
+        uint8_t m = 0;
+        m |= (uint8_t)ws_tab[s[i + 0]] << 0;
+        m |= (uint8_t)ws_tab[s[i + 1]] << 1;
+        m |= (uint8_t)ws_tab[s[i + 2]] << 2;
+        m |= (uint8_t)ws_tab[s[i + 3]] << 3;
+        m |= (uint8_t)ws_tab[s[i + 4]] << 4;
+        m |= (uint8_t)ws_tab[s[i + 5]] << 5;
+        m |= (uint8_t)ws_tab[s[i + 6]] << 6;
+        m |= (uint8_t)ws_tab[s[i + 7]] << 7;
+
+        /* bit j set iff byte j-1 was whitespace and byte j is not */
+        w += (long long)__builtin_popcount((unsigned)((uint8_t)~m & (m << 1)));
+        if (prev && !(m & 1))
+            w++;
+        prev = (m >> 7) & 1;
+    }
+
+    for (; i < n; i++) {
+        int ws = ws_tab[s[i]];
+        if (prev && !ws)
+            w++;
+        prev = ws;
+    }
+
+    *prev_ws = prev;
+    return w;
+}
+
+/*
+ * Multibyte (-m) path: decode each character with mbrtowc, carrying
+ * incomplete sequences across read boundaries. Only used when -m is
+ * requested, so it stays deliberately simple.
+ */
+static void count_stream_mb(FILE *fp, counts_t *c)
+{
+    static unsigned char buf[1 << 17];
+    mbstate_t st;
+    size_t pend = 0, nread;
+    int prev_ws = 1;
+
+    memset(&st, 0, sizeof st);
+
+    while ((nread = fread(buf + pend, 1, sizeof buf - pend, fp)) > 0) {
+        size_t n = nread + pend;
+        size_t i = 0;
+
+        c->bytes += (long long)nread;
+
+        while (i < n) {
+            wchar_t wc;
+            size_t r;
+
+            if (buf[i] < 0x80) {
+                wc = buf[i];
+                r = 1;
+            } else {
+                r = mbrtowc(&wc, (const char *)buf + i, n - i, &st);
+                if (r == (size_t)-2) {          /* incomplete: carry over */
+                    pend = n - i;
+                    memmove(buf, buf + i, pend);
+                    break;
+                }
+                if (r == (size_t)-1) {          /* invalid sequence */
+                    memset(&st, 0, sizeof st);
+                    wc = L'\xfffd';
+                    r = 1;
+                }
+            }
+
+            if (wc == L'\n')
+                c->lines++;
+            if (iswspace(wc)) {
+                prev_ws = 1;
+            } else if (prev_ws) {
+                c->words++;
+                prev_ws = 0;
+            }
+            c->chars++;
+            i += r;
+        }
+
+        if (i >= n)
+            pend = 0;
+    }
+
+    if (ferror(fp))
+        c->ok = 0;
+}
+
+/* Fast path (-l/-w/-c): one pass, per-chunk memchr + table counting. */
+static void count_stream(FILE *fp, counts_t *c)
+{
+    static unsigned char buf[1 << 17];   /* 128 KiB */
+    size_t nread;
+    int prev_ws = 1;    /* start of file: as if preceded by whitespace */
+
+    if (flags & F_CHARS) {
+        count_stream_mb(fp, c);
+        return;
+    }
+
+    while ((nread = fread(buf, 1, sizeof buf, fp)) > 0) {
+        c->bytes += (long long)nread;
+        if (flags & F_LINES)
+            c->lines += count_newlines(buf, nread);
+        if (flags & F_WORDS)
+            c->words += count_words(buf, nread, &prev_ws);
+    }
+
+    if (ferror(fp))
+        c->ok = 0;
+}
+
+static void count_file(const char *path, counts_t *c)
+{
+    FILE *fp;
+
+    if (strcmp(path, "-") == 0) {
+        fp = stdin;
+    } else {
+        fp = fopen(path, "rb");
+        if (fp == NULL) {
+            fprintf(stderr, "fastwc: %s: %s\n", path, strerror(errno));
+            c->ok = 0;
+            return;
+        }
+    }
+
+    count_stream(fp, c);
+
+    if (ferror(fp))
+        fprintf(stderr, "fastwc: %s: read error: %s\n",
+                strcmp(path, "-") == 0 ? "standard input" : path,
+                strerror(errno));
+
+    if (fp != stdin)
+        fclose(fp);
+}
+
+static int col_width(long long v)
+{
+    int w = 1;
+    while (v >= 10) {
+        v /= 10;
+        w++;
+    }
+    return w;
+}
+
+int main(int argc, char **argv)
+{
+    counts_t *rows;
+    int nfiles = 0;
+    int failed = 0;
+    int i, a;
+
+    init_ws_tab();
+
+    for (a = 1; a < argc; a++) {
+        const char *arg = argv[a];
+
+        if (arg[0] != '-' || arg[1] == '\0')
+            break;                          /* first file argument */
+        if (strcmp(arg, "--") == 0) {
+            a++;
+            break;
+        }
+        if (strcmp(arg, "--help") == 0) {
+            usage(stdout);
+            return 0;
+        }
+        if (strcmp(arg, "--version") == 0) {
+            printf("fastwc 0.1.0\n");
+            return 0;
+        }
+        for (const char *p = arg + 1; *p; p++) {
+            switch (*p) {
+            case 'l': flags |= F_LINES; break;
+            case 'w': flags |= F_WORDS; break;
+            case 'c': flags |= F_BYTES; break;
+            case 'm': flags |= F_CHARS; break;
+            default:
+                fprintf(stderr, "fastwc: invalid option -- '%c'\n", *p);
+                usage(stderr);
+                return 1;
+            }
+        }
+    }
+
+    if (flags == 0)
+        flags = F_LINES | F_WORDS | F_BYTES;    /* wc default: -l -w -c */
+
+    if (flags & F_CHARS)
+        setlocale(LC_CTYPE, "");
+
+    nfiles = argc - a;
+    if (nfiles == 0) {
+        rows = calloc(1, sizeof *rows);
+        rows[0].ok = 1;
+        count_stream(stdin, &rows[0]);
+        if (!rows[0].ok) {
+            fprintf(stderr, "fastwc: standard input: read error: %s\n",
+                    strerror(errno));
+            failed = 1;
+        }
+        nfiles = 1;
+    } else {
+        rows = calloc((size_t)nfiles, sizeof *rows);
+        for (i = 0; i < nfiles; i++) {
+            rows[i].ok = 1;
+            count_file(argv[a + i], &rows[i]);
+            if (!rows[i].ok)
+                failed = 1;
+        }
+    }
+
+    /* Column widths: widest count in each column across rows + total. */
+    int wl = 1, ww = 1, wm = 1, wb = 1;
+    long long tl = 0, tw = 0, tm = 0, tb = 0;
+
+    for (i = 0; i < nfiles; i++) {
+        counts_t *r = &rows[i];
+        int x;
+        tl += r->lines; tw += r->words; tm += r->chars; tb += r->bytes;
+        if ((flags & F_LINES) && (x = col_width(r->lines)) > wl) wl = x;
+        if ((flags & F_WORDS) && (x = col_width(r->words)) > ww) ww = x;
+        if ((flags & F_CHARS) && (x = col_width(r->chars)) > wm) wm = x;
+        if ((flags & F_BYTES) && (x = col_width(r->bytes)) > wb) wb = x;
+    }
+    if ((flags & F_LINES) && col_width(tl) > wl) wl = col_width(tl);
+    if ((flags & F_WORDS) && col_width(tw) > ww) ww = col_width(tw);
+    if ((flags & F_CHARS) && col_width(tm) > wm) wm = col_width(tm);
+    if ((flags & F_BYTES) && col_width(tb) > wb) wb = col_width(tb);
+
+    for (i = 0; i < nfiles; i++) {
+        counts_t *r = &rows[i];
+        if (flags & F_LINES) printf("%*lld ", wl, r->lines);
+        if (flags & F_WORDS) printf("%*lld ", ww, r->words);
+        if (flags & F_CHARS) printf("%*lld ", wm, r->chars);
+        if (flags & F_BYTES) printf("%*lld ", wb, r->bytes);
+        if (argc - a > 0)
+            printf("%s", argv[a + i]);
+        printf("\n");
+    }
+
+    if (argc - a > 1) {
+        if (flags & F_LINES) printf("%*lld ", wl, tl);
+        if (flags & F_WORDS) printf("%*lld ", ww, tw);
+        if (flags & F_CHARS) printf("%*lld ", wm, tm);
+        if (flags & F_BYTES) printf("%*lld ", wb, tb);
+        printf("total\n");
+    }
+
+    free(rows);
+    return failed ? 1 : 0;
+}