huntedbytheirs/fastwc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

perf: make gnu wc the slowest thing in the benchmark

Browse Source 
SIMD kernels (AVX-512/AVX-2/SSE2, runtime-dispatched) counting a
single pass over mmap'd files, split across cores past 8 MiB, with
exact GNU oracle parity (NBSP included, glibc's decoder fixed, the
whole -m path mirrored so counts agree at every boundary).

Result: 1ms vs 2ms at 1M lines, 8-9ms vs 22-24ms at 10M. Forty years
of dependencies, hand-tuned AVX-512 assembly, a translation team per
language — and gnu wc still needs a buffer to copy into before it
can count. We mapped the file and just counted. The benchmark suite
no longer has a losing row; the shame report file is going to rust.
This commit is contained in:
huntedbytheirs
2026-08-29 15:32:08 -04:00
parent 5fc2f3e668
commit ff465e981b
6 changed files with 859 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTRIBUTING.md
+4 -4
View File
@@ -17,10 +17,10 @@ single count — your change does not ship. We did not spend this much
effort being faster than a forty-year-old dependency museum just so you
could add a `strlen()` in the hot loop.
The one standing exception: the 1M-line case, where GNU wins by exactly
one millisecond because they ship hand-tuned AVX-512 assembly. Closing
that gap is the project's open goal, not your excuse to be slower
anywhere else.
The old standing exception — the 1M-line case, where GNU's hand-tuned
AVX-512 assembly squeaked out a one-millisecond win — is closed. We
beat them there too now (see docs/PERFORMANCE.md). There are no
exceptions left, and there is no excuse to be slower anywhere else.
### 2. C99, or don't bother
Makefile.am
+1 -1
View File
@@ -1,4 +1,4 @@
AM_CFLAGS = -Wall -Wextra -O2
AM_CFLAGS = -Wall -Wextra -O2 -pthread
bin_PROGRAMS = fastwc
fastwc_SOURCES = src/main.c
README.md
+11 -5
View File
@@ -11,7 +11,9 @@ up, it doesn't get faster — it gets *more dependencies*.
fastwc is what `wc` looks like when nobody is paying you to maintain the
museum. One file. One purpose. No translators. No gnulib. No AVX-512
kernels hand-tuned by people whose entire job is compensating for the
bloat around them. Just counting, correctly, at full speed.
bloat around them — just our own: AVX-512, AVX-2, and SSE2 intrinsics
with runtime dispatch, and a scalar SWAR fallback. Just counting,
correctly, at full speed.
## The scoreboard
@@ -24,11 +26,14 @@ and exits non-zero. These are the facts:
|-------|--------|
| words (6 cases) | **6/6 wins.** Never slower, never wrong. |
| lines (up to 100k lines) | **Wins.** GNU never sees us coming. |
| lines (1M lines) | **GNU squeaks past by 1ms** — by shipping hand-tuned AVX-512 assembly written by a team of people who get paid for it. We call that cheating. Our SIMD pass is coming, and it will not be subtle. |
| lines (10M lines) | Not yet run. The benchmark aborts at the first loss. Coward. |
| lines (1M lines) | **Win: 1ms vs 2ms.** GNU's AVX-512 assist can't beat a mapped file. |
| lines (10M lines) | **Win: 8-9ms vs 22-24ms (~2.5x).** GNU's lead never survives contact with the buffer. |
| busybox lines (10M) | **Win: 8-9ms vs ~165ms (~18x).** If you must. |
The moment fastwc is slower than GNU `wc`, this project has failed and
you should say so loudly in an issue. The benchmark is the contract.
The how and why of the speed, with receipts, lives in
[docs/PERFORMANCE.md](docs/PERFORMANCE.md).
## Why
@@ -37,8 +42,9 @@ you should say so loudly in an issue. The benchmark is the contract.
- **GNU wc is slow where it should be fast.** Counting bytes is not
supposed to be an architectural achievement.
- **GNU wc counts like it's 1985** — because it is. We count like it's
now: fixed-stride SWAR loops, lookup tables, zero function calls in the
hot path.
now: regular files are mapped and counted in parallel across cores,
with SIMD kernels (AVX-512, AVX-2, SSE2) dispatched at runtime —
zero function calls in the hot path.
## What it does
STYLEGUIDE.md
+9 -5
View File
@@ -41,17 +41,21 @@ designed it — it was. We're one file, one purpose, one opinion.
## The hot path
`count_stream()`, `count_newlines()`, `count_words()` are the product.
The rest of the file is just the packaging.
`count_stream()`, `count_newlines()`, `count_words()`, and the SIMD
kernels (`count_lw_avx512`, `count_lw_avx2`, `count_lw_sse2`) are the
product. The rest of the file is just the packaging.
- **Fixed stride.** No per-token function calls. No `isspace()` in a
loop — that's what the lookup table is for.
- **No allocation, no locks, no syscalls in the counting loop.** The
`fread` is the only syscall, and it's not yours to add to.
kernels are pure; parallel slices need no locks. The `fread` (or the
single `mmap`) is the only syscall, and it's not yours to add to.
- **Branchless where it costs nothing.** A predictable branch is fine;
a mispredicted one is a lie you told the CPU.
- **The buffer is `static`, 128 KiB, and never grows.** GNU's wc reads
in chunks too — ours just doesn't make a ceremony of it.
- **Regular files are mapped, not streamed.** The kernel hands us the
pages; we count them, split across cores for anything past 8 MiB.
Stdin and odd files fall back to the `static`, 128 KiB buffer that
never grows.
## Counts
docs/PERFORMANCE.md
+86
View File
@@ -0,0 +1,86 @@
# Performance
GNU wc has had forty years and a team of people whose entire job is
compensating for the bloat around them. Here is what they bought with
that time, and what we paid for it.
## The scoreboard, with receipts
Benchmarked on an Intel Core Ultra 7 265KF, min of 3 interleaved runs,
page cache warm. The benchmark suite fails the moment we lose a single
case, so every number below survived contact with the contract.
| Case | GNU coreutils | busybox | fastwc |
|------|--------------:|--------:|-------:|
| words, 100k lines | 1-2ms | 2ms | ≤1ms |
| lines, 100k | 1-2ms | 2ms | ≤1ms |
| lines, 1M | 2ms | 18ms | **1ms** |
| lines, 10M | 22-24ms | ~165ms | **8-9ms** |
| bytes, 1GB sparse | reads all of it | reads all of it | `st_size`, no read |
That is a ~2.5x win over GNU on 10M lines, a 2x win on 1M lines, and
about an 18x win over busybox — which, to be fair, was not the fight
anyone was worried about.
## Why it's fast
1. **SIMD kernels, dispatched at runtime.** One pass derives both the
newline mask and the whitespace mask from a single load: AVX-512
(64 bytes per step) when the CPU has it, else AVX-2 (32), else SSE2
(16), else scalar SWAR. The whitespace test is one unsigned compare,
`(x - 9) < 5`, plus equalities for space and NBSP — exactly GNU's
word-separator set, in three vector instructions.
2. **Regular files are mapped, not streamed.** `mmap` hands us the
pages; there is no `fread`, no kernel-to-user copy, no 840 syscalls
per 110 MB file. One syscall in, one count out.
3. **Parallel across cores.** Files over 8 MiB are split into 64-byte
aligned slices counted by up to 8 threads. The kernels are pure, so
the split needs no locks; word boundaries between slices are seeded
from the byte before the slice, which makes the split exact. Below
8 MiB the thread spawn would cost more than the counting, so we
don't bother.
4. **No work that isn't asked for.** `-c` on a regular file is
`st_size` from `fstat` — GNU figured that one out too, so we copied
the good idea. `-l` without `-w` skips the whitespace mask entirely.
## Correctness is the other half of the contract
The benchmark compares counts, not just clocks. Matching GNU byte for
byte took some archaeology:
- GNU counts U+00A0 (non-breaking space) as a word separator even in
the C locale, and the full Unicode white space set in `-m` mode —
including U+2007 and U+202F, which glibc's `iswspace` forgets.
- glibc's `mbrtowc` accepts code points above U+10FFFF; GNU's gnulib
rejects them. So do we.
- In a single-byte locale, GNU's `-m` counts bytes, not characters.
So do we now.
- The `-m` path mirrors GNU's read loop exactly — same 256 KiB buffer,
same pointer accounting — so counts agree even at read boundaries,
where GNU's own decoder has a few opinions.
The selftest (`cc -DFASTWC_SELFTEST src/main.c`) checks every kernel
against the scalar reference over all sizes, both carry states, and
both counting modes, plus the threaded slice split. A randomized fuzz
against GNU as oracle (both locales, binary and UTF-8 data, every flag
combination) passes 100%.
## Known divergences, stated plainly
- **GNU wc's count is transport-dependent.** The same bytes piped into
`wc -mw` can count differently than the same bytes read from a file,
because 64 KiB pipe chunks trigger a re-scan of its carried bytes.
We reproduced this, then declined to. fastwc counts the data, not
the plumbing.
- **Without `-m`, fastwc counts bytes with C-locale semantics.**
GNU silently switches to multibyte decoding for `-w` in UTF-8
locales. We don't — that's what `-m` is for, and it keeps the fast
path fast. Under `LC_ALL=C` we match GNU exactly.
## Reproducing
```sh
make release
./benchmarks/bench-coreutils.sh # the real fight
./benchmarks/bench-busybox.sh # if you must
```
src/main.c
+748 -49
View File
@@ -1,24 +1,33 @@
/*
* fastwc - a fast wc replacement.
*
* Kickstart stub: functionally correct, with the standard fast-counting
* tricks already in place (memchr for newlines, a whitespace lookup table
* plus popcount for words). The next level of speed (SIMD / SWAR bulk
* scanning) plugs into count_stream() below.
* Fast path (-l/-w/-c): SIMD kernels with runtime dispatch. Each kernel
* derives the newline mask and the whitespace mask from one load, so
* lines and words share a single pass over the buffer. AVX-512 when the
* CPU has it, then AVX-2, then SSE2, then the scalar SWAR reference.
*/
#define _POSIX_C_SOURCE 200809L
#include <ctype.h>
#include <errno.h>
#include <fcntl.h>
#include <locale.h>
#include <pthread.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <unistd.h>
#include <wchar.h>
#include <wctype.h>
#if defined(__x86_64__) || defined(__i386__)
#include <immintrin.h>
#endif
enum
{
F_LINES = 1 << 0, /* -l: count '\n' */
@@ -38,12 +47,17 @@ typedef struct
int ok; /* read succeeded */
} counts_t;
static unsigned char ws_tab[256]; /* ws_tab[c] = 1 if c is whitespace */
static unsigned char ws_tab[256]; /* ws_tab[c] = 1 if c separates words */
/*
* Word separators match GNU wc (the benchmark oracle): the six C-locale
* isspace() bytes plus U+00A0 (non-breaking space), which GNU counts
* unless POSIXLY_CORRECT is set.
*/
static void init_ws_tab(void)
{
for (int i = 0; i < 256; i++)
ws_tab[i] = isspace((unsigned char)i) ? 1 : 0;
ws_tab[i] = (isspace((unsigned char)i) || i == 0xa0) ? 1 : 0;
}
static void usage(FILE *out)
@@ -63,16 +77,15 @@ static void usage(FILE *out)
/*
* Count '\n' in fixed 8-byte SWAR chunks. XOR turns '\n' bytes into zero
* bytes, the classic has-zero-byte trick flags them, popcount sums them.
* Constant stride (no per-newline memchr calls), so it stays fast even on
* files with dense newlines.
* bytes, the borrow-free haszero() flags them, popcount sums them. The
* common (x - 0x01..) & ~x & 0x80.. trick is NOT used: its borrow chain
* falsely flags a 0x01 byte that follows a zero byte, which is fine for
* strlen (lowest set bit) but overcounts here.
*/
static long long count_newlines(const unsigned char *s, size_t n)
{
const uint64_t nl = 0x0a0a0a0a0a0a0a0aULL;
const uint64_t lo = 0x0101010101010101ULL;
const uint64_t hi = 0x8080808080808080ULL;
const uint64_t cl = 0x7f7f7f7f7f7f7f7fULL;
const uint64_t seven = 0x7f7f7f7f7f7f7f7fULL;
long long k = 0;
size_t i = 0;
@@ -80,8 +93,9 @@ static long long count_newlines(const unsigned char *s, size_t n)
{
uint64_t x;
memcpy(&x, s + i, 8);
x = (x ^ nl) & cl;
k += (long long)__builtin_popcountll((x - lo) & ~x & hi);
x ^= nl;
x = ~(((x & seven) + seven) | x | seven);
k += (long long)__builtin_popcountll(x);
}
for (; i < n; i++)
k += s[i] == '\n';
@@ -132,6 +146,245 @@ static long long count_words(const unsigned char *s, size_t n, int *prev_ws)
return w;
}
/* SSE2 predates POPCNT; count 16-bit masks with the classic bit trick. */
static unsigned popcount16(unsigned x)
{
x = x - ((x >> 1) & 0x5555);
x = (x & 0x3333) + ((x >> 2) & 0x3333);
x = (x + (x >> 4)) & 0x0f0f;
return (x + (x >> 8)) & 0xff;
}
typedef struct
{
long long lines;
long long words;
} lw_t;
typedef lw_t (*count_lw_fn)(const unsigned char *s, size_t n, int *prev_ws,
int need_words);
/*
* Word separators match GNU wc (the benchmark oracle): the six C-locale
* isspace() bytes plus U+00A0 (non-breaking space). The 0x09..0x0d range
* is one unsigned compare, (x - 9) < 5, plus equalities for ' ' and NBSP.
*/
#if defined(__x86_64__) || defined(__i386__)
__attribute__((target("avx512f,avx512bw"))) static lw_t
count_lw_avx512(const unsigned char *s, size_t n, int *prev_ws, int need_words)
{
const __m512i nl = _mm512_set1_epi8('\n');
const __m512i sp = _mm512_set1_epi8(' ');
const __m512i nb = _mm512_set1_epi8((char)0xa0);
const __m512i lo = _mm512_set1_epi8(9);
const __m512i four = _mm512_set1_epi8(4);
long long lines = 0, words = 0;
size_t i = 0;
uint64_t prev = (*prev_ws != 0);
for (; i + 64 <= n; i += 64)
{
__m512i v = _mm512_loadu_si512((const void *)(s + i));
uint64_t nl_mask = (uint64_t)_mm512_cmpeq_epi8_mask(v, nl);
lines += (long long)_mm_popcnt_u64(nl_mask);
if (need_words)
{
/* min(d, 4) == d <=> (x - 9) < 5 unsigned */
__m512i d = _mm512_sub_epi8(v, lo);
uint64_t ws =
nl_mask | (uint64_t)_mm512_cmpeq_epi8_mask(v, sp) |
(uint64_t)_mm512_cmpeq_epi8_mask(v, nb) |
(uint64_t)_mm512_cmpeq_epi8_mask(_mm512_min_epu8(d, four), d);
words += (long long)_mm_popcnt_u64(~ws & ((ws << 1) | prev));
prev = (ws >> 63) & 1;
}
}
for (; i < n; i++)
{
int ws = ws_tab[s[i]];
lines += s[i] == '\n';
if (need_words)
{
if (prev && !ws)
words++;
prev = (uint64_t)ws;
}
}
*prev_ws = (int)prev;
lw_t r = {lines, words};
return r;
}
__attribute__((target("avx2"))) static lw_t
count_lw_avx2(const unsigned char *s, size_t n, int *prev_ws, int need_words)
{
const __m256i nl = _mm256_set1_epi8('\n');
const __m256i sp = _mm256_set1_epi8(' ');
const __m256i nb = _mm256_set1_epi8((char)0xa0);
const __m256i lo = _mm256_set1_epi8(9);
const __m256i four = _mm256_set1_epi8(4);
long long lines = 0, words = 0;
size_t i = 0;
uint32_t prev = (*prev_ws != 0);
for (; i + 32 <= n; i += 32)
{
__m256i v = _mm256_loadu_si256((const void *)(s + i));
uint32_t nl_mask =
(uint32_t)_mm256_movemask_epi8(_mm256_cmpeq_epi8(v, nl));
lines += (long long)_mm_popcnt_u32(nl_mask);
if (need_words)
{
/* min(d, 4) == d <=> (x - 9) < 5 unsigned */
__m256i d = _mm256_sub_epi8(v, lo);
uint32_t ws =
nl_mask |
(uint32_t)_mm256_movemask_epi8(_mm256_cmpeq_epi8(v, sp)) |
(uint32_t)_mm256_movemask_epi8(_mm256_cmpeq_epi8(v, nb)) |
(uint32_t)_mm256_movemask_epi8(
_mm256_cmpeq_epi8(_mm256_min_epu8(d, four), d));
words += (long long)_mm_popcnt_u32(~ws & ((ws << 1) | prev));
prev = (ws >> 31) & 1;
}
}
for (; i < n; i++)
{
int ws = ws_tab[s[i]];
lines += s[i] == '\n';
if (need_words)
{
if (prev && !ws)
words++;
prev = (uint32_t)ws;
}
}
*prev_ws = (int)prev;
lw_t r = {lines, words};
return r;
}
__attribute__((target("sse2"))) static lw_t
count_lw_sse2(const unsigned char *s, size_t n, int *prev_ws, int need_words)
{
const __m128i nl = _mm_set1_epi8('\n');
const __m128i sp = _mm_set1_epi8(' ');
const __m128i nb = _mm_set1_epi8((char)0xa0);
const __m128i lo = _mm_set1_epi8(9);
const __m128i four = _mm_set1_epi8(4);
long long lines = 0, words = 0;
size_t i = 0;
uint32_t prev = (*prev_ws != 0);
for (; i + 16 <= n; i += 16)
{
__m128i v = _mm_loadu_si128((const void *)(s + i));
uint32_t nl_mask = (uint32_t)_mm_movemask_epi8(_mm_cmpeq_epi8(v, nl));
lines += (long long)popcount16(nl_mask);
if (need_words)
{
__m128i d = _mm_sub_epi8(v, lo);
uint32_t ws = nl_mask |
(uint32_t)_mm_movemask_epi8(_mm_cmpeq_epi8(v, sp)) |
(uint32_t)_mm_movemask_epi8(_mm_cmpeq_epi8(v, nb)) |
(uint32_t)_mm_movemask_epi8(
_mm_cmpeq_epi8(_mm_min_epu8(d, four), d));
words += (long long)popcount16(~ws & ((ws << 1) | prev));
prev = (ws >> 15) & 1;
}
}
for (; i < n; i++)
{
int ws = ws_tab[s[i]];
lines += s[i] == '\n';
if (need_words)
{
if (prev && !ws)
words++;
prev = (uint32_t)ws;
}
}
*prev_ws = (int)prev;
lw_t r = {lines, words};
return r;
}
#endif /* x86 */
/* Reference path: the two scalar SWAR counters, kept as the fallback. */
static lw_t count_lw_scalar(const unsigned char *s, size_t n, int *prev_ws,
int need_words)
{
lw_t r;
r.lines = count_newlines(s, n);
r.words = need_words ? count_words(s, n, prev_ws) : 0;
return r;
}
static count_lw_fn count_lw = count_lw_scalar; /* chosen by pick_kernel() */
#if defined(__x86_64__) || defined(__i386__)
static count_lw_fn pick_kernel(void)
{
__builtin_cpu_init();
if (__builtin_cpu_supports("avx512bw"))
return count_lw_avx512;
if (__builtin_cpu_supports("avx2"))
return count_lw_avx2;
if (__builtin_cpu_supports("sse2"))
return count_lw_sse2;
return count_lw_scalar;
}
#else
static count_lw_fn pick_kernel(void)
{
return count_lw_scalar;
}
#endif
/*
* Word separator exactly as GNU wc defines it: the Unicode white space
* set. glibc's iswspace covers it incompletely (U+2007 and U+202F are
* missing), which is why the table is explicit.
*/
static int is_wspace(wchar_t wc)
{
if (wc >= 0x2000 && wc <= 0x200a) /* en space .. hair space */
return 1;
switch (wc)
{
case 0x09: /* tab */
case 0x0a: /* newline */
case 0x0b: /* vertical tab */
case 0x0c: /* form feed */
case 0x0d: /* carriage return */
case 0x20: /* space */
case 0xa0: /* no-break space */
case 0x1680: /* ogham space mark */
case 0x2028: /* line separator */
case 0x2029: /* paragraph separator */
case 0x202f: /* narrow no-break space */
case 0x205f: /* medium mathematical space */
case 0x3000: /* ideographic space */
return 1;
default:
return 0;
}
}
/*
* Multibyte (-m) path: decode each character with mbrtowc, carrying
* incomplete sequences across read boundaries. Only used when -m is
@@ -139,50 +392,88 @@ static long long count_words(const unsigned char *s, size_t n, int *prev_ws)
*/
static void count_stream_mb(FILE *fp, counts_t *c)
{
static unsigned char buf[1 << 17];
/*
* GNU wc's read size and pointer accounting are replicated exactly
* (256 KiB buffer, p + prev decode pointer, p += charbytes), so
* counts agree byte for byte with the oracle - its boundary
* artifacts included. Only used when -m is requested.
*/
static unsigned char buf[1 << 18];
mbstate_t st;
size_t pend = 0, nread;
size_t pend = 0;
int in_shift = 0;
int prev_ws = 1;
memset(&st, 0, sizeof st);
while ((nread = fread(buf + pend, 1, sizeof buf - pend, fp)) > 0)
for (;;)
{
size_t n = nread + pend;
size_t i = 0;
size_t nread = fread(buf + pend, 1, sizeof buf - pend, fp);
if (nread == 0 && pend == 0)
break;
c->bytes += (long long)nread;
while (i < n)
unsigned char *p = buf;
unsigned char *plim = buf + pend + nread;
size_t prev = pend;
pend = 0;
do
{
wchar_t wc;
size_t r;
size_t charbytes;
if (buf[i] < 0x80)
if (!in_shift && *p < 0x80)
{
wc = buf[i];
r = 1;
charbytes = 1;
wc = *p;
}
else
{
r = mbrtowc(&wc, (const char *)buf + i, n - i, &st);
if (r == (size_t)-2)
{ /* incomplete: carry over */
pend = n - i;
memmove(buf, buf + i, pend);
break;
}
if (r == (size_t)-1)
{ /* invalid sequence */
size_t scanbytes = plim - (p + prev);
size_t r =
mbrtowc(&wc, (const char *)(p + prev), scanbytes, &st);
prev = 0;
if (scanbytes < r)
{
if (r == (size_t)-2 && plim - p < (long)sizeof buf && nread)
{ /* incomplete: carry over */
pend = (size_t)(plim - p);
memmove(buf, p, pend);
in_shift = 1;
break;
}
/* encoding error: a byte but not a character */
p++;
memset(&st, 0, sizeof st);
wc = L'\xfffd';
r = 1;
in_shift = 0;
if (prev_ws)
{
c->words++;
prev_ws = 0;
}
continue;
}
if (wc > 0x10ffff || (wc >= 0xd800 && wc <= 0xdfff))
{ /* out of range: gnulib rejects, glibc does not */
p++;
if (prev_ws)
{
c->words++;
prev_ws = 0;
}
continue;
}
charbytes = r + (r == 0);
in_shift = !mbsinit(&st);
}
if (wc == L'\n')
c->lines++;
if (iswspace(wc))
if (is_wspace(wc))
{
prev_ws = 1;
}
@@ -192,27 +483,48 @@ static void count_stream_mb(FILE *fp, counts_t *c)
prev_ws = 0;
}
c->chars++;
i += r;
}
if (i >= n)
pend = 0;
p += charbytes;
} while (p < plim);
}
if (ferror(fp))
c->ok = 0;
}
/* Fast path (-l/-w/-c): one pass, per-chunk memchr + table counting. */
static void count_stream(FILE *fp, counts_t *c)
{
static unsigned char buf[1 << 17]; /* 128 KiB */
static _Alignas(64) unsigned char buf[1 << 17]; /* 128 KiB, SIMD-aligned */
size_t nread;
int prev_ws = 1; /* start of file: as if preceded by whitespace */
if (flags & F_CHARS)
{
count_stream_mb(fp, c);
if (MB_CUR_MAX <= 1)
{ /* single-byte locale: every byte is a character, like GNU wc */
for (;;)
{
nread = fread(buf, 1, sizeof buf, fp); /* NOLINT */
if (nread == 0)
break;
c->bytes += (long long)nread;
c->chars += (long long)nread;
if (flags & (F_LINES | F_WORDS))
{
lw_t r =
count_lw(buf, nread, &prev_ws, (flags & F_WORDS) != 0);
if (flags & F_LINES)
c->lines += r.lines;
if (flags & F_WORDS)
c->words += r.words;
}
}
}
else
{
count_stream_mb(fp, c);
}
if (ferror(fp))
c->ok = 0;
return;
}
@@ -222,16 +534,126 @@ static void count_stream(FILE *fp, counts_t *c)
if (nread == 0)
break;
c->bytes += (long long)nread;
if (flags & F_LINES)
c->lines += count_newlines(buf, nread);
if (flags & F_WORDS)
c->words += count_words(buf, nread, &prev_ws);
if (flags & (F_LINES | F_WORDS))
{
lw_t r = count_lw(buf, nread, &prev_ws, (flags & F_WORDS) != 0);
if (flags & F_LINES)
c->lines += r.lines;
if (flags & F_WORDS)
c->words += r.words;
}
}
if (ferror(fp))
c->ok = 0;
}
typedef struct
{
const unsigned char *s;
size_t n;
int prev_ws;
int need_words;
lw_t r;
} mjob_t;
static void *map_worker(void *arg)
{
mjob_t *j = arg;
j->r = count_lw(j->s, j->n, &j->prev_ws, j->need_words);
return NULL;
}
/*
* Count a whole mapping in nt slices. Slice starts are 64-byte aligned
* so SIMD loads never straddle into a neighbor's slice; the word
* boundary between slices is seeded from the byte before the slice,
* which makes the split exact. The kernels are pure, so no locks.
*/
static void count_sliced(const unsigned char *p, size_t n, int nt,
int need_words, long long *lines, long long *words)
{
mjob_t jobs[8];
pthread_t th[8];
long long tl = 0, tw = 0;
size_t per = (n + (size_t)nt - 1) / (size_t)nt;
int i;
per = (per + 63) & ~(size_t)63;
if (per == 0)
per = 64;
for (i = 0; i < nt; i++)
{
size_t start = (size_t)i * per;
jobs[i].s = p + start;
jobs[i].n = (start + per <= n) ? per : (start < n ? n - start : 0);
jobs[i].prev_ws = (i == 0 || jobs[i].n == 0) ? 1 : ws_tab[p[start - 1]];
jobs[i].need_words = need_words;
if (jobs[i].n > 0)
pthread_create(&th[i], NULL, map_worker, &jobs[i]);
else
th[i] = 0;
}
for (i = 0; i < nt; i++)
{
if (th[i] == 0)
continue;
pthread_join(th[i], NULL);
tl += jobs[i].r.lines;
tw += jobs[i].r.words;
}
*lines = tl;
*words = tw;
}
static int pick_threads(size_t n)
{
long ncpu = sysconf(_SC_NPROCESSORS_ONLN);
int nt;
if (n >= (size_t)32 << 20)
nt = 8;
else if (n >= (size_t)8 << 20)
nt = 4;
else
nt = 1;
if (ncpu > 0 && nt > ncpu)
nt = (int)ncpu;
return nt;
}
static void count_mapped(const unsigned char *p, size_t n, counts_t *c)
{
int need_words = (flags & F_WORDS) != 0;
int nt = pick_threads(n);
long long lines = 0, words = 0;
c->bytes += (long long)n;
if (nt <= 1)
{
int prev_ws = 1;
lw_t r = count_lw(p, n, &prev_ws, need_words);
lines = r.lines;
words = r.words;
}
else
{
count_sliced(p, n, nt, need_words, &lines, &words);
}
if (flags & F_LINES)
c->lines += lines;
if (flags & F_WORDS)
c->words += words;
}
static void count_file(const char *path, counts_t *c)
{
FILE *fp;
@@ -253,6 +675,31 @@ static void count_file(const char *path, counts_t *c)
return;
}
/* Regular files: map instead of streaming - no copy, and the count
* can be split across cores. Falls back to streaming on any hitch. */
struct stat st;
if (fstat(fileno(fp), &st) == 0 && S_ISREG(st.st_mode))
{
if (flags == F_BYTES)
{ /* GNU wc does not read the file either */
c->bytes = (long long)st.st_size;
fclose(fp);
return;
}
if (!(flags & F_CHARS) && st.st_size > 0)
{
void *m = mmap(NULL, (size_t)st.st_size, PROT_READ, MAP_PRIVATE,
fileno(fp), 0);
if (m != MAP_FAILED)
{
count_mapped((const unsigned char *)m, (size_t)st.st_size, c);
munmap(m, (size_t)st.st_size);
fclose(fp);
return;
}
}
}
count_stream(fp, c);
if (ferror(fp))
@@ -283,6 +730,7 @@ static void widen(int *width, long long v, int enabled)
*width = w;
}
#ifndef FASTWC_SELFTEST
int main(int argc, char **argv)
{
counts_t *rows;
@@ -291,6 +739,7 @@ int main(int argc, char **argv)
int i, a;
init_ws_tab();
count_lw = pick_kernel();
for (a = 1; a < argc; a++)
{
@@ -422,3 +871,253 @@ int main(int argc, char **argv)
free(rows);
return failed ? 1 : 0;
}
#else /* FASTWC_SELFTEST */
/*
* Kernel self-test (cc -DFASTWC_SELFTEST): every kernel is checked
* against the scalar reference over deterministic inputs covering all
* chunk tail lengths and both carry states. Kernels this host cannot
* run (e.g. AVX-512) can be exercised under qemu-x86_64 -cpu max.
*/
static unsigned long long rng_state = 1;
static unsigned rng32(void)
{
rng_state = rng_state * 6364136223846793005ULL + 1442695040888963407ULL;
return (unsigned)(rng_state >> 33);
}
static long long ref_lines(const unsigned char *s, size_t n)
{
return count_newlines(s, n);
}
static long long ref_words(const unsigned char *s, size_t n, int *prev_ws)
{
return count_words(s, n, prev_ws);
}
/*
* Scalar mirror of the AVX-512 kernel: identical 64-byte chunking and
* 64-bit mask arithmetic (bit-63 carry, 64-bit shift/OR, popcnt) for
* hosts that cannot execute the real zmm code. AVX-2 already proves
* the algorithm; this proves the width.
*/
static lw_t count_lw_avx512_mirror(const unsigned char *s, size_t n,
int *prev_ws, int need_words)
{
long long lines = 0, words = 0;
size_t i = 0;
uint64_t prev = (*prev_ws != 0);
for (; i + 64 <= n; i += 64)
{
uint64_t nl_mask = 0, ws = 0;
for (size_t j = 0; j < 64; j++)
{
if (s[i + j] == '\n')
nl_mask |= (uint64_t)1 << j;
if (ws_tab[s[i + j]])
ws |= (uint64_t)1 << j;
}
lines += (long long)__builtin_popcountll(nl_mask);
if (need_words)
{
words += (long long)__builtin_popcountll(~ws & ((ws << 1) | prev));
prev = (ws >> 63) & 1;
}
}
for (; i < n; i++)
{
int ws = ws_tab[s[i]];
lines += s[i] == '\n';
if (need_words)
{
if (prev && !ws)
words++;
prev = (uint64_t)ws;
}
}
*prev_ws = (int)prev;
lw_t r = {lines, words};
return r;
}
static int check_kernel(const char *name, count_lw_fn fn)
{
static unsigned char buf[512];
int fails = 0;
size_t n, k;
for (n = 0; n <= 512; n++)
{
for (k = 0; k < n; k++)
buf[k] = (unsigned char)rng32();
for (int pw = 0; pw <= 1; pw++)
{
for (int nw = 0; nw <= 1; nw++)
{
int a = pw, b = pw;
lw_t got = fn(buf, n, &a, nw);
long long want_l = ref_lines(buf, n);
long long want_w = nw ? ref_words(buf, n, &b) : 0;
if (got.lines != want_l || got.words != want_w ||
a != (nw ? b : pw))
{
printf("%s: n=%zu pw=%d nw=%d lines %lld/%lld "
"words %lld/%lld state %d/%d\n",
name, n, pw, nw, got.lines, want_l, got.words,
want_w, a, b);
if (n <= 64)
{
for (k = 0; k < n; k++)
printf("%02x", buf[k]);
printf("\n");
}
fails++;
if (fails > 5)
return fails;
}
}
}
}
/* whitespace-heavy patterns exercise the boundary logic harder */
for (n = 1; n <= 200; n++)
{
for (k = 0; k < n; k++)
buf[k] = (unsigned char)" \t\n\v\f\rx"[rng32() % 7];
for (int pw = 0; pw <= 1; pw++)
{
int a = pw, b = pw;
lw_t got = fn(buf, n, &a, 1);
if (got.lines != ref_lines(buf, n) ||
got.words != ref_words(buf, n, &b) || a != b)
{
printf("%s: ws-pattern n=%zu pw=%d failed\n", name, n, pw);
for (k = 0; k < n; k++)
printf("%02x", buf[k]);
printf("\n");
fails++;
break;
}
}
}
printf("%s: %s\n", name, fails ? "FAIL" : "ok");
return fails;
}
/*
* Slice-path check: the mmap+threads split must agree with the scalar
* reference for every thread count and many shapes, since word starts
* at slice boundaries are seeded from the neighboring byte.
*/
static int check_sliced(void)
{
static unsigned char buf[9001];
int fails = 0;
size_t n, k;
for (n = 0; n <= 9000; n += (n < 300 ? 1 : 37))
{
for (k = 0; k < n; k++)
buf[k] = (unsigned char)rng32();
for (int nt = 1; nt <= 8; nt++)
{
long long tl = 0, tw = 0;
long long want_l = count_newlines(buf, n);
int pw = 1;
long long want_w = count_words(buf, n, &pw);
count_sliced(buf, n, nt, 1, &tl, &tw);
if (tl != want_l || tw != want_w)
{
printf("sliced: n=%zu nt=%d lines %lld/%lld "
"words %lld/%lld\n",
n, nt, tl, want_l, tw, want_w);
fails++;
if (fails > 5)
return fails;
}
}
}
/* whitespace-heavy data stresses the slice boundary seeding */
for (n = 64; n <= 3000; n += 31)
{
for (k = 0; k < n; k++)
buf[k] = (unsigned char)" \t\n\v\f\r\xa0x"[rng32() % 8];
for (int nt = 1; nt <= 8; nt++)
{
long long tl = 0, tw = 0;
long long want_l = count_newlines(buf, n);
int pw = 1;
long long want_w = count_words(buf, n, &pw);
count_sliced(buf, n, nt, 1, &tl, &tw);
if (tl != want_l || tw != want_w)
{
printf("sliced-ws: n=%zu nt=%d lines %lld/%lld "
"words %lld/%lld\n",
n, nt, tl, want_l, tw, want_w);
fails++;
if (fails > 5)
return fails;
}
}
}
printf("sliced: %s\n", fails ? "FAIL" : "ok");
return fails;
}
int main(void)
{
int fails = 0;
const char *force = getenv("FASTWC_SELFTEST_FORCE");
init_ws_tab();
if (force == NULL || strcmp(force, "scalar") == 0)
fails += check_kernel("scalar", count_lw_scalar);
#if defined(__x86_64__) || defined(__i386__)
if (force == NULL || strcmp(force, "sse2") == 0)
fails += check_kernel("sse2", count_lw_sse2);
if (force == NULL || strcmp(force, "avx2") == 0)
fails += check_kernel("avx2", count_lw_avx2);
if (force == NULL)
{
__builtin_cpu_init();
if (__builtin_cpu_supports("avx512bw"))
fails += check_kernel("avx512", count_lw_avx512);
else
fails += check_kernel("avx512(mirror)", count_lw_avx512_mirror);
}
else if (strcmp(force, "avx512") == 0)
fails += check_kernel("avx512", count_lw_avx512);
else if (strcmp(force, "mirror") == 0)
fails += check_kernel("avx512(mirror)", count_lw_avx512_mirror);
#endif
count_lw = count_lw_scalar;
fails += check_sliced();
#if defined(__x86_64__) || defined(__i386__)
count_lw = count_lw_avx2;
fails += check_sliced();
count_lw = count_lw_sse2;
fails += check_sliced();
count_lw = count_lw_avx512_mirror;
fails += check_sliced();
#endif
if (fails)
{
printf("selftest FAILED\n");
return 1;
}
printf("selftest passed\n");
return 0;
}
#endif /* FASTWC_SELFTEST */