huntedbytheirs/libvct-spec
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add libVCT public specification set (000-015)

Browse Source
This commit is contained in:
huntedbytheirs
2026-09-13 15:56:53 -04:00
parent 591fdf3107
commit 765c91dabc
16 changed files with 3769 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
spec/012-testing.md
+203
View File
@@ -0,0 +1,203 @@
# 012 — Testing
- **Status:** Draft
- **Normative language:** `MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are to be
interpreted as described in RFC 2119.
## 1. Purpose
This file defines how an implementation demonstrates conformance. It specifies the test layers,
the fixture strategy, the structural verifier, the differential oracle, the fuzzing obligations,
and the CI guarantees. It states **what must be tested and what must hold**, not how to write the
test code.
## 2. Scope
The obligations here apply to every stage: the HIR optimizer, lowering, the VIR optimizer, both
backends, the driver, and the diagnostics service. The determinism guarantee from
[000 — Overview](000-overview.md) is tested here, but it is defined there.
## 3. Definitions
| Term | Meaning |
|---|---|
| **Fixture** | A fully constructed, trait-complete HIR module used as input to a test. |
| **Fixture layer** | The harness that constructs fixtures without a frontend. |
| **Layer** | One category of test with a distinct input source and success criterion. |
| **Verifier** | The component that checks SSA well-formedness and trait consistency. |
| **Oracle** | A trusted implementation used to judge another implementation's output. |
| **Round-trip property** | `print(parse(print(m))) == print(m)` for a module `m`. |
| **FileCheck test** | A test whose expectation is expressed as ordered pattern directives. |
| **Regression corpus** | The checked-in set of input IR plus expected directives. |
| **Matrix** | The cross-product of compiler, optimization level, and backend under test. |
## 4. Core constraint: VIR cannot be tested without a valid HIR fixture
VIR is a strict consumer of traits and performs no discovery (see [006 — VIR](006-vir.md) and
[007 — VIR Optimizer](007-vir-optimizer.md)). A VIR module cannot be authored meaningfully by
hand, because its facts come from HIR. Testing therefore starts from HIR.
1. Every component that consumes traits **MUST** be exercised through HIR fixtures that carry
complete, epoch-valid traits. A test **MUST NOT** assert behavior on VIR whose traits are
incomplete or stale, except a test that deliberately targets the verifier or the
internal-compiler-error path.
2. `vct.test.hirbuild` **MUST** be the fixture layer for HIR optimization, lowering, VIR
optimization, and both backends.
3. `vct.test.hirbuild` **MUST** construct modules, functions, nodes, types, values, and traits
without a frontend, and the modules it produces **MUST** pass the verifier.
## 5. Test layers
| Layer | Input | Success criterion | Required in v1 |
|---|---|---|---|
| Unit | D `unittest` per module | Function-level assertions | Yes |
| Fixture | HIR builder modules | Component behaves as specified on annotated HIR | Yes |
| Round-trip | Textual IR | `print(parse(print(m))) == print(m)` | Yes |
| FileCheck | IR plus directives | Ordered pattern directives match | Yes |
| Verifier | Any IR at any stage | SSA and traits are consistent | Yes |
| Differential | VIR plus host harness | Interpreter and compiled object agree | Yes |
| Fuzz | Random HIR and traits | See the properties below | Yes |
| Matrix | Regression corpus | All matrix configurations pass | Yes |
| Performance | Benchmark programs | Codegen quality and compile time tracked | Yes |
### 5.1 Unit tests
4. Each module **MUST** ship D `unittest` blocks covering its public and internal entry points.
### 5.2 HIR test-builder harness
5. The harness **MUST** produce input that is indistinguishable, for the consuming stage, from a
frontend-produced module. It **MUST** allow a test to set or suggest traits explicitly so that
trait validation, overwrite, and staleness behavior can be tested.
6. Tests for lowering and VIR optimization **MUST** build their HIR through the harness and run
the HIR optimizer first, so VIR sees the same complete annotation it sees in production.
### 5.3 Textual round-trip
7. For every module in the regression corpus, the textual form **MUST** satisfy
`print(parse(print(m))) == print(m)`. Parsing followed by printing **MUST** be lossless with
respect to the printed form.
8. The textual form exists for tools and round-trip tests. It is not a supported authoring
surface (see [000 — Overview](000-overview.md)); tests **MUST NOT** rely on hand-authored
textual VIR by itself as a source of traits.
### 5.4 FileCheck-style matching
9. A test **MUST** express its expected output using `CHECK`, `CHECK-NOT`, and `CHECK-NEXT`
directives against captured stage output.
10. The matcher **MUST** enforce directive order: `CHECK` lines match in sequence, `CHECK-NEXT`
requires a match on the line immediately following the previous match, and `CHECK-NOT`
requires that its pattern does **not** occur between the surrounding matches.
11. CI **MUST** fail a test when a `CHECK-NOT` pattern matches. Exact golden files **SHOULD NOT**
be used, with the sole exception of byte-exact determinism comparisons.
### 5.5 Structural verifier
12. A structural verifier **MUST** check SSA well-formedness and trait consistency. It **MUST**
run after every pass in debug builds, and it **MUST** be invocable on demand through
`--verify` in release builds (see [010 — Driver & CLI](010-driver-cli.md)).
13. The verifier **MUST** check at least these SSA properties, as required by
[006 — VIR](006-vir.md):
- every use is dominated by its definition;
- every basic block is terminated;
- every phi node has exactly one operand per predecessor block;
- every value has a single definition.
14. The verifier **MUST** check trait consistency: traits are complete for the current epoch, and
no two retained traits contradict one another (see [002 — Traits](002-traits.md)).
15. A failed verification **MUST** be an internal compiler error: it **MUST** assert in debug
builds and **MUST** produce an ICE diagnostic plus a failure result in release builds (see
[011 — Diagnostics](011-diagnostics.md)). It **MUST NOT** be silently ignored.
### 5.6 Differential testing
16. A VIR reference interpreter **MUST** be built in v1. It is the differential oracle.
17. For each differential test, the interpreter **MUST** run the same VIR that the backend is
given, the compiled object **MUST** be run on the same inputs, and the two observable results
**MUST** match. Observable results include defined output and exit status; unobservable
internal state is excluded.
18. Where the LLVM path is exercised, the C path and the LLVM path **SHOULD** be compared
differentially for equivalent observable behavior.
### 5.7 Fuzzing
19. Random-HIR and random-trait fuzzers **MUST** exist in v1.
20. Every fuzz input **MUST** satisfy all of these properties:
- the pipeline terminates within its configured budget;
- the verifier reports clean after every pass;
- emitted C, and emitted LLVM IR when the LLVM path is active, compiles;
- for deterministic programs, the interpreter and the compiled object agree.
21. A crash, a verifier failure, non-termination, or non-compiling output **MUST** be treated as
a bug and **MUST** block merge.
### 5.8 Test matrix
22. CI **MUST** test at least GCC and Clang. Each compiler **MUST** be tested across `-O0`,
`-O1`, `-O2`, `-O3`, `-Ofast`, and `-Oz`, and across the C backend and the LLVM backend where
the backend applies.
### 5.9 Performance benchmarks
23. Benchmarks **MUST** track both codegen quality and compile time. Compile time is measured
through `--time-passes` and `--stats` (see [010 — Driver & CLI](010-driver-cli.md)).
24. Because codegen quality is the headline goal (see [000 — Overview](000-overview.md)), a
regression in generated-code quality **MUST** be surfaced even when compile time improves.
## 6. Determinism testing
25. Every pipeline run in CI **MUST** be executed at least twice with identical `Config` and
identical input, and the two outputs **MUST** be byte-identical.
26. CI **MUST** vary the thread count across runs, including at least one run with `-j1` and one
with `-j` greater than one, to demonstrate independence from thread count.
27. The determinism guarantee itself is defined in [000 — Overview](000-overview.md). A
nondeterministic result **MUST** be treated as a correctness bug even if the outputs are
semantically equivalent.
## 7. Regression corpus
28. The regression corpus **MUST** store input IR paired with expected FileCheck patterns.
29. CI **MUST** run the full matrix over the corpus. A failing corpus entry **MUST** block merge.
30. A bug fix **MUST** add a corpus entry that fails before the fix and passes after it.
## 8. Invariants
- A green suite **MUST** mean that every requirement in this file held for the tested revision.
- The verifier **MUST** be sound: it **MUST NOT** report clean on malformed SSA or on
contradictory traits. A false negative is itself a bug.
- The differential oracle **MUST** be conservative: disagreement between the interpreter and the
compiled object **MUST** be reported as a failure, not explained away.
## 9. Examples
A round-trip test asserts:
```
assert(print(parse(print(m))) == print(m));
```
A FileCheck test names the stage output it inspects and lists its directives:
```
; RUN: vct --dump-vir --verify %s | FileCheck %s
; CHECK: define i32 @main(
; CHECK-NEXT: entry:
; CHECK-NOT: phi
; CHECK: ret i32 0
```
The first `CHECK` matches the function header, `CHECK-NEXT` requires the entry label on the very
next line, `CHECK-NOT` asserts that no phi node appears before the return, and the final `CHECK`
matches the return instruction.
## 10. Cross-references
- [000 — Overview](000-overview.md) for the determinism guarantee.
- [002 — Traits](002-traits.md) for trait completeness, epochs, and the verifier's trait checks.
- [004 — HIR Optimizer](004-hir-optimizer.md) for the cascade under test.
- [005 — Lowering](005-lowering.md) for SSA output invariants.
- [006 — VIR](006-vir.md) for the SSA properties the verifier checks.
- [007 — VIR Optimizer](007-vir-optimizer.md) for request results under test.
- [008 — C Backend](008-c-backend.md) and [009 — LLVM Backend](009-llvm-backend.md) for the
emitted forms checked by FileCheck and the matrix.
- [010 — Driver & CLI](010-driver-cli.md) for `--verify`, `--time-passes`, and `--stats`.
- [011 — Diagnostics](011-diagnostics.md) for the ICE policy the verifier feeds.
- [013 — Build & Packaging](013-build-packaging.md) for how `vct.test.hirbuild` ships.