libvct-spec/spec/012-testing.md

012 — Testing

• Status: Draft
• Normative language: MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are to be interpreted as described in RFC 2119.

1. Purpose

This file defines how an implementation demonstrates conformance. It specifies the test layers, the fixture strategy, the structural verifier, the differential oracle, the fuzzing obligations, and the CI guarantees. It states what must be tested and what must hold, not how to write the test code.

2. Scope

The obligations here apply to every stage: the HIR optimizer, lowering, the VIR optimizer, both backends, the driver, and the diagnostics service. The determinism guarantee from 000 — Overview is tested here, but it is defined there.

3. Definitions

Term	Meaning
Fixture	A fully constructed, trait-complete HIR module used as input to a test.
Fixture layer	The harness that constructs fixtures without a frontend.
Layer	One category of test with a distinct input source and success criterion.
Verifier	The component that checks SSA well-formedness and trait consistency.
Oracle	A trusted implementation used to judge another implementation's output.
Round-trip property	print(parse(print(m))) == print(m) for a module m.
FileCheck test	A test whose expectation is expressed as ordered pattern directives.
Regression corpus	The checked-in set of input IR plus expected directives.
Matrix	The cross-product of compiler, optimization level, and backend under test.

4. Core constraint: VIR cannot be tested without a valid HIR fixture

VIR is a strict consumer of traits and performs no discovery (see 006 — VIR and 007 — VIR Optimizer). A VIR module cannot be authored meaningfully by hand, because its facts come from HIR. Testing therefore starts from HIR.

1. Every component that consumes traits MUST be exercised through HIR fixtures that carry complete, epoch-valid traits. A test MUST NOT assert behavior on VIR whose traits are incomplete or stale, except a test that deliberately targets the verifier or the internal-compiler-error path.
2. vct.test.hirbuild MUST be the fixture layer for HIR optimization, lowering, VIR optimization, and both backends.
3. vct.test.hirbuild MUST construct modules, functions, nodes, types, values, and traits without a frontend, and the modules it produces MUST pass the verifier.

5. Test layers

Layer	Input	Success criterion	Required in v1
Unit	D unittest per module	Function-level assertions	Yes
Fixture	HIR builder modules	Component behaves as specified on annotated HIR	Yes
Round-trip	Textual IR	print(parse(print(m))) == print(m)	Yes
FileCheck	IR plus directives	Ordered pattern directives match	Yes
Verifier	Any IR at any stage	SSA and traits are consistent	Yes
Differential	VIR plus host harness	Interpreter and compiled object agree	Yes
Fuzz	Random HIR and traits	See the properties below	Yes
Matrix	Regression corpus	All matrix configurations pass	Yes
Performance	Benchmark programs	Codegen quality and compile time tracked	Yes

5.1 Unit tests

4. Each module MUST ship D unittest blocks covering its public and internal entry points.

5.2 HIR test-builder harness

5. The harness MUST produce input that is indistinguishable, for the consuming stage, from a frontend-produced module. It MUST allow a test to set or suggest traits explicitly so that trait validation, overwrite, and staleness behavior can be tested.
6. Tests for lowering and VIR optimization MUST build their HIR through the harness and run the HIR optimizer first, so VIR sees the same complete annotation it sees in production.

5.3 Textual round-trip

7. For every module in the regression corpus, the textual form MUST satisfy print(parse(print(m))) == print(m). Parsing followed by printing MUST be lossless with respect to the printed form.
8. The textual form exists for tools and round-trip tests. It is not a supported authoring surface (see 000 — Overview); tests MUST NOT rely on hand-authored textual VIR by itself as a source of traits.

5.4 FileCheck-style matching

9. A test MUST express its expected output using CHECK, CHECK-NOT, and CHECK-NEXT directives against captured stage output.
10. The matcher MUST enforce directive order: CHECK lines match in sequence, CHECK-NEXT requires a match on the line immediately following the previous match, and CHECK-NOT requires that its pattern does not occur between the surrounding matches.
11. CI MUST fail a test when a CHECK-NOT pattern matches. Exact golden files SHOULD NOT be used, with the sole exception of byte-exact determinism comparisons.

5.5 Structural verifier

12. A structural verifier MUST check SSA well-formedness and trait consistency. It MUST run after every pass in debug builds, and it MUST be invocable on demand through --verify in release builds (see 010 — Driver & CLI).
13. The verifier MUST check at least these SSA properties, as required by 006 — VIR:
    • every use is dominated by its definition;
    • every basic block is terminated;
    • every phi node has exactly one operand per predecessor block;
    • every value has a single definition.
14. The verifier MUST check trait consistency: traits are complete for the current epoch, and no two retained traits contradict one another (see 002 — Traits).
15. A failed verification MUST be an internal compiler error: it MUST assert in debug builds and MUST produce an ICE diagnostic plus a failure result in release builds (see 011 — Diagnostics). It MUST NOT be silently ignored.

5.6 Differential testing

16. A VIR reference interpreter MUST be built in v1. It is the differential oracle.
17. For each differential test, the interpreter MUST run the same VIR that the backend is given, the compiled object MUST be run on the same inputs, and the two observable results MUST match. Observable results include defined output and exit status; unobservable internal state is excluded.
18. Where the LLVM path is exercised, the C path and the LLVM path SHOULD be compared differentially for equivalent observable behavior.

5.7 Fuzzing

19. Random-HIR and random-trait fuzzers MUST exist in v1.
20. Every fuzz input MUST satisfy all of these properties:
    • the pipeline terminates within its configured budget;
    • the verifier reports clean after every pass;
    • emitted C, and emitted LLVM IR when the LLVM path is active, compiles;
    • for deterministic programs, the interpreter and the compiled object agree.
21. A crash, a verifier failure, non-termination, or non-compiling output MUST be treated as a bug and MUST block merge.

5.8 Test matrix

22. CI MUST test at least GCC and Clang. Each compiler MUST be tested across -O0, -O1, -O2, -O3, -Ofast, and -Oz, and across the C backend and the LLVM backend where the backend applies.

5.9 Performance benchmarks

23. Benchmarks MUST track both codegen quality and compile time. Compile time is measured through --time-passes and --stats (see 010 — Driver & CLI).
24. Because codegen quality is the headline goal (see 000 — Overview), a regression in generated-code quality MUST be surfaced even when compile time improves.

6. Determinism testing

25. Every pipeline run in CI MUST be executed at least twice with identical Config and identical input, and the two outputs MUST be byte-identical.
26. CI MUST vary the thread count across runs, including at least one run with -j1 and one with -j greater than one, to demonstrate independence from thread count.
27. The determinism guarantee itself is defined in 000 — Overview. A nondeterministic result MUST be treated as a correctness bug even if the outputs are semantically equivalent.

7. Regression corpus

28. The regression corpus MUST store input IR paired with expected FileCheck patterns.
29. CI MUST run the full matrix over the corpus. A failing corpus entry MUST block merge.
30. A bug fix MUST add a corpus entry that fails before the fix and passes after it.

8. Invariants

• A green suite MUST mean that every requirement in this file held for the tested revision.
• The verifier MUST be sound: it MUST NOT report clean on malformed SSA or on contradictory traits. A false negative is itself a bug.
• The differential oracle MUST be conservative: disagreement between the interpreter and the compiled object MUST be reported as a failure, not explained away.

9. Examples

A round-trip test asserts:

assert(print(parse(print(m))) == print(m));

A FileCheck test names the stage output it inspects and lists its directives:

; RUN: vct --dump-vir --verify %s | FileCheck %s
; CHECK: define i32 @main(
; CHECK-NEXT: entry:
; CHECK-NOT: phi
; CHECK: ret i32 0

The first CHECK matches the function header, CHECK-NEXT requires the entry label on the very next line, CHECK-NOT asserts that no phi node appears before the return, and the final CHECK matches the return instruction.

10. Cross-references

• 000 — Overview for the determinism guarantee.
• 002 — Traits for trait completeness, epochs, and the verifier's trait checks.
• 004 — HIR Optimizer for the cascade under test.
• 005 — Lowering for SSA output invariants.
• 006 — VIR for the SSA properties the verifier checks.
• 007 — VIR Optimizer for request results under test.
• 008 — C Backend and 009 — LLVM Backend for the emitted forms checked by FileCheck and the matrix.
• 010 — Driver & CLI for --verify, --time-passes, and --stats.
• 011 — Diagnostics for the ICE policy the verifier feeds.
• 013 — Build & Packaging for how vct.test.hirbuild ships.