diff --git a/spec/000-overview.md b/spec/000-overview.md new file mode 100644 index 0000000..9716e97 --- /dev/null +++ b/spec/000-overview.md @@ -0,0 +1,117 @@ +# 000 — Overview + +- **Status:** Draft +- **Normative language:** `MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are to be + interpreted as described in RFC 2119. + +## 1. Purpose + +libVCT (the **Vox C Transpiler Library**) is a compiler library written in D. Given a program's +IR, libVCT optimizes that IR, transpiles it to C17, and drives a system C compiler to produce an +object file. An alternate backend emits textual LLVM IR instead of C. + +libVCT is a **library**, not a compiler. A frontend embeds it, builds IR through its builder API, +and receives either C source or LLVM IR back. libVCT never links object files into an executable; +that is the caller's job, with the single opt-in exception of `-mangled` mode +(see [010 — Driver & CLI](010-driver-cli.md)). + +## 2. The two-stage IR thesis + +libVCT is organized around **two IRs**: + +- **HIR**: a high-level, tree-shaped AST. The frontend constructs it; libVCT annotates it *in + place* with **traits**. HIR performs discovery and small, cheap folds. +- **VIR**: a low-level SSA/CFG instruction IR. VIR performs the heavy transformations and + performs **no discovery**; it is a strict consumer of the traits HIR produced. + +The design thesis: because HIR hands VIR a complete, annotated "dictionary" of facts, VIR can skip +analysis and go straight to transformation. HIR's cheap folds cascade and parallelize per function, +and the VIR that results is smaller and faster to optimize. The intended result is LLVM-class +codegen quality at materially lower compile time. + +## 3. Goals + +1. **Codegen quality is the headline.** Generated code quality takes priority over compile speed. +2. **Two-stage optimization.** HIR performs small folds and trait derivation; VIR performs large + transformations. +3. **Trait-driven VIR.** VIR never re-analyzes. Every fact it uses is an attribute, relationship, or + request supplied by HIR. +4. **Parallel, cascading HIR.** HIR passes run per function in parallel and cascade to a fixpoint. +5. **Compiler-friendly C.** The C backend always emits C written to be pattern-matched by GCC and + Clang. +6. **Deterministic output.** The same input and configuration produce byte-identical output, + regardless of thread count. + +## 4. Non-goals (v1) + +- **Linking.** libVCT produces object files; linking is the caller's job. The sole exception is + `-mangled` mode. +- **Windows / MSVC.** v1 is Linux/POSIX-first and supports GCC and Clang only. +- **LLVM bitcode.** The LLVM backend emits textual `.ll` only; the caller runs `llvm-as`. +- **LLVM debug metadata.** There is no `!llvm.dbg` in v1. Debug mapping exists on the C path via + `#line` directives. +- **A hand-authored VIR.** VIR is internal and intentionally hostile to hand-authoring. A textual + form exists solely for tools and round-trip tests; it is not a supported authoring surface. +- **Consuming LLVM IR or Vox's existing IR.** libVCT owns its IR definition. + +## 5. Glossary + +| Term | Meaning | +|---|---| +| **HIR** | High-level IR: the frontend AST, annotated in place with traits. | +| **VIR** | Vox IR: a low-level SSA/CFG instruction IR. | +| **Trait** | The union of an Attribute, Requests, and Relationships attached to a node. | +| **Attribute** | A fact about a node (type, value, mutability, …). | +| **Request** | An attribute-derived suggestion from HIR to VIR. | +| **Relationship** | A directed edge between nodes, traceable up and down. | +| **Cascade** | The chain of HIR rewrites triggered by a single fold. | +| **Lowering** | The "middleman" that translates annotated HIR into VIR SSA. | +| **Remark** | An optimizer decision record emitted for diagnostics. | + +## 6. Conformance + +An implementation conforms to this specification if it satisfies every `MUST` and `MUST NOT` +requirement in every specification file listed in §8. `SHOULD` requirements may be violated only +with a documented, deliberate reason. `MAY` requirements are optional. + +Where this specification and an implementation disagree, this specification is authoritative. Where +a later specification file refines an earlier one, the later file is authoritative for its subject. + +## 7. Determinism guarantee + +For a fixed `Config` and a fixed input, libVCT **MUST** produce byte-identical output on every run, +independent of thread count. This guarantee binds every stage: HIR optimization, lowering, VIR +optimization, and both backends. Determinism is a correctness property, not an optimization. + +## 8. Specification index + +| File | Subject | +|---|---| +| [000 — Overview](000-overview.md) | Purpose, thesis, goals, glossary, conformance, determinism. | +| [001 — Architecture](001-architecture.md) | Modules, data flow, public interfaces, boundaries. | +| [002 — Traits](002-traits.md) | Attributes, requests, relationships, representation, staleness. | +| [003 — HIR](003-hir.md) | HIR node model, builder contract, textual format, `defer`. | +| [004 — HIR Optimizer](004-hir-optimizer.md) | Pass framework, cascade, parallelism, comptime, budgets. | +| [005 — Lowering](005-lowering.md) | HIR → VIR translation, SSA construction, memory model. | +| [006 — VIR](006-vir.md) | VIR SSA/CFG model, type system, instructions, invariants. | +| [007 — VIR Optimizer](007-vir-optimizer.md) | Pass framework, analyses, pipeline, request handling. | +| [008 — C Backend](008-c-backend.md) | VIR → C17: out-of-SSA, compiler-friendly C, layout, ABI. | +| [009 — LLVM Backend](009-llvm-backend.md) | VIR → textual LLVM IR. | +| [010 — Driver & CLI](010-driver-cli.md) | Pipeline orchestration, flags, C compiler invocation. | +| [011 — Diagnostics](011-diagnostics.md) | Diagnostics service, source maps, channels, ICE policy. | +| [012 — Testing](012-testing.md) | Test layers, fixture harness, oracle, fuzzing, matrix. | +| [013 — Build & Packaging](013-build-packaging.md) | Toolchain, artifacts, API surfaces, memory ownership. | +| [014 — Worked Example](014-worked-example.md) | End-to-end trait/request walkthrough (normative). | +| [015 — Open Questions](015-open-questions.md) | Deferred items and future work. | + +## 9. How to read this specification + +Each file is a self-contained specification of one subject. Cross-references name the target file +directly (for example, "see [002 — Traits](002-traits.md)") so that the set can be read out of +order. + +Specifications state **observable and required behavior** (what a component must do and what +guarantees it must uphold) rather than prescribing internal implementation. Where a component's +internal design is load-bearing for correctness (for example, the SSA construction algorithm in +[005 — Lowering](005-lowering.md)), the required properties are stated normatively and the named +algorithm is given as the reference method. diff --git a/spec/001-architecture.md b/spec/001-architecture.md new file mode 100644 index 0000000..99fe713 --- /dev/null +++ b/spec/001-architecture.md @@ -0,0 +1,94 @@ +# 001 — Architecture + +- **Status:** Draft +- **Normative language:** `MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are to be + interpreted as described in RFC 2119. + +## 1. Overview + +libVCT is decomposed into modules with explicit responsibilities and boundaries. This file defines +the module map, the data flow between stages, the three public interfaces, and the boundaries that +carry the design's weight. + +## 2. Modules + +| Module | Responsibility | Allocation | Visibility | +|---|---|---|---| +| `vct.ir.hir` | HIR AST node types, the trait attachment point, builder/emitter API | arena | **public** | +| `vct.ir.vir` | VIR SSA/CFG types, VIR type system, instruction set | arena | internal | +| `vct.traits` | Attribute / Request / Relationship definitions and queries | arena | **public** | +| `vct.comptime` | Comptime evaluator / constant-folding engine | arena | internal | +| `vct.hir.opt` | HIR pass framework and passes, parallel cascade | arena | internal | +| `vct.lower` | HIR → VIR lowering and SSA construction | arena | internal | +| `vct.vir.opt` | VIR pass framework and heavy passes | arena | internal | +| `vct.backend.c` | VIR → C17 emitter (out-of-SSA, compiler-friendly C) | arena + GC | **public entry** | +| `vct.backend.llvm` | VIR → textual LLVM IR emitter | arena + GC | **public entry** | +| `vct.driver` | C compiler invocation, flag assembly, `.o` emission, LTO, cosmopolitan | GC | **public** | +| `vct.cli` | Thin command-line wrapper | GC | **public** | +| `vct.diag` | Diagnostics, source maps, error reporting | GC | **public** | +| `vct.context` | Context and arena ownership, `Config` | GC + arena | **public** | +| `vct.test.hirbuild` | Fluent HIR test-builder harness | arena | **public** | + +## 3. Data flow + +The default pipeline is: + +``` +frontend --build--> HIR + --hir.opt (parallel, cascading)--> annotated HIR + --lower--> VIR (SSA/CFG) + --vir.opt--> optimized VIR + --backend.c--> C17 source --driver--> .o (default path) + +optimized VIR --backend.llvm--> textual LLVM IR (caller links LLVM) +``` + +Each stage consumes the previous stage's output and is specified independently: +[003 — HIR](003-hir.md), [004 — HIR Optimizer](004-hir-optimizer.md), +[005 — Lowering](005-lowering.md), [006 — VIR](006-vir.md), +[007 — VIR Optimizer](007-vir-optimizer.md), [008 — C Backend](008-c-backend.md), and +[009 — LLVM Backend](009-llvm-backend.md). + +## 4. Public interfaces + +libVCT exposes exactly three interfaces to embedders: + +1. **HIR builder (frontend-facing).** Construct nodes, set types and values, attach + frontend-supplied traits (such as `is_static` and optimization hints), and finish a module. See + [003 — HIR](003-hir.md). +2. **Trait query (VIR-facing).** Read attributes and walk relationships. This is the **only** + channel through which VIR obtains facts. See [002 — Traits](002-traits.md). +3. **Backend (driver-facing).** Consume optimized VIR and emit C or textual LLVM IR. See + [008 — C Backend](008-c-backend.md) and [009 — LLVM Backend](009-llvm-backend.md). + +The native D API is the full-feature surface. A thin `extern(C)` C API (opaque handles for +`Context`/`Module`/`Builder`/`Config`) is the primary v1 public surface and is a shim over the D +API; no logic lives in the shim. See [013 — Build & Packaging](013-build-packaging.md). + +## 5. Load-bearing boundaries + +- **`vct.traits` is the contract.** Both HIR and VIR depend on it. VIR depends on nothing else from + HIR except **node identity**. A change to the trait vocabulary is a cross-cutting change and is + versioned accordingly (see [013 — Build & Packaging](013-build-packaging.md)). +- **`vct.lower` is the SSA construction site.** Phi-nodes, dominance, and the memory model all land + there; it is the most algorithmically dense module. Its correctness invariants are specified in + [005 — Lowering](005-lowering.md) and [006 — VIR](006-vir.md). + +## 6. Memory and ownership model + +- A `Context` owns one or more **arenas**. An arena is the unit of reclamation; the default is one + arena per `Module`, freed wholesale. +- All IR and optimizer objects are **arena-owned and non-GC**. Passes mutate in place under epoch + guards (see [002 — Traits](002-traits.md)). +- **GC is permitted only on cold paths**: the driver, diagnostics, and CLI. +- The arena interface is exposed so embedders can supply their own backing memory; a default bump + allocator ships in-tree. + +Full lifecycle, ownership, and threading rules are specified in +[013 — Build & Packaging](013-build-packaging.md). + +## 7. Threading + +libVCT is thread-safe when each compilation unit has its own `Context` and arenas. There is no +shared mutable global state. Intra-module parallelism is internal and bounded by the `-j` setting. +The determinism guarantee in [000 — Overview](000-overview.md) applies regardless of thread count. diff --git a/spec/002-traits.md b/spec/002-traits.md new file mode 100644 index 0000000..680d569 --- /dev/null +++ b/spec/002-traits.md @@ -0,0 +1,344 @@ +# 002 — Traits + +- **Status:** Draft +- **Normative language:** `MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are to be + interpreted as described in RFC 2119. + +## 1. Purpose + +The **trait model** is the vocabulary libVCT uses to describe facts about IR nodes. It is the +contract between the two optimization halves of the library: HIR discovers and derives traits, and +VIR reads them and transforms accordingly. VIR performs no discovery of its own. Everything VIR +may rely on is an attribute, a request, or a relationship defined here. + +## 2. Scope + +This file specifies: + +- the `Trait` union of an **Attribute**, a set of **Requests**, and a set of **Relationships**; +- the complete v1 attribute set and request catalog; +- relationship flags and relation attributes; +- the query API semantics a trait consumer may assume; +- the trait representation and staleness model; +- request conflict resolution. + +Out of scope: how HIR derives attributes (see [004 — HIR Optimizer](004-hir-optimizer.md)), how +lowering copies traits onto VIR entities (see [005 — Lowering](005-lowering.md)), and how the VIR +optimizer acts on requests (see [007 — VIR Optimizer](007-vir-optimizer.md)). + +Traits are **public**. Frontends embed trait vocabulary; frontends are not required to derive +traits themselves. + +## 3. Definitions + +| Term | Definition | +|---|---| +| **Node** | An arena-owned HIR entity carrying one `Trait`. | +| **Attribute** | A fact about a node (its type, value, mutability, and so on). | +| **Request** | An attribute-derived suggestion from HIR to VIR. Never ad-hoc. | +| **Relationship** | A directed, traceable edge from a node to another node. | +| **Suggested** | An attribute or request set by the frontend at build time. | +| **Derived** | A trait produced by the HIR optimizer. Authoritative. | +| **Epoch** | A monotonically increasing generation counter used to invalidate stale traits. | +| **Consumer** | Any component that reads traits. VIR is the primary consumer. | + +The `Trait` is the union of the three sub-structures: + +```d +struct Trait { + Attribute attr; + Request[] reqs; + Relation[] rels; +} +``` + +A trait attaches to exactly one node. Trait values are per-node and, after HIR optimization, are +complete for every node that lowering consumes. + +## 4. Attribute model + +Attributes are facts about a node. They share **one namespace with two origins**: + +- **Suggested**: set by the frontend at build time through the builder API. +- **Derived**: produced by the HIR optimizer during the cascade. Derived attributes are + authoritative. + +Every attribute carries an `attr_source` marker recording whether its current value is `suggested` +or `derived`. + +### 4.1 Attribute validation rule + +This rule is normative and applies to every suggested attribute: + +1. When the frontend suggests an attribute, HIR **MUST** validate the suggestion against what it + can prove. +2. If the suggested value is wrong or unprovable-in-the-suggested-direction, HIR **MUST** emit a + warning, **MUST** overwrite the attribute with the correct value, and **MUST** mark + `attr_source = derived`. +3. The warning **MUST** be promoted to an error under `-Werror`. A frontend **MAY** opt out for a + specific category with `-Wno-error=`, or for every category with a bare + `-Wno-error`. +4. A correct suggestion **MUST** be preserved and continue to report `attr_source = suggested` + until HIR derives a replacement value. + +HIR **MUST NOT** silently accept a wrong suggestion. A consumer **MUST** treat derived attributes +as authoritative and **MUST NOT** re-derive them. + +### 4.2 v1 attribute set + +| Attribute | Meaning | Typical source | +|---|---|---| +| `ty` | The node's VIR type. | Both | +| `const_value` | Present if and only if the value is known at compile time. | Derived | +| `is_static` | Frontend-declared compile-time value (the frontend-facing `static` concept). Seeds the comptime evaluator. | Suggested | +| `is_comptime` | HIR-proven compile-time value. | Derived | +| `is_constant` | Value is constant; not necessarily compile-time-evaluable. | Derived | +| `is_used` | Node has at least one use. | Derived | +| `is_mutably_used` | Node is mutated through at least one use. | Derived | +| `is_addressed` | Address taken via pointer or reference. | Derived | +| `escapes` | Value escapes its defining scope. | Derived | +| `is_runtime_mutable` | Storage is written at runtime. | Derived | +| `may_change_at_runtime` | The observable value may differ between reads or executions (external state). Distinct from `is_runtime_mutable`. | Derived | +| `is_volatile` | Accesses are volatile: not eliminated, not reordered, and not promoted to SSA. | Suggested | +| `is_atomic` | Accesses are atomic and carry a declared memory ordering. | Suggested | +| `complex` | Frontend-provided; gates the `NoOptimize` request. | Suggested | +| `layout` | Pinned aggregate ABI: field offsets, total size, and alignment. | Suggested | + +`is_runtime_mutable` describes storage writes; `may_change_at_runtime` describes externally +observable value change. They are independent and both **MUST** be tracked. + +`is_static` is trusted by HIR as "this intends to be compile-time", but it is still validated: HIR +**MUST** verify the claim before marking `is_comptime`. + +`is_volatile` and `is_atomic` force a value to memory; lowering and later passes **MUST NOT** +promote such a value to SSA (see [005 — Lowering](005-lowering.md)). The `layout` attribute pins +an aggregate's ABI; the C backend **MUST** emit explicit padding and static assertions for a +layout-pinned aggregate (see [008 — C Backend](008-c-backend.md)). + +## 5. Request model + +Requests are attribute-derived suggestions. A frontend suggesting a request that no attribute +supports is invalid and HIR **MUST** reject it with a diagnostic. + +Each request has a **strength**: + +- **Soft**: a suggestion VIR **MAY** decline. +- **Strong**: violating it is likely a bug. A capable consumer **MUST** honor it or report an + error with a `DenyReason` of `Illegality` or `ContradictsTrait` (see + [007 — VIR Optimizer](007-vir-optimizer.md)). + +Every attempted request produces a `RequestResult`: + +```d +struct RequestResult { + Request req; + bool accepted; + DenyReason reason; + string note; // one-line human-readable explanation +} +``` + +A denial **MUST** carry a `DenyReason` and a one-line `note`. On acceptance, `reason` is +unspecified and `note` **MAY** be empty. + +### 5.1 DenyReason + +| Enumerator | Meaning | +|---|---| +| `CostModel` | Profitable only under a different cost model. | +| `Illegality` | The transform would be incorrect. | +| `AlreadyDone` | No-op; the goal already holds. | +| `Unsupported` | VIR does not implement it yet. | +| `ContradictsTrait` | Conflicts with a stronger fact. | +| `TooLarge` | Exceeds a size budget. | +| `NoTargetSupport` | The backend or target cannot express it. | + +## 6. Request catalog + +The v1 catalog contains **42** requests. Requests marked **Strong** are noted; all others are +**Soft**. + +### 6.1 Inlining and call edges (10) + +| Request | Strength | Meaning | +|---|---|---| +| `Inline` | Soft | Callee is a candidate for inlining. | +| `AlwaysInline` | **Strong** | Callee must be inlined. | +| `NoInline` | **Strong** | Callee must not be inlined. | +| `TailCall` | Soft | Prefer tail-call formation. | +| `MustTail` | **Strong** | Tail-call formation is required. | +| `NoTail` | **Strong** | Tail-call formation must not occur. | +| `Devirtualize` | Soft | Call target is expected to resolve to one target. | +| `ColdCall` | Soft | Call is unlikely; optimize for size. | +| `LikelyCall` | Soft | Call is likely executed. | +| `UnlikelyCall` | Soft | Call is unlikely executed. | + +### 6.2 Loops (12) + +| Request | Strength | Meaning | +|---|---|---| +| `Vectorize` | Soft | Vectorize the loop. | +| `Unroll(factor)` | Soft | Unroll by `factor`. | +| `Interleave(count)` | Soft | Interleave `count` iterations. | +| `Peel(count)` | Soft | Peel `count` iterations. | +| `Distribute` | Soft | Distribute loop bodies. | +| `Fuse` | Soft | Fuse adjacent loops. | +| `Jam` | Soft | Fuse loops by jamming. | +| `Unswitch` | Soft | Unswitch loop-invariant conditions. | +| `Rotate` | Soft | Rotate the loop. | +| `LICM` | Soft | Explicit request to hoist loop-invariant code (an action). | +| `LoopInvariant` | Soft | A proven invariant property VIR may exploit (a property). | +| `MustProgress` | Soft | The loop is guaranteed to make progress. | + +`LICM` and `LoopInvariant` are distinct: one requests an action, the other asserts a property. Both +**MUST** be retained by HIR. + +### 6.3 Memory effects (13) + +| Request | Strength | Meaning | +|---|---|---| +| `NoAlias` | Soft | The value aliases no other relevant value. | +| `Restrict` | Soft | The value may be marked `restrict`. | +| `NonNull` | Soft | The value is not null. | +| `Align(n)` | Soft | The value is aligned to `n`. | +| `ReadOnly` | Soft | Region-level read-only fact. | +| `WriteOnly` | Soft | Region-level write-only fact. | +| `NoRead` | Soft | Primitive fact on an access edge: no read occurs. | +| `NoWrite` | Soft | Primitive fact on an access edge: no write occurs. | +| `NoCapture` | Soft | The value is not captured by a callee. | +| `Dereferenceable(n)` | Soft | At least `n` bytes are dereferenceable. | +| `Constant` | Soft | The value is immutable. | +| `NoUndef` | Soft | Reserved; meaningful only on the LLVM path. | +| `Prefetch` | Soft | Insert a prefetch hint. | + +`NoRead`/`NoWrite` are **primitives on an access edge**; `ReadOnly`/`WriteOnly` are **region-level +facts** derived from them. All four are retained, and their scope difference is part of their +meaning. + +### 6.4 Control flow, assumptions, and optimization control (7) + +| Request | Category | Strength | Meaning | +|---|---|---|---| +| `HotPath` | Control flow | Soft | Block is likely executed. | +| `ColdPath` | Control flow | Soft | Block is unlikely executed. | +| `Unreachable` | Control flow | Soft | Execution never reaches this point. | +| `NoReturn` | Control flow | Soft | The call never returns. | +| `Assume(pred)` | Assumptions | Soft | `pred` holds at this point. | +| `Range(lo, hi)` | Assumptions | Soft | The value lies in `[lo, hi]`. | +| `NoOptimize` | Optimization control | **Strong** | The region must not be rewritten. Requested only when the frontend sets `complex`. | + +HIR **MUST NOT** issue `NoOptimize` unless `complex` is set on the node. The resulting region is an +optimization barrier (see [005 — Lowering](005-lowering.md)). + +## 7. Relationships + +Relationships are directed edges that HIR leaves for the non-constant world. They are traceable +**up and down** to a terminal node. + +### 7.1 Relationship flags + +| Flag | Meaning | +|---|---| +| `is_offspring` | The edge cannot be traced up. | +| `is_ancestor` | The edge cannot be traced down. | +| `is_common` | Branch point; a consumer must choose a direction. | +| `has_siblings` | The node connects down to two or more nodes. | +| `is_apex` | The node participates in a cycle (`a → b → c → d → a`), usually from polymorphism. | + +An apex **SHOULD** normally be ironed out by HIR. A residual apex after the HIR cascade typically +arises from polymorphism, and a consumer that walks into one **MUST** handle the cycle rather than +loop forever. + +### 7.2 Relation attributes + +Each relationship edge carries a small attribute set: + +| Attribute | Meaning | +|---|---| +| `was_changed` | The related value changed since the relationship was recorded. | +| `is_mutable` | The related value is mutable. | +| `is_pointer` | The relationship passes through a pointer. | + +## 8. Query API semantics + +The trait query interface is the **only** channel through which VIR obtains facts. Its guaranteed +semantics are: + +1. A query **MUST** return the trait values valid for the node's **current epoch**. A query against + a node whose trait is stale for the current epoch **MUST** fail rather than return stale data. +2. A query **MUST NOT** trigger analysis or re-derivation. It is a read of already-derived facts. +3. Reading `const_value` when absent **MUST** be reported as "not compile-time known" and **MUST + NOT** be interpreted as zero or any other value. +4. Walking a relationship in the available direction(s) **MUST** terminate at a terminal node or + report that the walk hit an apex cycle. +5. A consumer **MUST NOT** invent, weaken, or generalize a fact absent from the trait. VIR must not + assume a property that no trait states. +6. Query results **MUST** be deterministic: for a fixed module and epoch, the same query returns the + same result regardless of thread count. + +## 9. Representation + +Representation has two parts: an **embedded dense `Trait` struct** on every node is primary, so +common attributes are read without indirection; an **optional sparse side-channel** holds rare or +frontend-extensible attributes and **MAY** be absent. A consumer **MUST** observe the same logical +trait regardless of which part carries it. The dense struct is the reference representation; the +side-channel is an optimization, not a semantic difference. + +## 10. Staleness + +Traits are annotated in place, so a rewrite can invalidate dependent traits. Staleness is resolved +with **epoch/generation counters**: + +1. Each node carries an **epoch**. A trait is valid only for the node's **current epoch**. +2. A rewrite that changes a trait **MUST** advance the affected node's epoch. +3. A rewrite that can affect a related node **MUST** mark that relationship dirty and propagate the + invalidation along the relationship edge. +4. A derived trait computed from a node whose epoch has advanced **MUST** be recomputed before it is + consumed. +5. Epoch counters **MUST** be sufficient to keep parallel HIR workers from reading stale traits. + This is why the model survives parallel passes. + +## 11. Request conflict resolution + +Requests from independent attributes can conflict. Resolution is normative: + +1. **Strength ordering.** `Strong` beats `Soft`. When a Strong and a Soft request conflict, the + Strong request wins and the Soft request is discarded without a diagnostic. +2. **Conflicting Strongs.** When two **Strong** requests conflict, HIR **MUST** emit a diagnostic + and **MUST** reject the frontend suggestion. The derived, authoritative request wins. +3. **Canonical opposing pairs** are `{Inline, AlwaysInline}` against `{NoInline}`, and + `{TailCall, MustTail}` against `{NoTail}`. In the named pairs `Inline`/`NoInline` and + `TailCall`/`NoTail`, the `No*` member is Strong, so those resolve by rule 1; the corresponding + Strong-versus-Strong pairs (`AlwaysInline`/`NoInline`, `MustTail`/`NoTail`) resolve by rule 2. +4. A rejected suggestion **MUST NOT** silently disappear; the diagnostic and a remark **MUST** + record the rejection. + +## 12. Invariants + +- Every node carries exactly one `Trait`. +- Every `RequestResult` denial carries both a `DenyReason` and a non-empty `note`. +- A derived attribute is authoritative; consumers **MUST NOT** re-derive it. +- `const_value` is present if and only if `is_comptime` holds for the node. +- No consumer reads a trait whose epoch is not the node's current epoch. +- The trait vocabulary is versioned independently of the textual IR format (see + [013 — Build & Packaging](013-build-packaging.md)). + +## 13. Example + +For `y = x + 4` where HIR proves `x` is compile-time `4`, the node `x` carries `is_comptime`, +`is_constant`, and `const_value = 4` with `attr_source = derived`; the fold marks downstream nodes +`is_comptime` and issues an `Inline` request when the call is proven small. A frontend that had +suggested `is_comptime = false` on `x` receives a warning and sees HIR overwrite it. The full +walkthrough is normative in [014 — Worked Example](014-worked-example.md). + +## 14. Cross-references + +- [000 — Overview](000-overview.md) +- [001 — Architecture](001-architecture.md) +- [003 — HIR](003-hir.md) +- [004 — HIR Optimizer](004-hir-optimizer.md) +- [005 — Lowering](005-lowering.md) +- [007 — VIR Optimizer](007-vir-optimizer.md) +- [011 — Diagnostics](011-diagnostics.md) +- [013 — Build & Packaging](013-build-packaging.md) +- [014 — Worked Example](014-worked-example.md) diff --git a/spec/003-hir.md b/spec/003-hir.md new file mode 100644 index 0000000..3f25e7a --- /dev/null +++ b/spec/003-hir.md @@ -0,0 +1,255 @@ +# 003 — HIR + +- **Status:** Draft +- **Normative language:** `MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are to be + interpreted as described in RFC 2119. + +## 1. Purpose + +HIR (High-level IR) is the tree-shaped abstract syntax that a frontend builds and libVCT annotates +in place. It is the discovery half of the two-stage IR: HIR establishes node identity, attaches +traits, and performs cheap folds. Lowering consumes annotated HIR and produces VIR (see +[005 — Lowering](005-lowering.md)). + +This file specifies the HIR node model, the node kinds, node identity, the builder/emitter contract, +the `defer` statement, and the textual HIR format. + +## 2. Scope + +In scope: the HIR data model and its observable contract toward frontends, the HIR optimizer, and +lowering. Out of scope: the trait vocabulary itself (see [002 — Traits](002-traits.md)), the +optimizer's pass framework and cascade (see +[004 — HIR Optimizer](004-hir-optimizer.md)), and the HIR-to-VIR mapping (see +[005 — Lowering](005-lowering.md)). + +A frontend that embeds libVCT interacts with HIR through the builder API and the trait query API. +The textual HIR format is a tool surface, not a frontend surface. + +## 3. Node model + +Key properties, all normative: + +1. HIR is **tree-shaped**. Every node except the module root has exactly one parent. A node has an + ordered list of children whose meaning is determined by the node kind. +2. HIR is **arena-owned**. Nodes are allocated from the module's arena and are never individually + freed. An arena is reclaimed wholesale (see [001 — Architecture](001-architecture.md) and + [013 — Build & Packaging](013-build-packaging.md)). +3. HIR is **annotated in place**. Every node carries one embedded `Trait` (see + [002 — Traits](002-traits.md)) that the HIR optimizer mutates as it derives facts. There is no + separate side table for the common case. +4. HIR is **high-level**. It preserves structured control flow (`if`/`while`/`for`/`switch`/`block`) + and expressions; it is not SSA and has no phi-nodes. +5. Nodes are mutable during the HIR optimizer's cascade and become read-only to a consumer once + traits are epoch-valid. A frontend **MUST NOT** mutate a module after finishing it. + +Every node exposes at least: + +```d +struct HirNode { + NodeId id; // stable identity, unique within the module + NodeKind kind; + Span span; // optional frontend source span + Trait trait; // attribute + requests + relationships + NodeId[] children; +} +``` + +`span` **MAY** be absent. Absence **MUST** degrade diagnostics gracefully and **MUST NOT** affect +optimization correctness. + +## 4. Node kinds + +The following is the **reference node set** for v1. A frontend **MUST** build modules only from +these kinds. Additional kinds are reserved and require a specification change. A frontend **MAY** +use any subset of the set. + +### 4.1 Structural kinds + +| Kind | Description | Children / payload | +|---|---|---| +| `Module` | Root of one translation unit. | functions, globals | +| `Function` | A function definition or declaration. | params, body `Block` | +| `Param` | A function parameter. | none | +| `Block` | An ordered statement scope. | statements | + +### 4.2 Declaration kinds + +| Kind | Description | Children / payload | +|---|---|---| +| `GlobalVar` | A module-level storage object. | optional init expression | +| `LocalVar` | A local binding with optional initializer. | optional init expression | + +A declaration **MUST** carry a `ty` attribute. A declaration whose initializer is compile-time +known **MUST** be eligible for `is_comptime`/`const_value` annotation by the optimizer. + +### 4.3 Statement kinds + +| Kind | Description | Children / payload | +|---|---|---| +| `ExprStmt` | Evaluate an expression for effect. | expression | +| `Return` | Return from the enclosing function. | optional value | +| `If` | Two-way branch. | condition, then-block, optional else-block | +| `While` | Pre-test loop. | condition, body | +| `For` | C-style loop. | optional init, optional condition, optional step, body | +| `Switch` | Multi-way branch. | subject, cases | +| `Case` | One switch arm. | optional value, body | +| `Break` | Exit the innermost loop or switch. | none | +| `Continue` | Continue the innermost loop. | none | +| `Defer` | Scope-exit action; see the `defer` section below. | deferred expression or statement | + +### 4.4 Expression kinds + +| Kind | Description | Children / payload | +|---|---|---| +| `Literal` | Integer, float, string, boolean, or null constant. | literal payload | +| `NameRef` | Reference to a declared name. | resolved target identity | +| `Unary` | Prefix or postfix unary operator. | operand | +| `Binary` | Binary operator. | left, right | +| `Assign` | Assignment to an lvalue. | target, value | +| `Call` | Function or intrinsic call. | callee, arguments | +| `Member` | Aggregate field access. | base, field index | +| `Index` | Pointer or array element access. | base, index | +| `Cast` | Type conversion. | operand, target type | + +Every expression node **MUST** carry a `ty` attribute once HIR has validated it. `Literal` nodes +**MUST** additionally carry `const_value` when their value is representable in the type. + +## 5. Node identity + +1. Every node **MUST** have a `NodeId` that is unique within its `Module`. +2. Identity **MUST** be assigned deterministically from construction order, so that two runs over + the same input produce the same identity assignment. +3. Identity **MUST** remain stable across trait annotation and optimizer rewrites of the same node. +4. A deleted node's identity **MUST NOT** be reused for a different node. +5. VIR depends on HIR for **traits and node identity only** (see + [001 — Architecture](001-architecture.md)). Lowering records the mapping from HIR identity to + VIR entities (see [005 — Lowering](005-lowering.md)). + +Identity is the join key that lets diagnostics, relationships, and the lowering map refer to the +same node across stages. + +## 6. The `defer` statement + +HIR carries `defer` as a **real high-level statement**. Its observable semantics: + +1. A `Defer` node names a body that runs at **every scope exit** of the enclosing `Block`. +2. Scope exits include normal fall-through, `Break`, `Continue`, and `Return`. +3. Deferred bodies within one scope run in **reverse declaration order** (last declared runs first). +4. A backend **MUST NOT** ever observe a `Defer` node. Before VIR is produced, every `Defer` marker + **MUST** be erased with zero runtime residue. The behavior **MUST** be identical to writing the + deferred body textually at each exit point. +5. Expansion is performed by the HIR optimizer when HIR optimization is enabled. If HIR + optimization is disabled (`-O0`, see + [004 — HIR Optimizer](004-hir-optimizer.md)), **lowering itself MUST expand `defer` at every + scope exit** (see [005 — Lowering](005-lowering.md)). +6. Expansion **MUST** be deterministic. When a scope has multiple exits, the emitted bodies **MUST** + appear in a stable, specified order so that output is byte-identical across runs. + +Because expansion is always resolved before VIR, `defer` costs nothing at runtime. + +## 7. Builder and emitter contract + +### 7.1 Frontend-facing builder + +The builder is the frontend's only construction surface. It provides operations to create each +node kind, link children, set a node's type and value, and attach frontend-suggested traits +(including `is_static` and optimization hints). Its contract: + +1. The builder **MUST** assign node identity per the node identity section above. +2. The builder **MUST** allow a frontend to construct a module without supplying derived + attributes. Derived facts are HIR's responsibility, not the frontend's. +3. The builder **MUST** accept a frontend-suggested attribute or request, record it with + `attr_source = suggested`, and defer validation to HIR. +4. `finish(module)` **MUST** validate every suggested trait. Validation follows the rule in + [002 — Traits](002-traits.md): on a wrong or unprovable suggestion HIR emits a warning (an error + under `-Werror`), **overwrites** the value with the correct one, and marks it `derived`. A + correct suggestion is preserved. +5. After `finish`, the builder **MUST** reject further structural mutation of that module by the + frontend. Optimization mutates in place internally, under epoch guards. + +### 7.2 Trait attachment + +The frontend attaches a `Trait` to each node. The trait is incomplete at build time: the frontend +supplies `ty`, `is_static`, and `complex` where it knows them, and the HIR optimizer fills in the +rest. A node lacking a derived attribute is not an error at build time; it becomes one only if +lowering finds the trait incomplete (see [005 — Lowering](005-lowering.md)). + +HIR **MUST NOT** require a frontend to compute `is_used`, `escapes`, `is_addressed`, +`const_value`, or any other derived attribute. + +### 7.3 Textual emitter + +The textual emitter is a distinct, non-authoring surface (see the textual HIR format section). It +**MUST** be able to serialize a finished HIR module and to parse that serialization back into an +equivalent module. + +## 8. Textual HIR format + +HIR has a textual form used **solely for tools and round-trip tests**. It is explicitly **NOT** an +authoring surface: frontends embed the library and build HIR through the builder API. The format is +versioned independently of the library and of the trait vocabulary (see +[013 — Build & Packaging](013-build-packaging.md)). + +The format carries, per node: kind, identity, child references, type, and the node's trait (with +each attribute's value and `attr_source`, its requests and strengths, and its relationships). The +reference round-trip property is: + +``` +print(parse(print(m))) == print(m) +``` + +A conforming implementation **MUST** satisfy this property for any module it can emit. Parsing +**MUST NOT** be required to accept hand-written input beyond what the emitter produces. + +Illustrative fragment (shape only, not a complete grammar): + +``` +fn @main() -> i32 { + %v1: i32 = literal 4 ; attr: const_value=4 is_comptime=true (derived) + %v2: i32 = binary add %v1, 4 ; req: Inline (soft) + defer %v3 + ret %v2 +} +``` + +## 9. Invariants + +- Every non-root node has exactly one parent; the module root has none. +- Every node has a unique `NodeId` within its module, and ids are not reused. +- Every node carries exactly one `Trait`. +- After `finish`, every suggested attribute has been validated and either preserved or overwritten. +- No `Defer` node survives into VIR. +- HIR is not SSA and **MUST NOT** be required to answer SSA questions. +- Frontend-supplied types and values are suggestions, never authoritative facts. + +## 10. Example + +A frontend builds: + +``` +fn main() -> int { + x = 4; + y = x + 4; + z = foo(x, y); + a = sqrt(z); + println(a); +} +``` + +It creates `Function`, `LocalVar`, `Binary`, `Call`, and `Return` nodes, sets each node's `ty`, +attaches `is_static` where known, and calls `finish`. HIR then validates the suggestions, derives +`const_value` for `x`, `y`, and the folded `sqrt` result, and issues an `Inline` request for +`foo` (see [004 — HIR Optimizer](004-hir-optimizer.md)). Content that `defer` would add is expanded +at scope exits, so the emitted VIR contains no defer marker. The full walkthrough is normative in +[014 — Worked Example](014-worked-example.md). + +## 11. Cross-references + +- [000 — Overview](000-overview.md) +- [001 — Architecture](001-architecture.md) +- [002 — Traits](002-traits.md) +- [004 — HIR Optimizer](004-hir-optimizer.md) +- [005 — Lowering](005-lowering.md) +- [011 — Diagnostics](011-diagnostics.md) +- [013 — Build & Packaging](013-build-packaging.md) +- [014 — Worked Example](014-worked-example.md) diff --git a/spec/004-hir-optimizer.md b/spec/004-hir-optimizer.md new file mode 100644 index 0000000..76d58ba --- /dev/null +++ b/spec/004-hir-optimizer.md @@ -0,0 +1,253 @@ +# 004 — HIR Optimizer + +- **Status:** Draft +- **Normative language:** `MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are to be + interpreted as described in RFC 2119. + +## 1. Purpose + +The HIR optimizer is the **discovery and small-fold** stage. It validates frontend-suggested traits, +derives attributes and relationships, evaluates compile-time expressions, runs cheap folds to a +fixpoint, and expands `defer`. Its output is annotated HIR whose traits are complete and +epoch-valid, and which hands VIR a "dictionary" of facts. + +The HIR optimizer **MUST NOT** perform heavy transformations. In particular, it **MUST NOT** inline +code (see the "HIR never inlines" requirement below). + +## 2. Scope + +In scope: the HIR pass framework, triggers, the worklist scheduler and cascade-to-fixpoint, per- +function parallelism and the module-level fixpoint, epoch-based staleness, the comptime evaluator, +`defer` expansion, optimization-level behavior, and default budgets. + +Out of scope: the trait vocabulary (see [002 — Traits](002-traits.md)), the node model and builder +contract (see [003 — HIR](003-hir.md)), and the HIR-to-VIR translation (see +[005 — Lowering](005-lowering.md)). + +## 3. Definitions + +| Term | Definition | +|---|---| +| **Pass** | A unit of discovery or folding over one function's HIR. | +| **Trigger** | A record that a rewrite succeeded and may enable further work. | +| **Worklist** | The deduplicated queue of `(pass, node)` work items. | +| **Cascade** | The chain of rewrites triggered by one fold. | +| **Fixpoint** | The state in which the worklist drains with no new work. | +| **Epoch** | The per-node generation counter that invalidates stale traits. | +| **Step budget** | The bounded amount of work allowed before the scheduler stops. | + +## 4. Pass framework + +A pass is a named unit with the following shape: + +```d +interface HirPass { + string name(); + void run(HirFunction fn, HirContext ctx); +} +``` + +Normative requirements: + +1. A pass **MUST** mutate the annotated AST **in place**. It **MUST NOT** rebuild the tree or + return a replacement. +2. A pass **MUST** be deterministic: for a fixed input and epoch, it produces the same mutations in + the same order regardless of thread count. +3. A pass **MUST** emit a **trigger** for every successful rewrite (see the triggers section). +4. A pass **MUST** respect the step budgets in the budgets section. On exhaustion it **MUST** stop + and leave the tree in a consistent, verifiable state. +5. A pass **MUST NOT** read a trait whose epoch is not the node's current epoch (see the + epoch-based staleness section). + +## 5. Triggers + +Successful rewrites emit triggers, which the scheduler uses to enqueue dependent work: + +| Trigger | Meaning | +|---|---| +| `ConstantUnfolded` | A node's value became compile-time known. | +| `UsesReplaced` | Uses of a node were replaced, potentially making the node or its users constant or dead. | +| `NodeDeleted` | A node was removed; its former users may now be foldable. | +| `TraitChanged` | A trait changed and dependent traits may be stale. | +| `StaticDiscovered` | A frontend `is_static` claim was confirmed or a new compile-time value was found. | + +A rewrite that changes no fact **MUST NOT** emit a trigger. Emitting a trigger for a no-op risks a +non-terminating cascade and is a correctness defect. + +## 6. Worklist scheduler and cascade + +### 6.1 Scheduler + +1. The scheduler collects `(pass, node)` pairs into a **worklist**, **deduplicates** them, and runs + until the worklist drains to a **fixpoint**: a state in which no scheduled work produces a new + trigger. +2. Order of processing within one function **MUST** be deterministic, so cascade order is + reproducible. +3. A **per-function step budget** guarantees termination even if triggers keep firing. When the + budget is exhausted the scheduler **MUST** stop scheduling further passes for that function and + report the exhaustion. +4. The scheduler **MUST NOT** schedule work for a node whose governing function is not yet ready in + the current phase. + +### 6.2 Canonical cascade + +The reference cascade for a fold: + +``` +4 * 16 + -> const 64 + -> mark is_comptime / is_constant / const_value = 64 + -> prove not addressed / not runtime-mutable + -> replace all uses with literal 64 + -> node dead -> delete + -> downstream nodes now constant -> enqueue +``` + +Each hop emits a trigger. The worklist drains until nothing new is provable. The cascade is the +mechanism by which one fold can make a whole subtree constant; it **MUST** reach the same fixpoint +regardless of the order in which independent items are processed. + +## 7. Parallelism and determinism + +1. HIR passes **MUST** parallelize **per function**. Independent function trees may be processed + concurrently. +2. Within one function, passes **MUST** run **sequentially**, so cascade order is deterministic. +3. Cross-function effects, such as cross-module constant propagation and requests that depend on + callees, **MUST** use a **module-level fixpoint**: + - run per-function passes in parallel; + - run a deterministic module phase; + - re-enqueue only the functions whose inputs changed. +4. The module phase **MUST** be deterministic: it processes functions in a stable order and folds + their results in a stable order. +5. The determinism guarantee of [000 — Overview](000-overview.md) binds the HIR optimizer: for a + fixed `Config` and input, output **MUST** be byte-identical independent of thread count and of + `-j`. + +## 8. Epoch-based staleness + +1. Each node carries an epoch counter. A trait is valid only for the node's **current epoch**. +2. A rewrite that changes a trait **MUST** advance the affected node's epoch. +3. A rewrite that can affect a related node **MUST** mark the relationship dirty and propagate + invalidation along the relationship edge. +4. A pass in a parallel worker **MUST NOT** consume a trait that became stale after the worker read + it. Epoch checks **MUST** detect this. +5. A stale derived trait **MUST** be recomputed before it is consumed. Recomputing it **MUST** be + part of the cascade, not a silent fallback to a default. + +This model is what makes parallel HIR passes safe without a global lock. + +## 9. Comptime evaluator + +The comptime evaluator is an interpreter over HIR subgraphs. It computes compile-time values and +marks them on the nodes. + +### 9.1 Eligibility + +An HIR subgraph is eligible for comptime evaluation **only if** all of the following hold: + +1. It has **no side effects**. +2. It involves **primitives only** (no aggregate or opaque operations that cannot be evaluated + element-wise). +3. It performs **no global mutation**. +4. It performs **no I/O**. +5. It stays within the loop and recursion budgets (see the budgets section below). + +If any condition fails, the evaluator **MUST** bail out cleanly and defer the computation to VIR. +A bailout **MUST NOT** mark any partial result as compile-time known. + +### 9.2 Outputs + +On successful evaluation, the evaluator writes, in order: + +1. `const_value`: the computed value. +2. `is_comptime` and `is_constant`: the proven properties. + +`const_value` **MUST** be present if and only if `is_comptime` holds. + +### 9.3 Seeding + +The frontend `is_static` attribute (the frontend-facing `static` concept) **seeds** the evaluator: +HIR treats `is_static` as the claim "this is compile-time" and **MUST** validate the claim before +setting `is_comptime`. A false `is_static` claim is handled by the attribute validation rule of +[002 — Traits](002-traits.md): warning, overwrite, mark derived. + +## 10. `defer` expansion + +1. When HIR optimization is enabled, the optimizer **MUST** expand every `defer` body at every + scope-exit path and erase the `defer` marker. This leaves zero runtime residue. +2. Expansion **MUST** obey the ordering and determinism rules of [003 — HIR](003-hir.md). +3. When HIR optimization is disabled (`-O0`), the optimizer does not expand `defer`; lowering + performs the same expansion. Either way, no `Defer` node reaches VIR. + +## 11. Optimization-level behavior + +| Level | HIR behavior | VIR | +|---|---|---| +| `-O0` | Traits validated only. | off | +| `-O1` | Builtin folds and discovery. | off | +| `-O2` | Full cascade plus comptime evaluation. | **on** | +| `-O3` | `-O2` plus special AST transforms (loop-unfold/vectorize, then constant-unfold, then comptime). | on | +| `-Ofast` | `-O3` plus `-march=native` and fast-math. | on | +| `-Oz` | Size-tuned discovery and folds. | on | + +Compiler-friendly C is emitted at **all** optimization levels. `-O3` only adds more aggressive AST +transforms; it does not change the trait contract. + +## 12. HIR never inlines + +HIR **MUST NOT** inline code. It proves that a call is small and eligible, then issues an `Inline` +request (see [002 — Traits](002-traits.md)). Inlining is a VIR/IPA transform (see +[007 — VIR Optimizer](007-vir-optimizer.md)). This separation keeps HIR's work cheap and +parallelizable. + +## 13. Default budgets + +All budgets are configurable through the API and CLI. The defaults are: + +| Budget | Default | +|---|---| +| `comptime.max_iterations` | `4096` | +| `comptime.max_recursion_depth` | `256` | +| `comptime.max_steps` | `1000000` | +| `comptime.max_aggregate_elements` | `65536` | +| `hir.max_rewrites_per_function` | `100000` | +| `hir.max_rewrites_per_module` | `1000000` | + +A budget exhaustion is a normal, reported outcome; it **MUST NOT** be treated as a crash and +**MUST NOT** produce partially annotated output that lowering would accept as complete. + +## 14. Invariants + +- The worklist drains to a fixpoint or stops at an explicit budget: the scheduler never loops + forever. +- Every successful rewrite emits exactly the triggers needed to re-enqueue affected work. +- A no-op rewrite emits no trigger. +- Every pass validates the epochs of traits it reads. +- After HIR optimization, every trait consumed by lowering is complete and epoch-valid. +- HIR performs no inlining. +- No `Defer` node survives HIR optimization when HIR optimization is enabled. +- Output is deterministic for a fixed `Config` and input, independent of thread count. + +## 15. Example + +For `y = x + 4` with `x = 4`: + +1. The comptime evaluator folds `x` to `const_value = 4`, marking `is_comptime` and `is_constant`. +2. `ConstantUnfolded` triggers the fold of `y` to `8`. +3. `foo(x, y)` is proven small; HIR issues a soft `Inline` request instead of inlining it. +4. The `sqrt` call folds to a floating-point constant. +5. The worklist drains; the module reaches a fixpoint with `println` left as the only real call. + +The normative end-to-end walkthrough is [014 — Worked Example](014-worked-example.md). + +## 16. Cross-references + +- [000 — Overview](000-overview.md) +- [001 — Architecture](001-architecture.md) +- [002 — Traits](002-traits.md) +- [003 — HIR](003-hir.md) +- [005 — Lowering](005-lowering.md) +- [007 — VIR Optimizer](007-vir-optimizer.md) +- [011 — Diagnostics](011-diagnostics.md) +- [013 — Build & Packaging](013-build-packaging.md) +- [014 — Worked Example](014-worked-example.md) diff --git a/spec/005-lowering.md b/spec/005-lowering.md new file mode 100644 index 0000000..bd18e4b --- /dev/null +++ b/spec/005-lowering.md @@ -0,0 +1,304 @@ +# 005 — Lowering + +- **Status:** Draft +- **Normative language:** `MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are to be + interpreted as described in RFC 2119. + +## 1. Purpose + +Lowering translates annotated HIR into VIR. It is the bridge between the frontend's tree and the +SSA/CFG form that the optimizer and both backends consume. It turns structured control flow into +basic blocks and terminators, turns variables into SSA values or stack slots according to their +traits, and records the correspondence between every source node and the VIR entities it produced. + +Lowering is a **pure translation**. It discovers nothing and proves nothing. Every type, value, +mutability fact, aliasing property, and call hint it needs is already present as a trait that the +HIR optimizer left on the tree (see [004 — HIR Optimizer](004-hir-optimizer.md)). When a required +fact is absent or contradicts another, lowering does not guess: it reports an internal compiler +error. + +## 2. Scope + +This file specifies the lowering contract, the translation of structured HIR constructs into a +control-flow graph, SSA construction, the trait-driven hybrid memory model, the `Phi` instruction +shape, the `LoweringMap`, region optimization barriers, the `defer` fallback, the VIR types that +lowering emits, and the invariants the output must satisfy. + +It does not specify the HIR node model (see [003 — HIR](003-hir.md)), the trait vocabulary (see +[002 — Traits](002-traits.md)), VIR instruction semantics (see [006 — VIR](006-vir.md)), or the +optimizer passes (see [007 — VIR Optimizer](007-vir-optimizer.md)). + +## 3. Definitions and model + +| Term | Meaning | +|---|---| +| **Annotated HIR** | An HIR tree on which every relevant node carries a complete trait set. | +| **Epoch-valid trait** | A trait whose recorded epoch equals the node's current epoch. Stale traits are not readable. A trait is invalidated when a rewrite makes it stale (see [002 — Traits](002-traits.md)). | +| **Lowering unit** | One HIR function. Lowering runs independently per function. | +| **Basic block** | A maximal straight-line sequence of VIR instructions with one entry and one terminator. | +| **Sealed block** | A block for which all predecessors are known. A block with a pending back-edge is unsealed. | +| **Current-definition map** | Per-block mapping from a source variable to the SSA value most recently written to it on the path into that point. | +| **LoweringMap** | The bidirectional correspondence between HIR nodes and the VIR entities produced from them. | + +Lowering consumes HIR and produces VIR. It reads only the trait contract and HIR node identity +(see [001 — Architecture](001-architecture.md)). It MUST NOT read HIR structure to recover a fact +that the trait model already carries. + +## 4. Normative requirements + +### 4.1 Contract + +- **L-1.** The input MUST be annotated HIR whose traits are complete and epoch-valid. +- **L-2.** The output MUST be a VIR module in which every function is well-formed SSA (see + [006 — VIR](006-vir.md)). +- **L-3.** Lowering MUST NOT perform discovery. It MUST NOT infer types, constant values, + mutability, aliasing, or call effects that are not stated by traits. +- **L-4.** Lowering MUST validate trait completeness before it translates a function. A required + trait that is missing, stale, or self-contradictory MUST raise an internal-compiler-error + diagnostic (see [011 — Diagnostics](011-diagnostics.md)). Lowering MUST NOT substitute a default, + a guess, or a conservative fallback for a missing trait. +- **L-5.** Lowering MUST be a pure function of the annotated HIR and the `Config`. The same input + and `Config` MUST produce byte-identical VIR regardless of thread count, per the determinism + guarantee in [000 — Overview](000-overview.md). +- **L-6.** Lowering MUST be parallelizable per function. Two functions MUST NOT share mutable + lowering state. The only cross-function state is read-only, namely the module's trait graph and + the `LoweringMap` for already-lowered functions where a relationship endpoint crosses functions. +- **L-7.** Emission order within a function MUST be deterministic. Where a construct admits + multiple equivalent orderings (for example, the order of predecessor operands in a `Phi`), the + order MUST be fixed by a documented rule, not by hash iteration. + +### 4.2 Structured control flow to CFG + +Lowering is a streaming recursive interpreter over the HIR tree. Structured constructs become +blocks connected by terminators. Lowering MUST emit blocks and terminators, not a flat instruction +stream with implicit jumps. + +| HIR construct | Required VIR emission | +|---|---| +| Statement sequence, `block` | Instructions appended to the current block in source order. | +| `if (c) T else E` | `condbr c, %then, %else`; a `%then` block, an `%else` block, and a `%join` block. Values live past the construct receive a `Phi` in `%join`. | +| `while (c) B` | A `%header` block that evaluates `c`, ending in `condbr c, %body, %exit`; a `%body` block; a `%latch` block ending in `br %header`; an `%exit` block. | +| `for (init; c; step) B` | An `%init` block; a `%header` block for `c`; a `%body` block; a `%step` block that runs `step` then branches to `%header`; an `%exit` block. | +| `switch (v)` | A `switch` terminator whose case targets and default target are the arm entry blocks. | +| `break` | `br` to the exit block of the innermost enclosing loop or `switch`. | +| `continue` | `br` to the latch or step block of the innermost enclosing loop. | +| `return e` | `ret` of the lowered value of `e`; `ret void` when the function returns void. | +| End of a void function | `ret void`. | +| End of a non-void function without an explicit return | `unreachable`. | + +- **L-8.** Every basic block MUST be terminated by exactly one terminator. +- **L-9.** A block that is not reachable by fallthrough MAY remain unreachable from the entry block + as long as it is well-formed; unreachable blocks MAY persist until dead-code elimination removes + them (see [006 — VIR](006-vir.md), WF-11). +- **L-10.** The entry block MUST have no predecessors. + +### 4.3 Expression translation + +Expressions lower to temporaries and instructions. + +- **L-11.** Each HIR expression that produces a runtime value MUST lower to exactly one VIR value: + either an existing constant or a newly emitted instruction result. Aggregate-producing + expressions MAY lower to a memory location instead. +- **L-12.** Operands MUST be lowered before the instruction that consumes them, in source order + (left to right for binary operators, receiver before arguments for calls). +- **L-13.** A HIR constant that is comptime must lower to a VIR constant operand and MUST NOT + produce a runtime instruction. +- **L-14.** A call MUST attach the call-site traits (`Inline`, `AlwaysInline`, `NoInline`, + `TailCall`, `MustTail`, `NoTail`, `ColdCall`, `LikelyCall`, `UnlikelyCall`, and the memory-effect + requests) to the emitted call entity. Lowering MUST NOT decide whether to inline; that decision + belongs to the VIR optimizer (see [007 — VIR Optimizer](007-vir-optimizer.md)). +- **L-15.** Reading a variable MUST follow the representation chosen in §4.5: a pure SSA read from + the current-definition map, or an explicit `load` from its stack slot. + +### 4.4 SSA construction + +SSA MUST be constructed with the sealed-block algorithm of Braun, Buchwald, Hack, Leißa, Mallon, +and Zwinkau, "Simple and Efficient Construction of Static Single Assignment Form" (CC 2013). That +algorithm is the reference method; the requirements below are the observable properties it +guarantees. + +- **L-16.** Construction MUST be a single pass interleaved with the streaming translation. Lowering + MUST NOT run a separate dominance-frontier computation followed by iterated phi placement. +- **L-17.** Lowering MUST maintain a current-definition map per block. A write records the new + value for the variable in the current block. A read consults the current block first. +- **L-18.** On a read for which the current block has no definition, lowering MUST: + 1. if the block is sealed, recurse to its predecessors, place a `Phi` at the nearest join, and + return the phi result; + 2. if the block is not sealed, create an **incomplete phi** placeholder for the variable in that + block and return it; the placeholder is resolved when the block is sealed. +- **L-19.** Lowering MUST apply trivial-phi elimination: a phi whose operands all resolve to the same + value MUST be replaced by that value, and a phi that references only itself MUST be removed. This + keeps construction linear in practice. +- **L-20.** Sealing a block MUST fill each incomplete phi with one operand per predecessor, each + operand obtained by reading the variable in that predecessor, and MUST then recursively seal + successors whose predecessors are now all known. +- **L-21.** A loop header MUST NOT be sealed until its back-edge has been emitted. Sealing early is + the one mistake that makes the algorithm produce unsound phis. +- **L-22.** Phi operand order MUST follow a deterministic predecessor order (for example, ascending + block identifier), so that output is reproducible. + +### 4.5 Trait-driven hybrid memory model + +Lowering MUST choose each value's representation from its traits alone. + +| Condition | Representation | +|---|---| +| `ty` is scalar AND NOT `is_addressed` AND NOT `escapes` AND NOT `is_runtime_mutable` | Pure SSA value held in a register. | +| `is_addressed` OR `escapes` OR `is_runtime_mutable` OR the type is an aggregate | Stack slot: an `alloca` plus explicit `load`/`store`. | +| `is_volatile` or `is_atomic` holds | Forced to memory, never promoted to SSA. | + +- **L-23.** A scalar that is not addressed, does not escape, and is not runtime-mutable MUST be + kept as a pure SSA value. +- **L-24.** A value that is addressed, escapes, is runtime-mutable, or has aggregate type MUST be + given a stack slot with explicit `load`/`store`. +- **L-25.** A value carrying `is_volatile` or `is_atomic` MUST be forced to memory and MUST NOT be + promoted to SSA at any point, including by later optimizations. +- **L-26.** Lowering's choice is not final for values that carry neither `is_volatile` nor + `is_atomic`. `mem2reg` and `SROA` MAY promote a memory value back to SSA when traits confirm + that the promotion is safe (see [007 — VIR Optimizer](007-vir-optimizer.md)). Such a promotion + MUST preserve the program's observable behavior. +- **L-27.** `is_runtime_mutable` means the storage is written at runtime. `may_change_at_runtime` + means the observable value may differ between reads because of external state. Lowering MUST + consult `is_runtime_mutable` for the representation decision and MUST NOT conflate the two. + +### 4.6 Phi + +VIR has an explicit, LLVM-style `Phi` instruction. + +- **L-28.** A `Phi` MUST record one operand per CFG predecessor of its block. Each operand MUST be a + `(value, predecessor-block)` pair. +- **L-29.** The number of operands MUST equal the number of predecessors, and each named + predecessor MUST be an actual predecessor of the phi's block. + +Example: + +``` +%loop.header: + %i = phi i32 [ 0, %entry ], [ %i.next, %loop.latch ] +``` + +### 4.7 LoweringMap and trait transfer + +- **L-30.** Lowering MUST build a `LoweringMap` from each HIR node to the one or more VIR entities + produced from it, and from each VIR entity back to its source HIR node. +- **L-31.** Every VIR entity MUST carry its own `Trait` instance, populated from the source HIR + node's trait. Attributes and requests MUST be transferred; the VIR optimizer reads them through + the trait contract and does not consult HIR. +- **L-32.** Every HIR relationship endpoint MUST be rewritten to the corresponding VIR entity via + the `LoweringMap`, so that VIR relationship walks terminate in VIR entities without touching HIR. +- **L-33.** The `LoweringMap` MUST be total on HIR nodes that produce a VIR entity and MUST be + stable for the lifetime of the module. The diagnostics service uses it to resolve VIR locations + back to source (see [011 — Diagnostics](011-diagnostics.md)). +- **L-34.** After lowering, VIR MUST depend on the trait contract and HIR node identity only. It + MUST NOT depend on HIR structure. + +### 4.8 Region optimization barriers + +- **L-35.** A HIR node marked `complex` or carrying the `NoOptimize` request MUST be lowered inside + a **region-level** optimization barrier. The barrier covers the node and the region it dominates + as defined by the source construct, and is recorded on the VIR entities in that region. +- **L-36.** VIR passes MUST NOT rewrite, reorder, delete, or hoist across a region barrier, and + MUST treat the region as an opaque, side-effecting operation for alias and memory-effect + analyses. Backends MAY emit a compiler fence for it (see [008 — C Backend](008-c-backend.md)). + +### 4.9 Defer fallback + +- **L-37.** When HIR optimization is disabled (`-O0`), HIR does not expand `defer`, and + lowering MUST expand each deferred call at every scope-exit path and then erase the `defer` + marker. Expansion order MUST follow the reverse order of registration at each exit path. +- **L-38.** When HIR optimization is enabled, HIR has already expanded `defer`; lowering MUST NOT + expand it a second time. A residual `defer` marker reaching lowering when HIR optimization is on + MUST be treated as an internal inconsistency under L-4. + +### 4.10 VIR type system overview + +The `ty` attribute already holds the node's VIR type (see [002 — Traits](002-traits.md)). It +is authoritative: lowering MUST read it and MUST NOT recompute a type from HIR structure. + +| VIR type | Meaning | +|---|---| +| `void` | No value; the result type of effect-only calls and functions. | +| `iN` | An `N`-bit integer. `N` MUST be at least 1. `i1` is the boolean type. | +| `f32`, `f64` | IEEE-754 binary32 and binary64. Implementations MAY support `f16`, `f80`, `f128`. | +| `ptr` | A pointer to `T`. The LLVM backend maps this to opaque `ptr` (see [009 — LLVM Backend](009-llvm-backend.md)). | +| `[N x T]` | A fixed-size array of `N` elements of `T`. | +| `{T1, T2, ...}` | A struct with the named field types. | +| `union {T1, ...}` | A union of the named member types. | +| `fn(T1, ...) -> R` | A function type with parameter types and return type `R`. | + +- **L-39.** Aggregate values are first-class: they MAY be loaded, stored, passed, returned, and + named by a `Phi`. +- **L-40.** Aggregate indexing MUST use field indices, not byte offsets. The C backend leaves + aggregate layout to the C compiler except where the `layout` attribute pins an ABI + (see [008 — C Backend](008-c-backend.md)). + +## 5. Invariants + +The VIR module released by lowering MUST satisfy all of the following. They are checked by the +verifier after lowering and, in debug builds, after every subsequent pass. + +1. **Single definition.** Every SSA value has exactly one defining instruction. +2. **Def dominates use.** Every use is dominated by its definition. A phi operand used in + predecessor `P` MUST either dominate `P` or be the phi result itself (the loop case). +3. **Terminated blocks.** Every basic block ends in exactly one terminator. +4. **Phi arity.** Every `Phi` has one operand per predecessor, and each named predecessor is real. +5. **CFG consistency.** Every successor edge from a block has a matching predecessor edge on the + target, and the entry block has no predecessors. +6. **Type consistency.** Every instruction's operands and result match the VIR type system. +7. **LoweringMap total.** Every value-producing HIR node maps to at least one VIR entity, and every + VIR entity maps back to a source HIR node. +8. **Determinism.** Re-lowering the same input produces byte-identical VIR. + +## 6. Example + +Source: + +``` +while (i < n) { + s = s + a[i]; + i = i + 1; +} +``` + +Lowered shape (values elided): + +``` +entry: + br %loop.header + +loop.header: + %i = phi i32 [ 0, %entry ], [ %i.next, %loop.latch ] + %s = phi i32 [ 0, %entry ], [ %s.next, %loop.latch ] + %cmp = icmp slt i32 %i, %n + condbr %cmp, %loop.body, %loop.exit + +loop.body: + %addr = gep [4 x i32] %a, 0, %i + %elem = load i32 %addr + %s.mid = add i32 %s, %elem + br %loop.latch + +loop.latch: + %s.next = add i32 %s.mid, 0 + %i.next = add i32 %i, 1 + br %loop.header + +loop.exit: + ret void +``` + +`%i` and `%s` are scalars that are neither addressed, escaping, nor runtime-mutable, so they are +pure SSA values; their merge points in `loop.header` are phis. If `s` were addressed by a pointer, +lowering would instead allocate a slot for it and emit `load`/`store` around each access. + +## 7. Cross-references + +- [000 — Overview](000-overview.md): determinism guarantee and conformance. +- [001 — Architecture](001-architecture.md): module boundaries; lowering is the SSA construction site. +- [002 — Traits](002-traits.md): attribute, request, and relationship vocabulary; epochs. +- [003 — HIR](003-hir.md): the tree lowering consumes, including `defer`. +- [004 — HIR Optimizer](004-hir-optimizer.md): who produces the annotated HIR. +- [006 — VIR](006-vir.md): the target IR, its instructions, and its invariants. +- [007 — VIR Optimizer](007-vir-optimizer.md): who consumes lowered VIR. +- [008 — C Backend](008-c-backend.md), [009 — LLVM Backend](009-llvm-backend.md): consumers of VIR. +- [011 — Diagnostics](011-diagnostics.md): internal-compiler-error policy and source mapping. diff --git a/spec/006-vir.md b/spec/006-vir.md new file mode 100644 index 0000000..5757c17 --- /dev/null +++ b/spec/006-vir.md @@ -0,0 +1,298 @@ +# 006 — VIR + +- **Status:** Draft +- **Normative language:** `MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are to be + interpreted as described in RFC 2119. + +## 1. Purpose + +VIR (Vox IR) is the low-level instruction IR of libVCT. It is an SSA form over a control-flow +graph. Every optimization above the backends operates on VIR, and both the C backend and the LLVM +backend consume VIR. Lowering produces it (see [005 — Lowering](005-lowering.md)); the VIR +optimizer transforms it (see [007 — VIR Optimizer](007-vir-optimizer.md)). + +VIR is internal and is intended to be machine-generated. A textual form exists for tools, +dumps, and round-trip tests, but it is not a supported authoring surface: programs are built +through the HIR builder or as fixtures for tests (see [003 — HIR](003-hir.md), +[012 — Testing](012-testing.md)). + +## 2. Scope + +This file specifies the VIR SSA/CFG model, the VIR type system, the instruction set, and the +well-formedness invariants every VIR module must satisfy. It does not specify how VIR is produced +(see [005 — Lowering](005-lowering.md)), how it is optimized (see +[007 — VIR Optimizer](007-vir-optimizer.md)), or how it is emitted (see +[008 — C Backend](008-c-backend.md), [009 — LLVM Backend](009-llvm-backend.md)). + +## 3. SSA/CFG model + +A **VIR module** contains functions. A **function** has a name, a signature, and a non-empty list +of basic blocks; one block is the **entry block**. Every function is either a definition or a +declaration. A declaration has a signature and no blocks. + +A **basic block** has a label, unique within its function, and an ordered list of instructions +ending in exactly one **terminator**. The terminator is the last instruction of the block. + +An **SSA value** has a type and is defined by exactly one instruction. Values are referenced by +name. Following the LLVM convention, SSA names written `%name` are numbered values; unnumbered +temporaries are assigned deterministic numbers at print time. + +The **CFG** is the directed graph whose nodes are blocks and whose edges are taken from +terminators. Edge `b -> c` means c is a **successor** of b and b is a **predecessor** of c. The +entry block has no predecessors. + +**Domination.** Block `a` dominates block `b` if every path from the entry block to `b` passes +through `a`. Domination is reflexive; `a` strictly dominates `b` when `a` and `b` differ. An +instruction `d` dominates a use `u` when the block containing `d` dominates the block containing +`u`, or `d` and `u` are in the same block and `d` appears before `u`. + +**SSA form.** Every value has exactly one definition, and every use is dominated by its definition. +The only relaxation is the `Phi` instruction, whose operands are conceptually evaluated at the end +of the corresponding predecessor. + +**Def-use.** Each use of a value is a `(user-instruction, operand-index)` pair. The def-use graph +is the union of those pairs over all instructions. Passes that rewrite an operand MUST update both +sides of the edge. + +## 4. Type system + +VIR types are strictly typed. There are no implicit conversions; a conversion is an explicit cast +instruction. + +| Type | Meaning | +|---|---| +| `void` | No value. The result of effect-only calls and functions. | +| `iN` | An `N`-bit two's-complement integer, `N >= 1`. `i1` is boolean. | +| `f32`, `f64` | IEEE-754 binary32 and binary64. `f16`, `f80`, `f128` MAY be supported. | +| `ptr` | A pointer to `T`. | +| `[N x T]` | An array of `N` elements of type `T`. | +| `{T1, ..., Tn}` | A struct with the listed field types in order. A packed variant MAY exist. | +| `union {T1, ...}` | A union of the listed member types. | +| `fn(T1, ...) -> R` | A function type: parameter types and return type `R`. A variadic variant MAY exist. | + +- **V-1.** `iN` MUST accept any width `N >= 1`; the implementation MUST NOT restrict integers to a + fixed set of power-of-two widths. +- **V-2.** `void` MUST NOT be used as the type of a value. It is valid only as a function result + type and as the result type of a call that produces no value. +- **V-3.** Aggregate types (`[N x T]`, struct, union) are first-class: they MAY be operands and + results of `load`, `store`, `phi`, `select`, and calls. +- **V-4.** Pointer element types are used for typing `load`, `store`, and indexed `gep`. They are + a convenience for the C backend; the LLVM backend emits opaque `ptr` and relies on the + instructions' operand types for correctness (see [009 — LLVM Backend](009-llvm-backend.md)). + +## 5. Instruction set + +An instruction has an opcode, zero or more operands, an optional result type and value, and +optional flags. The catalog below is normative for opcode meaning; the exact textual spelling is +specified by the dump format and is not an authoring interface. + +### 5.1 Terminators + +| Opcode | Syntax (conceptual) | Meaning | +|---|---|---| +| `br` | `br %target` | Unconditional edge. | +| `condbr` | `condbr %cond, %iftrue, %iffalse` | Conditional edge on an `i1` value. | +| `switch` | `switch %value, %default [ case ... ]` | Multi-way edge on an integer value. | +| `ret` | `ret void` / `ret %value` | Return from the function. | +| `unreachable` | `unreachable` | Marks a point that cannot be reached at runtime. | + +- **V-5.** Every block MUST end in exactly one terminator. +- **V-6.** A `condbr` condition MUST have type `i1`. +- **V-7.** A `switch` value MUST have integer type; each case constant MUST fit that type. +- **V-8.** A `ret` operand MUST match the function's declared return type, and a void function + MUST use `ret void`. + +### 5.2 Memory instructions + +| Opcode | Syntax (conceptual) | Meaning | +|---|---|---| +| `alloca` | `%p = alloca T [, align A]` | Allocate `T` on the stack; result type is `ptr`. | +| `load` | `%v = load T, ptr %p [, align A] [, volatile]` | Read a `T` from memory. | +| `store` | `store T %v, ptr %p [, align A] [, volatile]` | Write a `T` to memory. | +| `gep` | `%p = gep T, ptr %base, idx...` | Compute a pointer to an indexed element or field. | +| `memcpy` | `call ...` | Copy memory; also allowed as a dedicated opcode. | +| `memset` | `call ...` | Fill memory; also allowed as a dedicated opcode. | +| `memmove` | `call ...` | Overlapping copy; also allowed as a dedicated opcode. | + +- **V-9.** A `load` of type `T` MUST take a pointer of type `ptr`; a `store` MUST agree on `T`. +- **V-10.** A `gep` MUST index into aggregates by **field index**, not by byte offset. The first + index traverses the pointer; subsequent indices traverse arrays and struct fields. +- **V-11.** A `volatile` load or store MUST NOT be reordered with respect to any other volatile or + atomic access, and MUST NOT be eliminated even when its result is unused. + +### 5.3 Arithmetic and bitwise instructions + +| Category | Opcodes | +|---|---| +| Integer binary | `add`, `sub`, `mul`, `udiv`, `sdiv`, `urem`, `srem` | +| Integer bitwise | `and`, `or`, `xor` | +| Shifts | `shl`, `lshr`, `ashr` | +| Floating binary | `fadd`, `fsub`, `fmul`, `fdiv`, `frem` | +| Floating unary | `fneg` | + +- **V-12.** Integer arithmetic MAY carry `nsw` (no signed wrap) and `nuw` (no unsigned wrap) + flags. A result on which overflow occurs under an asserted flag is poison; an implementation + that does not model poison MUST NOT emit such a flag unless the overflow is provably impossible. +- **V-13.** Floating-point instructions MAY carry the fast-math flags `nnan`, `ninf`, `nsz`, + `arcp`, `contract`, `afn`, and `reassoc`. A flag MUST only be set when the corresponding trait or + `Config` licenses it. Fast-math flags MUST NOT be applied at optimization levels below `-Ofast` + unless explicitly configured. +- **V-14.** `udiv`/`sdiv`/`urem`/`srem` by zero is undefined behavior. An implementation that + models this MUST NOT fold such a division to an arbitrary value. + +### 5.4 Comparison instructions + +| Opcode | Predicates | +|---|---| +| `icmp` | `eq`, `ne`, `slt`, `sle`, `sgt`, `sge`, `ult`, `ule`, `ugt`, `uge` | +| `fcmp` | `oeq`, `one`, `olt`, `ole`, `ogt`, `oge`, `ord`, `uno`, `ueq`, `une`, `ult`, `ule`, `ugt`, `uge`, `true`, `false` | + +- **V-15.** The result of `icmp` and `fcmp` MUST have type `i1`. +- **V-16.** Integer comparison predicates beginning with `s` are signed; those beginning with `u` + are unsigned. + +### 5.5 Conversion instructions + +| Opcode | Meaning | +|---|---| +| `trunc` | Narrow an integer. | +| `zext`, `sext` | Widen an integer, zero- or sign-extending. | +| `fptrunc`, `fpext` | Narrow or widen a float. | +| `fptoui`, `fptosi` | Float to integer. | +| `uitofp`, `sitofp` | Integer to float. | +| `ptrtoint`, `inttoptr` | Pointer/integer conversion. | +| `bitcast` | Reinterpret bits; requires equal size and no pointer-to-pointer element change in the C backend. | + +- **V-17.** A conversion MUST be explicit. There MUST be no implicit widening, narrowing, or + signedness change anywhere else in the instruction set. + +### 5.6 Other instructions + +| Opcode | Syntax (conceptual) | Meaning | +|---|---|---| +| `phi` | `%v = phi T [ v1, %b1 ], [ v2, %b2 ], ...` | SSA merge at a join. | +| `select` | `%v = select %cond, %a, %b` | Choose `%a` or `%b` on an `i1`. | +| `call` | `%v = call R @callee(args...) [flags]` | Call a function or intrinsic. | +| `atomicrmw` | `%v = atomicrmw op, ptr %p, %x ordering` | Atomic read-modify-write, result is the old value. | +| `cmpxchg` | `%v = cmpxchg ptr %p, %expected, %new ordering` | Atomic compare-and-exchange. | +| `fence` | `fence ordering` | Memory ordering fence. | + +- **V-18.** A `phi` has one operand per predecessor, each a `(value, predecessor-block)` pair. Its + result type MUST equal each operand's type. +- **V-19.** A `select` condition MUST have type `i1`, and its two value operands MUST share a type + that equals the result type. +- **V-20.** A `call` result value MAY be absent when the callee returns `void`. Call-site attributes + (`Inline`, `NoInline`, `TailCall`, `MustTail`, `NoTail`, `ColdCall`, `NoReturn`, and the + memory-effect requests) MUST be carried on the call and read from its trait, not re-derived. +- **V-21.** Non-atomic accesses MUST NOT be reordered across an atomic access or a fence in a way + that violates the ordering the access declares. + +### 5.7 Intrinsics + +A VIR intrinsic is a well-known callee referenced by a `call` instruction. The v1 intrinsic set is +fixed: + +| Intrinsic | Signature | Meaning | +|---|---|---| +| `memcpy` | `(ptr, ptr, iN) -> void` | Copy `N` bytes between non-overlapping regions. | +| `memset` | `(ptr, i8, iN) -> void` | Fill `N` bytes with the given byte value. | +| `memmove` | `(ptr, ptr, iN) -> void` | Copy `N` bytes between possibly overlapping regions. | +| `sqrt` | `(f32) -> f32`, `(f64) -> f64` | IEEE-754 square root. | +| `fabs` | `(f32) -> f32`, `(f64) -> f64` | Absolute value. | +| `ctpop` | `(iN) -> iN` | Count of set bits. | +| `fshl` | `(iN, iN, iN) -> iN` | Funnel shift left. | +| `sadd_sat`, `uadd_sat`, `ssub_sat`, `usub_sat` | `(iN, iN) -> iN` | Saturating add and subtract. | +| `sadd_overflow`, `uadd_overflow`, `ssub_overflow`, `usub_overflow`, `smul_overflow`, `umul_overflow` | `(iN, iN) -> {iN, i1}` | Wrapping result plus an overflow flag. | + +`memcpy`, `memset`, and `memmove` MAY also appear as dedicated opcodes (§5.2); the other intrinsics +appear only as calls. + +- **V-25.** An intrinsic `call` MUST use the canonical name in the table above and operands that + match its signature. Its result type follows the signature; the overflow intrinsics return the + struct `{iN, i1}`. +- **V-26.** The intrinsic names are reserved. An implementation MUST recognize them as intrinsics + rather than as user-defined callees, and MUST NOT allow a module to redefine them. + +## 6. Well-formedness invariants + +A VIR module is **well-formed** when it satisfies all of the following. The verifier checks them +after lowering and, in debug builds, after every pass (see [012 — Testing](012-testing.md)). + +- **WF-1 (single definition).** Every SSA value is defined exactly once in its function. +- **WF-2 (def dominates use).** Every use is dominated by its definition, with the phi exception: + an operand `(v, P)` of a phi requires `v` to dominate `P`, or to be the phi's own result when `P` + is a loop back-edge predecessor. +- **WF-3 (termination).** Every basic block ends in exactly one terminator, and no terminator + appears before the end of a block. +- **WF-4 (phi arity).** The number of phi operands equals the number of the phi block's + predecessors, and each named predecessor is an actual predecessor. +- **WF-5 (CFG consistency).** For every successor edge `b -> c` from a terminator, `c` lists `b` + as a predecessor. The entry block has no predecessors. +- **WF-6 (operand types).** Every instruction's operands and result conform to §4 and §5. +- **WF-7 (return type).** Every `ret` agrees with the function's declared return type. +- **WF-8 (function isolation).** No instruction references a value defined in another function. +- **WF-9 (declared callees).** Every `call` names a function that is either defined in the module + or declared in it. +- **WF-10 (alloca result).** An `alloca` of `T` produces a `ptr`, and every `load`/`store` + through it agrees on `T`. +- **WF-11 (dead blocks).** A block unreachable from the entry block MAY remain, but it MUST itself + be well-formed. + +Verification policy: + +- **V-22.** In debug builds, the verifier MUST run after lowering and after every optimization pass. + A violation MUST assert. +- **V-23.** In release builds, verification MUST be available behind a flag (the driver's + `--verify`) and MUST NOT run by default. A violation detected in release MUST raise an + internal-compiler-error diagnostic as specified in [011 — Diagnostics](011-diagnostics.md). +- **V-24.** An implementation MUST NOT "repair" a detected violation silently. + +## 7. Textual form + +The textual form prints functions, blocks, instructions, and types in a stable order. It exists +for dumps, tooling, and round-trip tests. The round-trip property is +`print(parse(print(m))) == print(m)` (see [012 — Testing](012-testing.md)). The parser is not a +supported frontend: it MAY assume the input was produced by the printer. + +## 8. Example + +``` +define i32 @sum(i32 %n) { +entry: + br %loop.header + +loop.header: + %i = phi i32 [ 0, %entry ], [ %i.next, %loop.latch ] + %s = phi i32 [ 0, %entry ], [ %s.next, %loop.latch ] + %cmp = icmp slt i32 %i, %n + condbr %cmp, %loop.body, %loop.exit + +loop.body: + %s.mid = add i32 %s, %i + br %loop.latch + +loop.latch: + %i.next = add i32 %i, 1 + %s.next = add i32 %s.mid, 0 + br %loop.header + +loop.exit: + ret i32 %s +} +``` + +The two phis are required because `loop.header` has two predecessors, `entry` and `loop.latch`. +WF-4 requires both phis to carry exactly two operands, and WF-2 requires each operand to dominate +its named predecessor. + +## 9. Cross-references + +- [000 — Overview](000-overview.md): determinism, non-goals (no hand-authored VIR). +- [001 — Architecture](001-architecture.md): VIR module placement and visibility. +- [002 — Traits](002-traits.md): the only channel by which VIR receives facts. +- [005 — Lowering](005-lowering.md): constructs and validates VIR. +- [007 — VIR Optimizer](007-vir-optimizer.md): transforms VIR under the invariants here. +- [008 — C Backend](008-c-backend.md): out-of-SSA and C emission. +- [009 — LLVM Backend](009-llvm-backend.md): 1:1 VIR to LLVM mapping. +- [011 — Diagnostics](011-diagnostics.md): internal-compiler-error policy. +- [012 — Testing](012-testing.md): verifier, round-trip, and differential tests. diff --git a/spec/007-vir-optimizer.md b/spec/007-vir-optimizer.md new file mode 100644 index 0000000..4b3484f --- /dev/null +++ b/spec/007-vir-optimizer.md @@ -0,0 +1,311 @@ +# 007 — VIR Optimizer + +- **Status:** Draft +- **Normative language:** `MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are to be + interpreted as described in RFC 2119. + +## 1. Purpose + +The VIR optimizer is the heavy-transformation half of libVCT. It consumes lowered VIR and applies +large, whole-function and whole-module transformations: inlining, scalar and memory +optimization, loop transformation, vectorization, control-flow cleanup, and code-generation +preparation. It is where the design's codegen-quality goal is realized. + +The VIR optimizer is a **strict trait consumer**. It discovers nothing. Every fact that enables or +forbids a transform is an attribute or relationship left by HIR, or a Request that HIR issued. +When a fact is not present in a trait, the optimizer MUST assume the conservative answer and move +on. + +## 2. Scope + +This file specifies the optimizer contract, the pass framework, the analysis set and its +invalidation discipline, the default `-O2` pipeline, request handling, alias analysis, region +barriers, budgets, and the rules that relate optimization level to aggression. + +It does not specify VIR itself (see [006 — VIR](006-vir.md)), how VIR was produced (see +[005 — Lowering](005-lowering.md)), or how optimized VIR is emitted (see +[008 — C Backend](008-c-backend.md), [009 — LLVM Backend](009-llvm-backend.md)). + +## 3. Definitions and model + +| Term | Meaning | +|---|---| +| **VIR pass** | A named transform over a function or a module. | +| **Pass manager** | The driver that schedules passes and owns analyses. | +| **Analysis** | A derived fact about a function or module, cached and invalidated. | +| **Preserved set** | The analyses a pass declares it does not invalidate. | +| **Request** | An attribute-derived suggestion from HIR, with a strength (see [002 — Traits](002-traits.md)). | +| **Action request** | A Request that asks for a transform (`Inline`, `Unroll(factor)`, `LICM`, ...). It yields a `RequestResult` when the responsible pass runs. | +| **Property request** | A proven property the optimizer MAY exploit (`LoopInvariant`, `Range`, `Assume`, ...). It is consumed as a fact and does not by itself produce a `RequestResult`. | +| **DenyReason** | The machine-readable reason a request was declined. | +| **Region barrier** | An opaque region produced by lowering for `complex`/`NoOptimize` code. | +| **Remark** | A per-decision record emitted for diagnostics. | + +## 4. Normative requirements + +### 4.1 Contract + +- **O-1.** The input MUST be well-formed VIR (see [006 — VIR](006-vir.md)) with complete, + epoch-valid traits. +- **O-2.** The output MUST be well-formed VIR that is still in SSA form. The optimizer MUST NOT + break any invariant in [006 — VIR](006-vir.md). +- **O-3.** The optimizer MUST be a strict trait consumer. A transform MUST be enabled by an + attribute, a relationship, or a Request. The optimizer MUST NOT assume a fact that is absent from + traits, and MUST NOT infer one from program structure. +- **O-4.** The optimizer MUST run only at `-O2`, `-O3`, `-Ofast`, and `-Oz`. At `-O0` and `-O1` it + MUST NOT run at all (see §4.9). +- **O-5.** The optimizer MUST be deterministic. For fixed input and `Config`, it MUST produce + byte-identical output regardless of thread count, per [000 — Overview](000-overview.md). Where a + pass has a choice that affects output (for example, the order of two independent transforms), the + pass MUST fix the order deterministically. +- **O-6.** The optimizer MUST preserve the observable behavior of the program. A transform that + would change behavior is legal only when the traits license that change (for example, fast-math + flags). +- **O-7.** The optimizer MUST terminate on every input. Budgets under §4.8 guarantee this. + +### 4.2 Pass framework + +A pass is a named unit of work. A function pass runs on one function; a module (IPA) pass runs on +the whole module. Passes mutate VIR in place. + +``` +interface VirPass { + string name(); + void run(VirModule m, PassContext ctx); // module pass +} +``` + +- **O-8.** Every pass MUST expose a stable name. The name MUST be usable as a `-f` / + `-fno-` toggle (see §4.9). +- **O-9.** A pass MUST declare the analyses it preserves before it runs. The pass manager uses the + declaration to invalidate only the analyses the pass might destroy (see §4.3). +- **O-10.** A pass MUST update the def-use graph for every operand it rewrites (see + [006 — VIR](006-vir.md)). +- **O-11.** A pass MUST NOT run when a required analysis cannot be built, when a region barrier + forbids it (see §4.7), or when a request that gates it is denied. +- **O-12.** A pass that reads a trait MUST ensure the trait is epoch-valid. If the epoch advanced + since it read the trait, it MUST re-read rather than reuse the stale value. + +### 4.3 Analyses and invalidation + +The pass manager owns a cache of analyses. Analyses are built lazily on first request and reused +until invalidated. + +| Analysis | Subject | +|---|---| +| `DominatorTree` | Immediate dominators and the dominator tree of each function. | +| `PostDominatorTree` | Immediate post-dominators, used by control-flow transforms. | +| `LoopInfo` | Natural loops, headers, latches, preheaders, and nesting. | +| `AliasInfo` | May-alias answers synthesized from traits (§4.6). | +| `DefUse` | The def-use graph over SSA values. | +| `CallGraph` | Caller/callee edges and call-site metadata. | +| `RangeInfo` | Lightweight value ranges, seeded by `Range`/`Assume` traits. | + +- **O-13.** Invalidation MUST be fine-grained. The pass manager MUST invalidate exactly the + analyses that a pass does not declare preserved. It MUST NOT invalidate all analyses as a + default. An invalidated analysis is rebuilt lazily on next request. +- **O-14.** A pass MUST NOT read an analysis after the pass manager has invalidated it; it MUST + request a fresh build. +- **O-15.** An analysis whose result depends on traits MUST be invalidated when a trait epoch + advances in a way that can change that result. The epoch counter is the mechanism (see + [002 — Traits](002-traits.md)). +- **O-16.** `RangeInfo` MUST be lightweight. It MUST be seeded by `Range` and `Assume` traits and + MUST NOT implement a full scalar-evolution analysis. A pass MUST NOT expect exact ranges. +- **O-17.** `AliasInfo` MUST NOT guess. Its answers are defined in §4.6. + +### 4.4 Default `-O2` pipeline + +At `-O2` the pass manager runs the following canonical pipeline. The order is fixed; the pipeline +is not reordered adaptively by requests. Requests gate and parameterize passes, they do not resequence +them. + +| Stage | Passes | +|---|---| +| 1. Canonicalize | `mem2reg`, `SROA`, `instcombine`, `simplifyCFG`, `early-CSE`, `DCE` | +| 2. Scalar | `GVN`, `SCCP`, `LICM`, `indvars`, `reassociation` | +| 3. IPA | inliner (gated by `Inline`/`AlwaysInline`/`NoInline`/`ColdCall`/`LikelyCall`), global `DCE`, `IPSCCP`, function-attribute propagation | +| 4. Loops | `unroll`/`interleave`/`peel`/`rotate`/`unswitch`/`distribute`/`fuse`/`jam`, each gated by its Request | +| 5. Vectorize | loop and SLP vectorization, gated by `Vectorize`, `LoopInvariant`, `Range`, `NoAlias`, `Restrict` | +| 6. Memory | alias-driven `DSE`, GEP simplification, load widening | +| 7. Control flow | block layout (`HotPath`/`ColdPath`), tail-call formation (`TailCall`/`MustTail`/`NoTail`), unreachable pruning, jump threading | +| 8. Codegen prep | backend-directed canonicalization; output remains SSA | + +- **O-18.** At `-O2` the optimizer MUST run all eight stages in the order shown, subject to pass + toggles, region barriers, and request denials. +- **O-19.** Stage 3 (IPA) MUST be driven by the call-edge requests. A callee carrying `AlwaysInline` + MUST be inlined where legal. A callee carrying `NoInline` MUST NOT be inlined. +- **O-20.** Stage 4 loop transforms MUST each be gated by their own Request. For example, `unroll` + runs only under an `Unroll(factor)` request, and `LICM` hoists only under an `LICM` request or + when a `LoopInvariant` property licenses it. +- **O-21.** Stage 5 vectorization MUST be gated by `Vectorize` and MUST consult `NoAlias`, + `Restrict`, `Range`, and `LoopInvariant` before reordering memory operations. +- **O-22.** Stage 8 MUST NOT perform out-of-SSA translation and MUST NOT lower `Phi` + instructions. Out-of-SSA is a backend concern (see [008 — C Backend](008-c-backend.md)). + +### 4.5 Request handling + +Every action request that the responsible pass evaluates MUST produce a `RequestResult`. + +``` +struct RequestResult { + Request req; + bool accepted; + DenyReason reason; + string note; // one-line human-readable explanation +} +``` + +A denial MUST carry a `DenyReason` and a one-line `note`. + +| DenyReason | Meaning | +|---|---| +| `CostModel` | Profitable only under a different cost model. | +| `Illegality` | The transform would be incorrect. | +| `AlreadyDone` | No-op; the goal already holds. | +| `Unsupported` | The optimizer does not implement it yet. | +| `ContradictsTrait` | Conflicts with a stronger fact. | +| `TooLarge` | Exceeds a size or count budget. | +| `NoTargetSupport` | The backend or target cannot express the result. | + +- **O-23.** When a pass evaluates a request, it MUST record a `RequestResult` with a `DenyReason` + and a one-line `note` on denial. It MUST NOT deny silently. +- **O-24.** A `Soft` request MAY be denied for any `DenyReason`. A denied `Soft` request MUST NOT + produce an error; it SHOULD produce a remark. +- **O-25.** A `Strong` request denied with `Illegality` or `ContradictsTrait` MUST raise an + internal-compiler-error diagnostic (see [011 — Diagnostics](011-diagnostics.md)). A `Strong` + request should be legally satisfiable; failing to satisfy it indicates a compiler inconsistency. + This rule applies only when the target is capable of the transform; a capability gap MUST instead + be reported as `NoTargetSupport`. +- **O-26.** A `Strong` request denied with `Unsupported` or `NoTargetSupport` MUST produce a warning, + not an error, unless the target genuinely cannot express semantics the request requires for + correctness or ABI. Such a semantic-capability failure (for example, `MustTail` on GCC; see + [008 — C Backend](008-c-backend.md)) MUST be a hard error. +- **O-27.** A `Strong` request denied for any other reason (`CostModel`, `AlreadyDone`, `TooLarge`) + MUST NOT be a hard error and SHOULD produce a warning. +- **O-28.** The optimizer MUST NOT drop a request without a `RequestResult`, and MUST NOT flip a + request to accepted when it did not perform the transform. +- **O-29.** `LoopInvariant` is a proven property and `LICM` is an explicit hoist request. Both MUST + be retained and handled distinctly: a pass MAY exploit `LoopInvariant` without an `LICM` request, + and a pass MUST attempt the hoist when `LICM` is requested. +- **O-30.** `NoRead`/`NoWrite` are access-edge primitives; `ReadOnly`/`WriteOnly` are region-level + facts. A pass MUST respect the scope difference when it consumes either form. + +### 4.6 Alias analysis + +`AliasInfo` answers may-alias queries. It is synthesized from traits and provenance only. + +- **O-31.** `AliasInfo` MUST derive its answers from `NoAlias`, `Restrict`, `NoCapture`, + `Dereferenceable`, `ReadOnly`, and `WriteOnly` traits, together with the provenance relationships + HIR left on the values. +- **O-32.** When no trait establishes a relationship between two memory references, `AliasInfo` + MUST answer `MayAlias`. It MUST NOT infer `NoAlias` from program structure, from the fact that + two allocas are distinct, or from any fact not present in traits. +- **O-33.** A pass MAY use a `NoAlias` answer only for the exact pair and scope the trait licenses. + It MUST NOT generalize an answer to other pairs. +- **O-34.** A region barrier is an alias barrier: no reference inside one is known not to alias a + reference outside it unless a trait says so (see §4.7). + +### 4.7 Region barriers + +- **O-35.** Lowering marks `complex`/`NoOptimize` code with a region-level barrier (see + [005 — Lowering](005-lowering.md)). VIR passes MUST treat the region as opaque. +- **O-36.** A pass MUST NOT rewrite, reorder, delete, hoist, or sink an instruction across a region + barrier. It MUST NOT propagate a value or a fact into or out of the region. +- **O-37.** Memory and alias analyses MUST model a barrier region as a single opaque, + side-effecting operation. +- **O-38.** A pass that cannot make progress because of a barrier MUST skip the affected code + without error and MUST NOT force a transform through it. + +### 4.8 Budgets and termination + +- **O-39.** The optimizer MUST honor `vir.max_iterations` (per-pass or per-analysis iteration cap) + and `vir.max_pipeline_rounds` (number of times the pipeline may re-run). Both MUST be + configurable through the API and CLI and MUST have finite defaults. +- **O-40.** When a budget is exhausted, the optimizer MUST stop and return the current, well-formed + VIR. It MUST NOT loop indefinitely and MUST NOT return malformed VIR. +- **O-41.** Budget exhaustion SHOULD be reported as a warning and MAY be reported as a remark. It + MUST NOT be an internal compiler error. + +The default budgets are: + +| Budget | Default | +|---|---| +| `vir.max_iterations` | `100000` | +| `vir.max_pipeline_rounds` | `4` | + +An implementation MUST use these defaults unless the caller overrides them through the API or CLI. +The values are finite (O-39) and are chosen at the same order of magnitude as the HIR budgets (see +[004 — HIR Optimizer](004-hir-optimizer.md)); `vir.max_pipeline_rounds` bounds how many times the +eight-stage pipeline of §4.4 may be re-run. + +### 4.9 Optimization levels and pass toggles + +| Level | VIR optimization | Behavior | +|---|---|---| +| `-O0` | off | None. Only lowering's own translation. | +| `-O1` | off | None. HIR folds only. | +| `-O2` | on | The eight-stage default pipeline of §4.4. | +| `-O3` | on | The `-O2` pipeline with raised aggression (see below). | +| `-Ofast` | on | `-O3` plus fast-math and `-march=native`. | +| `-Oz` | on | The `-O2` pipeline tuned for size. | + +- **O-42.** The optimizer MUST be entirely disabled at `-O0` and `-O1`. In particular, `mem2reg`, + `SROA`, and `DCE` MUST NOT run at those levels. +- **O-43.** `-O3` MUST raise aggression relative to `-O2`: higher inlining thresholds, larger + unroll/interleave factors, and wider vectorization factors, all still gated by their requests. +- **O-44.** `-Ofast` MUST enable fast-math flags and `-march=native`. Fast-math flags MUST NOT be + set at any lower level unless the caller explicitly configured them. +- **O-45.** `-Oz` MUST be size-first: most unrolling and vectorization MUST be disabled, inlining + limited to very small callees, while canonicalization and dead-code elimination remain enabled. +- **O-46.** Every pass MUST be individually controllable through a `-f` / `-fno-` + toggle. A disabled pass MUST be omitted from the pipeline rather than run and then reversed. + +### 4.10 Diagnostics and remarks + +- **O-47.** A pass MUST emit its decisions through the diagnostics service; it MUST NOT print + directly (see [011 — Diagnostics](011-diagnostics.md)). +- **O-48.** Remarks MUST be off by default and MUST be enabled by `--remarks`; they MUST be + enabled automatically at `-O3`. +- **O-49.** A remark that reports a request decision SHOULD include the request, the subject, the + outcome, and the one-line note. Example: + + `VIR: denied Inline on @foo, CostModel (callee 2.4x size budget)` + +## 5. Invariants + +The output of the optimizer MUST satisfy all of the following. + +1. **Well-formed VIR.** All invariants in [006 — VIR](006-vir.md) hold after every pass. +2. **Still SSA.** The output is in SSA form; `Phi` instructions remain; no out-of-SSA lowering + happens here. +3. **Trait fidelity.** No transform may invent a fact. Every fact used is traceable to a trait. +4. **Barrier respect.** No transform crosses a region barrier. +5. **Request accountability.** Every evaluated action request has a `RequestResult`. +6. **Termination.** The pipeline terminates within its configured budgets. +7. **Determinism.** Fixed input and `Config` produce byte-identical output. + +## 6. Example + +Given a callee annotated `AlwaysInline` and a call site annotated `NoInline`: + +- The call-edge requests conflict. HIR resolves strong conflicts before lowering (see + [002 — Traits](002-traits.md)), so lowering should not present both to the optimizer. If a + contradiction still reaches the optimizer, the inliner denies the `Strong` request with + `ContradictsTrait`, which is an internal compiler error under O-25. + +Given a callee with a `ColdCall` request and a size estimate above the inline threshold: + +- The inliner denies `Inline` with `CostModel` and records a `RequestResult` with the note + `callee 3.1x size budget`. No error is raised; a remark is emitted when remarks are enabled. + +## 7. Cross-references + +- [000 — Overview](000-overview.md): determinism guarantee and conformance. +- [001 — Architecture](001-architecture.md): `vct.vir.opt` module boundary. +- [002 — Traits](002-traits.md): requests, strengths, conflict rules, epochs. +- [004 — HIR Optimizer](004-hir-optimizer.md): who produces the traits the optimizer consumes. +- [005 — Lowering](005-lowering.md): who produces the VIR and the region barriers. +- [006 — VIR](006-vir.md): the IR and its invariants. +- [008 — C Backend](008-c-backend.md), [009 — LLVM Backend](009-llvm-backend.md): out-of-SSA on the C path and 1:1 mapping on the LLVM path. +- [011 — Diagnostics](011-diagnostics.md): internal-compiler-error policy and remarks. +- [012 — Testing](012-testing.md): verifier and differential testing. diff --git a/spec/008-c-backend.md b/spec/008-c-backend.md new file mode 100644 index 0000000..5894758 --- /dev/null +++ b/spec/008-c-backend.md @@ -0,0 +1,280 @@ +# 008 — C Backend + +- **Status:** Draft +- **Normative language:** `MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are to + be interpreted as described in RFC 2119. + +## 1. Purpose + +The C backend (`vct.backend.c`) translates optimized VIR into C17 source. It is the +default backend: the driver hands each translation unit to a system C compiler that +produces an object file. The backend's target is **compiler-friendly C**, meaning the +emitted text is shaped so that GCC and Clang recognize it and optimize it well. libVCT +does not try to pre-optimize away everything the C compiler can do itself. + +## 2. Scope + +This file specifies the backend's contract and the required shape of its output: +translation-unit granularity, out-of-SSA lowering, control-flow reconstruction, +trait-to-C mappings, type and layout emission, name mangling, intrinsic selection, +debug line mapping, and ABI guarantees. The VIR input is specified in +[006 — VIR](006-vir.md); the pipeline that feeds the backend is specified in +[007 — VIR Optimizer](007-vir-optimizer.md) and [010 — Driver & CLI](010-driver-cli.md). + +## 3. Definitions + +| Term | Meaning | +|---|---| +| TU | Translation unit: one C17 source file. | +| Out-of-SSA | Conversion of SSA values into C variables with explicit assignments on every control-flow path. | +| Critical edge | A CFG edge whose source has multiple successors and whose destination has multiple predecessors. | +| Reducible CFG | A control-flow graph in which every cycle has exactly one loop header. | +| Loop header | The unique block by which control enters a loop. | +| `layout` attribute | The attribute that pins an aggregate's ABI: field offsets, total size, and alignment. | +| Opaque barrier call | A `noinline` function whose body the C compiler cannot inspect, used to stop optimization across a region. | + +## 4. Model + +The backend is a pure translation stage. It consumes optimized VIR that satisfies the +SSA invariants of [006 — VIR](006-vir.md) and emits C17. It performs no optimization +and no discovery; every fact it uses is read from traits (see +[002 — Traits](002-traits.md)). + +Emission is **per function**. A VIR module with N functions produces N independent C +translation units. Each TU carries one function definition plus the declarations it +needs. That independence is what lets emission run in parallel, and it is what lets +the driver concatenate the results in a deterministic order. + +## 5. Normative requirements + +### 5.1 Translation-unit structure + +**R1.** The backend MUST emit exactly one C17 translation unit per VIR function. + +**R2.** Emitting one function's TU MUST NOT depend on emitting any other function's TU. +Two functions emitted in any order MUST yield identical per-function text. + +**R3.** Each TU MUST be emitted in two passes: first all forward declarations, then all +definitions. Every declaration MUST precede the definition that uses it. + +**R4.** The backend MUST assign each function a stable emission index derived only from +module content, so the driver can order TUs deterministically (see +[010 — Driver & CLI](010-driver-cli.md)). + +**R5.** For a fixed module and `Config`, the concatenation of per-function TUs MUST be +byte-identical across runs and independent of thread count (see +[000 — Overview](000-overview.md)). + +### 5.2 Out-of-SSA + +**R6.** The backend MUST convert SSA to C using the algorithm of Boissinot, Brisk, +Caron, and Ferrand, "Revisiting Out-of-SSA Translation for Correctness, Code Quality, +and Efficiency" (CGO 2009), or an algorithm observably equivalent to it. The +conversion MUST handle the lost-copy and swap problems. + +**R7.** The backend MUST split all critical edges **before** running out-of-SSA +conversion. + +**R8.** After conversion, every SSA value MUST map to exactly one C variable or +expression, and every `Phi` MUST lower to an assignment on each incoming edge, or to a +semantically equivalent construct. + +### 5.3 Control-flow reconstruction + +**R9.** For reducible CFGs, the backend MUST reconstruct structured C control flow using +`if`/`else`, `while`, `for`, `do`, and `switch` wherever the shape is recognized. + +**R10.** For irreducible CFGs, or for regions where no structured shape is recognized, +the backend MUST fall back to `goto` and labels. The fallback MUST preserve semantics. +This is a correctness requirement, not a quality preference. + +**R11.** Structured reconstruction SHOULD be preferred over `goto` fallback whenever a +recognized shape exists, because it improves downstream optimization quality. + +### 5.4 Compiler-friendly C + +**R12.** The backend MUST map traits to C qualifiers, hints, and pragmas as specified +in Table 1. When a trait is absent, the corresponding construct MUST NOT be emitted. + +**Table 1. Trait to C mapping** + +| Trait | C construct | Notes | +|---|---|---| +| `Restrict`, `NoAlias` | `restrict` | On pointer parameters and local pointers. | +| `ReadOnly`, `NoWrite` | `const` | On the pointee type of a pointer or reference parameter. | +| `NoWrite` | `__attribute__((pure))` | Function level; reads allowed, writes not allowed. | +| `NoRead` + `NoWrite` | `__attribute__((const))` | Function level; no memory access. | +| `NoRead` | No portable C17 construct | No GCC/Clang attribute means "does not read"; omit it. | +| `NonNull` | `nonnull` attribute or `__builtin_unreachable` guard | | +| `NoReturn` | `_Noreturn` | | +| `ColdPath` | `__attribute__((cold))` | | +| `HotPath` | `__attribute__((hot))` | | +| `LikelyCall` | `__builtin_expect(cond, 1)` | On the branch condition. | +| `UnlikelyCall`, `ColdCall` | `__builtin_expect(cond, 0)` | On the branch condition. | +| `AlwaysInline` | `static inline` plus `always_inline` attribute | | +| `NoInline` | `__attribute__((noinline))` | | +| `Align(n)` | `_Alignas(n)` and/or `__builtin_assume_aligned` | | +| `Assume(pred)` | `__builtin_assume(pred)` (Clang) | | +| `Unreachable` | `__builtin_unreachable()` | | +| `Vectorize` | `#pragma GCC ivdep` or `#pragma clang loop vectorize(enable)` | Selected by probed compiler dialect. | +| `NoOptimize`, `complex` | Optimization barrier, see R14 | | + +**R13.** The backend MUST select between GCC and Clang spellings of pragmas and +builtins from the compiler dialect probed once by the driver (see +[010 — Driver & CLI](010-driver-cli.md)). When the probed compiler does not support a +construct, the backend MUST omit it rather than emit something that fails to compile, +unless the corresponding request is `Strong`, in which case the denial rules of +[007 — VIR Optimizer](007-vir-optimizer.md) apply. + +**R14 (Optimization barrier).** For a region marked `complex` or carrying a `NoOptimize` +request, the backend MUST emit a compiler barrier: a memory fence +(`__asm__ __volatile__("" ::: "memory")` on GCC and Clang) followed by a call to an +opaque function marked `noinline`. The call MUST be a real call that the C compiler +cannot see through. + +### 5.5 Types and layout + +**R15.** The backend MUST map VIR types to C types as specified in Table 2. + +**Table 2. VIR type to C type** + +| VIR type | C type | +|---|---| +| Signed integer with a C17 fixed-width type | `intN_t` | +| Unsigned integer with a C17 fixed-width type | `uintN_t` | +| `f32` | `float` | +| `f64` | `double` | +| Pointer `T*` | `T *` | +| Struct | `struct` | +| Union | `union` | +| Array `[N x T]` | `T[N]` | +| Function type | Function declarator or function pointer | +| `void` | `void` | + +**R16.** An integer width with no C17 fixed-width type (any width other than 8, 16, 32, +or 64 bits) MUST be diagnosed as unsupported on the C path unless the probed compiler +provides an N-bit integer type, in which case the backend MAY use it. + +**R17.** Aggregate GEPs MUST use field indices, never byte offsets. The backend MUST +let the C compiler choose struct layout unless the `layout` attribute pins it. + +**R18.** When the `layout` attribute pins an aggregate's ABI, the backend MUST emit explicit +padding members to force the specified offsets and total size, and MUST emit +`_Static_assert` checks on `sizeof` and on `offsetof` for every pinned field. + +### 5.6 Naming + +**R19.** Exported names MUST be preserved exactly as the frontend supplied them. +Internal names MUST be sanitized into valid C identifiers and uniquified +deterministically. + +**R20.** Sanitization and uniquification MUST NOT produce an identifier that collides +with a C keyword, a standard library identifier, or a compiler-reserved identifier. The +reserved-word and collision prefix policy of Table 3 MUST be applied. + +**Table 3. Reserved identifier prefix policy for internal names** + +| Category | Action | +|---|---| +| C keywords (`int`, `while`, ...) | Prefix with `vct_` | +| C17 standard library names (`memcpy`, `printf`, ...) | Prefix with `vct_` | +| Implementation-reserved names (`_x`, `__x`, `_[A-Z]...`) | Prefix with `vct_` | +| Compiler builtins (`__builtin_*`) | Prefix with `vct_` | +| Post-mangle collisions | Append a stable numeric suffix | + +**R21.** The disambiguator used for uniquification MUST be derived only from +deterministic module content. It MUST NOT depend on allocation addresses, hash-table +iteration order, or thread scheduling. + +### 5.7 Intrinsics + +**R22.** The backend MUST lower the VIR intrinsics of [006 — VIR](006-vir.md) to the matching +`__builtin_*` function where GCC and Clang provide one. + +**R23.** For intrinsics with no portable builtin, the backend MUST emit a call to a +portable helper with a stable name. A helper definition MUST be emitted per function, +deterministically, with internal linkage, into every TU whose function references it, so +that per-function text remains a pure function of that function's VIR (R2, I2). A helper +definition MUST NOT be emitted once per module or into a shared, module-level TU. + +### 5.8 Debug mapping + +**R24.** When debug mapping is enabled, the backend MUST emit `#line` directives that map +emitted C lines to frontend source positions, using the source map registered with the +diagnostics service (see [011 — Diagnostics](011-diagnostics.md)). + +**R25.** Debug mapping MUST be flag-gated: enabled when debug mode is on, disabled in +release. When disabled, the backend MUST NOT emit any `#line` directive. + +### 5.9 ABI + +**R26.** Exported functions MUST match the platform C ABI. Internal functions MUST be +emitted `static`. + +**R27.** Struct-by-value arguments and returns MUST follow the C compiler's ABI, +including any constraint introduced by the `layout` attribute. + +**R28.** A `MustTail` request MUST be honored with `[[clang::musttail]]` when the probed +compiler is Clang. When the probed compiler is GCC, a `MustTail` request MUST be a hard +error (see [011 — Diagnostics](011-diagnostics.md)). This is the semantic-capability +exception of O-26 in [007 — VIR Optimizer](007-vir-optimizer.md): GCC cannot express the +required tail-call guarantee, so the `Strong` request is fatal rather than a warning. + +**R29.** The backend MUST NOT emit implementation-defined constructs beyond those +explicitly permitted here. In particular it MUST NOT emit byte-offset aggregate +accesses, alignment-violating pointer casts, or identifiers outside the reserved prefix +policy. + +## 6. Invariants + +- **I1.** Each emitted TU is a valid C17 translation unit given the shared declarations. +- **I2.** Per-function text, including any helper definitions it carries, is a pure function of + that function's VIR and the traits it references. +- **I3.** Concatenated output is byte-identical for a fixed input and `Config`, at any + thread count. +- **I4.** Every definition is preceded by a declaration. +- **I5.** No two emitted identifiers collide after mangling. +- **I6.** Every layout-pinned aggregate satisfies its static assertions at compile time. +- **I7.** Every structured-to-`goto` fallback preserves the CFG's reachability and + semantics. + +## 7. Examples + +A `Phi` merging `%a` and `%b` on the true and false edges of a conditional branch lowers +to a temporary assigned on each edge before out-of-SSA bookkeeping: + +```c +int vct_phi0; /* out-of-SSA temporary for a Phi */ +if (cond) { vct_phi0 = a; } else { vct_phi0 = b; } +return vct_phi0; +``` + +A `NoOptimize` region emits a fence followed by an opaque call: + +```c +__asm__ __volatile__("" ::: "memory"); +vct_opaque_barrier(x); +``` + +A layout-pinned struct emits padding plus compile-time checks: + +```c +struct vct_pair { + int32_t lo; + int32_t vct_pad0; + int64_t hi; +}; +_Static_assert(sizeof(struct vct_pair) == 16, "layout"); +_Static_assert(offsetof(struct vct_pair, hi) == 8, "layout"); +``` + +## 8. Cross-references + +- [000 — Overview](000-overview.md) for determinism and conformance. +- [001 — Architecture](001-architecture.md) for module boundaries. +- [002 — Traits](002-traits.md) for the trait vocabulary consumed here. +- [006 — VIR](006-vir.md) for the input IR and its invariants. +- [007 — VIR Optimizer](007-vir-optimizer.md) for the pass pipeline. +- [009 — LLVM Backend](009-llvm-backend.md) for the alternate backend. +- [010 — Driver & CLI](010-driver-cli.md) for compiler probing and TU concatenation. +- [011 — Diagnostics](011-diagnostics.md) for error reporting and `#line`. diff --git a/spec/009-llvm-backend.md b/spec/009-llvm-backend.md new file mode 100644 index 0000000..ab478f0 --- /dev/null +++ b/spec/009-llvm-backend.md @@ -0,0 +1,267 @@ +# 009 — LLVM Backend + +- **Status:** Draft +- **Normative language:** `MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are to + be interpreted as described in RFC 2119. + +## 1. Purpose + +The LLVM backend (`vct.backend.llvm`) translates optimized VIR into textual LLVM IR +(the `.ll` assembly format). It is the alternate backend. It runs **after** VIR +optimization and is a pure translation: it performs no optimization and no discovery. +libVCT never invokes `llvm-as`, `opt`, or `llc`, and never links against LLVM. +Converting `.ll` to bitcode, an object file, or an executable is the caller's job. + +## 2. Scope + +This file specifies the backend's contract: 1:1 SSA mapping, the type mapping, the full +trait-to-attribute and trait-to-metadata mapping, intrinsic selection, module +scaffolding, the LLVM version pin, and determinism. The VIR input is specified in +[006 — VIR](006-vir.md); the optimizer that runs before this backend is specified in +[007 — VIR Optimizer](007-vir-optimizer.md). The C backend is specified in +[008 — C Backend](008-c-backend.md). + +## 3. Definitions + +| Term | Meaning | +|---|---| +| `.ll` | Textual LLVM assembly. | +| Opaque pointer | The LLVM `ptr` type, which carries no pointee type. | +| Target triple | The architecture/vendor/OS string that selects codegen. | +| Datalayout | The LLVM string describing primitive sizes, alignments, and endianness. | +| Metadata | `!name` nodes attached to instructions or the module. | +| `!range` | Metadata constraining the possible values of a load or call result. | + +## 4. Model + +VIR is already in SSA form, so the mapping to LLVM is **1:1**. There is no out-of-SSA +step here; that step exists only on the C path (see [008 — C Backend](008-c-backend.md)). +Each VIR block becomes one LLVM basic block, each VIR instruction becomes one LLVM +instruction, and each `Phi` becomes an LLVM `phi`. + +The output is a **self-contained module**. It carries its own target triple and +datalayout, and it declares every external symbol it references before it defines +anything. A consumer can pass the module straight to `llvm-as` with no additional +environment. + +## 5. Normative requirements + +### 5.1 Purity and output + +**R1.** The LLVM backend MUST perform pure translation. It MUST NOT run optimization, +inference, or discovery passes, and it MUST NOT modify the VIR it reads. + +**R2.** The backend MUST emit textual `.ll` only. It MUST NOT invoke `llvm-as`, `opt`, +`llc`, or any other LLVM tool, and it MUST NOT link against LLVM libraries. + +**R3.** The backend MUST emit a self-contained module that includes a target triple and +a datalayout derived from that triple. + +**R4.** The backend MUST emit all declarations before any definition. + +**R5.** For a fixed input and `Config`, the emitted module MUST be byte-identical across +runs and independent of thread count (see [000 — Overview](000-overview.md)). + +### 5.2 SSA mapping + +**R6.** The backend MUST map VIR constructs to LLVM constructs 1:1: + +**Table 1. VIR to LLVM structural mapping** + +| VIR | LLVM | +|---|---| +| Basic block | Basic block with a label | +| Instruction | Instruction, in the same order | +| `Phi(value, pred)` | `phi` with one incoming pair per predecessor | +| Conditional branch | `br i1 , label , label ` | +| Switch | `switch` | +| Unconditional branch | `br label ` | +| Return | `ret` | +| Unreachable | `unreachable` | +| `alloca` | `alloca` | +| `load` | `load` | +| `store` | `store` | + +**R7.** The backend MUST preserve block order given by the VIR module, and MUST preserve +instruction order within a block, so that the 1:1 mapping is observable in the text. + +**R8.** Every VIR value MUST map to exactly one LLVM SSA name. Names MUST be generated +deterministically. + +### 5.3 Types + +**R9.** The backend MUST map VIR types to LLVM types as specified in Table 2. + +**Table 2. VIR type to LLVM type** + +| VIR type | LLVM type | +|---|---| +| Integer of N bits, signed or unsigned | `iN` | +| `f32` | `float` | +| `f64` | `double` | +| Pointer `T*` | `ptr` | +| Struct | `%struct.Name = type { ... }` | +| Union | `%union.Name = type { ... }` | +| Array `[N x T]` | `[N x T]` | +| `void` | `void` | +| Function type | ` ()` | + +**R10.** The backend MUST pin LLVM 18 or later and MUST use **opaque pointers**. It MUST +NOT emit typed pointers such as `i32*`. + +**R11.** Aggregate GEPs MUST use VIR field indices directly as LLVM struct indices. The +backend MUST NOT translate them to byte offsets. + +**R12.** The backend MUST emit no `!llvm.dbg` metadata in v1. Debug mapping exists only +on the C path (see [008 — C Backend](008-c-backend.md)). + +### 5.4 Traits to attributes and metadata + +**R13.** The backend MUST map traits to LLVM attributes and metadata as specified in +Table 3. When a trait is absent, the matching attribute or metadata MUST NOT be emitted. + +**Table 3. Trait to LLVM attribute and metadata mapping** + +| Trait | LLVM attribute or metadata | +|---|---| +| `Restrict`, `NoAlias` | `noalias` parameter attribute plus `!alias.scope` and `!noalias` | +| `ReadOnly`, `NoWrite` | `readonly` | +| `WriteOnly`, `NoRead` | `writeonly` | +| `NoRead` + `NoWrite` | `readnone` (spelled `memory(none)` on LLVM 18) | +| `NonNull` | `nonnull` | +| `Align(n)` | `align n` | +| `Range(lo, hi)` | `!range` on loads and calls | +| `Assume(pred)` | `llvm.assume` | +| `NoInline`, `AlwaysInline` | `noinline` / `alwaysinline` | +| `TailCall`, `MustTail`, `NoTail` | `tail` / `musttail` / `none` | +| `NoReturn` | `noreturn` | +| `HotPath`, `ColdPath` | `!prof` | +| `Vectorize` | `!llvm.loop.vectorize.enable` | +| `Unroll(n)` | `!llvm.loop.unroll.count` | +| `Unreachable` | `unreachable` terminator | +| `NoUndef` | Reserved; see R14 | + +The pairing mirrors the C mapping in [008 — C Backend](008-c-backend.md): a no-write fact takes a +read-only annotation, a no-read fact takes a write-only annotation, and their combination takes +`readnone`. `ReadOnly`/`WriteOnly` are region-level facts and `NoRead`/`NoWrite` are access-edge +primitives (see [002 — Traits](002-traits.md)); the emitted attribute follows the entity the trait +attaches to, whether a function or a pointer parameter. + +**R14.** `NoUndef` is reserved and is only meaningful on the LLVM path. The backend MUST +NOT interpret it on the C path. When present here, it MUST translate to the +corresponding `noundef` attribute. + +**R15.** The backend MUST emit real `llvm.assume` calls and real `!range` metadata; it +MUST NOT emit placeholders that a consumer would have to fill in. + +### 5.5 Intrinsics + +**R16.** The backend MUST lower the VIR intrinsics of [006 — VIR](006-vir.md) to the LLVM +intrinsics listed in Table 4. + +**Table 4. VIR intrinsic to LLVM intrinsic** + +| VIR intrinsic | LLVM intrinsic | +|---|---| +| `memcpy` | `llvm.memcpy.p0.p0.i64` | +| `memset` | `llvm.memset.p0.i64` | +| `memmove` | `llvm.memmove.p0.p0.i64` | +| `sqrt(f32/f64)` | `llvm.sqrt.f32` / `llvm.sqrt.f64` | +| `fabs(f32/f64)` | `llvm.fabs.f32` / `llvm.fabs.f64` | +| `ctpop(iN)` | `llvm.ctpop.iN` | +| `fshl(iN)` | `llvm.fshl.iN` | +| `sadd_sat`, `uadd_sat`, `ssub_sat`, `usub_sat` | `llvm.sadd.sat.iN`, `llvm.uadd.sat.iN`, `llvm.ssub.sat.iN`, `llvm.usub.sat.iN` | +| `sadd_overflow`, `uadd_overflow`, `ssub_overflow`, `usub_overflow`, `smul_overflow`, `umul_overflow` | `llvm.sadd.with.overflow.iN`, `llvm.uadd.with.overflow.iN`, `llvm.ssub.with.overflow.iN`, `llvm.usub.with.overflow.iN`, `llvm.smul.with.overflow.iN`, `llvm.umul.with.overflow.iN` | + +**R17.** The backend MUST emit the declaration of every intrinsic it references before +the definition that calls it. + +### 5.6 Module scaffolding + +**R18.** The backend MUST emit a `target triple` line taken from the configured target +(see [010 — Driver & CLI](010-driver-cli.md)). + +**R19.** The backend MUST emit a `target datalayout` line derived from that triple, so +the module is self-contained. + +**R20.** The backend MUST emit declarations for all external functions and globals before +any definition. + +**R21.** The backend SHOULD emit deterministic type and global names so that two runs +over the same module produce identical text. + +## 6. Invariants + +- **I1.** The emitted module is parseable by `llvm-as` for LLVM 18 or later. +- **I2.** There is a 1:1 correspondence between VIR blocks and instructions and the + emitted LLVM blocks and instructions. +- **I3.** The module is self-contained: triple, datalayout, and all referenced + declarations are present. +- **I4.** No typed pointers appear anywhere in the output. +- **I5.** Declarations precede definitions. +- **I6.** Output is deterministic for a fixed input and `Config`, at any thread count. +- **I7.** No `!llvm.dbg` metadata appears in v1 output. + +## 7. Examples + +Module scaffolding with a declaration before a definition, opaque pointers, and a `phi`: + +```llvm +; ModuleID = 'demo' +target triple = "x86_64-unknown-linux-gnu" +target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128" + +%struct.Pair = type { i32, i64 } + +declare i32 @extern_fn(ptr) + +define i32 @add(i32 %a, i32 %b) { +entry: + %r = add i32 %a, %b + ret i32 %r +} + +define i32 @loop_sum(ptr %p, i32 %n) { +entry: + br label %loop + +loop: + %i = phi i32 [ 0, %entry ], [ %next, %body ] + %acc = phi i32 [ 0, %entry ], [ %acc.next, %body ] + %cond = icmp slt i32 %i, %n + br i1 %cond, label %body, label %exit + +body: + %v = load i32, ptr %p, align 4, !range !0 + %acc.next = add i32 %acc, %v + %next = add i32 %i, 1 + br label %loop + +exit: + ret i32 %acc +} + +!0 = !{i32 0, i32 100} +``` + +A `Restrict` parameter and an `Assume` lower like this: + +```llvm +define void @copy(ptr noalias %dst, ptr noalias %src) { +entry: + %ok = icmp ne ptr %dst, null + call void @llvm.assume(i1 %ok) + ret void +} +``` + +## 8. Cross-references + +- [000 — Overview](000-overview.md) for determinism and conformance. +- [001 — Architecture](001-architecture.md) for module boundaries. +- [002 — Traits](002-traits.md) for the trait vocabulary consumed here. +- [006 — VIR](006-vir.md) for the input IR and its invariants. +- [007 — VIR Optimizer](007-vir-optimizer.md) for the pass pipeline. +- [008 — C Backend](008-c-backend.md) for the default C path. +- [010 — Driver & CLI](010-driver-cli.md) for the `-target` flag and mode selection. +- [011 — Diagnostics](011-diagnostics.md) for error reporting. diff --git a/spec/010-driver-cli.md b/spec/010-driver-cli.md new file mode 100644 index 0000000..432c524 --- /dev/null +++ b/spec/010-driver-cli.md @@ -0,0 +1,223 @@ +# 010 — Driver & CLI + +- **Status:** Draft +- **Normative language:** `MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are to + be interpreted as described in RFC 2119. + +## 1. Purpose + +libVCT splits orchestration across two layers. The driver (`vct.driver`) is the +library-level orchestrator: it takes emitted C translation units, invokes the system C +compiler, and collects object files. The CLI (`vct.cli`) is a thin wrapper that parses +command-line flags into a `Config`, drives the whole pipeline, formats diagnostics, and +sets exit codes. The CLI holds no optimizer state and adds no compilation logic of its +own. + +## 2. Scope + +This file specifies the driver/CLI role split, the pipeline order, the full flag +surface, the exact C compiler invocation, compiler discovery and probing, error +handling, determinism, `-mangled` mode, and object-file emission and merging. The +compiler-facing backends are specified in [008 — C Backend](008-c-backend.md) and +[009 — LLVM Backend](009-llvm-backend.md). Diagnostics, remarks, and exit codes are +specified in [011 — Diagnostics](011-diagnostics.md). + +## 3. Definitions + +| Term | Meaning | +|---|---| +| `Config` | One struct holding opt level, target, budgets, pass toggles, warning policy, compiler selection, and mode. | +| TU | Translation unit: one emitted C file. | +| `cc` | The discovered system C compiler. | +| APE | Actually Portable Executable, the output of cosmopolitan tooling. | +| Fat LTO object | An object that carries both native machine code and LTO bitcode. | +| Response file | A file listing arguments for a linker or compiler, referenced by `@file`. | + +## 4. Model + +`vct.cli` is GC-allocated and cold-path only. It converts `argv` into a `Config`, runs +the pipeline, forwards every diagnostic to the configured consumer, and returns an exit +code. `vct.driver` owns process invocation. Any embedder that has already constructed a +`Config` can call the driver directly and skip the CLI entirely. + +The driver also owns the `-llvm` path, which returns textual LLVM IR and never touches +`cc`, and the `-mangled` path, which switches to the cosmopolitan toolchain. + +## 5. Normative requirements + +### 5.1 Role split + +**R1.** `vct.cli` MUST be a thin wrapper. It MUST NOT contain optimization, lowering, or +backend logic, and it MUST NOT hold optimizer state. + +**R2.** `vct.cli` MUST be a pure function from `argv` to `Config` plus pipeline +invocation. The library entry point MUST accept a `Config`; programmatic callers MUST be +able to bypass the CLI. + +**R3.** `vct.driver` MUST own all invocation of the system C compiler. + +**R4.** `vct.cli` MUST be GC-allocated and MUST be usable on the cold path only. + +### 5.2 Pipeline + +**R5.** The default pipeline order MUST be: read input, then HIR optimization if enabled, +then lowering, then VIR optimization if enabled, then the C backend, then the driver. + +``` +read input + -> (HIR opt if enabled) + -> lower + -> (VIR opt if enabled) + -> backend.c -> driver -> .o +``` + +**R6.** `-S` MUST stop after C emission and MUST NOT invoke `cc`. + +**R7.** `-emit-llvm` and `-llvm` MUST stop after LLVM IR emission and MUST NOT invoke +`cc`. + +### 5.3 Flag surface + +**R8.** The CLI MUST accept the flags listed in Table 1. + +**Table 1. v1 flag surface** + +| Group | Flags | Meaning | +|---|---|---| +| Optimize | `-O0` `-O1` `-O2` `-O3` `-Ofast` `-Oz` | Optimization level. | +| Target | `-march=`, `-mcpu=`, `-target ` | Target selection; `-march`/`-mcpu` forward to `cc` as cflags. | +| Modes | `-S`, `-emit-llvm`, `-llvm`, `-mangled` | Stop point or alternate mode. | +| Toolchain | `-cc=`, `-cflags="..."`, `-j`, `-save-temps` | Compiler selection, extra flags, parallelism, temp retention. | +| LTO | `-flto[=full\|thin]`, `-ffat-lto-objects` | Link-time optimization control. | +| Debug | `-g`, `--dump-hir`, `--dump-vir`, `--verify`, `--time-passes`, `--stats` | Debug output and instrumentation. | +| Output | `-o ` | Output path. | +| Warnings | `-Wall`, `-Werror`, `-Wextra`, `-Wno-error[=]`, `-w` | Warning policy. | +| Diagnostics | `--remarks`, `--diagnostics=json` | Remark and diagnostic channel selection. | +| Pass toggles | `-f`, `-fno-` | Enable or disable an individual pass. | + +**R9.** `-Wall`, `-Werror`, and `-Wextra` MUST be on by default. A bare `-Wno-error` MUST +disable every promotion, `-Wno-error=` MUST disable the promotion of that +category, and `-w` MUST silence warnings. + +**R10.** `-Werror` MUST apply to the C source that libVCT generates, not only to +frontend input. + +**R11.** Every pass MUST be exposed as both `-f` and `-fno-`. + +### 5.4 C compiler invocation + +**R12.** For each emitted TU, the driver MUST invoke the compiler as: + +``` +cc -std=c17 -c -o -Wall -Werror -Wextra +``` + +where `` is `-g` when debug is enabled, and `` is the LTO flag when LTO is +requested. + +**R13.** The driver MUST run one compiler process per TU, with concurrency bounded by +`-j`. + +**R14.** The driver MUST discover the compiler via the `$CC` environment variable first, +then by searching `PATH`, unless `-cc=` forces a selection. + +**R15.** The driver MUST probe the discovered compiler's version and dialect exactly +once per run. The probed dialect MUST drive the backend's choice of pragmas and builtins +(see [008 — C Backend](008-c-backend.md)). + +**R16.** The driver MUST NOT pass `-march`/`-mcpu` values to `cc` that the probed +compiler does not accept; it MUST diagnose an unsupported target value. + +### 5.5 Error handling + +**R17.** The driver MUST capture the compiler's stderr. + +**R18.** The driver MUST use the `#line` directives emitted by the C backend (see +[008 — C Backend](008-c-backend.md)) to map C compiler errors back to VIR and frontend +source positions, then re-emit them through the diagnostics service (see +[011 — Diagnostics](011-diagnostics.md)). + +**R19.** A compiler failure MUST produce a diagnostic and MUST cause a nonzero exit. + +**R20.** The driver MUST NOT discard compiler diagnostics silently. + +### 5.6 Determinism + +**R21.** The driver MUST order TUs and flags deterministically. For a fixed `Config` and +input, the sequence of compiler invocations and their argument order MUST be identical +across runs and independent of thread count. + +**R22.** The driver MUST place temporary files under a GC-managed temporary directory. + +**R23.** The temporary directory MUST be cleaned up after the run unless `-save-temps` +is given. + +### 5.7 `-mangled` mode + +**R24.** `-mangled` MUST switch the toolchain to `cosmocc` and the cosmopolitan libc. + +**R25.** `-mangled` MUST produce an Actually Portable Executable. + +**R26.** `-mangled` is the one mode in which libVCT links. In every other mode libVCT +produces object files and leaves linking to the caller (see +[000 — Overview](000-overview.md)). + +### 5.8 Object emission and merging + +**R27.** In the default path, the driver MUST produce object files only. It MUST NOT link +a final executable. + +**R28.** When a single `-o` path is requested and the module produced multiple +per-function objects, the driver MUST merge them with a relocatable link +(`ld -r -o .o ...`). + +**R29.** The driver MUST emit, for the caller, a recommended link line or response file +listing the produced objects and any needed libraries. + +## 6. Invariants + +- **I1.** A fixed `Config` and input produce byte-identical outputs and an identical + sequence of compiler invocations. +- **I2.** `vct.cli` holds no optimizer state. +- **I3.** The `-llvm` path never spawns `cc`. +- **I4.** The temporary directory is removed unless `-save-temps` is set. +- **I5.** Flag assembly order is stable. +- **I6.** Every captured compiler error is re-emitted through the diagnostics service. +- **I7.** `-mangled` is the only mode that links. + +## 7. Examples + +Default compile of a single source to an object: + +``` +vct -O2 -march=native -cc=clang -j8 -o out.o input.vir +``` + +Stop after emitting C, without invoking `cc`: + +``` +vct -O2 -S -o out.c input.vir +``` + +Emit textual LLVM IR instead of C: + +``` +vct -O3 -emit-llvm -target x86_64-unknown-linux-gnu -o out.ll input.vir +``` + +Produce a portable executable: + +``` +vct -O2 -mangled -o app.com input.vir +``` + +## 8. Cross-references + +- [000 — Overview](000-overview.md) for determinism and the no-linking rule. +- [001 — Architecture](001-architecture.md) for module boundaries. +- [002 — Traits](002-traits.md) for the trait vocabulary. +- [007 — VIR Optimizer](007-vir-optimizer.md) for pass toggles and the pipeline. +- [008 — C Backend](008-c-backend.md) for `#line` and compiler dialect use. +- [009 — LLVM Backend](009-llvm-backend.md) for `-target` and the `-llvm` path. +- [011 — Diagnostics](011-diagnostics.md) for diagnostics, remarks, and exit codes. +- [013 — Build & Packaging](013-build-packaging.md) for `Config` and the C API. diff --git a/spec/011-diagnostics.md b/spec/011-diagnostics.md new file mode 100644 index 0000000..93f0c3c --- /dev/null +++ b/spec/011-diagnostics.md @@ -0,0 +1,246 @@ +# 011 — Diagnostics + +- **Status:** Draft +- **Normative language:** `MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are to + be interpreted as described in RFC 2119. + +## 1. Purpose + +libVCT has a single central diagnostics service (`vct.diag`). Every module emits +diagnostics through it; no module prints directly to a stream. This gives embedders one +uniform surface for errors, warnings, notes, optimizer remarks, and internal compiler +errors, and it lets the CLI present them in several formats without every module knowing +which format is active. + +## 2. Scope + +This file specifies the diagnostics service, the `Diagnostic` structure and its +severity and code vocabulary, source maps, output channels, optimizer remarks, the +internal-compiler-error policy, and process exit codes. The pipeline that produces these +diagnostics is specified in [010 — Driver & CLI](010-driver-cli.md); the backends that +consume source maps are specified in [008 — C Backend](008-c-backend.md) and +[009 — LLVM Backend](009-llvm-backend.md). + +## 3. Definitions + +| Term | Meaning | +|---|---| +| Diagnostic | One emitted message with severity, code, span, and optional notes and suggestions. | +| Severity | Error, Warning, Note, Remark, or Ice. | +| `DiagCode` | A stable, namespaced identifier such as `VCT1002`. | +| Span | A source range: file, line, column, and length. | +| Source map | The association of HIR nodes with frontend source locations, resolved for VIR through the lowering map. | +| Consumer | A `DiagnosticConsumer` callback that receives emitted diagnostics. | +| ICE | Internal compiler error. | + +## 4. Model + +The service is **emit-only** from the perspective of compilation modules. A module +builds a `Diagnostic` and hands it to the service. The service forwards it to the +configured consumer. Fatal conditions (an error diagnostic, or an ICE) stop the +pipeline at the next safe boundary; they never let compilation continue on invalid +state. + +The service is GC-allocated, consistent with the cold path rule in +[001 — Architecture](001-architecture.md). + +```d +struct Diagnostic { + Severity severity; // Error | Warning | Note | Remark | Ice + DiagCode code; // stable namespaced, e.g. VCT1002 + string message; + Span primary; + Span[] notes; + Suggestion[] suggestions; +} +``` + +## 5. Normative requirements + +### 5.1 Central service + +**R1.** Every module MUST emit diagnostics through the central service. No module MUST +print to standard output or standard error directly. + +**R2.** The service MUST be GC-allocated. + +**R3.** The service MUST forward each emitted diagnostic to the configured consumer in +emission order. + +**R4.** An `Error` or `Ice` diagnostic MUST prevent further compilation on the invalid +path. The pipeline MUST stop at the next safe boundary after such a diagnostic. + +### 5.2 Diagnostic structure + +**R5.** Every diagnostic MUST carry a `Severity` and a `DiagCode`. A diagnostic with no +code MUST NOT be emitted. + +**R6.** `Severity` MUST be one of the values in Table 1. + +**Table 1. Severity levels** + +| Severity | Meaning | +|---|---| +| Error | Compilation cannot proceed correctly. | +| Warning | Suspicious but continuable. | +| Note | Supporting context attached to another diagnostic. | +| Remark | An optimizer decision record. | +| Ice | Internal compiler error. | + +**R7.** A `DiagCode` MUST be a stable, namespaced identifier in the form `VCT` followed +by digits. Once published, a code's meaning MUST NOT change and the code MUST NOT be +renumbered. + +**R8.** A diagnostic MUST carry a primary `Span` whenever a source location is known, and it MAY +carry additional note spans and suggestions. When no location can be resolved, the diagnostic MUST +carry no location instead (see R12). + +**R9.** A `Note` diagnostic MUST be attached to a parent diagnostic rather than emitted +as a standalone top-level message. + +### 5.3 Source maps + +**R10.** The frontend MUST be able to register a `SourceLocation` (file, line, column, +length) against any HIR node. + +**R11.** For VIR entities, the service MUST resolve source locations through the +lowering map that associates HIR nodes with the VIR entities they became (see +[005 — Lowering](005-lowering.md)). + +**R12.** When a location cannot be resolved, the service MUST degrade gracefully to a +diagnostic with no location. It MUST NOT fabricate a location. + +**R13.** The C backend MUST emit `#line` directives from the registered source map (see +[008 — C Backend](008-c-backend.md)), and the driver MUST re-map compiler errors through +it (see [010 — Driver & CLI](010-driver-cli.md)). + +### 5.4 Channels + +**R14.** The service MUST expose a single canonical `DiagnosticConsumer` callback for +embedders. + +**R15.** The CLI MUST provide a human-readable channel that enables color when standard +error is a terminal. + +**R16.** The CLI MUST provide a JSON channel selected by `--diagnostics=json`. + +**R17.** The JSON channel MUST use a versioned schema. The schema version MUST appear in +the output, and the schema MUST be stable within a major version. + +**R18.** A single diagnostic MUST render equivalently in content across all channels; the +channels differ only in presentation. + +Example JSON shape: + +```json +{ + "version": 1, + "diagnostics": [ + { + "severity": "error", + "code": "VCT1002", + "message": "missing trait at lowering", + "file": "input.vox", + "line": 12, + "column": 5, + "length": 3, + "notes": [] + } + ] +} +``` + +### 5.5 Optimizer remarks + +**R19.** Optimizer remarks MUST be off by default and MUST be enabled by `--remarks`. + +**R20.** Optimizer remarks MUST be enabled automatically at `-O3`. + +**R21.** Every remark MUST be tied to a specific trait-pipeline decision, naming the +request or attribute that drove it. + +Examples: + +``` +HIR: replaced 3 uses of node#412 (comptime 64) +VIR: denied Inline on @foo - CostModel (callee 2.4x size budget) +``` + +### 5.6 Internal compiler error policy + +**R22.** The following conditions MUST be treated as internal compiler errors: + +- a missing or contradictory trait encountered at lowering (see + [005 — Lowering](005-lowering.md)); +- a VIR invariant violation detected in a debug build (see [006 — VIR](006-vir.md)); +- a `Strong` request denied for reason `Illegality` or `ContradictsTrait` (see + [007 — VIR Optimizer](007-vir-optimizer.md)). + +**R23.** In release builds, an internal compiler error MUST return a failure result and +emit an `Ice` diagnostic that requests a module dump. + +**R24.** In debug builds, an internal compiler error MUST assert. + +**R25.** An internal compiler error MUST NEVER be silently continued past. + +### 5.7 Warning policy + +**R26.** The trait-mismatch warning raised when a suggested trait conflicts with the +derived value MUST be promoted to an error under `-Werror`. + +**R27.** A frontend MUST be able to opt out of warning promotions: `-Wno-error=` MUST +disable the promotion of that category, and a bare `-Wno-error` MUST disable every promotion. + +### 5.8 Exit codes + +**R28.** The CLI MUST exit with the codes in Table 2. + +**Table 2. Exit codes** + +| Code | Meaning | +|---|---| +| 0 | Success. | +| 1 | Diagnostics present. | +| 2 | Usage or configuration error. | +| 3 | Internal compiler error. | + +**R29.** Exit code 3 MUST take precedence over exit code 1 when an ICE has been emitted. + +## 6. Invariants + +- **I1.** Every emitted diagnostic has a severity and a `DiagCode`. +- **I2.** No compilation module writes directly to a stream. +- **I3.** Diagnostics are emitted and rendered in a deterministic order for a fixed + input and `Config`. +- **I4.** An ICE never resolves to a success exit code. +- **I5.** A location is either resolved or omitted; it is never invented. +- **I6.** The JSON channel's schema version is always present. + +## 7. Examples + +A frontend registers a location and a later lowering failure reports it: + +```d +diag.registerLocation(node, SourceLocation("input.vox", 12, 5, 3)); +// later, during lowering: +emit(Diagnostic(Severity.Error, DiagCode(1002), + "missing trait at lowering", span)); +``` + +A `MustTail` request on GCC becomes a hard error rather than a silent fallback: + +``` +error[VCT1044]: MustTail requires [[clang::musttail]]; current compiler is GCC + --> input.vox:7:5 +``` + +## 8. Cross-references + +- [000 — Overview](000-overview.md) for conformance and determinism. +- [001 — Architecture](001-architecture.md) for the cold-path GC rule. +- [002 — Traits](002-traits.md) for the trait vocabulary and mismatch handling. +- [005 — Lowering](005-lowering.md) for the lowering map and ICE conditions. +- [006 — VIR](006-vir.md) for the invariants checked in debug. +- [007 — VIR Optimizer](007-vir-optimizer.md) for request denials and `DenyReason`. +- [008 — C Backend](008-c-backend.md) for `#line` emission. +- [010 — Driver & CLI](010-driver-cli.md) for `--remarks`, `--diagnostics`, and exit codes. diff --git a/spec/012-testing.md b/spec/012-testing.md new file mode 100644 index 0000000..9422a62 --- /dev/null +++ b/spec/012-testing.md @@ -0,0 +1,203 @@ +# 012 — Testing + +- **Status:** Draft +- **Normative language:** `MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are to be + interpreted as described in RFC 2119. + +## 1. Purpose + +This file defines how an implementation demonstrates conformance. It specifies the test layers, +the fixture strategy, the structural verifier, the differential oracle, the fuzzing obligations, +and the CI guarantees. It states **what must be tested and what must hold**, not how to write the +test code. + +## 2. Scope + +The obligations here apply to every stage: the HIR optimizer, lowering, the VIR optimizer, both +backends, the driver, and the diagnostics service. The determinism guarantee from +[000 — Overview](000-overview.md) is tested here, but it is defined there. + +## 3. Definitions + +| Term | Meaning | +|---|---| +| **Fixture** | A fully constructed, trait-complete HIR module used as input to a test. | +| **Fixture layer** | The harness that constructs fixtures without a frontend. | +| **Layer** | One category of test with a distinct input source and success criterion. | +| **Verifier** | The component that checks SSA well-formedness and trait consistency. | +| **Oracle** | A trusted implementation used to judge another implementation's output. | +| **Round-trip property** | `print(parse(print(m))) == print(m)` for a module `m`. | +| **FileCheck test** | A test whose expectation is expressed as ordered pattern directives. | +| **Regression corpus** | The checked-in set of input IR plus expected directives. | +| **Matrix** | The cross-product of compiler, optimization level, and backend under test. | + +## 4. Core constraint: VIR cannot be tested without a valid HIR fixture + +VIR is a strict consumer of traits and performs no discovery (see [006 — VIR](006-vir.md) and +[007 — VIR Optimizer](007-vir-optimizer.md)). A VIR module cannot be authored meaningfully by +hand, because its facts come from HIR. Testing therefore starts from HIR. + +1. Every component that consumes traits **MUST** be exercised through HIR fixtures that carry + complete, epoch-valid traits. A test **MUST NOT** assert behavior on VIR whose traits are + incomplete or stale, except a test that deliberately targets the verifier or the + internal-compiler-error path. +2. `vct.test.hirbuild` **MUST** be the fixture layer for HIR optimization, lowering, VIR + optimization, and both backends. +3. `vct.test.hirbuild` **MUST** construct modules, functions, nodes, types, values, and traits + without a frontend, and the modules it produces **MUST** pass the verifier. + +## 5. Test layers + +| Layer | Input | Success criterion | Required in v1 | +|---|---|---|---| +| Unit | D `unittest` per module | Function-level assertions | Yes | +| Fixture | HIR builder modules | Component behaves as specified on annotated HIR | Yes | +| Round-trip | Textual IR | `print(parse(print(m))) == print(m)` | Yes | +| FileCheck | IR plus directives | Ordered pattern directives match | Yes | +| Verifier | Any IR at any stage | SSA and traits are consistent | Yes | +| Differential | VIR plus host harness | Interpreter and compiled object agree | Yes | +| Fuzz | Random HIR and traits | See the properties below | Yes | +| Matrix | Regression corpus | All matrix configurations pass | Yes | +| Performance | Benchmark programs | Codegen quality and compile time tracked | Yes | + +### 5.1 Unit tests + +4. Each module **MUST** ship D `unittest` blocks covering its public and internal entry points. + +### 5.2 HIR test-builder harness + +5. The harness **MUST** produce input that is indistinguishable, for the consuming stage, from a + frontend-produced module. It **MUST** allow a test to set or suggest traits explicitly so that + trait validation, overwrite, and staleness behavior can be tested. +6. Tests for lowering and VIR optimization **MUST** build their HIR through the harness and run + the HIR optimizer first, so VIR sees the same complete annotation it sees in production. + +### 5.3 Textual round-trip + +7. For every module in the regression corpus, the textual form **MUST** satisfy + `print(parse(print(m))) == print(m)`. Parsing followed by printing **MUST** be lossless with + respect to the printed form. +8. The textual form exists for tools and round-trip tests. It is not a supported authoring + surface (see [000 — Overview](000-overview.md)); tests **MUST NOT** rely on hand-authored + textual VIR by itself as a source of traits. + +### 5.4 FileCheck-style matching + +9. A test **MUST** express its expected output using `CHECK`, `CHECK-NOT`, and `CHECK-NEXT` + directives against captured stage output. +10. The matcher **MUST** enforce directive order: `CHECK` lines match in sequence, `CHECK-NEXT` + requires a match on the line immediately following the previous match, and `CHECK-NOT` + requires that its pattern does **not** occur between the surrounding matches. +11. CI **MUST** fail a test when a `CHECK-NOT` pattern matches. Exact golden files **SHOULD NOT** + be used, with the sole exception of byte-exact determinism comparisons. + +### 5.5 Structural verifier + +12. A structural verifier **MUST** check SSA well-formedness and trait consistency. It **MUST** + run after every pass in debug builds, and it **MUST** be invocable on demand through + `--verify` in release builds (see [010 — Driver & CLI](010-driver-cli.md)). +13. The verifier **MUST** check at least these SSA properties, as required by + [006 — VIR](006-vir.md): + - every use is dominated by its definition; + - every basic block is terminated; + - every phi node has exactly one operand per predecessor block; + - every value has a single definition. +14. The verifier **MUST** check trait consistency: traits are complete for the current epoch, and + no two retained traits contradict one another (see [002 — Traits](002-traits.md)). +15. A failed verification **MUST** be an internal compiler error: it **MUST** assert in debug + builds and **MUST** produce an ICE diagnostic plus a failure result in release builds (see + [011 — Diagnostics](011-diagnostics.md)). It **MUST NOT** be silently ignored. + +### 5.6 Differential testing + +16. A VIR reference interpreter **MUST** be built in v1. It is the differential oracle. +17. For each differential test, the interpreter **MUST** run the same VIR that the backend is + given, the compiled object **MUST** be run on the same inputs, and the two observable results + **MUST** match. Observable results include defined output and exit status; unobservable + internal state is excluded. +18. Where the LLVM path is exercised, the C path and the LLVM path **SHOULD** be compared + differentially for equivalent observable behavior. + +### 5.7 Fuzzing + +19. Random-HIR and random-trait fuzzers **MUST** exist in v1. +20. Every fuzz input **MUST** satisfy all of these properties: + - the pipeline terminates within its configured budget; + - the verifier reports clean after every pass; + - emitted C, and emitted LLVM IR when the LLVM path is active, compiles; + - for deterministic programs, the interpreter and the compiled object agree. +21. A crash, a verifier failure, non-termination, or non-compiling output **MUST** be treated as + a bug and **MUST** block merge. + +### 5.8 Test matrix + +22. CI **MUST** test at least GCC and Clang. Each compiler **MUST** be tested across `-O0`, + `-O1`, `-O2`, `-O3`, `-Ofast`, and `-Oz`, and across the C backend and the LLVM backend where + the backend applies. + +### 5.9 Performance benchmarks + +23. Benchmarks **MUST** track both codegen quality and compile time. Compile time is measured + through `--time-passes` and `--stats` (see [010 — Driver & CLI](010-driver-cli.md)). +24. Because codegen quality is the headline goal (see [000 — Overview](000-overview.md)), a + regression in generated-code quality **MUST** be surfaced even when compile time improves. + +## 6. Determinism testing + +25. Every pipeline run in CI **MUST** be executed at least twice with identical `Config` and + identical input, and the two outputs **MUST** be byte-identical. +26. CI **MUST** vary the thread count across runs, including at least one run with `-j1` and one + with `-j` greater than one, to demonstrate independence from thread count. +27. The determinism guarantee itself is defined in [000 — Overview](000-overview.md). A + nondeterministic result **MUST** be treated as a correctness bug even if the outputs are + semantically equivalent. + +## 7. Regression corpus + +28. The regression corpus **MUST** store input IR paired with expected FileCheck patterns. +29. CI **MUST** run the full matrix over the corpus. A failing corpus entry **MUST** block merge. +30. A bug fix **MUST** add a corpus entry that fails before the fix and passes after it. + +## 8. Invariants + +- A green suite **MUST** mean that every requirement in this file held for the tested revision. +- The verifier **MUST** be sound: it **MUST NOT** report clean on malformed SSA or on + contradictory traits. A false negative is itself a bug. +- The differential oracle **MUST** be conservative: disagreement between the interpreter and the + compiled object **MUST** be reported as a failure, not explained away. + +## 9. Examples + +A round-trip test asserts: + +``` +assert(print(parse(print(m))) == print(m)); +``` + +A FileCheck test names the stage output it inspects and lists its directives: + +``` +; RUN: vct --dump-vir --verify %s | FileCheck %s +; CHECK: define i32 @main( +; CHECK-NEXT: entry: +; CHECK-NOT: phi +; CHECK: ret i32 0 +``` + +The first `CHECK` matches the function header, `CHECK-NEXT` requires the entry label on the very +next line, `CHECK-NOT` asserts that no phi node appears before the return, and the final `CHECK` +matches the return instruction. + +## 10. Cross-references + +- [000 — Overview](000-overview.md) for the determinism guarantee. +- [002 — Traits](002-traits.md) for trait completeness, epochs, and the verifier's trait checks. +- [004 — HIR Optimizer](004-hir-optimizer.md) for the cascade under test. +- [005 — Lowering](005-lowering.md) for SSA output invariants. +- [006 — VIR](006-vir.md) for the SSA properties the verifier checks. +- [007 — VIR Optimizer](007-vir-optimizer.md) for request results under test. +- [008 — C Backend](008-c-backend.md) and [009 — LLVM Backend](009-llvm-backend.md) for the + emitted forms checked by FileCheck and the matrix. +- [010 — Driver & CLI](010-driver-cli.md) for `--verify`, `--time-passes`, and `--stats`. +- [011 — Diagnostics](011-diagnostics.md) for the ICE policy the verifier feeds. +- [013 — Build & Packaging](013-build-packaging.md) for how `vct.test.hirbuild` ships. diff --git a/spec/013-build-packaging.md b/spec/013-build-packaging.md new file mode 100644 index 0000000..4e71e61 --- /dev/null +++ b/spec/013-build-packaging.md @@ -0,0 +1,185 @@ +# 013 — Build & Packaging + +- **Status:** Draft +- **Normative language:** `MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are to be + interpreted as described in RFC 2119. + +## 1. Purpose + +This file specifies how libVCT is built, what it produces, what it exposes to embedders, and who +owns memory across the API boundary. It also fixes the configuration, threading, versioning, and +determinism contracts that the rest of the specification depends on. + +## 2. Scope + +Covers the toolchain, the build system, the shipped artifacts, the two API surfaces, module +visibility, arena ownership, the `Config` struct, the threading contract, versioning, and the +determinism guarantee. It does not restate stage semantics; those live in the stage files. + +## 3. Definitions + +| Term | Meaning | +|---|---| +| **C API** | The `extern(C)` surface exposed to non-D callers. | +| **D API** | The native D surface; the implementation itself. | +| **Shim** | The thin translation layer implementing the C API over the D API. | +| **Arena** | A region of memory reclaimed as a unit, not per object. | +| **Bump allocator** | The default in-tree allocator that serves requests by advancing a pointer. | +| **Cold path** | Code that runs at most a few times per compilation: driver, diagnostics, CLI. | + +## 4. Toolchain + +1. The library **MUST** be written in D. +2. Release builds **MUST** use LDC2, chosen for generated-code quality. +3. Debug builds **MUST** use DMD, chosen for compile and iteration speed. +4. libVCT **MUST** have zero runtime dependencies beyond its own code. It **MUST NOT** require a + third-party library at build time or at load time. +5. A D compiler is required to build libVCT. A system C compiler is required only when the C path + is used, and it **MUST** be discovered at runtime (see [010 — Driver & CLI](010-driver-cli.md)). + Prebuilt binaries **MUST NOT** require a D compiler at load time. + +## 5. Build system + +6. The build system **MUST** be xmake. A single `xmake.lua` **MUST** drive every target. +7. The build system **MUST NOT** be dub. +8. The `release` profile **MUST** select LDC2. The `debug` profile **MUST** select DMD. +9. Building libVCT **MUST NOT** require an external package manager. + +## 6. Artifacts + +10. A conforming build **MUST** produce all of the following: + +| Artifact | Kind | Purpose | +|---|---|---| +| `libvct.a` | static library | Static linking of the full library. | +| `libvct.so` | shared library | Dynamic linking; embedders load it at runtime. | +| `vct` | executable | The command-line driver (see [010 — Driver & CLI](010-driver-cli.md)). | +| `vctc.h` | C header | The declarations of the C API. | +| `vct.test.hirbuild` | D module | The fixture harness, shipped publicly (see [012 — Testing](012-testing.md)). | + +11. The static and shared libraries **MUST** expose the same symbols and behavior; choosing one + over the other **MUST NOT** change observable results. + +## 7. Public API surfaces + +12. The **C API** **MUST** be the primary public surface in v1. It **MUST** be declared + `extern(C)` so that C, C++, and other FFI callers can use it without a D toolchain. +13. The C API **MUST** represent library state through opaque handles for `Context`, `Module`, + `Builder`, and `Config`. Callers **MUST NOT** depend on the layout of these types, and the + header **MUST NOT** expose their internal fields. +14. The C API **MUST** provide operations to build IR, run the pipeline, and query diagnostics. +15. The **D API** **MUST** be the full-feature implementation surface. The C API is a shim over + it. +16. The shim **MUST NOT** contain logic beyond argument marshalling, error translation, and + handle conversion. Every behavior reachable through the C API **MUST** be implemented in the + D API, so that every C API call exercises the D API. +17. An implementation **MUST NOT** place optimizer, lowering, or backend logic in the header or + the shim. + +An illustrative header shape (not a complete declaration set): + +```c +typedef struct vct_context vct_context; +typedef struct vct_module vct_module; +typedef struct vct_builder vct_builder; +typedef struct vct_config vct_config; + +vct_context *vct_context_create(const vct_config *cfg); +void vct_context_destroy(vct_context *ctx); +``` + +## 8. Module visibility + +18. The following modules **MUST** be public: `vct.ir.hir`, `vct.traits`, `vct.context`, the + entry points of `vct.backend.c` and `vct.backend.llvm`, `vct.driver`, `vct.diag`, and + `vct.test.hirbuild`. +19. The following modules **MUST** be internal: `vct.ir.vir`, `vct.lower`, `vct.hir.opt`, + `vct.vir.opt`, and the internals of `vct.comptime`. +20. A public module **MUST NOT** expose internal types in its public signatures in a way that + forces embedders to depend on internal layout. The trait vocabulary and node identity cross + this boundary and are versioned accordingly (see section 12). + +## 9. Memory ownership and lifecycle + +21. A `Context` **MUST** own one or more arenas. An arena is the unit of reclamation. The default + is one arena per `Module`, freed wholesale. +22. All IR and optimizer objects **MUST** be arena-owned and non-GC. Passes **MUST** mutate them + in place under epoch guards (see [002 — Traits](002-traits.md)). +23. Garbage collection **MUST** be permitted only on cold paths: the driver, diagnostics, and the + CLI. The optimizer, lowering, and backends **MUST NOT** depend on the GC for IR objects. +24. The `Context` **MUST** accept an `ArenaAllocator` interface so embedders can supply their own + backing memory. The interface **MUST** provide exactly these operations: + - `allocate(size, alignment)` returns uninitialized storage owned by the arena; callers MUST + NOT assume zeroed storage; + - `reset()` returns the arena to its initial state so it can be reused; + - `destroy()` releases the arena's backing memory. +25. A default bump allocator **MUST** ship in-tree and **MUST** be the allocator used when the + embedder supplies none. +26. Ownership rules: + - Objects allocated into a `Context` or arena live until that arena is reset or destroyed. + - A caller **MUST NOT** free an individual arena-owned object. Freeing happens only through + `reset` or `destroy`. + - A pointer or handle returned across the API **MUST** remain valid until the owning arena is + reset or destroyed. + - Use of an object after its arena has been reset or destroyed is undefined, and the + implementation **MUST NOT** be required to detect it. + +## 10. Configuration + +27. A single `Config` **MUST** carry every setting that affects output. At minimum it **MUST** + carry: + +| Field group | Contents | +|---|---| +| Optimization level | `-O0`, `-O1`, `-O2`, `-O3`, `-Ofast`, `-Oz`. | +| Target | `-march`, `-mcpu`, `-target `. | +| Budgets | comptime and HIR budgets (see [004 — HIR Optimizer](004-hir-optimizer.md)) and VIR budgets (see [007 — VIR Optimizer](007-vir-optimizer.md)). | +| Pass toggles | enable and disable for every pass exposed as `-f` and `-fno-`. | +| Warning policy | `-Wall`, `-Wextra`, `-Werror`, `-Wno-error[=]`, `-w`. | +| C compiler selection | `-cc=gcc` or `-cc=clang`, or automatic discovery. | +| Mode | `-S`, `-emit-llvm`/`-llvm`, or `-mangled`. | + +28. The CLI **MUST** be a pure function from `argv` to a `Config`. The library **MUST** take a + `Config` and **MUST NOT** read process arguments itself. +29. Programmatic callers **MUST** be able to bypass the CLI and construct a `Config` directly. +30. The determinism guarantee (section 13) is stated relative to a fixed `Config`. + +## 11. Threading + +31. libVCT **MUST** be thread-safe when each compilation unit has its own `Context` and arenas. +32. libVCT **MUST NOT** hold shared mutable global state. +33. Intra-module parallelism is internal and **MUST** be bounded by the `-j` setting. The + observed output **MUST NOT** depend on `-j` (see [000 — Overview](000-overview.md)). + +## 12. Versioning + +34. The library **MUST** follow semantic versioning. +35. The textual IR format **MUST** carry its own version, independent of the library version. +36. The trait vocabulary **MUST** carry its own version. Adding an attribute or request is a + backward-compatible change; changing the meaning of an existing one is a major change. +37. Experimental passes **MUST** be gated behind feature flags so that default behavior stays + stable within a minor version. + +## 13. Determinism guarantee + +38. For a fixed `Config` and a fixed input, libVCT **MUST** produce byte-identical output on every + run, independent of thread count. This binds HIR optimization, lowering, VIR optimization, + and both backends. It is a correctness property, not an optimization. +39. Determinism **MUST** be demonstrated in CI as specified in [012 — Testing](012-testing.md). + +## 14. Invariants + +- The C API and the D API **MUST** expose the same behavior; a divergence is a bug in the shim. +- An arena **MUST** be reclaimable as a unit; no arena-owned object may outlive its arena. +- A stage **MUST NOT** depend on shared mutable global state. +- The same `Config` and input **MUST** yield the same bytes regardless of `-j`. + +## 15. Cross-references + +- [000 — Overview](000-overview.md) for goals, non-goals, and the determinism guarantee. +- [001 — Architecture](001-architecture.md) for the module map and interfaces. +- [002 — Traits](002-traits.md) for epochs and the trait vocabulary version. +- [004 — HIR Optimizer](004-hir-optimizer.md) for budget names and defaults. +- [007 — VIR Optimizer](007-vir-optimizer.md) for VIR budget names and defaults. +- [010 — Driver & CLI](010-driver-cli.md) for flag semantics and C compiler discovery. +- [012 — Testing](012-testing.md) for determinism tests and the fixture harness. diff --git a/spec/014-worked-example.md b/spec/014-worked-example.md new file mode 100644 index 0000000..953ae21 --- /dev/null +++ b/spec/014-worked-example.md @@ -0,0 +1,230 @@ +# 014 — Worked Example + +- **Status:** Draft +- **Normative language:** `MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are to be + interpreted as described in RFC 2119. + +## 1. Purpose + +This file walks one small program from the HIR builder through the comptime cascade, trait +derivation, request handling, lowering, VIR optimization, and C emission. It is **normative for +trait and request semantics**: a conforming implementation **MUST** reach the trait states, +request outcomes, and observable result described here. Where a stage file gives the general +rule, this file gives the concrete instance that fixes its meaning. + +## 2. Scope + +The example covers the HIR optimizer, the trait vocabulary, lowering, the VIR optimizer's +handling of one request, and the C backend. It does not restate the full grammar of HIR or VIR, +and it is not a conformance test by itself. It is the reference walkthrough that other files cite. + +## 3. The input program + +``` +fn foo(x: int, y: int) -> int { + return x + y; +} + +fn main() -> int { + x = 4; + y = x + 4; + z = foo(x, y); + a = sqrt(z); + println(a); +} +``` + +A frontend builds this through the HIR builder. The builder calls are illustrative; their shapes +show how nodes and values enter HIR: + +```d +auto m = ModuleBuilder("foo"); +auto fn = m.function("foo", [intTy, intTy], intTy); +auto body = fn.block(); +body.ret(body.add(fn.param(0), fn.param(1))); + +auto main_ = m.function("main", [], intTy); +auto b = main_.block(); +auto x = b.local("x", b.literal(4)); +auto y = b.local("y", b.add(x, b.literal(4))); +auto z = b.local("z", b.call("foo", x, y)); +auto a = b.local("a", b.call("sqrt", z)); +b.expr(b.call("println", a)); +``` + +The frontend **MAY** suggest traits while building. HIR validates each suggestion and overwrites a +wrong value with the derived one, promoting the mismatch to an error under `-Werror` (see +[002 — Traits](002-traits.md)). + +## 4. Rules exercised + +| Rule | Statement | Defined in | +|---|---|---| +| **R1** | When every operand of a pure primitive is `is_comptime`, the evaluator computes `const_value` and sets `is_comptime` and `is_constant`. | [004 — HIR Optimizer](004-hir-optimizer.md) | +| **R2** | Before substituting a constant for a node, HIR **MUST** prove the node is not addressed and not runtime-mutable. | [002 — Traits](002-traits.md), [004 — HIR Optimizer](004-hir-optimizer.md) | +| **R3** | A `is_comptime` node with `const_value = v` has every use replaced by the literal `v`. | [004 — HIR Optimizer](004-hir-optimizer.md) | +| **R4** | A node with no remaining uses and no side effects **MUST** be deleted. | [004 — HIR Optimizer](004-hir-optimizer.md) | +| **R5** | Every node whose operands became constant **MUST** be re-enqueued on the worklist. | [004 — HIR Optimizer](004-hir-optimizer.md) | +| **R6** | HIR **MUST NOT** inline. It proves smallness and issues an `Inline` request to VIR. | [004 — HIR Optimizer](004-hir-optimizer.md), [007 — VIR Optimizer](007-vir-optimizer.md) | +| **R7** | Every request **MUST** yield a `RequestResult`; an accepted request needs no reason, and a denial **MUST** carry a `DenyReason` and a one-line note. | [002 — Traits](002-traits.md) | +| **R8** | Each VIR entity **MUST** carry the `Trait` of its source HIR node; relationship endpoints are rewritten through the `LoweringMap`. | [005 — Lowering](005-lowering.md) | +| **R9** | A scalar that is not addressed, not escaping, and not runtime-mutable **MUST** lower to a pure SSA value. | [005 — Lowering](005-lowering.md) | +| **R10** | A comptime value **MUST** be emitted as a literal, not recomputed. | [008 — C Backend](008-c-backend.md) | + +## 5. The canonical cascade shape + +The cascade is the chain of rewrites triggered by one fold. Its canonical textbook shape uses +`4 * 16`: + +``` +n = 4 * 16 + -> const 64 (R1; trigger ConstantUnfolded) + -> mark is_comptime / is_constant / const_value = 64 + -> prove not addressed / not runtime-mutable (R2) + -> replace all uses with literal 64 (R3; trigger UsesReplaced) + -> node dead -> delete (R4; trigger NodeDeleted) + -> downstream nodes now constant -> enqueue (R5; trigger TraitChanged) +``` + +Each hop emits a trigger. The worklist scheduler dedupes `(pass, node)` pairs and runs to a +fixpoint, bounded by the per-function and per-module rewrite budgets (see +[004 — HIR Optimizer](004-hir-optimizer.md)). Step 6 of the walkthrough below is exactly this +shape applied to `y = x + 4`, and the same shape recurs for `z`, `a`, and the argument of +`println`. + +## 6. Worked cascade for `foo` / `main` + +The steps below run per function, in deterministic order. Each step names the rule it exercises. + +1. **Build.** `main` binds `x` to the literal `4`. The literal already carries + `const_value = 4`; HIR sets `is_comptime = true` and `is_constant = true` (R1). +2. **Prove `x`.** HIR proves `x` is not addressed and not runtime-mutable: `is_addressed = false`, + `is_runtime_mutable = false`, `escapes = false`, `is_mutably_used = false` (R2). +3. **Replace uses of `x`.** Every use of `x` becomes the literal `4`. Trigger `UsesReplaced` + (R3). +4. **Delete `x`.** `x` has no uses and no side effects; it is deleted. Trigger `NodeDeleted` + (R4). +5. **Re-enqueue.** `y = x + 4` is now `y = 4 + 4`, so `y` is enqueued. Trigger `TraitChanged` + (R5). +6. **Fold `y`.** The evaluator computes `4 + 4 = 8`, and sets `y.const_value = 8`, + `y.is_comptime = true`, `y.is_constant = true`. Trigger `ConstantUnfolded` (R1). +7. **Prove and replace `y`.** R2 holds for `y`; R3 rewrites its uses to `8`; R4 deletes `y`. +8. **Re-enqueue `z`.** `z = foo(x, y)` is now `z = foo(4, 8)`; `z` is enqueued (R5). +9. **Fold `z`.** `foo` is pure, non-recursive, and its body evaluates with primitives only, so + the evaluator interprets the call with arguments `4` and `8` and computes `12`. `z` becomes + `is_comptime = true`, `is_constant = true`, `const_value = 12` (R1). This is constant + evaluation, not inlining: HIR computes a value and does not rewrite `foo` into `main` + (R6). +10. **Request inlining.** Separately, HIR proves `foo` is small and issues an `Inline` request on + the call edge. HIR does not perform the inlining (R6). +11. **Prove, replace, delete `z`.** R2 holds; R3 rewrites the use in `a = sqrt(z)` to + `a = sqrt(12)`; R4 deletes `z`. +12. **Fold `a`.** `sqrt` is a primitive and its argument is `is_comptime`, so the evaluator + computes a `double` result: `a.const_value = 3.4641016151377544`, `a.is_comptime = true`, + `a.is_constant = true` (R1). The other float arithmetic in this example is exact; the square + root is not, and the field holds the IEEE 754 binary64 value of `sqrt(12)`, printed with the + shortest decimal string that round-trips. +13. **Replace and delete `a`.** R3 rewrites the argument of `println` to + `3.4641016151377544`; R4 deletes `a`. +14. **Collapse.** What remains of `main` is `println(3.4641016151377544)`. The `foo` function now + has no call site. HIR does not delete functions; that is a VIR/IPA concern. + +The worklist drains when no node changes, which here is after step 14. + +## 7. Traits at completion + +The derived traits that survive the cascade are: + +| Node | `ty` | `const_value` | `is_comptime` | `is_constant` | `is_addressed` | `is_runtime_mutable` | `escapes` | +|---|---|---|---|---|---|---|---| +| `x` (deleted) | `int` | `4` | true | true | false | false | false | +| `y` (deleted) | `int` | `8` | true | true | false | false | false | +| `z` (deleted) | `int` | `12` | true | true | false | false | false | +| `a` (deleted) | `double` | `3.4641016151377544` | true | true | false | false | false | +| `println` arg | `double` | `3.4641016151377544` | true | true | false | false | false | +| `foo` | function | absent | false | false | false | false | false | + +`foo.is_used` falls to `false` once its only call is folded. `is_constant` without `is_comptime` +is possible in general; every value in this example is comptime. + +## 8. Requests and their results + +| Request | Target | Strength | Requested because | Result | +|---|---|---|---|---| +| `Inline` | call edge of `foo` in `main` | Soft | `foo` proven small (R6) | denied: `AlreadyDone` (call folded in HIR; no live call remains) (R7) | + +The `Inline` request is soft, so VIR **MAY** decline it; here it declines with `AlreadyDone` +because HIR already folded the only call site, so no live call remains to inline. A denial +**MUST** carry a `DenyReason` and a one-line note, for example +`denied Inline on @foo, reason CostModel, note "callee exceeds size budget"`. No conflicting +`Strong` request is issued in this example, and `NoInline` **MUST NOT** be requested for `foo`. +Recording the request as accepted would violate O-28 of +[007 — VIR Optimizer](007-vir-optimizer.md), which forbids marking a request accepted when no +transform was performed. VIR removes the now-unused `foo` through global dead-code elimination. + +## 9. Lowering to VIR + +Lowering is a pure translation. It builds a `LoweringMap` from HIR nodes to VIR entities and +copies each node's `Trait` onto its VIR entity (R8). Each local here satisfies R9, so it lowers +to a pure SSA value rather than an `alloca`. An illustrative VIR listing for the optimized +`main`, plus the still-present `foo`, is: + +``` +define i32 @foo(i32 %x, i32 %y) { +entry: + %s = add i32 %x, %y + ret i32 %s +} + +define i32 @main() { +entry: + %r = call i32 @println(double 3.4641016151377544) + ret i32 %r +} +``` + +The textual form is shown for clarity only. It is not an authoring surface (see +[000 — Overview](000-overview.md)); the same module can be expressed with the builder or as +opaque binary structures. + +VIR optimization resolves the `Inline` request (denied `AlreadyDone`), runs global DCE, and +leaves the call to `println`. It does not re-derive traits; every fact it uses came from HIR (see +[007 — VIR Optimizer](007-vir-optimizer.md)). + +## 10. Emitted C + +The C backend emits C17. Because the argument is comptime, it is emitted as a literal (R10), and +the irreducible shape collapses to a single call: + +```c +extern int println(double); + +int main(void) { + return println(3.4641016151377544); +} +``` + +The exact spelling of the float literal is implementation-defined as long as it round-trips to +the same binary64 value; the value `3.4641016151377544` is normative. + +## 11. Invariants + +- A conforming implementation **MUST** derive the trait values in section 7 from the section 3 + input. +- A conforming implementation **MUST** issue an `Inline` request for `foo` and **MUST NOT** + inline it in HIR. +- A conforming implementation **MUST** reach the observable behavior of `println` called once + with the binary64 value of `sqrt(12)`, whatever its internal pipeline order. +- Running the example twice with the same `Config` **MUST** produce byte-identical output (see + [000 — Overview](000-overview.md)). + +## 12. Cross-references + +- [000 — Overview](000-overview.md) for the two-stage thesis and the determinism guarantee. +- [002 — Traits](002-traits.md) for attributes, requests, `RequestResult`, and `DenyReason`. +- [004 — HIR Optimizer](004-hir-optimizer.md) for the cascade, triggers, and budgets. +- [005 — Lowering](005-lowering.md) for the `LoweringMap`, SSA construction, and the memory + model behind R9. +- [006 — VIR](006-vir.md) for the SSA and CFG shapes used in section 9. +- [007 — VIR Optimizer](007-vir-optimizer.md) for request handling and global DCE. +- [008 — C Backend](008-c-backend.md) for emission and literal formatting. diff --git a/spec/015-open-questions.md b/spec/015-open-questions.md new file mode 100644 index 0000000..2c3e067 --- /dev/null +++ b/spec/015-open-questions.md @@ -0,0 +1,159 @@ +# 015 — Open Questions + +- **Status:** Draft +- **Normative language:** `MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are to be + interpreted as described in RFC 2119. + +## 1. Purpose + +This file records the capabilities that v1 deliberately defers. Each item is stated as an open +question with the reason for deferral and the work needed to resolve it. The purpose is to +separate "not yet designed" from "required and specified", so that readers and implementors do +not mistake a gap for a guarantee. + +## 2. Scope and conformance status + +Everything in this file is **out of scope for v1**. These items are **not** conformance +requirements, and they add no behavior an implementation must provide. + +1. An implementation **MUST NOT** be judged non-conforming for omitting any item in this file. +2. An implementation **MAY** implement any item here. Doing so **MUST NOT** change the behavior + required by the stage files for the features that are in scope. +3. The absence statements already in [000 — Overview](000-overview.md), such as the absence of + LLVM debug metadata on the LLVM path, describe the v1 scope. They **MUST NOT** be read as + forbidding a future extension. +4. When an item is resolved, it **MUST** be promoted into a stage file with its own normative + requirements; it **MUST NOT** remain only here. + +## 3. Summary + +| ID | Open question | v1 status | Would touch | +|---|---|---|---| +| OQ1 | LLVM debug info (`!llvm.dbg`) | Deferred | [009 — LLVM Backend](009-llvm-backend.md) | +| OQ2 | Windows and MSVC support | Out of scope | [008 — C Backend](008-c-backend.md), [010 — Driver & CLI](010-driver-cli.md) | +| OQ3 | LLVM bitcode emission | Deferred | [009 — LLVM Backend](009-llvm-backend.md) | +| OQ4 | Incremental and cached compilation | Out of scope | [010 — Driver & CLI](010-driver-cli.md), [013 — Build & Packaging](013-build-packaging.md) | +| OQ5 | Cross-compilation beyond passthrough | Future work | [009 — LLVM Backend](009-llvm-backend.md), [010 — Driver & CLI](010-driver-cli.md) | + +## 4. OQ1: LLVM debug information + +**Question.** Should the LLVM backend emit `!llvm.dbg` metadata so that debuggers can map machine +state back to frontend source? + +**v1 behavior.** No. The LLVM backend emits no debug metadata. Debug mapping exists only on the C +path, through `#line` directives (see [008 — C Backend](008-c-backend.md)). + +**Why deferred.** The C path already provides source mapping for the v1 target, and LLVM debug +metadata is a large, version-sensitive surface. Emitting it correctly needs `DICompileUnit`, +`DIFile`, `DISubprogram`, `DILocation`, and scope chain construction, plus a verifier pass that +checks the metadata graph. That is a subsystem of its own, and getting it half-right produces +misleading debuggers, which is worse than no debug info. + +**What resolution requires.** + +- Define how VIR source locations resolve through the `LoweringMap` and the registered source + map (see [011 — Diagnostics](011-diagnostics.md)). +- Lower those locations to LLVM metadata on the LLVM 18+ schema, including lexical block scopes. +- Add a verification step over the metadata graph, and test it with `llvm-dwarfdump` and a + debugger round-trip in CI. +- Decide whether debug metadata is emitted only under `-g` and how it interacts with the + determinism guarantee. + +## 5. OQ2: Windows and MSVC support + +**Question.** Should libVCT support Windows as a host or target, and MSVC as the C compiler? + +**v1 behavior.** No. v1 is Linux/POSIX-first and supports GCC and Clang only (see +[000 — Overview](000-overview.md)). + +**Why deferred.** The C backend leans on GCC and Clang constructs: `__builtin_expect`, +`__builtin_assume_aligned`, `__builtin_assume`, `__asm__` fences, `#[[]]` attributes, and +`#pragma` loop hints. MSVC lacks most of these and spells the rest differently. Beyond the C +surface, the driver's compiler discovery, response-file handling, path rules, and ABI +(`dllexport`/`dllimport`, COFF) all differ. Cosmopolitan mode covers a subset of Windows +execution but not MSVC as a backend compiler. + +**What resolution requires.** + +- Define an MSVC dialect layer: per-construct mappings for hints, attributes, fences, and + pragmas, with a portable fallback where no equivalent exists. +- Extend the driver with `cl.exe` discovery, argument syntax, and error parsing. +- Specify the Windows ABI surface, including exported symbol decoration and the C runtime. +- Add a Windows CI matrix, which also forces decisions about the xmake build on Windows. + +## 6. OQ3: LLVM bitcode emission + +**Question.** Should the LLVM backend be able to emit bitcode (`.bc`) directly instead of +textual `.ll`? + +**v1 behavior.** No. The backend emits textual `.ll` only, and the caller runs `llvm-as` (see +[009 — LLVM Backend](009-llvm-backend.md)). libVCT never links LLVM. + +**Why deferred.** The two obvious routes both carry cost or conflict. Linking LLVM would add a +large native dependency and contradict the zero-runtime-dependency rule (see +[013 — Build & Packaging](013-build-packaging.md)). Emitting the LLVM bitstream by hand would +require implementing and version-tracking the bitstream container format, which is stable but +large and not the intended value of the project. Textual output keeps the backend a pure +translator. + +**What resolution requires.** + +- Choose a route: an optional LLVM-linked build, a bundled bitstream writer, or a supported + external `llvm-as` invocation driven by the library. +- If the library invokes `llvm-as`, specify discovery and failure handling the way the C compiler + is handled in [010 — Driver & CLI](010-driver-cli.md). +- Keep textual emission available, since it is the format the round-trip and FileCheck tests use + (see [012 — Testing](012-testing.md)). + +## 7. OQ4: Incremental and cached compilation + +**Question.** Should libVCT cache compiled objects keyed by a content hash so that repeated +compilations skip work? + +**v1 behavior.** No. Content-hash `.o` caching is out of scope. Every invocation of the pipeline +compiles from the input it is given. + +**Why deferred.** Caching is only safe when the cache key captures everything that affects +output. The deterministic-output guarantee makes caching feasible, but it also raises the bar: +a stale hit would silently violate determinism. The key must cover the input, the full `Config`, +the library version, the trait vocabulary version, the textual IR format version, and the chosen +C compiler and its version. The storage, eviction, and concurrency design is a separate feature. + +**What resolution requires.** + +- Specify the cache key and prove that no input affecting output is omitted. +- Specify cache storage, invalidation, and locking, including concurrent invocations. +- Expose the feature through the CLI and the API without making it the default. +- Test that a cache hit and a cache miss produce byte-identical output. + +## 8. OQ5: Cross-compilation + +**Question.** Should libVCT support cross-compilation to targets other than the host? + +**v1 behavior.** Partial. `-target ` passes through to the toolchain, and `-mangled` +mode produces an Actually Portable Executable, but there is no general cross-compilation support. +Compiler discovery is host-oriented, and there is no sysroot or header management. + +**Why deferred.** Real cross-compilation needs a target model, not just a flag: target data +layout for the LLVM backend, an ABI definition for the C backend, discovery of a target C +compiler, and a sysroot with target headers and libraries. The v1 pipeline assumes a host +compiler that already knows its target. + +**What resolution requires.** + +- Define the supported target matrix and the meaning of a target triple in `Config`. +- Derive the LLVM target data layout and triple, and validate the module against it. +- Specify cross C compiler discovery, sysroot layout, and header resolution. +- Define the target ABI surface for the C backend and test it on at least one non-host target. + +## 9. Cross-references + +- [000 — Overview](000-overview.md) for the v1 non-goals these questions extend. +- [008 — C Backend](008-c-backend.md) for the GCC and Clang constructs behind OQ2 and OQ5. +- [009 — LLVM Backend](009-llvm-backend.md) for the emission surface behind OQ1, OQ3, and OQ5. +- [010 — Driver & CLI](010-driver-cli.md) for compiler discovery behind OQ2, OQ3, OQ4, and OQ5. +- [011 — Diagnostics](011-diagnostics.md) for source maps behind OQ1. +- [012 — Testing](012-testing.md) for the regression and determinism obligations new features + must inherit. +- [013 — Build & Packaging](013-build-packaging.md) for the dependency and versioning rules + these features must respect.