diff --git a/docs/extensions.md b/docs/extensions.md new file mode 100644 index 0000000..c05423f --- /dev/null +++ b/docs/extensions.md @@ -0,0 +1,159 @@ +# Extensions: adding a language without touching core + +stupidtools is deliberately not a one-trick C compiler. The core binary +knows only about a generic extension ABI (`src/ext/abi.h`); the C/C++ +languages are themselves just the first modules over it (`src/ext/lang_c.c`, +`src/ext/lang_cpp.c`). Any other language — Fortran, Rust, D, a toy DSL — +is added by dropping a Lua module into an extension directory. No core +changes, no recompiling. + +The worked example this document walks through is +[`examples/ext_fortran.lua`](../examples/ext_fortran.lua), a complete +third-party Fortran stub. Read that file first; it is heavily commented. + +--- + +## 1. Where extensions live (discovery) + +Extension discovery (`src/ext/discovery.h`) looks in this order: + +1. **`--ext-dir DIR`** — repeatable command-line flag (`stupidtools + --ext-dir DIR [--ext-dir DIR2 ...] `). +2. **`STUPIDTOOLS_EXT`** — colon-separated directory list + (`STUPIDTOOLS_EXT=/a:/b stupidtools `); empty entries are + skipped. +3. **User directory** — `$XDG_DATA_HOME/stupidtools/ext`, falling back to + `$HOME/.local/share/stupidtools/ext` when `XDG_DATA_HOME` is unset or + empty. Skipped entirely when neither variable yields a path. +4. **Builtin directory** — `builtin-ext/` relative to the working + directory (compile-time configurable via + `-DSTUPIDTOOLS_BUILTIN_EXT_DIR` for installed layouts). + +Scanning rules: only **regular files ending in `.lua`** are loaded +(dotfiles excluded); load order within a directory is bytewise- +lexicographic; a missing or empty directory is a silent no-op; an empty +`.lua` file is a clean no-op. Discovery **fail-fasts**: the first module +that fails to load (syntax error, runtime error, sandbox violation, +unreadable file) aborts with `extension load failed: : `, +and later modules are not attempted. + +Each loaded module logs one line built from its **actual** registrations: + +``` +loaded extension: /tmp/extdemo/ext_fortran.lua — checks: 1 (fortran_compiler_flag) languages: 1 (fortran) +``` + +## 2. The sandbox contract + +Extension code runs in a stripped Lua 5.4 runtime (`src/ext/lua.h`). The +module is **declarative** — it registers capabilities; it cannot execute +or touch the outside world: + +| Capability | Status | +| --- | --- | +| `st.register_language`, `st.register_check` | the **only** interface | +| base / table / string / curated `math` / hand-built `os` | available | +| `os.execute`, `os.exit`, `os.remove`, `os.rename` | **present but blocked** — calling them raises `sandbox blocked: ...` and fails discovery | +| `dofile`, `loadfile`, `require`, `package`, `io`, `debug`, `coroutine` | **nil** — their C libraries are not even linked | + +A module that violates the sandbox is rejected with a clear load error; +the binary never runs untrusted program execution. + +## 3. The registration API + +Two functions on the `st` table. Call shapes (exact — see +`src/ext/api.h` and `src/ext/lua.c`): + +```lua +st.register_language(name, [spec_table]) +st.register_check(name, { kind = "compile"|"link"|"run", source = "", link = { "-lfoo", ... } }) +``` + +- **`register_language(name [, table])`** — registers a language *name*. + The optional second argument, if present, must be a table; v1 records + the name only (the table's contents are reserved for future language + detection). +- **`register_check(name, spec)`** — registers a custom check. `kind` is + the probe **mode** (compile / link / run), `source` is the C snippet + the probe compiles, and `link` is an optional list of extra argv + elements for the link step. The check's *name* becomes a new check kind + a build file can reference, exactly like the builtin `header` kind. +- Names must be non-empty strings with no NUL bytes and must not collide + with an already-registered name (builtin or earlier module). + +The Lua tables are materialized into a `struct st_lua_probe_spec` +(`src/ext/api.h`): all fields strdup'd, owned by the runtime's registry, +freed with the runtime. + +## 4. v1 limitations (documented, deliberate) + +- **Custom checks compile C snippets against the C toolchain.** The + `source` in a probe spec is a *C* snippet driven by the detected C + compiler (`st_lua_probe_run` in `src/ext/api.h`). There is no generic + per-language probe runner yet; the Fortran example's check is a C + probe whose name says "fortran". (This is why the example keeps its + source libm-free and trivially true.) +- **Lua language detection is a stub.** `register_language` records the + name only; nothing yet detects a `fortran` compiler on the host, and + the DSL has no language-scoped feature resolution wired to it. The + table argument exists so a later release can add + `register_language("fortran", { detect = ... })` without breaking + existing modules. +- **The bridge maps every Lua check to the `"c"` language** internally + (`ST_LUA_CHECK_LANGUAGE` in `src/ext/api.c`) — again, v1 custom checks + are C snippets, by design. + +## 5. Step-by-step: adding a new language + +Copy this recipe for any language, not just Fortran. + +1. **Write a `.lua` module.** Model it on `examples/ext_fortran.lua`: + + ```lua + -- mylang.lua + st.register_language("mylang", { kind = "compile" }) + st.register_check("mylang_compiler_flag", { + kind = "compile", + source = "int main(void) { return 0; }", + link = { "-lmylang" }, + }) + ``` + + Only `st.*` calls are allowed; no `os.execute`, no `io`, no + `require` — the sandbox enforces it. + +2. **Drop it in a discovery directory.** Either: + + ```sh + mkdir -p /tmp/extdemo + cp mylang.lua /tmp/extdemo/ + ./src/stupidtools --ext-dir /tmp/extdemo + ``` + + or: + + ```sh + STUPIDTOOLS_EXT=/tmp/extdemo ./src/stupidtools + ``` + + or install it to `$XDG_DATA_HOME/stupidtools/ext/` (default user dir). + +3. **Check the load line.** You should see your module named with its + actual registrations: + + ``` + loaded extension: /tmp/extdemo/mylang.lua — checks: 1 (mylang_compiler_flag) languages: 1 (mylang) + ``` + + A malformed module aborts discovery with `extension load failed: + : :: ` — fix and rerun. + +4. **Reference the new check kind from a build file.** Once a module has + registered `mylang_compiler_flag`, a `feature` block in the KDL build + file can name that kind like any builtin check kind. (Full fixture- + driven resolution lands with the buildfile pipeline; discovery + + registration is what this version proves.) + +That is the whole contract: **discover → sandbox → register → reference.** +Languages are data to core, and the builtin C/C++ modules are just the +first extensions. diff --git a/examples/ext_fortran.lua b/examples/ext_fortran.lua new file mode 100644 index 0000000..2a2b4a0 --- /dev/null +++ b/examples/ext_fortran.lua @@ -0,0 +1,103 @@ +-- ext_fortran.lua -- an example third-party language extension. +-- +-- WHAT THIS MODULE DOES +-- +-- Fortran is NOT built into stupidtools: the builtin modules are only the +-- C/C++ language pair (src/ext/lang_c.c / src/ext/lang_cpp.c). This file +-- shows how a third party adds another language -- and a language-specific +-- compiler-flag check -- without touching core or recompiling the binary. +-- +-- HOW DISCOVERY LOADS IT +-- +-- Any regular file whose name ends in ".lua" is discovered (src/ext/ +-- discovery.h), from any of these sources, in this order: +-- +-- 1. each --ext-dir DIR command-line flag (repeatable); +-- 2. each directory in the colon-separated STUPIDTOOLS_EXT env var; +-- 3. the per-user directory: $XDG_DATA_HOME/stupidtools/ext, falling +-- back to $HOME/.local/share/stupidtools/ext; +-- 4. the builtin directory ("builtin-ext", compile-time configured). +-- +-- Each module is read into memory and run as ONE sandboxed Lua chunk with +-- the full file path as its chunkname (so diagnostics name file AND line). +-- Load order within a directory is bytewise-lexicographic, and a module +-- that fails to load aborts discovery (fail-fast). Try it: +-- +-- cp examples/ext_fortran.lua /tmp/extdemo/ +-- ./src/stupidtools --ext-dir /tmp/extdemo +-- +-- or, equivalently: +-- +-- STUPIDTOOLS_EXT=/tmp/extdemo ./src/stupidtools +-- +-- Discovery prints one line per loaded module, built from the ACTUAL +-- registrations the module performed: +-- +-- loaded extension: /tmp/extdemo/ext_fortran.lua - checks: 1 +-- (fortran_compiler_flag) languages: 1 (fortran) +-- +-- THE SANDBOX CONTRACT (what a module CANNOT do) +-- +-- The runtime is deliberately stripped (src/ext/lua.h). Extension code +-- runs with: +-- * os.execute / os.exit / os.remove / os.rename PRESENT but raising +-- a "sandbox blocked:" error when called; +-- * dofile / loadfile / require / package / io / debug / coroutine +-- REMOVED (the globals are nil; their C libraries are not even +-- linked); +-- * only the base, table, string, a curated libm-free `math`, and the +-- hand-built `os` tables available, plus the `st` registration table. +-- +-- So a module can declare capabilities but cannot execute programs, touch +-- the filesystem, or load other code. Violating the sandbox fails +-- discovery with a clear error. Note the probe SOURCE strings below are +-- inert text compiled by the host C toolchain, not executed here. +-- +-- THE EXTENSION POINTS (the only API) +-- +-- st.register_language(name [, spec-table]) +-- Registers a language. v1 registers the NAME only; the optional +-- table is accepted (and must BE a table) for forward compatibility +-- -- no language detection or toolchain is wired to it yet (see +-- "v1 limitations" below). +-- +-- st.register_check(name, { kind=..., source=..., link={...} }) +-- Registers a custom check. `kind` is the probe MODE +-- ("compile" | "link" | "run"), `source` is the C snippet the probe +-- compiles, and `link` (optional) is a list of extra argv elements +-- for the link step. v1 custom checks are C snippets compiled with +-- the detected C toolchain (see "v1 limitations" below). +-- +-- Both names must be non-empty strings free of NUL bytes and must not +-- collide with an already-registered name (builtin or earlier module). + +-- --------------------------------------------------------------------------- +-- 1. Register the language itself. "fortran" is arbitrary from core's +-- point of view -- any name works. The table argument is validated +-- (must be a table) but its contents are NOT consumed in v1. +-- --------------------------------------------------------------------------- +st.register_language("fortran", { kind = "compile" }) + +-- --------------------------------------------------------------------------- +-- 2. Register a language-specific check. The name becomes a new check kind +-- a build file can reference (like the builtin "header" kind is). The +-- probe spec says: compile `source` with the C toolchain and require +-- the compile (and, with `link`, the link) to succeed. +-- +-- The source below is deliberately trivially-true C and libm-free: it +-- links nothing, calls nothing, includes nothing -- a probe that must +-- pass even on the smallest toolchain. A realistic flag probe would +-- compile a C snippet with the flag in question and FAIL when the +-- compiler rejects the flag. The `-lfortran` link argument is inert +-- while the probe compiles (not links) but shows how extra link argv +-- is carried through for "link"/"run" mode probes. +-- --------------------------------------------------------------------------- +st.register_check("fortran_compiler_flag", { + kind = "compile", + source = "int main(void) { return 0; }", + link = { "-lfortran" }, +}) + +-- That is the whole module. Everything else -- detection, probing, the +-- generated ./configure -- is core's job; an extension only declares +-- what languages and checks exist.