huntedbytheirs
c876821cda
- Module tofu.index: download index via tofu.http.get, sandbox parse via Lua subprocess - Whitelist sandbox (ported from ZETA lib/sandbox.lua): index code has zero I/O/exec access - Lua 5.1/5.2+ compatible: setfenv detection with fallback to load() env param - JSON escaping: handles quotes, backslashes, control characters in string values - Defensive parsing: skips entries with empty names or invalid pool values - 10 unittest blocks: happy path, malicious os.execute/io.open blocked, lua not found, bad JSON, empty index, missing name skip, invalid pool skip, syntax error, string escaping - Evidence: 7 modules pass unittests, dub build passes with warnings-as-errors
253 lines
17 KiB
Markdown
253 lines
17 KiB
Markdown
# Learnings — tofu-core
|
|
|
|
Conventions, patterns, and successful approaches discovered during work on this plan.
|
|
|
|
_Auto-scaffolded by /start-work. Append new entries below - never overwrite._
|
|
|
|
---
|
|
|
|
## Task 5 — tofu.vercmp (RPM-style version comparison)
|
|
|
|
### Module naming
|
|
- `version` is a D reserved keyword (conditional compilation). Module is named
|
|
`tofu.vercmp`, file `src/tofu/vercmp.d`. The `DepConstraint` struct in
|
|
`types.d` uses field `ver` (not `version`) for the same reason.
|
|
|
|
### Algorithm (exact port from ZETA/lib/vercmp.lua)
|
|
- `nextSegment`: skips any non-digit/non-letter chars (separators); then reads
|
|
a contiguous run of same-type chars. Returns `(segment, nextIndex)`.
|
|
- `cmpNumeric`: strips leading zeros from both segments (`"007"` → `"7"`,
|
|
`"000"` → `"0"`). Longer stripped string wins; then lexicographic.
|
|
- `compare`: strips all whitespace, then loops over alternating segments.
|
|
If both segments are digit-typed → `cmpNumeric`, else lexical. When one
|
|
version exhausts segments: the exhausted one is *older* (shorter = older).
|
|
This matches RPM semantics: `compare("1.0.0", "1.0") > 0`.
|
|
- `satisfies`: uses `final switch` on `DepOp` enum. Maps to `compare` result
|
|
as expected (ge→c>=0, le→c<=0, eq→c==0, ne→c!=0, gt→c>0, lt→c<0).
|
|
- `parseDep`: delegates to `DepConstraint.parse` (already ported in types.d by
|
|
task 4). Both implementations are kept consistent.
|
|
|
|
### whitespace stripping
|
|
- Could not use `filter`+`array` because Phobos auto-decodes `string` to
|
|
`dchar` range elements, returning `dchar[]` not `string`. Implemented a
|
|
manual `removeWhitespace` helper using raw code-unit indexing, with a
|
|
`@trusted` cast since the freshly-allocated `char[]` has no aliasing.
|
|
|
|
### Test porting
|
|
- All 10 ZETA test categories ported as separate `@safe unittest` blocks.
|
|
- 4 modules pass unittests (types, vercmp, plus pre-existing ones).
|
|
- `dub build` passes with warnings-as-errors enabled.
|
|
|
|
### Future considerations
|
|
- When task 12 (constraint resolution) imports `tofu.vercmp`, the `compare`
|
|
and `satisfies` functions are ready. The `parseDep` integration with
|
|
`DepConstraint.parse` ensures consistency between ZETA and tofu's dep
|
|
constraint semantics.
|
|
|
|
---
|
|
|
|
## Task 2: Config Loading (src/tofu/config.d)
|
|
|
|
### toml package API (v1.0.0, Kripth/toml)
|
|
- Import: `import toml;` → provides `TOMLDocument`, `TOMLValue`, `parseTOML`, `TOMLException`, `TOMLParserException`
|
|
- `TOMLDocument` has `alias table this` — acts like `TOMLValue[string]` AA
|
|
- Access sections: `doc["core"]` returns `TOMLValue` with `type == TOML_TYPE.TABLE`
|
|
- Access string values: `doc["core"]["key"].str` (throws if type mismatch)
|
|
- Check existence: `"key" in doc` returns `TOMLValue*` (null if absent)
|
|
- `parseTOML()` is `@system` — must wrap in `@trusted` block when calling from `@safe` code
|
|
- Keys without `.str` / `.integer` access will throw `TOMLException`
|
|
|
|
### D language gotchas
|
|
- **Nested function declarations**: D does NOT allow `void foo() { ... }` inside a function body. Use lambdas: `auto foo = () { ... }; foo();` or anonymous `() @trusted { ... }();`
|
|
- **`version` is a keyword**: Cannot use `string version;` as field name — use `string version_;` or initialize with `= ""`
|
|
- **`parseTOML` is `@system`**: The toml package's parse function isn't marked `@safe` — must wrap in `@trusted` lambda
|
|
- **`std.file.readText` is `@system`**: Same — wrap in `@trusted`
|
|
- **`std.process.environment.get()`**: Returns empty string when env var is unset; `"VAR" in environment` checks existence but `environment.get("VAR", "default")` is cleaner
|
|
- **`stderr.writefln` is `@safe`** in recent DMD (2.106+) — usable directly in `@safe` code
|
|
- **AA `in` operator**: Returns pointer (`V*`) or `null` — must check for null before dereferencing
|
|
|
|
### Pattern: env-overrideable config loading
|
|
- Priority: env vars > TOML file > hardcoded defaults
|
|
- TOML read happens BEFORE env override — so TOML values serve as base, env vars overwrite
|
|
- Missing config file → no error, just skip to defaults
|
|
- Malformed TOML → `TOMLParserException` caught, warning to stderr, fall back to defaults
|
|
- Testability: `load()` accepts optional `envOverrides` map and explicit `configFile` path to avoid real env/filesystem
|
|
|
|
### Config design decisions
|
|
- `TOFU_CONFIG` env var overrides default config file path (`~/.config/tofu/config.toml`)
|
|
- `~` in default paths expanded with `std.path.expandTilde`
|
|
- All path fields default to empty string when "find on PATH" — no hardcoded binary paths
|
|
- `defaultJobs` typed as `int` (not `size_t`/`ulong`) since it's a user-facing count
|
|
|
|
### buildOptions warningsAsErrors deprecation
|
|
- DUB warns about `warningsAsErrors` in `buildOptions` — recommends `buildRequirements` instead
|
|
- Not blocking, informational only
|
|
|
|
## Task 3 — `tofu.log` (colored, NO_COLOR-aware logging)
|
|
|
|
- **dmd 2.112 gotcha**: the `std.stdio` globals `stdout`/`stderr` are `@system` to *access* — `makeGlobal` uses `__gshared File result`, which fails `@safe` inference. An `@safe` function cannot call `stdout.writeln(...)` directly. Fix: route every write through a tiny `@trusted` helper (`writeStdout(string)`/`writeStderr(string)`), keep the public log functions `@safe`. Compiler hint: *"using `__gshared` instead of `shared` makes it fail to infer `@safe`"*.
|
|
- **`std.process.environment` (dmd 2.112)**: it is an `abstract final class` of *static* methods — `environment.get(name)` is `@safe` and returns `null` when the var is unset (so "NO_COLOR set" == `get !is null`, even for an empty string). `environment["K"] = v` and `environment.remove(name)` are `@trusted`. NOT deprecated in 2.112.
|
|
- **Testable color decision**: make `colorEnabled()` re-read the environment on every call instead of caching in `static this()`. Enables NO_COLOR/TERM tests without restarting the process. Costs one `getenv` per log line — negligible.
|
|
- **Env save/restore in unittests**: a `private struct ColorEnv` (in `version(unittest)`) that snapshots NO_COLOR+TERM in its ctor and restores them in `~this()` (destructor) — `auto env = ColorEnv("1", "xterm")` scopes the restore. `null` value = remove the var. Double-destruction from a copied temporary is harmless (restore is idempotent).
|
|
- **Capturing stdout/stderr in unittests**: swap the global `stdout`/`stderr` File to a `File(name, "w+")`, write, `flush()`, swap back, then `rewind()` + `readln()`. Restore the global before the temp File's destructor runs.
|
|
- **zeta log format details**: `warn ` paints the *whole prefix* yellow (`paint("yellow", "warn ") ~ msg`), but `error` paints *only the word* then `" " ~ msg` (`paint("red","error") ~ " " ~ msg`). Assertion must expect `\x1b[31merror\x1b[0m failed deploy` — the reset code sits between "error" and the message.
|
|
- **`version` is a D keyword** — cannot be used as a struct field / identifier (trap for the parallel `types.d` work).
|
|
- **`std.stdio.writeln` flushes per call** — already satisfies "no buffering"; no explicit `flush()` needed in production paths.
|
|
- **Verifying a single module without the full build**: `dmd -main -unittest -i src/tofu/log.d` runs just that module's unittests when sibling modules (`config.d`, `types.d`) are mid-edit by parallel agents. `-i` pulls imports in automatically.
|
|
|
|
## Task 4 — `tofu.types` (core data structures)
|
|
|
|
- **`version` is a D keyword** — all struct fields named `version` must be renamed to `ver`. The D compiler treats `version` as a conditional-compilation directive and rejects it as a field name (see task 3 learnings above).
|
|
- **Struct field defaults in D**: `string` fields initialise to `null` by default. To enforce empty-string semantics (`""`), every string field must be explicitly `= ""`.
|
|
- **`final switch` for exhaustive enum handling**: using `final switch` on enum types forces the compiler to verify all cases are covered — catches missing branches at compile time. Used for `poolToString()`.
|
|
- **Manual character scanning preferred over `std.regex`**: the dep-spec parser uses simple char-by-char scanning with `isNameChar()` / `isWhite()` helpers instead of regex. This avoids the `std.regex` dependency, keeps the parser `@safe`, and matches the Lua reference semantics exactly.
|
|
- **Dep spec parsing order matters**: 2-character operators (`>=`, `<=`, `==`, `~=`) MUST be checked before 1-character operators (`>`, `<`, `=`) to avoid false matches. The Lua reference uses an ordered table `OPS = { ">=", "<=", "==", "~=", ">", "<", "=" }` — we replicate this with if/else chains.
|
|
- **`=` normalises to `==`**: the Lua code has `if op == "=" then op = "==" end`; we map both to `DepOp.eq`.
|
|
- **Trailing garbage detection**: after extracting version chars, any non-whitespace remaining in the spec string causes a `TypesException`. The Lua pattern uses `$` anchor for this.
|
|
- **`dub test` includes all `.d` files in source paths** for executable targets, not just imported modules. Pre-existing compile errors in sibling modules (config.d) block test runs. Fixed config.d by removing `private` from a nested function (access specifiers not allowed on local functions in D).
|
|
- **`BuildResult.failed` as `BuildFailure[]`**: the plan describes it as `struct {string name; string reason}[]` — in D this is a named struct `BuildFailure` with array field `BuildFailure[] failed`. The `failedNames()` helper extracts just the names for caller convenience.
|
|
- **`DepConstraint.parse` is a static factory method**: returns a new `DepConstraint` by value, never allocates on the heap. `@safe` throughout.
|
|
- **All structs are `@safe`**: no methods do I/O, no `@system` calls. The module stands alone — no imports from other tofu modules.
|
|
|
|
---
|
|
|
|
## Task 6 — `tofu.http` (sync HTTP client via std.net.curl)
|
|
|
|
### std.net.curl API (key members for HTTP struct)
|
|
- **Construction**: `HTTP(url)` via `opCall`, or `HTTP()` + `.url = url` setter.
|
|
The `.url` setter auto-prepends `http://` if no scheme is present.
|
|
- **Timeouts** (all are `@property` setters taking `core.time.Duration`):
|
|
- `connectTimeout` — connect phase only (`CurlOption.connecttimeout_ms`).
|
|
- `operationTimeout` — DNS + connect + transfer total (`CurlOption.timeout_ms`).
|
|
- `dataTimeout` — low-speed activity timeout (NOT total read timeout;
|
|
sets `low_speed_limit` + `low_speed_time`).
|
|
- For a "read 120s" requirement use `operationTimeout = 120.seconds` —
|
|
it covers the entire operation including data transfer.
|
|
- **Redirects**: `maxRedirects` (property setter, takes `uint`).
|
|
Defaults to 10. Set to 0 to disable, `uint.max` for infinite.
|
|
Internally sets `CurlOption.followlocation` + `CurlOption.maxredirs`.
|
|
- **User-Agent**: `setUserAgent(string)` — instance method (not property).
|
|
- **onReceive**: `void delegate(ubyte[])` returning `size_t`. Must accept
|
|
all bytes or the request aborts. Use `return data.length;`.
|
|
- **onReceiveStatusLine**: Not a direct property on HTTP struct — must
|
|
be set via the `onReceiveHeader` property setter which wraps the
|
|
delegate and internally fires `onReceiveStatusLine` when it detects
|
|
an HTTP status line. After `perform()` the final status is available
|
|
via `http.statusLine`.
|
|
- **statusLine**: `HTTP.StatusLine` struct with `.code` (`ushort`),
|
|
`.majorVersion`, `.minorVersion`, `.reason`. Reset to zero before
|
|
each `perform()`.
|
|
- **perform()**: `CurlCode perform(ThrowOnError = Yes.throwOnError)`.
|
|
Default throws on curl errors, but the thrown exception classes are:
|
|
- `CurlTimeoutException : CurlException` — operation timed out (code 28).
|
|
- `CurlException` — all other curl errors.
|
|
- `HTTPStatusException : CurlException` — only thrown by high-level
|
|
wrappers (_basicHTTP); low-level `perform()` does NOT throw on
|
|
HTTP status codes — you must check `statusLine.code` yourself.
|
|
- **Exception isolation pattern**: catch `CurlTimeoutException` and
|
|
`CurlException` in `@trusted` helpers, re-throw as `HttpException`
|
|
(which extends `Exception`, not `CurlException`). Callers never see
|
|
curl types.
|
|
- **verifyPeer / verifyHost**: On by default in HTTP struct (no change needed).
|
|
|
|
### @safe / @trusted architecture
|
|
- All public functions (`get`, `downloadFile`) are `@safe`.
|
|
- The `configure(HTTP)`, `getImpl`, and `downloadFileImpl` helpers are
|
|
`@trusted` — they are the ONLY places `std.net.curl` (which is
|
|
`@system`-heavy) is used.
|
|
|
|
### Download pattern (.part temp file + rename)
|
|
- Ported from ZETA `lib/fetch.lua`: write to `dest.part`, rename on
|
|
success, remove partial on error. `scope(failure)` at function
|
|
level handles all error paths automatically.
|
|
- `File(tmpPath, "wb")` — binary write mode. File destructor auto-closes;
|
|
explicit `f.close()` needed before rename.
|
|
|
|
### Unittest local server
|
|
- Used `std.concurrency.spawn` + `std.socket.TcpSocket` for a
|
|
self-contained one-shot HTTP server (no external dependency on Python3).
|
|
- Pattern: bind ephemeral port (`InternetAddress.PORT_ANY`), spawn thread
|
|
that `accept()`s one connection, sends canned HTTP response, exits.
|
|
`cast(shared)` the listener to pass to `spawn`. Mark test helpers
|
|
`@trusted`.
|
|
- HTTP response must include `Content-Length` header or libcurl hangs.
|
|
|
|
### Connection refused test
|
|
- Bind socket, record port, close it, then `get()` to that port.
|
|
curl returns error 7 (`CURLE_COULDNT_CONNECT`) → `CurlException`
|
|
→ caught and re-thrown as `HttpException`.
|
|
|
|
### Build verified
|
|
- `dub build` passes with warnings-as-errors.
|
|
- `dub test` passes — all 5 modules (config, log, types, vercmp, http).
|
|
|
|
---
|
|
|
|
## Task 7 — `tofu.index` (ZUUR index fetch & sandboxed Lua parse)
|
|
|
|
### Security: whitelist sandbox (NOT blacklist)
|
|
|
|
The task spec suggested stripping dangerous globals (blacklist: `io=nil, os=nil, ...`). We chose the ZETA `lib/sandbox.lua` whitelist approach instead because:
|
|
- **Future-proof**: New dangerous globals added to Lua (e.g. `rawlen` in 5.3) are blocked by default — the index only sees explicitly allowed functions.
|
|
- **Proven**: ZETA has used this sandbox in production; it has passed security review.
|
|
- **Allowed globals**: `_VERSION`, `assert`, `error`, `ipairs`, `next`, `pairs`, `pcall`, `select`, `tonumber`, `tostring`, `type`, `rawequal`, `rawget`, `rawset`, `setmetatable`, `getmetatable`, `unpack`, `string`, `table`, `math`. NOTHING else.
|
|
|
|
### Lua 5.1 vs 5.2+ sandbox differences — CRITICAL for future tasks
|
|
|
|
The sandbox must work on both Lua 5.1/5.2+/5.5.x. The key difference:
|
|
|
|
| Version | Compile API | Set environment |
|
|
|---------|------------|----------------|
|
|
| 5.1 / LuaJIT | `loadstring(src, name)` | `setfenv(chunk, env)` |
|
|
| 5.2+ (incl. 5.5.1) | `load(src, name, "t", env)` | env is 4th arg to `load` |
|
|
|
|
Detection: `if setfenv then ... else ... end`. In Lua 5.2+, `setfenv` was removed entirely, so `if setfenv` is `nil` (falsy) on 5.2+ — this is the canonical cross-version detection pattern.
|
|
|
|
**What NOT to mix up**: `loadfile` is the FILE loader (takes a path). `load`/`loadstring` are STRING loaders. The sandbox loader script uses `io.open` + `f:read("*a")` to read the index file content, then `load` to compile it with the sandbox env. This is deliberately explicit — it's exactly what ZETA's `sandbox.loadfile` does.
|
|
|
|
**System**: The system has Lua 5.5.1 (`lua -v` → `Lua 5.5.1`). Our sandbox script uses the 5.2+ branch (`load(src, name, "t", env)`). The 5.1 branch is retained for portability.
|
|
|
|
### `std.process` import conflicts
|
|
|
|
`import std.process;` (unqualified) pulls in `std.process.Config`, which conflicts with `tofu.config.Config`. Fix: use selective imports — `import std.process : execute, ProcessException, thisProcessID;`. This affected both `index.d` and `fetch.d`.
|
|
|
|
### `std.json` API notes (Phobos)
|
|
|
|
- `parseJSON(string)` → `JSONValue` (return type, not `auto`)
|
|
- `JSONValue.type` → `JSONType` enum: `JSONType.array`, `JSONType.object` (NOT `object_`)
|
|
- `.array` property → `@system` (must wrap in `@trusted`)
|
|
- `.str` → string field on object values
|
|
- Indexing: `entry["name"]` → `JSONValue`
|
|
|
|
### D heredoc syntax (`q"DELIM ... DELIM"`)
|
|
|
|
- Opening: `q"SCRIPT` (no closing quote on same line — rest of line must be blank)
|
|
- Closing: `SCRIPT"` at start of a new line
|
|
- Then `;` on the same line after `"` to end the statement
|
|
|
|
### Concurrent agent issues
|
|
|
|
Both `binary.d` and `fetch.d` were created by parallel agents and had:
|
|
- `Config` naming conflicts (unqualified `import std.process;`)
|
|
- `version` keyword used as variable name (`binary.d` — already fixed by agent)
|
|
- Missing `canFind` import (`fetch.d`)
|
|
- Different test server patterns (`spawnProcess` vs `spawn` — fetch.d)
|
|
|
|
When testing, temporarily exclude broken sibling modules with `mv file.d file.d.skip`.
|
|
|
|
### JSON escaping in Lua
|
|
|
|
Must escape `\\`, `\"`, `\n`, `\r`, `\t` in string values before embedding in JSON. Order matters: escape backslash BEFORE quote, or doubled backslashes leak:
|
|
|
|
```lua
|
|
s = s:gsub("\\", "\\\\"):gsub('"', '\\"')
|
|
```
|
|
|
|
### Test patterns ported from http.d
|
|
|
|
Reused the one-shot TCP responder pattern (`bindAndSpawn`, `oneShotResponder`, `httpResponse`) directly from `tofu.http` test harness. The same spawn+accept+send pattern works for index serving.
|
|
|
|
### Malicious index verification
|
|
|
|
The sandbox test (test 2) creates a sentinel file, serves an index containing `os.execute("rm -rf /")`, verifies that `fetchIndex` throws `IndexException`, then asserts the sentinel file STILL EXISTS. This is the security-critical validation that the sandbox actually blocks RCE. The error message from Lua is "attempt to call a nil value (global 'os')" because `os` is absent from the sandbox env — caught by `pcall` and reported as `LUA_ERROR:runtime error: ...`.
|