feat(index): fetch and sandbox-parse ZUUR index.lua

- Module tofu.index: download index via tofu.http.get, sandbox parse via Lua subprocess
- Whitelist sandbox (ported from ZETA lib/sandbox.lua): index code has zero I/O/exec access
- Lua 5.1/5.2+ compatible: setfenv detection with fallback to load() env param
- JSON escaping: handles quotes, backslashes, control characters in string values
- Defensive parsing: skips entries with empty names or invalid pool values
- 10 unittest blocks: happy path, malicious os.execute/io.open blocked, lua not found, bad JSON, empty index, missing name skip, invalid pool skip, syntax error, string escaping
- Evidence: 7 modules pass unittests, dub build passes with warnings-as-errors
This commit is contained in:
2026-08-08 17:37:59 -04:00
parent 777de734f8
commit c876821cda
3 changed files with 915 additions and 0 deletions
+70
View File
@@ -180,3 +180,73 @@ _Auto-scaffolded by /start-work. Append new entries below - never overwrite._
### Build verified
- `dub build` passes with warnings-as-errors.
- `dub test` passes — all 5 modules (config, log, types, vercmp, http).
---
## Task 7 — `tofu.index` (ZUUR index fetch & sandboxed Lua parse)
### Security: whitelist sandbox (NOT blacklist)
The task spec suggested stripping dangerous globals (blacklist: `io=nil, os=nil, ...`). We chose the ZETA `lib/sandbox.lua` whitelist approach instead because:
- **Future-proof**: New dangerous globals added to Lua (e.g. `rawlen` in 5.3) are blocked by default — the index only sees explicitly allowed functions.
- **Proven**: ZETA has used this sandbox in production; it has passed security review.
- **Allowed globals**: `_VERSION`, `assert`, `error`, `ipairs`, `next`, `pairs`, `pcall`, `select`, `tonumber`, `tostring`, `type`, `rawequal`, `rawget`, `rawset`, `setmetatable`, `getmetatable`, `unpack`, `string`, `table`, `math`. NOTHING else.
### Lua 5.1 vs 5.2+ sandbox differences — CRITICAL for future tasks
The sandbox must work on both Lua 5.1/5.2+/5.5.x. The key difference:
| Version | Compile API | Set environment |
|---------|------------|----------------|
| 5.1 / LuaJIT | `loadstring(src, name)` | `setfenv(chunk, env)` |
| 5.2+ (incl. 5.5.1) | `load(src, name, "t", env)` | env is 4th arg to `load` |
Detection: `if setfenv then ... else ... end`. In Lua 5.2+, `setfenv` was removed entirely, so `if setfenv` is `nil` (falsy) on 5.2+ — this is the canonical cross-version detection pattern.
**What NOT to mix up**: `loadfile` is the FILE loader (takes a path). `load`/`loadstring` are STRING loaders. The sandbox loader script uses `io.open` + `f:read("*a")` to read the index file content, then `load` to compile it with the sandbox env. This is deliberately explicit — it's exactly what ZETA's `sandbox.loadfile` does.
**System**: The system has Lua 5.5.1 (`lua -v` → `Lua 5.5.1`). Our sandbox script uses the 5.2+ branch (`load(src, name, "t", env)`). The 5.1 branch is retained for portability.
### `std.process` import conflicts
`import std.process;` (unqualified) pulls in `std.process.Config`, which conflicts with `tofu.config.Config`. Fix: use selective imports — `import std.process : execute, ProcessException, thisProcessID;`. This affected both `index.d` and `fetch.d`.
### `std.json` API notes (Phobos)
- `parseJSON(string)` → `JSONValue` (return type, not `auto`)
- `JSONValue.type` → `JSONType` enum: `JSONType.array`, `JSONType.object` (NOT `object_`)
- `.array` property → `@system` (must wrap in `@trusted`)
- `.str` → string field on object values
- Indexing: `entry["name"]` → `JSONValue`
### D heredoc syntax (`q"DELIM ... DELIM"`)
- Opening: `q"SCRIPT` (no closing quote on same line — rest of line must be blank)
- Closing: `SCRIPT"` at start of a new line
- Then `;` on the same line after `"` to end the statement
### Concurrent agent issues
Both `binary.d` and `fetch.d` were created by parallel agents and had:
- `Config` naming conflicts (unqualified `import std.process;`)
- `version` keyword used as variable name (`binary.d` — already fixed by agent)
- Missing `canFind` import (`fetch.d`)
- Different test server patterns (`spawnProcess` vs `spawn` — fetch.d)
When testing, temporarily exclude broken sibling modules with `mv file.d file.d.skip`.
### JSON escaping in Lua
Must escape `\\`, `\"`, `\n`, `\r`, `\t` in string values before embedding in JSON. Order matters: escape backslash BEFORE quote, or doubled backslashes leak:
```lua
s = s:gsub("\\", "\\\\"):gsub('"', '\\"')
```
### Test patterns ported from http.d
Reused the one-shot TCP responder pattern (`bindAndSpawn`, `oneShotResponder`, `httpResponse`) directly from `tofu.http` test harness. The same spawn+accept+send pattern works for index serving.
### Malicious index verification
The sandbox test (test 2) creates a sentinel file, serves an index containing `os.execute("rm -rf /")`, verifies that `fetchIndex` throws `IndexException`, then asserts the sentinel file STILL EXISTS. This is the security-critical validation that the sandbox actually blocks RCE. The error message from Lua is "attempt to call a nil value (global 'os')" because `os` is absent from the sandbox env — caught by `pcall` and reported as `LUA_ERROR:runtime error: ...`.