feat(index): fetch and sandbox-parse ZUUR index.lua
- Module tofu.index: download index via tofu.http.get, sandbox parse via Lua subprocess - Whitelist sandbox (ported from ZETA lib/sandbox.lua): index code has zero I/O/exec access - Lua 5.1/5.2+ compatible: setfenv detection with fallback to load() env param - JSON escaping: handles quotes, backslashes, control characters in string values - Defensive parsing: skips entries with empty names or invalid pool values - 10 unittest blocks: happy path, malicious os.execute/io.open blocked, lua not found, bad JSON, empty index, missing name skip, invalid pool skip, syntax error, string escaping - Evidence: 7 modules pass unittests, dub build passes with warnings-as-errors
This commit is contained in:
@@ -180,3 +180,73 @@ _Auto-scaffolded by /start-work. Append new entries below - never overwrite._
|
||||
### Build verified
|
||||
- `dub build` passes with warnings-as-errors.
|
||||
- `dub test` passes — all 5 modules (config, log, types, vercmp, http).
|
||||
|
||||
---
|
||||
|
||||
## Task 7 — `tofu.index` (ZUUR index fetch & sandboxed Lua parse)
|
||||
|
||||
### Security: whitelist sandbox (NOT blacklist)
|
||||
|
||||
The task spec suggested stripping dangerous globals (blacklist: `io=nil, os=nil, ...`). We chose the ZETA `lib/sandbox.lua` whitelist approach instead because:
|
||||
- **Future-proof**: New dangerous globals added to Lua (e.g. `rawlen` in 5.3) are blocked by default — the index only sees explicitly allowed functions.
|
||||
- **Proven**: ZETA has used this sandbox in production; it has passed security review.
|
||||
- **Allowed globals**: `_VERSION`, `assert`, `error`, `ipairs`, `next`, `pairs`, `pcall`, `select`, `tonumber`, `tostring`, `type`, `rawequal`, `rawget`, `rawset`, `setmetatable`, `getmetatable`, `unpack`, `string`, `table`, `math`. NOTHING else.
|
||||
|
||||
### Lua 5.1 vs 5.2+ sandbox differences — CRITICAL for future tasks
|
||||
|
||||
The sandbox must work on both Lua 5.1/5.2+/5.5.x. The key difference:
|
||||
|
||||
| Version | Compile API | Set environment |
|
||||
|---------|------------|----------------|
|
||||
| 5.1 / LuaJIT | `loadstring(src, name)` | `setfenv(chunk, env)` |
|
||||
| 5.2+ (incl. 5.5.1) | `load(src, name, "t", env)` | env is 4th arg to `load` |
|
||||
|
||||
Detection: `if setfenv then ... else ... end`. In Lua 5.2+, `setfenv` was removed entirely, so `if setfenv` is `nil` (falsy) on 5.2+ — this is the canonical cross-version detection pattern.
|
||||
|
||||
**What NOT to mix up**: `loadfile` is the FILE loader (takes a path). `load`/`loadstring` are STRING loaders. The sandbox loader script uses `io.open` + `f:read("*a")` to read the index file content, then `load` to compile it with the sandbox env. This is deliberately explicit — it's exactly what ZETA's `sandbox.loadfile` does.
|
||||
|
||||
**System**: The system has Lua 5.5.1 (`lua -v` → `Lua 5.5.1`). Our sandbox script uses the 5.2+ branch (`load(src, name, "t", env)`). The 5.1 branch is retained for portability.
|
||||
|
||||
### `std.process` import conflicts
|
||||
|
||||
`import std.process;` (unqualified) pulls in `std.process.Config`, which conflicts with `tofu.config.Config`. Fix: use selective imports — `import std.process : execute, ProcessException, thisProcessID;`. This affected both `index.d` and `fetch.d`.
|
||||
|
||||
### `std.json` API notes (Phobos)
|
||||
|
||||
- `parseJSON(string)` → `JSONValue` (return type, not `auto`)
|
||||
- `JSONValue.type` → `JSONType` enum: `JSONType.array`, `JSONType.object` (NOT `object_`)
|
||||
- `.array` property → `@system` (must wrap in `@trusted`)
|
||||
- `.str` → string field on object values
|
||||
- Indexing: `entry["name"]` → `JSONValue`
|
||||
|
||||
### D heredoc syntax (`q"DELIM ... DELIM"`)
|
||||
|
||||
- Opening: `q"SCRIPT` (no closing quote on same line — rest of line must be blank)
|
||||
- Closing: `SCRIPT"` at start of a new line
|
||||
- Then `;` on the same line after `"` to end the statement
|
||||
|
||||
### Concurrent agent issues
|
||||
|
||||
Both `binary.d` and `fetch.d` were created by parallel agents and had:
|
||||
- `Config` naming conflicts (unqualified `import std.process;`)
|
||||
- `version` keyword used as variable name (`binary.d` — already fixed by agent)
|
||||
- Missing `canFind` import (`fetch.d`)
|
||||
- Different test server patterns (`spawnProcess` vs `spawn` — fetch.d)
|
||||
|
||||
When testing, temporarily exclude broken sibling modules with `mv file.d file.d.skip`.
|
||||
|
||||
### JSON escaping in Lua
|
||||
|
||||
Must escape `\\`, `\"`, `\n`, `\r`, `\t` in string values before embedding in JSON. Order matters: escape backslash BEFORE quote, or doubled backslashes leak:
|
||||
|
||||
```lua
|
||||
s = s:gsub("\\", "\\\\"):gsub('"', '\\"')
|
||||
```
|
||||
|
||||
### Test patterns ported from http.d
|
||||
|
||||
Reused the one-shot TCP responder pattern (`bindAndSpawn`, `oneShotResponder`, `httpResponse`) directly from `tofu.http` test harness. The same spawn+accept+send pattern works for index serving.
|
||||
|
||||
### Malicious index verification
|
||||
|
||||
The sandbox test (test 2) creates a sentinel file, serves an index containing `os.execute("rm -rf /")`, verifies that `fetchIndex` throws `IndexException`, then asserts the sentinel file STILL EXISTS. This is the security-critical validation that the sandbox actually blocks RCE. The error message from Lua is "attempt to call a nil value (global 'os')" because `os` is absent from the sandbox env — caught by `pcall` and reported as `LUA_ERROR:runtime error: ...`.
|
||||
|
||||
Reference in New Issue
Block a user