feat(fetch): download ZUUR recipe directories to cache

Implements tofu.fetch module with fetchRecipe() that downloads
ZUUR recipe directories using a known-file strategy (no directory
listing assumed). Downloads .recipe (required), package.lua (optional),
build.sh (optional), and custom build_script references.

Key features:
- Light recipe scanning for build_system/build_script discovery
- 404 on .recipe → 'package not found' user-friendly error
- 404 on optional files → logged and skipped
- 404 on referenced build_script → real error
- Partial file cleanup on failure
- 5 unittests using python3 http.server for multi-file scenarios

8/8 modules pass, dub build succeeds with warnings-as-errors.
This commit is contained in:
2026-08-08 17:38:41 -04:00
parent b367f21d29
commit d7d0d2ec6d
3 changed files with 812 additions and 0 deletions
+73
View File
@@ -183,6 +183,79 @@ _Auto-scaffolded by /start-work. Append new entries below - never overwrite._
---
## Task 8 — `tofu.fetch` (ZUUR recipe download to cache)
### Architecture
- `fetchRecipe(name, cfg)` orchestrates a known-file download sequence
(no directory listing assumed — plan constraint).
- Download order: .recipe (required) → package.lua (optional) → build.sh
(optional) → custom build_script (if referenced).
- 404 on .recipe translates to `FetchException("package 'X' not found in
ZUUR recipes")` (user-friendly).
- 404 on optional files (package.lua, build.sh) is logged and skipped.
- 404 on a referenced `build_script` is a real error (build cannot proceed).
- Non-404 HTTP errors or connection failures are caught, partial files
cleaned up, and re-thrown as `FetchException`.
### Light recipe scanning
- Full Lua parsing is task 9/10's job. Task 8 only needs to discover
`build_system = "custom"` + `build_script = "..."` to fetch referenced
build scripts.
- Manual string scanning: find key, skip whitespace/`=`, read quoted value.
Handles arbitrary whitespace variations. Avoids `std.regex` dependency.
- `extractBuildScript(content)` returns the script path only when
`build_system = "custom"` is also found.
### `@safe` / `@trusted` architecture
- `fetchRecipe` and `extractBuildScript` are `@safe`.
- Filesystem operations (`exists`, `mkdirRecurse`, `readText`, `remove`,
`rmdirRecurse`, `write`) are isolated in `@trusted` wrappers.
- Follows the same pattern as `http.d` and `config.d`.
- `std.file.readText` and friends are `@system` in DMD 2.112.
### Name conflicts between imports
- `tofu.config.Config` conflicts with `std.process.Config` (both imported
in `version(unittest)` blocks). Solution: fully qualify as
`tofu.config.Config` in function signatures where `std.process` is
also imported.
### `std.string.indexOf` vs `std.algorithm.canFind`
- `"string".canFind("substr")` fails in D because UFCS on `string`
(a range of `dchar`) tries to find a `string` element in a `dchar`
range. Use `e.msg.indexOf("pattern") >= 0` instead.
- `std.algorithm.searching.canFind(haystack, needle)` as a free function
works but is less readable.
### Test strategy
- Used `python3 -m http.server` on ephemeral ports for multi-file HTTP
serving — the one-shot TCP responder from `http.d` handles only one
connection per spawn.
- Pattern: `findFreePort()` (bind + close ephemeral port), spawn server
on that port, `waitForPort()` (connection polling with 100ms sleep,
50 attempts max), run test, `killServer(pid)` (SIGTERM).
- `spawnProcess` returns `Pid` (a class, not a struct with `.pid` field).
Store the `Pid` object directly, not `.pid`.
- `Socket.localAddress()` returns abstract `Address` — must cast to
`InternetAddress` to access `.port`.
- Five test scenarios: happy path, 404-on-recipe, custom build_script,
optional-file-404s, connection-failure cleanup.
### Partial file cleanup
- `downloadFile` internally cleans up `.part` files on failure (scope(failure)
removes temp file). Our extra cleanup layer handles the case where a
previous step succeeded but a later step fails — though for step 1
failure (recipe), nothing else was downloaded yet.
- Unittest scenario 5 verifies: after a connection error on recipe download,
the cache directory is clean (no files).
### DUB details
- `dub.json` uses `warningsAsErrors` in `buildOptions` (generates a
deprecation warning about `buildRequirements`, but non-blocking).
- All 8 modules pass unittests with warnings-as-errors enabled.
- `dub build` produces the `tofu` binary successfully.
---
## Task 10 — `tofu.binary` (zuur/binary package.lua version checking)
### Architecture