feat(fetch): download ZUUR recipe directories to cache
Implements tofu.fetch module with fetchRecipe() that downloads ZUUR recipe directories using a known-file strategy (no directory listing assumed). Downloads .recipe (required), package.lua (optional), build.sh (optional), and custom build_script references. Key features: - Light recipe scanning for build_system/build_script discovery - 404 on .recipe → 'package not found' user-friendly error - 404 on optional files → logged and skipped - 404 on referenced build_script → real error - Partial file cleanup on failure - 5 unittests using python3 http.server for multi-file scenarios 8/8 modules pass, dub build succeeds with warnings-as-errors.
This commit is contained in:
@@ -183,6 +183,79 @@ _Auto-scaffolded by /start-work. Append new entries below - never overwrite._
|
||||
|
||||
---
|
||||
|
||||
## Task 8 — `tofu.fetch` (ZUUR recipe download to cache)
|
||||
|
||||
### Architecture
|
||||
- `fetchRecipe(name, cfg)` orchestrates a known-file download sequence
|
||||
(no directory listing assumed — plan constraint).
|
||||
- Download order: .recipe (required) → package.lua (optional) → build.sh
|
||||
(optional) → custom build_script (if referenced).
|
||||
- 404 on .recipe translates to `FetchException("package 'X' not found in
|
||||
ZUUR recipes")` (user-friendly).
|
||||
- 404 on optional files (package.lua, build.sh) is logged and skipped.
|
||||
- 404 on a referenced `build_script` is a real error (build cannot proceed).
|
||||
- Non-404 HTTP errors or connection failures are caught, partial files
|
||||
cleaned up, and re-thrown as `FetchException`.
|
||||
|
||||
### Light recipe scanning
|
||||
- Full Lua parsing is task 9/10's job. Task 8 only needs to discover
|
||||
`build_system = "custom"` + `build_script = "..."` to fetch referenced
|
||||
build scripts.
|
||||
- Manual string scanning: find key, skip whitespace/`=`, read quoted value.
|
||||
Handles arbitrary whitespace variations. Avoids `std.regex` dependency.
|
||||
- `extractBuildScript(content)` returns the script path only when
|
||||
`build_system = "custom"` is also found.
|
||||
|
||||
### `@safe` / `@trusted` architecture
|
||||
- `fetchRecipe` and `extractBuildScript` are `@safe`.
|
||||
- Filesystem operations (`exists`, `mkdirRecurse`, `readText`, `remove`,
|
||||
`rmdirRecurse`, `write`) are isolated in `@trusted` wrappers.
|
||||
- Follows the same pattern as `http.d` and `config.d`.
|
||||
- `std.file.readText` and friends are `@system` in DMD 2.112.
|
||||
|
||||
### Name conflicts between imports
|
||||
- `tofu.config.Config` conflicts with `std.process.Config` (both imported
|
||||
in `version(unittest)` blocks). Solution: fully qualify as
|
||||
`tofu.config.Config` in function signatures where `std.process` is
|
||||
also imported.
|
||||
|
||||
### `std.string.indexOf` vs `std.algorithm.canFind`
|
||||
- `"string".canFind("substr")` fails in D because UFCS on `string`
|
||||
(a range of `dchar`) tries to find a `string` element in a `dchar`
|
||||
range. Use `e.msg.indexOf("pattern") >= 0` instead.
|
||||
- `std.algorithm.searching.canFind(haystack, needle)` as a free function
|
||||
works but is less readable.
|
||||
|
||||
### Test strategy
|
||||
- Used `python3 -m http.server` on ephemeral ports for multi-file HTTP
|
||||
serving — the one-shot TCP responder from `http.d` handles only one
|
||||
connection per spawn.
|
||||
- Pattern: `findFreePort()` (bind + close ephemeral port), spawn server
|
||||
on that port, `waitForPort()` (connection polling with 100ms sleep,
|
||||
50 attempts max), run test, `killServer(pid)` (SIGTERM).
|
||||
- `spawnProcess` returns `Pid` (a class, not a struct with `.pid` field).
|
||||
Store the `Pid` object directly, not `.pid`.
|
||||
- `Socket.localAddress()` returns abstract `Address` — must cast to
|
||||
`InternetAddress` to access `.port`.
|
||||
- Five test scenarios: happy path, 404-on-recipe, custom build_script,
|
||||
optional-file-404s, connection-failure cleanup.
|
||||
|
||||
### Partial file cleanup
|
||||
- `downloadFile` internally cleans up `.part` files on failure (scope(failure)
|
||||
removes temp file). Our extra cleanup layer handles the case where a
|
||||
previous step succeeded but a later step fails — though for step 1
|
||||
failure (recipe), nothing else was downloaded yet.
|
||||
- Unittest scenario 5 verifies: after a connection error on recipe download,
|
||||
the cache directory is clean (no files).
|
||||
|
||||
### DUB details
|
||||
- `dub.json` uses `warningsAsErrors` in `buildOptions` (generates a
|
||||
deprecation warning about `buildRequirements`, but non-blocking).
|
||||
- All 8 modules pass unittests with warnings-as-errors enabled.
|
||||
- `dub build` produces the `tofu` binary successfully.
|
||||
|
||||
---
|
||||
|
||||
## Task 10 — `tofu.binary` (zuur/binary package.lua version checking)
|
||||
|
||||
### Architecture
|
||||
|
||||
Reference in New Issue
Block a user