feat(fetch): download ZUUR recipe directories to cache
Implements tofu.fetch module with fetchRecipe() that downloads ZUUR recipe directories using a known-file strategy (no directory listing assumed). Downloads .recipe (required), package.lua (optional), build.sh (optional), and custom build_script references. Key features: - Light recipe scanning for build_system/build_script discovery - 404 on .recipe → 'package not found' user-friendly error - 404 on optional files → logged and skipped - 404 on referenced build_script → real error - Partial file cleanup on failure - 5 unittests using python3 http.server for multi-file scenarios 8/8 modules pass, dub build succeeds with warnings-as-errors.
This commit is contained in:
@@ -0,0 +1,101 @@
|
||||
Warning
|
||||
Warning ## Warning for package tofu ##
|
||||
Warning
|
||||
Warning The following compiler flags have been specified in the package description
|
||||
Warning file. They are handled by DUB and direct use in packages is discouraged.
|
||||
Warning Alternatively, you can set the DFLAGS environment variable to pass custom flags
|
||||
Warning to the compiler, or use one of the suggestions below:
|
||||
Warning
|
||||
Warning warningsAsErrors: Use "buildRequirements" to control the warning level
|
||||
Warning
|
||||
Generating test runner configuration 'tofu-test-application' for 'application' (executable).
|
||||
Warning Excluding main source file src/main.d from test.
|
||||
Starting Performing "unittest" build using /usr/bin/dmd for x86_64.
|
||||
Up-to-date toml 1.0.0: target for configuration [library] is up to date.
|
||||
Up-to-date tofu ~main: target for configuration [tofu-test-application] is up to date.
|
||||
Finished To force a rebuild of up-to-date targets, run again with --force
|
||||
Running tofu-test-application
|
||||
Warning: malformed TOML config at /tmp/tofu-test-config-bad-191318.toml: Invalid table key declaration (2:0)
|
||||
Warning: invalid TOFU_DEFAULT_JOBS 'not-a-number', using default 1
|
||||
127.0.0.1 - - [08/Aug/2026 17:38:05] "GET /recipes/hello/hello.recipe HTTP/1.1" 200 -
|
||||
127.0.0.1 - - [08/Aug/2026 17:38:05] "GET /recipes/hello/package.lua HTTP/1.1" 200 -
|
||||
127.0.0.1 - - [08/Aug/2026 17:38:05] "GET /recipes/hello/build.sh HTTP/1.1" 200 -
|
||||
127.0.0.1 - - [08/Aug/2026 17:38:05] code 404, message File not found
|
||||
127.0.0.1 - - [08/Aug/2026 17:38:05] "GET /recipes/hello/hello.recipe HTTP/1.1" 404 -
|
||||
127.0.0.1 - - [08/Aug/2026 17:38:05] "GET /recipes/hello/hello.recipe HTTP/1.1" 200 -
|
||||
127.0.0.1 - - [08/Aug/2026 17:38:05] code 404, message File not found
|
||||
127.0.0.1 - - [08/Aug/2026 17:38:05] "GET /recipes/hello/package.lua HTTP/1.1" 404 -
|
||||
127.0.0.1 - - [08/Aug/2026 17:38:05] code 404, message File not found
|
||||
127.0.0.1 - - [08/Aug/2026 17:38:05] "GET /recipes/hello/build.sh HTTP/1.1" 404 -
|
||||
127.0.0.1 - - [08/Aug/2026 17:38:05] "GET /recipes/hello/scripts/build.sh HTTP/1.1" 200 -
|
||||
127.0.0.1 - - [08/Aug/2026 17:38:05] "GET /recipes/hello/hello.recipe HTTP/1.1" 200 -
|
||||
127.0.0.1 - - [08/Aug/2026 17:38:05] code 404, message File not found
|
||||
127.0.0.1 - - [08/Aug/2026 17:38:05] "GET /recipes/hello/package.lua HTTP/1.1" 404 -
|
||||
127.0.0.1 - - [08/Aug/2026 17:38:05] code 404, message File not found
|
||||
127.0.0.1 - - [08/Aug/2026 17:38:05] "GET /recipes/hello/build.sh HTTP/1.1" 404 -
|
||||
[33mwarn skipping index entry with empty name[0m
|
||||
[33mwarn skipping index entry with empty name[0m
|
||||
[33mwarn skipping index entry 'badpool': invalid pool 'bad_pool_value'[0m
|
||||
8 modules passed unittests
|
||||
[2m . checking binary firefox ...[0m
|
||||
- binary firefox-2.1.0 satisfies firefox>=2.0
|
||||
[2m . checking binary libfoo ...[0m
|
||||
- binary libfoo-1.9 does not satisfy libfoo>=2.0
|
||||
[2m . checking binary noexist ...[0m
|
||||
[2m . checking binary brokenpkg ...[0m
|
||||
[2m . checking binary badpkg ...[0m
|
||||
[2m . checking binary testpkg ...[0m
|
||||
- binary testpkg-2.1.0 satisfies testpkg
|
||||
[36m==> fetching recipe hello[0m
|
||||
[2m . downloading hello.recipe[0m
|
||||
[2m . downloading package.lua[0m
|
||||
[2m . downloading build.sh[0m
|
||||
[32m ok recipe hello downloaded[0m
|
||||
[36m==> fetching recipe hello[0m
|
||||
[2m . downloading hello.recipe[0m
|
||||
[36m==> fetching recipe hello[0m
|
||||
[2m . downloading hello.recipe[0m
|
||||
[2m . downloading package.lua[0m
|
||||
- no package.lua for hello (optional manifest)
|
||||
[2m . downloading build.sh[0m
|
||||
- no build.sh for hello (optional build script)
|
||||
[2m . custom build system: fetching scripts/build.sh[0m
|
||||
[32m ok recipe hello downloaded[0m
|
||||
[36m==> fetching recipe hello[0m
|
||||
[2m . downloading hello.recipe[0m
|
||||
[2m . downloading package.lua[0m
|
||||
- no package.lua for hello (optional manifest)
|
||||
[2m . downloading build.sh[0m
|
||||
- no build.sh for hello (optional build script)
|
||||
[32m ok recipe hello downloaded[0m
|
||||
[36m==> fetching recipe hello[0m
|
||||
[2m . downloading hello.recipe[0m
|
||||
[2m . fetching index http://127.0.0.1:45641//index.lua[0m
|
||||
[32m ok index loaded: 3 packages[0m
|
||||
[2m . fetching index http://127.0.0.1:48821//index.lua[0m
|
||||
[2m . fetching index http://127.0.0.1:60693//index.lua[0m
|
||||
[2m . fetching index http://127.0.0.1:35357//index.lua[0m
|
||||
[2m . fetching index http://127.0.0.1:41973//index.lua[0m
|
||||
[32m ok index loaded: 0 packages[0m
|
||||
[2m . fetching index http://127.0.0.1:50313//index.lua[0m
|
||||
[32m ok index loaded: 1 packages[0m
|
||||
[2m . fetching index http://127.0.0.1:54449//index.lua[0m
|
||||
[32m ok index loaded: 2 packages[0m
|
||||
[2m . fetching index http://127.0.0.1:38951//index.lua[0m
|
||||
[2m . fetching index http://127.0.0.1:40753//index.lua[0m
|
||||
[32m ok index loaded: 1 packages[0m
|
||||
Warning
|
||||
Warning ## Warning for package tofu ##
|
||||
Warning
|
||||
Warning The following compiler flags have been specified in the package description
|
||||
Warning file. They are handled by DUB and direct use in packages is discouraged.
|
||||
Warning Alternatively, you can set the DFLAGS environment variable to pass custom flags
|
||||
Warning to the compiler, or use one of the suggestions below:
|
||||
Warning
|
||||
Warning warningsAsErrors: Use "buildRequirements" to control the warning level
|
||||
Warning
|
||||
Starting Performing "debug" build using /usr/bin/dmd for x86_64.
|
||||
Up-to-date toml 1.0.0: target for configuration [library] is up to date.
|
||||
Building tofu ~main: building configuration [application]
|
||||
Linking tofu
|
||||
Finished To force a rebuild of up-to-date targets, run again with --force
|
||||
@@ -183,6 +183,79 @@ _Auto-scaffolded by /start-work. Append new entries below - never overwrite._
|
||||
|
||||
---
|
||||
|
||||
## Task 8 — `tofu.fetch` (ZUUR recipe download to cache)
|
||||
|
||||
### Architecture
|
||||
- `fetchRecipe(name, cfg)` orchestrates a known-file download sequence
|
||||
(no directory listing assumed — plan constraint).
|
||||
- Download order: .recipe (required) → package.lua (optional) → build.sh
|
||||
(optional) → custom build_script (if referenced).
|
||||
- 404 on .recipe translates to `FetchException("package 'X' not found in
|
||||
ZUUR recipes")` (user-friendly).
|
||||
- 404 on optional files (package.lua, build.sh) is logged and skipped.
|
||||
- 404 on a referenced `build_script` is a real error (build cannot proceed).
|
||||
- Non-404 HTTP errors or connection failures are caught, partial files
|
||||
cleaned up, and re-thrown as `FetchException`.
|
||||
|
||||
### Light recipe scanning
|
||||
- Full Lua parsing is task 9/10's job. Task 8 only needs to discover
|
||||
`build_system = "custom"` + `build_script = "..."` to fetch referenced
|
||||
build scripts.
|
||||
- Manual string scanning: find key, skip whitespace/`=`, read quoted value.
|
||||
Handles arbitrary whitespace variations. Avoids `std.regex` dependency.
|
||||
- `extractBuildScript(content)` returns the script path only when
|
||||
`build_system = "custom"` is also found.
|
||||
|
||||
### `@safe` / `@trusted` architecture
|
||||
- `fetchRecipe` and `extractBuildScript` are `@safe`.
|
||||
- Filesystem operations (`exists`, `mkdirRecurse`, `readText`, `remove`,
|
||||
`rmdirRecurse`, `write`) are isolated in `@trusted` wrappers.
|
||||
- Follows the same pattern as `http.d` and `config.d`.
|
||||
- `std.file.readText` and friends are `@system` in DMD 2.112.
|
||||
|
||||
### Name conflicts between imports
|
||||
- `tofu.config.Config` conflicts with `std.process.Config` (both imported
|
||||
in `version(unittest)` blocks). Solution: fully qualify as
|
||||
`tofu.config.Config` in function signatures where `std.process` is
|
||||
also imported.
|
||||
|
||||
### `std.string.indexOf` vs `std.algorithm.canFind`
|
||||
- `"string".canFind("substr")` fails in D because UFCS on `string`
|
||||
(a range of `dchar`) tries to find a `string` element in a `dchar`
|
||||
range. Use `e.msg.indexOf("pattern") >= 0` instead.
|
||||
- `std.algorithm.searching.canFind(haystack, needle)` as a free function
|
||||
works but is less readable.
|
||||
|
||||
### Test strategy
|
||||
- Used `python3 -m http.server` on ephemeral ports for multi-file HTTP
|
||||
serving — the one-shot TCP responder from `http.d` handles only one
|
||||
connection per spawn.
|
||||
- Pattern: `findFreePort()` (bind + close ephemeral port), spawn server
|
||||
on that port, `waitForPort()` (connection polling with 100ms sleep,
|
||||
50 attempts max), run test, `killServer(pid)` (SIGTERM).
|
||||
- `spawnProcess` returns `Pid` (a class, not a struct with `.pid` field).
|
||||
Store the `Pid` object directly, not `.pid`.
|
||||
- `Socket.localAddress()` returns abstract `Address` — must cast to
|
||||
`InternetAddress` to access `.port`.
|
||||
- Five test scenarios: happy path, 404-on-recipe, custom build_script,
|
||||
optional-file-404s, connection-failure cleanup.
|
||||
|
||||
### Partial file cleanup
|
||||
- `downloadFile` internally cleans up `.part` files on failure (scope(failure)
|
||||
removes temp file). Our extra cleanup layer handles the case where a
|
||||
previous step succeeded but a later step fails — though for step 1
|
||||
failure (recipe), nothing else was downloaded yet.
|
||||
- Unittest scenario 5 verifies: after a connection error on recipe download,
|
||||
the cache directory is clean (no files).
|
||||
|
||||
### DUB details
|
||||
- `dub.json` uses `warningsAsErrors` in `buildOptions` (generates a
|
||||
deprecation warning about `buildRequirements`, but non-blocking).
|
||||
- All 8 modules pass unittests with warnings-as-errors enabled.
|
||||
- `dub build` produces the `tofu` binary successfully.
|
||||
|
||||
---
|
||||
|
||||
## Task 10 — `tofu.binary` (zuur/binary package.lua version checking)
|
||||
|
||||
### Architecture
|
||||
|
||||
Reference in New Issue
Block a user