# taskforge A local DAG task runner with content-hash caching: a Make/Airflow hybrid for data and build pipelines on one machine. Declare tasks with inputs and outputs in YAML; taskforge runs the graph in parallel, skips work whose inputs have not changed (by content hash, never by mtime), restores outputs from a cache when you flip back to a known state, retries flaky steps, and keeps a queryable run history in SQLite. ## Features - **YAML pipelines**: shell (`cmd`) or Python (`python: module:function`) tasks, `inputs` (globs, other tasks' outputs, params, env vars), `outputs` (files or directories), `deps`, `retries` with `retry_delay` and exponential `backoff`, `timeout`, `env`, `cwd`, `priority`. - **Matrix expansion**: `matrix: {region: [in, us]}` creates one instance per combination, with `{{ region }}` and `{{ params.x }}` templating in every string field. - **Conditions**: `when` expressions (safe, no `eval`) enable or disable tasks from params, environment and matrix values; `always_run` for tasks that must run every time. - **Content-hash cache**: cache key = hash(command, env subset, input file content hashes, upstream cache keys, params). Outputs are stored under `.taskforge/cache//` and restored on a hit. File hashing is incremental, backed by a persistent stat cache. - **Scheduler**: topological, parallel worker pool; custom resource pools (`cpu: 2`, `mem: 4G`) cap concurrency; critical-path-first priorities from historical durations; fail-fast or `--keep-going`; timeouts kill the whole process group; Ctrl-C stops cleanly. - **Run history** in SQLite: `history`, `stats` (p50/p95, cache hit rate, flaky tasks), `logs`, and `why` (a diff of cache-key components explaining a re-run). - **CLI**: `run`, `graph` (dot, mermaid, ascii), `list`, `clean`, `logs`, `history`, `stats`, `why`, `watch`. - **Live terminal UI** (rich): running tasks with elapsed time, progress counter, cache hits, summary table, log tail for failures. ## Install Requires Python 3.12+. ```bash cd taskforge uv venv .venv && uv pip install -p .venv/bin/python -e '.[dev]' source .venv/bin/activate # or call .venv/bin/taskforge directly ``` ## Quickstart ```bash cd examples/pipeline taskforge run # first run: everything executes taskforge run # second run: 13 cache hits taskforge why report # explain what would re-run, and why ``` A minimal `taskforge.yaml`: ```yaml tasks: build: cmd: gcc -o app main.c inputs: [main.c] outputs: [app] test: cmd: ./app --self-test deps: [build] retries: 2 retry_delay: 1s ``` ## Usage All commands accept `-f FILE` (default `taskforge.yaml`) and `-p KEY=VALUE` to override top-level `params`. The project root is the directory of the YAML file; all input/output paths are relative to it and state lives in `.taskforge/`. ``` taskforge run [TARGETS] [-j N] [--dry-run] [--force TASK] [--keep-going] taskforge graph [TARGETS] [--format dot|mermaid|ascii] taskforge list taskforge clean [--cache] taskforge logs TASK [--run ID] taskforge history [--run ID] [--task NAME] [-n N] taskforge stats taskforge why TASK taskforge watch [TARGETS] [--interval SECONDS] ``` `TARGETS` are task names or matrix base names (`transform` selects every instance); their dependencies are included automatically. Exit codes: 0 success, 1 task failure, 2 configuration error, 130 interrupted. ### Walkthrough on the example pipeline `examples/pipeline` simulates a download (one CSV per region), cleans it, converts per-region revenue with a Python task, aggregates, draws an SVG chart and renders an HTML report. `fx_rates` is deliberately flaky: it fails on its first attempt and is retried. Output below is captured from real runs (the live status board is replaced by plain lines when output is piped). **1. First run** - every task executes; `fx_rates` is retried once: ``` $ taskforge run taskforge run #1: 13 task(s) success fetch[region=eu] 21ms success fetch[region=in] 22ms success fetch[region=us] 22ms retry fx_rates: attempt 1 failed (exit code 3), retrying in 500ms success clean[region=eu] 19ms ... success fx_rates 566ms after 2 attempts success transform[region=us] 25ms success aggregate 18ms success chart 17ms success report 19ms OK 13 success in 690ms ``` **2. No-op re-run** - everything is a cache hit: ``` $ taskforge run taskforge run #2: 13 task(s) cached (hit) fx_rates 1ms cached (hit) fetch[region=us] 1ms ... cached (hit) report 0ms OK 13 cached in 28ms ``` Touching a file without changing its content is also a hit: keys depend on content, not mtime. **3. Edit one input** - only the dependent task runs: ``` $ sed -i 's|

Revenue report

|

Revenue report v2

|' templates/report.html $ taskforge why report report: would re-run, 1 component(s) changed since run #2 changed file:templates/report.html: 763cfbf2dc -> 6fb0558593 $ taskforge run ... cached (hit) chart 0ms success report 17ms OK 1 success, 12 cached in 41ms ``` **4. Partial re-run after a parameter change** - `why` walks the upstream chain down to the root cause, `--dry-run` previews the plan: ``` $ taskforge why aggregate -p currency=EUR aggregate: would re-run, 3 component(s) changed since run #3 changed upstream:transform[region=eu]: 5ee636db09 -> dfdeeb61ec transform[region=eu]: would re-run, 1 component(s) changed since run #3 changed param:currency: "USD" -> "EUR" ... $ taskforge run --dry-run -p currency=EUR Dry run (nothing executed) ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓ ┃ Task ┃ Action ┃ Why ┃ Key ┃ ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩ │ fetch[region=in] │ cached │ up to date │ 9fd7f76e1c89 │ │ clean[region=us] │ cached │ up to date │ d26e26247efe │ │ transform[region=in] │ run │ inputs changed or never run │ a6c1c59b44f0 │ │ aggregate │ run │ inputs changed or never run │ d740fcc97911 │ │ report │ run │ inputs changed or never run │ b62b1376e8b3 │ └──────────────────────┴────────┴─────────────────────────────┴──────────────┘ 6 to run, 7 cached ``` (rows abbreviated). Reverting to `currency=USD` afterwards costs nothing: the old keys are still in the cache, so outputs are restored without executing anything. ### History and stats ``` $ taskforge history Runs ┏━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓ ┃ Run ┃ Started ┃ Status ┃ Duration ┃ Tasks ┃ ┡━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩ │ 4 │ 2026-10-16 21:15:22 │ success │ 136ms │ 7 cached 6 success │ │ 3 │ 2026-10-16 21:15:21 │ success │ 32ms │ 13 cached │ │ 1 │ 2026-10-16 21:15:20 │ success │ 694ms │ 13 success │ └─────┴─────────────────────┴─────────┴──────────┴────────────────────┘ $ taskforge stats # (abbreviated) │ Task │ Executed │ Cache hits │ Hit rate │ p50 │ p95 │ Failures │ Flaky │ │ aggregate │ 1 │ 2 │ 67% │ 17ms │ 17ms │ 0 │ - │ │ fx_rates │ 1 │ 2 │ 67% │ 550ms │ 550ms │ 0 │ yes (100%) │ overall cache hit rate: 64% (25/39 task runs) flaky tasks: fx_rates ``` `taskforge logs fx_rates --run 1` prints the captured log, including the `--- attempt 1/3 ---` headers. When a task fails, its last 15 log lines are printed in the summary. ### Graphs `taskforge graph --format mermaid` prints Mermaid source (paste into any Markdown renderer), `dot` prints Graphviz, `ascii` prints dependency levels: ``` [0] fetch[region=in] fx_rates [1] clean[region=in] <- fetch[region=in] [2] transform[region=in] <- clean[region=in], fx_rates ... ``` ## Configuration Summary below; the full reference is in [docs/config-reference.md](docs/config-reference.md). | Key | Meaning | | --- | --- | | `params` | Values for `{{ params.x }}`, `{param: x}` inputs and `when`. Override with `-p`. | | `pools` | Capacity of resource pools. `cpu` defaults to the CPU count; sizes like `8G` are parsed. | | `tasks..cmd` / `python` | Shell command, or `module:function` run in a subprocess (gets a `ctx` dict). | | `inputs` | Path/glob strings, `{param: n}`, `{env: NAME}`, `{task: name}`. Paths another task outputs create dependencies automatically. | | `outputs` | Files or directories, relative to the project root. | | `deps` | Extra dependencies; a matrix base name means all its instances. | | `retries`, `retry_delay`, `backoff` | Attempt `n` waits `retry_delay * backoff^(n-1)` before the next one. | | `timeout` | Per attempt; process group is killed (`SIGTERM`, then `SIGKILL`). | | `resources` | e.g. `{cpu: 2, mem: 4G}`, drawn from `pools` while the task runs. | | `matrix`, `when`, `always_run`, `priority`, `env`, `cwd` | See the reference. | Tasks see `TASKFORGE_TASK`, `TASKFORGE_ATTEMPT` (1-based) and `TASKFORGE_ROOT`; the interpreter taskforge runs under is first on `PATH`, so `python` in a `cmd` resolves to the same environment. ## Architecture ``` src/taskforge/ config.py YAML loading, schema validation, matrix expansion, templating, `when` expr.py safe AST evaluator for `when` graph.py DAG construction, implicit edges from outputs, cycle/producer/duplicate checks hashing.py sha256 file hashing with persistent stat cache plan.py cache-key components, dry-run plan, `why` explanations cache.py key-addressed output store, verify and restore engine.py scheduler, worker pool, pools, retries, timeouts, process-group handling history.py SQLite run history, stats queries, stat cache and result tables ui.py rich live display and summary render.py dot / mermaid / ascii graph output watch.py polling watcher pyrunner.py entry point for `python:` tasks cli.py click commands ``` Design notes with worked examples: [docs/caching-model.md](docs/caching-model.md), [docs/scheduler-design.md](docs/scheduler-design.md), [docs/config-reference.md](docs/config-reference.md). ## Tests ```bash .venv/bin/pytest -q ``` 90 tests cover schema validation, matrix expansion, cycle/producer/duplicate detection, scheduling (dependency order, job limit, resource pools, critical-path and explicit priority using timestamps from instrumented tasks), cache-key stability, content-vs-mtime invalidation, output restore, retries and backoff, timeouts and process-group kill, keep-going vs fail-fast, SIGINT handling, history queries, `why` diffs, `watch`, and the CLI end to end against the example pipeline. ## Limitations - Single machine only; no remote cache or distributed execution. - Change detection relies on `(size, mtime)` to skip re-hashing. A file rewritten with identical size **and** identical mtime within an unchanged cache is not noticed, except that files modified in the last 2 seconds are always re-hashed. - Undeclared inputs are invisible: if a command reads a file that is not listed under `inputs`, changes to it will not invalidate the cache. - Declared outputs are deleted before a task runs, so tasks cannot update their outputs in place. - `--force` forces the named tasks only; dependents re-run only if the forced task's outputs actually changed. - Resource pools are bookkeeping for concurrency limits; taskforge does not measure or enforce real CPU or memory use. - `watch` polls; there is no inotify backend. - Windows is not supported (process groups, `/bin/sh`). ## License MIT, see [LICENSE](LICENSE).