taskforge

A local DAG task runner with content-hash caching: a Make/Airflow hybrid for data and build pipelines on one machine. Declare tasks with inputs and outputs in YAML; taskforge runs the graph in parallel, skips work whose inputs have not changed (by content hash, never by mtime), restores outputs from a cache when you flip back to a known state, retries flaky steps, and keeps a queryable run history in SQLite.

Features

YAML pipelines: shell (cmd) or Python (python: module:function) tasks, inputs (globs, other tasks' outputs, params, env vars), outputs (files or directories), deps, retries with retry_delay and exponential backoff, timeout, env, cwd, priority.
Matrix expansion: matrix: {region: [in, us]} creates one instance per combination, with {{ region }} and {{ params.x }} templating in every string field.
Conditions: when expressions (safe, no eval) enable or disable tasks from params, environment and matrix values; always_run for tasks that must run every time.
Content-hash cache: cache key = hash(command, env subset, input file content hashes, upstream cache keys, params). Outputs are stored under .taskforge/cache/<key>/ and restored on a hit. File hashing is incremental, backed by a persistent stat cache.
Scheduler: topological, parallel worker pool; custom resource pools (cpu: 2, mem: 4G) cap concurrency; critical-path-first priorities from historical durations; fail-fast or --keep-going; timeouts kill the whole process group; Ctrl-C stops cleanly.
Run history in SQLite: history, stats (p50/p95, cache hit rate, flaky tasks), logs, and why (a diff of cache-key components explaining a re-run).
CLI: run, graph (dot, mermaid, ascii), list, clean, logs, history, stats, why, watch.
Live terminal UI (rich): running tasks with elapsed time, progress counter, cache hits, summary table, log tail for failures.

Install

Requires Python 3.12+.

cd taskforge
uv venv .venv && uv pip install -p .venv/bin/python -e '.[dev]'
source .venv/bin/activate        # or call .venv/bin/taskforge directly

Quickstart

cd examples/pipeline
taskforge run                    # first run: everything executes
taskforge run                    # second run: 13 cache hits
taskforge why report             # explain what would re-run, and why

A minimal taskforge.yaml:

tasks:
  build:
    cmd: gcc -o app main.c
    inputs: [main.c]
    outputs: [app]
  test:
    cmd: ./app --self-test
    deps: [build]
    retries: 2
    retry_delay: 1s

Usage

All commands accept -f FILE (default taskforge.yaml) and -p KEY=VALUE to override top-level params. The project root is the directory of the YAML file; all input/output paths are relative to it and state lives in .taskforge/.

taskforge run [TARGETS] [-j N] [--dry-run] [--force TASK] [--keep-going]
taskforge graph [TARGETS] [--format dot|mermaid|ascii]
taskforge list
taskforge clean [--cache]
taskforge logs TASK [--run ID]
taskforge history [--run ID] [--task NAME] [-n N]
taskforge stats
taskforge why TASK
taskforge watch [TARGETS] [--interval SECONDS]

TARGETS are task names or matrix base names (transform selects every instance); their dependencies are included automatically. Exit codes: 0 success, 1 task failure, 2 configuration error, 130 interrupted.

Walkthrough on the example pipeline

examples/pipeline simulates a download (one CSV per region), cleans it, converts per-region revenue with a Python task, aggregates, draws an SVG chart and renders an HTML report. fx_rates is deliberately flaky: it fails on its first attempt and is retried. Output below is captured from real runs (the live status board is replaced by plain lines when output is piped).

1. First run - every task executes; fx_rates is retried once:

$ taskforge run
taskforge run #1: 13 task(s)
success          fetch[region=eu]  21ms
success          fetch[region=in]  22ms
success          fetch[region=us]  22ms
retry            fx_rates: attempt 1 failed (exit code 3), retrying in 500ms
success          clean[region=eu]  19ms
...
success          fx_rates  566ms after 2 attempts
success          transform[region=us]  25ms
success          aggregate  18ms
success          chart  17ms
success          report  19ms
OK 13 success in 690ms

2. No-op re-run - everything is a cache hit:

$ taskforge run
taskforge run #2: 13 task(s)
cached (hit)     fx_rates  1ms
cached (hit)     fetch[region=us]  1ms
...
cached (hit)     report  0ms
OK 13 cached in 28ms

Touching a file without changing its content is also a hit: keys depend on content, not mtime.

3. Edit one input - only the dependent task runs:

$ sed -i 's|<h1>Revenue report</h1>|<h1>Revenue report v2</h1>|' templates/report.html
$ taskforge why report
report: would re-run, 1 component(s) changed since run #2
  changed  file:templates/report.html: 763cfbf2dc -> 6fb0558593
$ taskforge run
...
cached (hit)     chart  0ms
success          report  17ms
OK 1 success, 12 cached in 41ms

4. Partial re-run after a parameter change - why walks the upstream chain down to the root cause, --dry-run previews the plan:

$ taskforge why aggregate -p currency=EUR
aggregate: would re-run, 3 component(s) changed since run #3
  changed  upstream:transform[region=eu]: 5ee636db09 -> dfdeeb61ec
    transform[region=eu]: would re-run, 1 component(s) changed since run #3
      changed  param:currency: "USD" -> "EUR"
  ...

$ taskforge run --dry-run -p currency=EUR
Dry run (nothing executed)
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Task                 ┃ Action ┃ Why                         ┃ Key          ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ fetch[region=in]     │ cached │ up to date                  │ 9fd7f76e1c89 │
│ clean[region=us]     │ cached │ up to date                  │ d26e26247efe │
│ transform[region=in] │ run    │ inputs changed or never run │ a6c1c59b44f0 │
│ aggregate            │ run    │ inputs changed or never run │ d740fcc97911 │
│ report               │ run    │ inputs changed or never run │ b62b1376e8b3 │
└──────────────────────┴────────┴─────────────────────────────┴──────────────┘
6 to run, 7 cached

(rows abbreviated). Reverting to currency=USD afterwards costs nothing: the old keys are still in the cache, so outputs are restored without executing anything.

History and stats

$ taskforge history
Runs
┏━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
┃ Run ┃ Started             ┃ Status  ┃ Duration ┃ Tasks              ┃
┡━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
│ 4   │ 2026-10-16 21:15:22 │ success │ 136ms    │ 7 cached 6 success │
│ 3   │ 2026-10-16 21:15:21 │ success │ 32ms     │ 13 cached          │
│ 1   │ 2026-10-16 21:15:20 │ success │ 694ms    │ 13 success         │
└─────┴─────────────────────┴─────────┴──────────┴────────────────────┘

$ taskforge stats          # (abbreviated)
│ Task      │ Executed │ Cache hits │ Hit rate │ p50   │ p95   │ Failures │ Flaky        │
│ aggregate │ 1        │ 2          │ 67%      │ 17ms  │ 17ms  │ 0        │ -            │
│ fx_rates  │ 1        │ 2          │ 67%      │ 550ms │ 550ms │ 0        │ yes (100%)   │
overall cache hit rate: 64% (25/39 task runs)
flaky tasks: fx_rates

taskforge logs fx_rates --run 1 prints the captured log, including the --- attempt 1/3 --- headers. When a task fails, its last 15 log lines are printed in the summary.

Graphs

taskforge graph --format mermaid prints Mermaid source (paste into any Markdown renderer), dot prints Graphviz, ascii prints dependency levels:

[0]
  fetch[region=in]
  fx_rates
[1]
  clean[region=in]      <- fetch[region=in]
[2]
  transform[region=in]  <- clean[region=in], fx_rates
...

Configuration

Summary below; the full reference is in docs/config-reference.md.

Key	Meaning
`params`	Values for `{{ params.x }}`, `{param: x}` inputs and `when`. Override with `-p`.
`pools`	Capacity of resource pools. `cpu` defaults to the CPU count; sizes like `8G` are parsed.
`tasks.<name>.cmd` / `python`	Shell command, or `module:function` run in a subprocess (gets a `ctx` dict).
`inputs`	Path/glob strings, `{param: n}`, `{env: NAME}`, `{task: name}`. Paths another task outputs create dependencies automatically.
`outputs`	Files or directories, relative to the project root.
`deps`	Extra dependencies; a matrix base name means all its instances.
`retries`, `retry_delay`, `backoff`	Attempt `n` waits `retry_delay * backoff^(n-1)` before the next one.
`timeout`	Per attempt; process group is killed (`SIGTERM`, then `SIGKILL`).
`resources`	e.g. `{cpu: 2, mem: 4G}`, drawn from `pools` while the task runs.
`matrix`, `when`, `always_run`, `priority`, `env`, `cwd`	See the reference.

Tasks see TASKFORGE_TASK, TASKFORGE_ATTEMPT (1-based) and TASKFORGE_ROOT; the interpreter taskforge runs under is first on PATH, so python in a cmd resolves to the same environment.

Architecture

src/taskforge/
  config.py    YAML loading, schema validation, matrix expansion, templating, `when`
  expr.py      safe AST evaluator for `when`
  graph.py     DAG construction, implicit edges from outputs, cycle/producer/duplicate checks
  hashing.py   sha256 file hashing with persistent stat cache
  plan.py      cache-key components, dry-run plan, `why` explanations
  cache.py     key-addressed output store, verify and restore
  engine.py    scheduler, worker pool, pools, retries, timeouts, process-group handling
  history.py   SQLite run history, stats queries, stat cache and result tables
  ui.py        rich live display and summary
  render.py    dot / mermaid / ascii graph output
  watch.py     polling watcher
  pyrunner.py  entry point for `python:` tasks
  cli.py       click commands

Design notes with worked examples: docs/caching-model.md, docs/scheduler-design.md, docs/config-reference.md.

Tests

.venv/bin/pytest -q

90 tests cover schema validation, matrix expansion, cycle/producer/duplicate detection, scheduling (dependency order, job limit, resource pools, critical-path and explicit priority using timestamps from instrumented tasks), cache-key stability, content-vs-mtime invalidation, output restore, retries and backoff, timeouts and process-group kill, keep-going vs fail-fast, SIGINT handling, history queries, why diffs, watch, and the CLI end to end against the example pipeline.

Limitations

Single machine only; no remote cache or distributed execution.
Change detection relies on (size, mtime) to skip re-hashing. A file rewritten with identical size and identical mtime within an unchanged cache is not noticed, except that files modified in the last 2 seconds are always re-hashed.
Undeclared inputs are invisible: if a command reads a file that is not listed under inputs, changes to it will not invalidate the cache.
Declared outputs are deleted before a task runs, so tasks cannot update their outputs in place.
--force forces the named tasks only; dependents re-run only if the forced task's outputs actually changed.
Resource pools are bookkeeping for concurrency limits; taskforge does not measure or enforce real CPU or memory use.
watch polls; there is no inotify backend.
Windows is not supported (process groups, /bin/sh).

License

MIT, see LICENSE.