Configuration reference

A pipeline is one YAML file (default taskforge.yaml). Unknown keys are rejected so typos surface immediately.

version: 1            # optional, must be 1
params: {rows: 400}   # free-form values
pools: {cpu: 4, mem: 8G}
tasks: { ... }

Top level

params: scalars or nested maps. Referenced as {{ params.rows }}, {param: rows} and in when. -p rows=500 overrides (value parsed as YAML, so -p notify=true is a bool).
pools: concurrency budgets. cpu always exists and defaults to the CPU count. Other names are free (mem, gpu, db). Values accept sizes (K, M, G, T, powers of 1024) so mem: 8G works; plain integers are counts.

Task keys

Key	Type	Notes
`cmd`	string	Run with `/bin/sh -c` in `cwd`. Exactly one of `cmd`/`python`.
`python`	`pkg.mod:func`	Run in a fresh interpreter. If `func` accepts an argument it receives `ctx = {task, matrix, params, inputs, outputs, attempt}`. The project root and `cwd` are put on `sys.path`.
`inputs`	list	See below.
`outputs`	list	Files or directories, relative to the root, no globs.
`deps`	list	Task names or matrix base names. Templated, so `clean[region={{ region }}]` selects the matching instance.
`retries`	int >= 0	Extra attempts after the first.
`retry_delay`	duration	`500ms`, `2s`, `1m`, `1h` or seconds as a number. Default 0.
`backoff`	number >= 1	Multiplier, default 2. Wait before retry n is `retry_delay * backoff^(n-1)`.
`timeout`	duration	Per attempt. A timeout counts as a failure and is retried.
`resources`	map	Amount drawn from each named pool while the task runs. Must not exceed the pool.
`env`	map	Extra environment variables; part of the cache key.
`cwd`	path	Working directory of the command, relative to the root. Inputs/outputs stay root-relative.
`matrix`	map of lists	Cartesian product; instance name is `task[var=value,...]`.
`when`	expression	Evaluated at load time; false removes the task (and `list` reports it as disabled).
`always_run`	bool	Ignore cache hits.
`priority`	int	Higher starts first among ready tasks, before critical-path ordering.
`description`	string	Free text.

Inputs

inputs:
  - scripts/clean.py          # file, directory (hashed recursively) or glob (** supported)
  - "data/out/*.json"         # glob; matches another task's outputs -> implicit dependency
  - {param: currency}         # a param's value is part of the key
  - {env: AWS_REGION}         # an environment variable's value (empty if unset)
  - {task: fx_rates}          # depend on a task without naming a file

A path that equals, contains or is contained in another task's output (or a glob matching it) creates a dependency edge and is not hashed: the upstream task's cache key stands in for its content. A literal path that does not exist and that no task produces is an error ("missing producer"). Globs matching nothing are allowed.

Templating

{{ name }} and {{ dotted.path }} in string fields. Variables: matrix variables and params. Unknown variables are errors. Quote templated scalars in YAML flow lists (["out/{{ r }}.csv"]) because { starts a mapping there.

`when` expressions

Supports and, or, not, comparisons (==, !=, <, in, ...), literals, lists, attribute/index lookup. Names: params, env (unset variables read as ""), platform (sys.platform) and matrix variables.

when: params.notify and env.CI == 'true'
when: region != 'eu'

Anything else (calls, arithmetic, imports) is rejected.

Validation errors

Reported with exit code 2: schema problems, unknown tasks in deps, dependency cycles (the cycle path is printed), duplicate outputs, outputs nested in other outputs, outputs outside the project, missing producers, unknown or oversubscribed pools.