Caching model

What goes into a cache key

For every task taskforge builds a flat map of named components, then hashes the sorted JSON of that map:

Component Value
action the rendered cmd, or python:module:func
cwd, outputs the task's working directory and declared outputs
env:NAME each entry of the task's env
envvar:NAME value of each {env: NAME} input
param:name JSON of each {param: name} input
file:path sha256 of the content of each input file (globs and directories expanded, sorted)
upstream:task the signature of each dependency

Because components are named, taskforge why can diff them instead of just saying "the hash changed". Every run stores its components in the history database.

Content, not mtime

A file's hash is its content digest. To avoid re-reading big files, (size, mtime_ns) -> sha is kept in the stat_cache table. If both match, the stored digest is reused; if either differs the file is read again, and if the content turns out identical the key is unchanged. So touch is free and rewriting a file with the same bytes does not invalidate anything.

Files modified in the last two seconds are never trusted from the stat cache, nor stored: an editor may write twice within one timestamp tick (the "racy git" problem).

Upstream keys, not upstream files

Files produced by another task are not hashed. Their stand-in is the producer's signature, normally its cache key. This has two consequences: keys can be computed for the whole DAG before anything runs (that is how --dry-run and why work), and a change propagates downstream through keys alone.

fetch ──▶ clean ──▶ transform ──▶ aggregate
 key=K1    key=hash(K1,…)  ...

Edit scripts/clean.py: clean's file: component changes, so its key changes to K2, transform sees upstream:clean = K2, and so on. fetch is untouched and stays cached.

Tasks that bypass the cache

For always_run tasks and --forced tasks the key says nothing about what they produce (they run regardless). Their signature is therefore the digest of their output content after the run. Downstream tasks stay cached when such a task regenerates byte-identical outputs, and re-run when the content differs. dry-run/why use the digest of the previous execution as an estimate.

The store

.taskforge/
  cache/<key>/manifest.json      declared outputs, output dirs, {file: sha256}
  cache/<key>/files/<relpath>    copies of every output file
  history.db                     runs, task_runs, stat_cache, results
  logs/<run id>/<task>.log

On success the outputs are copied into cache/<key>/ (written to a temp dir and renamed, so partial entries never appear). Failed attempts are never stored.

Before executing, a task consults the store:

  1. cache/<key> absent: run it.
  2. Present and all outputs on disk match the manifest hashes (and directory outputs contain no extra or missing files): cached (hit), nothing touched.
  3. Present but outputs are missing or modified: cached (restored), outputs are replaced from the store.

Worked example: change currency USD -> EUR, run, then change it back. The second run finds the old USD keys in the store and restores the old outputs for all affected tasks with no execution. why reports this as "components differ ... but this exact key is in the cache so it would be restored".

Declared outputs are removed before each real execution so that "declared output missing" is a reliable failure signal and stale files never get cached.

Known blind spots