Scheduler design

Graph construction

Graph (graph.py) turns the expanded tasks into edges from three sources: deps, {task: x} inputs, and path inputs that match another task's outputs (exact, directory containment, or glob). Validation happens at load time for every command, so list or graph also catch problems:

• duplicate outputs, or an output nested in another task's output directory
• literal inputs that neither exist nor have a producer
• unknown task references (with a hint when the target was disabled by when)
• cycles, reported as a concrete path (a -> c -> b -> a) found by DFS

The topological order is Kahn's algorithm with ties broken by definition order, so output is deterministic.

Priorities: critical path first

When more tasks are ready than the pool can start, the order matters for makespan. Each selected task gets

crit(t) = cost(t) + max(crit(d) for d in dependents(t) within the selection)
cost(t) = median duration of its last 10 real executions, or 1.0 if never run

Ready tasks are sorted by (-priority, -crit, definition order). The explicit priority field wins; otherwise the task on the longest remaining chain starts first. Worked example with -j 1: quick_leaf (crit 1) and head -> mid -> tail (crit of head = 3). head starts first even though quick_leaf is listed first, and the tests assert this using start timestamps. Cache hits are not counted in the duration history, since they say nothing about real cost.

Main loop

A single coordinating thread owns all scheduling state; workers only execute one task each:

1. Sort ready; for each task, start it if running < jobs and its resources fit in the pools (used + request <= capacity for every pool). Resources are reserved at submit time and released when the future completes. Tasks that do not fit are skipped for now, so a small task can overtake a blocked large one (it can delay the large one, but it is greedy by priority order).
2. Wait for the first completed future (200 ms tick so Ctrl-C is handled promptly).
3. On success, decrement each dependent's wait set and move newly ready ones into ready. On failure either stop launching (fail-fast) or mark all transitive dependents skipped and carry on (--keep-going).

Pool requests above a pool's capacity are rejected at load time, so the loop cannot deadlock on an unsatisfiable task.

Fail-fast lets already-running tasks finish, then every task that never started is recorded as cancelled. With --keep-going, unrelated branches finish and only descendants of failures are skipped; the exit code is 1 either way.

Executing one task

Inside a worker: compute components and key (upstream signatures are guaranteed available because dependencies have settled), consult the cache, otherwise run up to retries + 1 attempts. Each attempt:

• removes stale declared outputs
• spawns the command in a new session (start_new_session=True), stdout and stderr appended to .taskforge/logs/<run>/<task>.log under an --- attempt n/m --- header
• waits up to timeout; on expiry sends SIGTERM to the process group, waits 2 s, then SIGKILLs it. After every attempt, any remaining members of the group are killed so grandchildren never leak.
• on exit 0, checks declared outputs exist, otherwise fails with declared output missing

Between attempts the worker sleeps retry_delay * backoff^(attempt-1) (cancellable). A retry that exhausts its attempts leaves the last status (failed or timeout).

Ctrl-C

KeyboardInterrupt in the coordinator sets a cancel flag (which also aborts backoff sleeps), sends SIGTERM to every live process group, waits for the workers, records in-flight tasks and everything not yet started as cancelled, marks the run interrupted and exits with 130. Finished tasks keep their cache entries, so the next run resumes where this one stopped.

Watch mode

watch polls (mtime, size) of every non-generated input of the selected tasks and the config file. After a change it waits one interval for writes to settle, reloads the project (the config may have changed), and performs an ordinary run. There is no special incremental logic: the content-hash cache restricts execution to affected tasks.