When workers fail

What happens when your process function (or the node it runs on) is broken — and what daisy tells you about it.

A minimal failing task

import daisy


def process(block):
    raise RuntimeError("broken import on this node")


task = daisy.Task(
    "my-step",
    total_roi=daisy.Roi((0,), (80,)),
    read_roi=daisy.Roi((0,), (10,)),
    write_roi=daisy.Roi((0,), (10,)),
    process_function=process,
    max_workers=4,
    max_worker_restarts=3,
)

daisy.run_blockwise([task])

What daisy 1.x did

Daisy 1.x had no limit on worker restarts: a worker whose process died was reaped and immediately respawned, forever (reap_dead_workers + inc_num_workers in daisy 1.x’s task_worker_pools.py). For a worker that crashes on startup this meant an infinite respawn loop — tens of new worker processes per second, zero blocks processed, and a run_blockwise call that never returns. On a cluster, that’s requested (billed) nodes spinning until someone notices and kills the job by hand.

What daisy does now

The runner replaces a failed worker at most max_worker_restarts times (default 10). If the cap is exhausted while blocks remain, the task is abandoned: remaining blocks are accounted as orphaned, downstream tasks that depended on them are abandoned transitively, and — most importantly — the run terminates and tells you why:

RuntimeError: task 'my-step' was abandoned (worker restart cap exhausted):
workers failed 4 times, 3 restarts performed; 8 of 8 blocks were orphaned.
Last worker error: RuntimeError: broken import on this node
Fix the worker error or increase Task(max_worker_restarts=...); use
Server().run_blockwise(...) to inspect task states without raising.

The original worker exception is captured and carried into the error — you don’t have to dig through per-worker log files to learn what broke.

Inspecting instead of raising

run_blockwise raising on abandonment is the right default for pipelines: a run that produced no output should not look like a clean exit. If you want to handle abandonment programmatically, use the server API, which returns per-task states and never raises for abandonment:

server = daisy.Server()
states = server.run_blockwise([task])

state = states["my-step"]
state.abandoned  # True
state.abandon_reason  # "worker restart cap exhausted"
state.last_worker_error  # "RuntimeError: broken import on this node"
state.orphaned_count  # 8

See Task abandonment and the typestate lifecycle for the mechanics (counter invariants, the typestate lifecycle, and how in-flight messages from dying workers are handled during the transition).