Step

The otter.step package contains the step model and all the mechanics for the parallel execution of tasks.

model module

Step model.

class otter.step.model.Step(name: str, specs: list[Spec])

Bases: object

Step class.

This class represents a step in the pipeline.

name

The name of the step.

specs: list[Spec]

The list of specs to be processed in this step.

manifest: StepManifest

The manifest for the step.

tasks: dict[str, Task]

The dict of tasks that ran for this step.

start() → None

Update a step that has started running.

finish(tasks: dict[str, Task]) → None

Update a step that has finished running.

upsert_task_manifest(task: TaskReporter) → None

Update the step manifest with new task manifests.

coordinator module

Coordinator is the class that manages the execution of a step.

otter.step.coordinator.COORDINATOR_POLLING_INTERVAL = 0.5

Default polling interval for the coordinator loop, in seconds.

class otter.step.coordinator.Coordinator(step: Step, task_registry: TaskRegistry, config: Config)

Bases: object

Coordinates the execution of a step’s tasks.

step

The otter.step.model.Step to run.

task_registry

The task registry to build tasks from specs.

config

The configuration object.

async run() → None

Run the coordinator loop.

worker module

Worker process requests tasks from the coordinator and runs them.

otter.step.worker.WORKER_POLLING_INTERVAL = 0.5

Default polling interval for the worker loop, in seconds.

class otter.step.worker.Worker(worker_id: int)

Bases: object

Worker that executes tasks with a persistent event loop.

run(task_queue: Queue, result_queue: Queue, shutdown_event: Event) → None

Run the worker loop.

Executes tasks from task_queue, and puts the results in result_queue. The worker will stop when shutdown_event is set.

Parameters:

• task_queue (Queue) – Queue to get tasks from

• result_queue (Queue) – Queue to put results in

• shutdown_event (Event) – Event to signal shutdown

execute_task(task: Task, abort_event: Event) → Task

Execute a single task.

Parameters:

• task (Task) – Task to execute

• abort_event (Event) – Event to signal abort

Returns:

The executed task

Return type:

Task

otter.step.worker.worker_process(worker_id: int, task_queue: Queue, result_queue: Queue, shutdown_event: Event) → None

Worker process entry point.

Creates a Worker instance and runs it.

Parameters:

• worker_id (int) – Unique identifier for this worker

• task_queue (Queue) – Queue to get tasks from

• result_queue (Queue) – Queue to put results in

• shutdown_event (Event) – Event to signal shutdown

Module contents

Step module.