step package

step.step module

Step module.

class pis.step.step.Step(name: str)

Bases: StepReporter

Step class.

This class represents a step in the pipeline. A step is a collection of pretasks and tasks. The step will:

1. Initialize the pretasks.

2. Run the pretasks.

3. Initialize the tasks.

4. Send the tasks to the pool for parallel execution.

5. Validate the tasks.

6. Upload the tasks.

Variables:

name (str) – The name of the step.

execute()

Execute the step.

Raises:

StepFailedError – If the step execution fails at any point.

Returns:

The step instance itself.

Return type:

Step

class pis.step.step.XPool(processes=None, initializer=None, initargs=(), maxtasksperchild=None, context=None)

Bases: Pool

Extended Pool class.

This class extends the Pool class to add an xmap method that allows for the execution of a function on a list of tasks, while also providing an abort mechanism.

xmap(func_name: str, tasks: list[Task], abort: Event) → list[Task]

Execute a function on a list of tasks.

This is a wrapper for imap_unordered that allows for the execution of a function on a list of tasks with arbitrary parameters without having to clump them together in a single tuple. It also provides an abort mechanism.

Parameters:

• func_name (str) – The name of the function to execute on the tasks.

• tasks (list[Task]) – The list of tasks to execute the function on.

• abort (Event) – The abort event to signal the tasks to stop execution.

Returns:

The list of tasks after the function has been executed on them.

Return type:

list[Task]

Module contents

Step class.