Manifest
The otter.manifest package contains the manifest models and utilities. The
manifest file is a way to provide tracking inforamtion on a run of otter.
See also
Refer to the Manifest Specification for more information, although the schema has changed slightly since the blog post.
Useful JQ queries
TODO
manifest.model module
Manifest data models.
- class otter.manifest.model.Result(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]
Bases:
StrEnumResult enumeration.
The result of a Task, Step or the whole set of steps. Used in the manifest to track the status of the run.
See also
- PENDING = 'pending'
- SUCCESS = 'success'
- FAILURE = 'failure'
- ABORTED = 'aborted'
- pydantic model otter.manifest.model.Artifact[source]
Bases:
BaseModelArtifact model.
An Artifact is the resulting product of a task. Tasks can produce none, one or more artifacts. It is not necesarily a file, it could be a folder or a database. If a task creates an Artifact, it will be logged into the manifest for tracking.
The Artifact class can be subclassed to extend the data model with additional fields to add to the manifest.
Artifacts must have a source and a destination, which can be used to track them through the flow of the pipeline.
Show JSON schema
{ "title": "Artifact", "description": "Artifact model.\n\nAn `Artifact` is the resulting product of a task. Tasks can produce none, one\nor more artifacts. It is not necesarily a file, it could be a folder or a\ndatabase. If a task creates an `Artifact`, it will be logged into the manifest\nfor tracking.\n\nThe `Artifact` class can be subclassed to extend the data model with additional\nfields to add to the manifest.\n\nArtifacts must have a source and a destination, which can be used to track them\nthrough the flow of the pipeline.", "type": "object", "properties": { "source": { "title": "Source", "type": "string" }, "destination": { "title": "Destination", "type": "string" } }, "required": [ "source", "destination" ] }
- Fields:
- field destination: str [Required]
The destination of the resource.
- field source: str [Required]
The source of the resource.
- pydantic model otter.manifest.model.TaskManifest[source]
Bases:
BaseModelModel representing a task in a step of the manifest.
Show JSON schema
{ "title": "TaskManifest", "description": "Model representing a task in a step of the manifest.", "type": "object", "properties": { "name": { "title": "Name", "type": "string" }, "result": { "$ref": "#/$defs/Result", "default": "pending" }, "started_run_at": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "title": "Started Run At" }, "finished_run_at": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "title": "Finished Run At" }, "started_validation_at": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "title": "Started Validation At" }, "finished_validation_at": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "title": "Finished Validation At" }, "log": { "default": [], "items": { "type": "string" }, "title": "Log", "type": "array" }, "artifacts": { "default": [], "items": { "$ref": "#/$defs/Artifact" }, "title": "Artifacts", "type": "array" } }, "$defs": { "Artifact": { "description": "Artifact model.\n\nAn `Artifact` is the resulting product of a task. Tasks can produce none, one\nor more artifacts. It is not necesarily a file, it could be a folder or a\ndatabase. If a task creates an `Artifact`, it will be logged into the manifest\nfor tracking.\n\nThe `Artifact` class can be subclassed to extend the data model with additional\nfields to add to the manifest.\n\nArtifacts must have a source and a destination, which can be used to track them\nthrough the flow of the pipeline.", "properties": { "source": { "title": "Source", "type": "string" }, "destination": { "title": "Destination", "type": "string" } }, "required": [ "source", "destination" ], "title": "Artifact", "type": "object" }, "Result": { "description": "Result enumeration.\n\nThe result of a `Task`, `Step` or the whole set of steps. Used in the manifest\nto track the status of the run.\n\n.. seealso:: :class:`TaskManifest`, :class:`StepManifest` and :class:`RootManifest`.", "enum": [ "pending", "success", "failure", "aborted" ], "title": "Result", "type": "string" } }, "additionalProperties": true, "required": [ "name" ] }
- Config:
extra: str = allow
- Fields:
- field finished_run_at: datetime | None = None
- field finished_validation_at: datetime | None = None
- field log: list[str] = []
- field name: str [Required]
- field started_run_at: datetime | None = None
- field started_validation_at: datetime | None = None
- property elapsed: float | None
Calculate the elapsed time.
- property run_elapsed: float | None
Calculate the elapsed time for the run.
- property validation_elapsed: float | None
Calculate the elapsed time for the validation.
- pydantic model otter.manifest.model.StepManifest[source]
Bases:
BaseModelModel representing a step in the manifest.
Show JSON schema
{ "title": "StepManifest", "description": "Model representing a step in the manifest.", "type": "object", "properties": { "name": { "title": "Name", "type": "string" }, "result": { "$ref": "#/$defs/Result", "default": "pending" }, "started_run_at": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "title": "Started Run At" }, "finished_run_at": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "title": "Finished Run At" }, "log": { "default": [], "items": { "type": "string" }, "title": "Log", "type": "array" }, "tasks": { "default": [], "items": { "$ref": "#/$defs/TaskManifest" }, "title": "Tasks", "type": "array" }, "artifacts": { "default": [], "items": { "$ref": "#/$defs/Artifact" }, "title": "Artifacts", "type": "array" } }, "$defs": { "Artifact": { "description": "Artifact model.\n\nAn `Artifact` is the resulting product of a task. Tasks can produce none, one\nor more artifacts. It is not necesarily a file, it could be a folder or a\ndatabase. If a task creates an `Artifact`, it will be logged into the manifest\nfor tracking.\n\nThe `Artifact` class can be subclassed to extend the data model with additional\nfields to add to the manifest.\n\nArtifacts must have a source and a destination, which can be used to track them\nthrough the flow of the pipeline.", "properties": { "source": { "title": "Source", "type": "string" }, "destination": { "title": "Destination", "type": "string" } }, "required": [ "source", "destination" ], "title": "Artifact", "type": "object" }, "Result": { "description": "Result enumeration.\n\nThe result of a `Task`, `Step` or the whole set of steps. Used in the manifest\nto track the status of the run.\n\n.. seealso:: :class:`TaskManifest`, :class:`StepManifest` and :class:`RootManifest`.", "enum": [ "pending", "success", "failure", "aborted" ], "title": "Result", "type": "string" }, "TaskManifest": { "additionalProperties": true, "description": "Model representing a task in a step of the manifest.", "properties": { "name": { "title": "Name", "type": "string" }, "result": { "$ref": "#/$defs/Result", "default": "pending" }, "started_run_at": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "title": "Started Run At" }, "finished_run_at": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "title": "Finished Run At" }, "started_validation_at": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "title": "Started Validation At" }, "finished_validation_at": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "title": "Finished Validation At" }, "log": { "default": [], "items": { "type": "string" }, "title": "Log", "type": "array" }, "artifacts": { "default": [], "items": { "$ref": "#/$defs/Artifact" }, "title": "Artifacts", "type": "array" } }, "required": [ "name" ], "title": "TaskManifest", "type": "object" } }, "required": [ "name" ] }
- Fields:
- field finished_run_at: datetime | None = None
- field log: list[str] = []
- field name: str [Required]
- field started_run_at: datetime | None = None
- field tasks: list[TaskManifest] = []
- property elapsed: float | None
Calculate the elapsed time.
- pydantic model otter.manifest.model.RootManifest[source]
Bases:
BaseModelModel representing the root of the manifest.
Show JSON schema
{ "title": "RootManifest", "description": "Model representing the root of the manifest.", "type": "object", "properties": { "result": { "$ref": "#/$defs/Result", "default": "pending" }, "started_at": { "default": "2025-02-28T09:26:33.708342Z", "format": "date-time", "title": "Started At", "type": "string" }, "modified_at": { "default": "2025-02-28T09:26:33.708349Z", "format": "date-time", "title": "Modified At", "type": "string" }, "log": { "default": [], "items": { "type": "string" }, "title": "Log", "type": "array" }, "steps": { "additionalProperties": { "$ref": "#/$defs/StepManifest" }, "default": {}, "title": "Steps", "type": "object" } }, "$defs": { "Artifact": { "description": "Artifact model.\n\nAn `Artifact` is the resulting product of a task. Tasks can produce none, one\nor more artifacts. It is not necesarily a file, it could be a folder or a\ndatabase. If a task creates an `Artifact`, it will be logged into the manifest\nfor tracking.\n\nThe `Artifact` class can be subclassed to extend the data model with additional\nfields to add to the manifest.\n\nArtifacts must have a source and a destination, which can be used to track them\nthrough the flow of the pipeline.", "properties": { "source": { "title": "Source", "type": "string" }, "destination": { "title": "Destination", "type": "string" } }, "required": [ "source", "destination" ], "title": "Artifact", "type": "object" }, "Result": { "description": "Result enumeration.\n\nThe result of a `Task`, `Step` or the whole set of steps. Used in the manifest\nto track the status of the run.\n\n.. seealso:: :class:`TaskManifest`, :class:`StepManifest` and :class:`RootManifest`.", "enum": [ "pending", "success", "failure", "aborted" ], "title": "Result", "type": "string" }, "StepManifest": { "description": "Model representing a step in the manifest.", "properties": { "name": { "title": "Name", "type": "string" }, "result": { "$ref": "#/$defs/Result", "default": "pending" }, "started_run_at": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "title": "Started Run At" }, "finished_run_at": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "title": "Finished Run At" }, "log": { "default": [], "items": { "type": "string" }, "title": "Log", "type": "array" }, "tasks": { "default": [], "items": { "$ref": "#/$defs/TaskManifest" }, "title": "Tasks", "type": "array" }, "artifacts": { "default": [], "items": { "$ref": "#/$defs/Artifact" }, "title": "Artifacts", "type": "array" } }, "required": [ "name" ], "title": "StepManifest", "type": "object" }, "TaskManifest": { "additionalProperties": true, "description": "Model representing a task in a step of the manifest.", "properties": { "name": { "title": "Name", "type": "string" }, "result": { "$ref": "#/$defs/Result", "default": "pending" }, "started_run_at": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "title": "Started Run At" }, "finished_run_at": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "title": "Finished Run At" }, "started_validation_at": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "title": "Started Validation At" }, "finished_validation_at": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "title": "Finished Validation At" }, "log": { "default": [], "items": { "type": "string" }, "title": "Log", "type": "array" }, "artifacts": { "default": [], "items": { "$ref": "#/$defs/Artifact" }, "title": "Artifacts", "type": "array" } }, "required": [ "name" ], "title": "TaskManifest", "type": "object" } } }
- Fields:
- field log: list[str] = []
- field modified_at: datetime = datetime.datetime(2025, 2, 28, 9, 26, 33, 708349, tzinfo=datetime.timezone.utc)
- field started_at: datetime = datetime.datetime(2025, 2, 28, 9, 26, 33, 708342, tzinfo=datetime.timezone.utc)
- field steps: dict[str, StepManifest] = {}
manifest.manifest_manager module
Manifest class for managing the manifest file.
- class otter.manifest.manifest_manager.ManifestManager(runner_name: str, remote_uri: str | None, local_path: Path, relevant_step: Step, steps: list[str])[source]
Bases:
objectClass that manages the manifest file.
Module contents
Manifest module.