Configuration

The otter.config package contains the configuration models, parsers and utilities.

config.model module

Models for the config.

otter.config.model.LOG_LEVELS

The log levels.

alias of Literal[‘TRACE’, ‘DEBUG’, ‘INFO’, ‘SUCCESS’, ‘WARNING’, ‘ERROR’, ‘CRITICAL’]

pydantic model otter.config.model.BaseConfig

Bases: BaseModel

Base config model.

Show JSON schema

{
   "title": "BaseConfig",
   "description": "Base config model.",
   "type": "object",
   "properties": {
      "step": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Step"
      },
      "config_path": {
         "anyOf": [
            {
               "format": "path",
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Config Path"
      },
      "work_path": {
         "anyOf": [
            {
               "format": "path",
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Work Path"
      },
      "release_uri": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Release Uri"
      },
      "pool_size": {
         "anyOf": [
            {
               "type": "integer"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Pool Size"
      },
      "log_level": {
         "anyOf": [
            {
               "enum": [
                  "TRACE",
                  "DEBUG",
                  "INFO",
                  "SUCCESS",
                  "WARNING",
                  "ERROR",
                  "CRITICAL"
               ],
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Log Level"
      }
   }
}

Fields:

• config_path (pathlib._local.Path | None)

• log_level (Literal['TRACE', 'DEBUG', 'INFO', 'SUCCESS', 'WARNING', 'ERROR', 'CRITICAL'] | None)

• pool_size (int | None)

• release_uri (str | None)

• step (str | None)

• work_path (pathlib._local.Path | None)

field config_path: Path | None = None

field log_level: LOG_LEVELS | None = None

field pool_size: int | None = None

field release_uri: Annotated[str, AfterValidator(_validate_uri)] | None = None

field step: str | None = None

field work_path: Path | None = None

pydantic model otter.config.model.Defaultconfig

Bases: BaseModel

Default config model.

Show JSON schema

{
   "title": "Defaultconfig",
   "description": "Default config model.",
   "type": "object",
   "properties": {
      "config_path": {
         "default": "config.yaml",
         "format": "path",
         "title": "Config Path",
         "type": "string"
      },
      "work_path": {
         "default": "output",
         "format": "path",
         "title": "Work Path",
         "type": "string"
      },
      "release_uri": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Release Uri"
      },
      "pool_size": {
         "default": 5,
         "title": "Pool Size",
         "type": "integer"
      },
      "log_level": {
         "default": "INFO",
         "enum": [
            "TRACE",
            "DEBUG",
            "INFO",
            "SUCCESS",
            "WARNING",
            "ERROR",
            "CRITICAL"
         ],
         "title": "Log Level",
         "type": "string"
      }
   }
}

Fields:

• config_path (pathlib._local.Path)

• log_level (Literal['TRACE', 'DEBUG', 'INFO', 'SUCCESS', 'WARNING', 'ERROR', 'CRITICAL'])

• pool_size (int)

• release_uri (str | None)

• work_path (pathlib._local.Path)

field config_path: Path = PosixPath('config.yaml')

field log_level: LOG_LEVELS = 'INFO'

field pool_size: int = 5

field release_uri: str | None = None

field work_path: Path = PosixPath('output')

pydantic model otter.config.model.YamlConfig

Bases: BaseModel

YAML config model.

Show JSON schema

{
   "title": "YamlConfig",
   "description": "YAML config model.",
   "type": "object",
   "properties": {
      "work_path": {
         "anyOf": [
            {
               "format": "path",
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Work Path"
      },
      "release_uri": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Release Uri"
      },
      "pool_size": {
         "anyOf": [
            {
               "type": "integer"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Pool Size"
      },
      "log_level": {
         "anyOf": [
            {
               "enum": [
                  "TRACE",
                  "DEBUG",
                  "INFO",
                  "SUCCESS",
                  "WARNING",
                  "ERROR",
                  "CRITICAL"
               ],
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Log Level"
      },
      "steps": {
         "default": [],
         "items": {
            "type": "string"
         },
         "title": "Steps",
         "type": "array"
      }
   }
}

Fields:

• log_level (Literal['TRACE', 'DEBUG', 'INFO', 'SUCCESS', 'WARNING', 'ERROR', 'CRITICAL'] | None)

• pool_size (int | None)

• release_uri (str | None)

• steps (list[str])

• work_path (pathlib._local.Path | None)

field log_level: LOG_LEVELS | None = None

field pool_size: int | None = None

field release_uri: Annotated[str, AfterValidator(_validate_uri)] | None = None

field steps: list[str] = []

field work_path: Path | None = None

pydantic model otter.config.model.Config

Bases: BaseModel

Config model.

This model is used to define the config for the application.

It is constructed by merging the config from the defaults, env vars, CLI, and YAML config file. The config is loaded in order of precedence: 1. Command line arguments 2. Environment variables 3. YAML configuration file 4. Default settings

The model is frozen after validation to prevent further modification.

Show JSON schema

{
   "title": "Config",
   "description": "Config model.\n\nThis model is used to define the config for the application.\n\nIt is constructed by merging the config from the defaults, env vars, CLI, and\nYAML config file. The config is loaded in order of precedence:\n1. Command line arguments\n2. Environment variables\n3. YAML configuration file\n4. Default settings\n\nThe model is frozen after validation to prevent further modification.",
   "type": "object",
   "properties": {
      "step": {
         "title": "Step",
         "type": "string"
      },
      "steps": {
         "default": [],
         "items": {
            "type": "string"
         },
         "title": "Steps",
         "type": "array"
      },
      "config_path": {
         "default": "config.yaml",
         "format": "path",
         "title": "Config Path",
         "type": "string"
      },
      "work_path": {
         "default": "output",
         "format": "path",
         "title": "Work Path",
         "type": "string"
      },
      "release_uri": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Release Uri"
      },
      "pool_size": {
         "default": 5,
         "title": "Pool Size",
         "type": "integer"
      },
      "log_level": {
         "default": "INFO",
         "enum": [
            "TRACE",
            "DEBUG",
            "INFO",
            "SUCCESS",
            "WARNING",
            "ERROR",
            "CRITICAL"
         ],
         "title": "Log Level",
         "type": "string"
      }
   },
   "required": [
      "step"
   ]
}

Config:

• frozen: bool = True

Fields:

• config_path (pathlib._local.Path)

• log_level (Literal['TRACE', 'DEBUG', 'INFO', 'SUCCESS', 'WARNING', 'ERROR', 'CRITICAL'])

• pool_size (int)

• release_uri (str | None)

• step (str)

• steps (list[str])

• work_path (pathlib._local.Path)

Validators:

• _step_in_step_list » all fields

field config_path: Path = PosixPath('config.yaml')

The path to the configuration file.

Validated by:

• _step_in_step_list

field log_level: LOG_LEVELS = 'INFO'

See LOG_LEVELS.

Validated by:

• _step_in_step_list

field pool_size: int = 5

The maximum number of workers in the pool where tasks will run.

Validated by:

• _step_in_step_list

field release_uri: Annotated[str, AfterValidator(_validate_uri)] | None = None

The release URI. If present, this is where resources, logs and manifest will be uploaded to.

Validated by:

• _step_in_step_list

field step: str [Required]

The step to run. This is a required field.

Validated by:

• _step_in_step_list

field steps: list[str] = []

The list of steps defined in the configuration file.

Validated by:

• _step_in_step_list

field work_path: Path = PosixPath('output')

The local working path. This is where resources will be downloaded and the manifest and logs will be written to before upload to the GCS bucket.

Validated by:

• _step_in_step_list

config.cli module

CLI argument parser.

otter.config.cli.config_to_env(var: str, runner_name: str) → str

Convert a config variable name to its env var.

otter.config.cli.parse_cli(runner_name: str) → BaseConfig

Parse the command line arguments.

Parameters:

runner_name (str) – The name of the runner.

Returns:

The parsed command line arguments.

Return type:

BaseConfig

config.env module

Environment variable parser.

otter.config.env.env_to_config(env_var: str, env_prefix: str) → str

Convert an env var to its config variable name.

otter.config.env.parse_env(runner_name: str) → BaseConfig

Parses the environment variables and returns an BaseConfig object.

Parameters:

runner_name (str) – The name of the runner.

Returns:

The parsed environment variables.

Return type:

BaseConfig

config.yaml module

YAML config parser.

otter.config.yaml.parse_yaml(path: Path) → dict[str, Any]

Parse a yaml file.

Loads a yaml file, parses its content, and returns it as a dictionary. There is a cache, so the file is only read once.

Warning

If the file cannot be read or the yaml content cannot be parsed, the program will log an error and exit.

Parameters:

path (Path) – The path to the yaml file.

Returns:

The parsed yaml content.

Return type:

dict

Module contents

Configuration package.

otter.config.load_config(runner_name: str) → Config

Load the config.

See otter.config.model.Config for the Config object.

Parameters:

runner_name (str) – The name of the runner.

Returns:

The config object.

Return type:

Config