Storage

The otter.storage package contains classes for interacting with remote storage services.

storage.model module

Abstract base class for remote storage services.

class otter.storage.model.RemoteStorage[source]

Bases: ABC

Abstract base class for remote storage services.

abstract property name: str

The name of the storage service.

Returns:: The name of the storage service.
Return type:: str

abstract check(uri: str) → bool[source]

Check if the provided storage is valid.

This method should check if the storage exists and has proper permissions. On Google Cloud Storage, for example, this would check if the bucket exists and the service account has get, list and create permissions.

Parameters:: uri (str) – The URI to check.
Returns:: True if the storage exists, False otherwise.
Return type:: bool

abstract stat(uri: str) → dict[str, Any][source]

Get metadata for a file.

Currently, only the modification time is required, as it is used for the download_latest task. This method should be expanded as needed.

Parameters:: uri (str) – The URI to get metadata for.
Returns:: A dictionary containing metadata.
Return type:: dict
Raises:: NotFoundError – If the file does not exist.

abstract list(uri: str, pattern: str | None = None) → list[str][source]

List files in prefix URI.

Optionally, a pattern can be provided to match files against. The pattern should be a simple string match, preceded by an exclamation mark to exclude files. For example, ‘foo’ will match all files containing ‘foo’, while ‘!foo’ will exclude all files containing ‘foo’.

Parameters:

uri (str) – The prefix URI by which to list files.
pattern (str | None) – Optional. The pattern to match files against.

Returns:

A list of file URIs.

Return type:

list[str]

abstract glob(uri: str) → list[str][source]

List blobs matching a pattern.

Parameters:: uri (str) – The URI with a glob expression to match for.
Returns:: A list of blob URIs.
Return type:: list[str]

abstract download_to_file(uri: str, dst: Path) → int[source]

Download a file to the local filesystem.

Parameters:

uri (str) – The URI of the file to download.
dst (Path) – The destination path to download the file to.

Returns:

The revision number of the file.

Return type:

int

Raises:

NotFoundError – If the file does not exist.
DownloadError – If an error occurs during download.

abstract download_to_string(uri: str) → tuple[str, int][source]

Download a file and return its contents as a string.

Parameters:

uri (str) – The URI of the file to download.

Returns:

A tuple containing the file contents and the revision number.

Return type:

tuple[str, int]

Raises:

NotFoundError – If the file does not exist.
DownloadError – If an error occurs during download

abstract upload(src: Path, uri: str, revision: int | None = None) → int[source]

Upload a file to the remote storage.

Optionally, a revision number can be provided to ensure that the file has not been modified since the last time it was read.

Parameters:

src (Path) – The source path of the file to upload.
uri (str) – The URI to upload the file to.
revision (int | None) – Optional. The expected revision number of the file.

Returns:

The new revision number of the file.

Return type:

int

Raises:

UploadError – If an error occurs during upload.
PreconditionFailedError – If the revision number does not match.

abstract get_session() → Any[source]

Return a session for making requests.

Returns:: The session.

storage.google module

Google Cloud Storage class.

class otter.storage.google.GoogleStorage[source]

Bases: RemoteStorage

Google Cloud Storage helper class.

This class implements the RemoteStorage interface for Google Cloud Storage.

Variables:

credentials (google.auth.credentials.Credentials) – The Google Cloud Storage credentials.
client (google.cloud.storage.client.Client) – The Google Cloud Storage client.

property name: str: The name of the storage provider.

check(uri: str) → bool[source]

Check if a bucket exists in Google Cloud Storage.

Parameters:: uri (str) – The URI of the file to check.
Returns:: True if the file exists, False otherwise.
Return type:: bool

stat(uri: str) → dict[str, float | None][source]

Get metadata for a file in Google Cloud Storage.

Parameters:: uri (str) – The URI of the file to get metadata for.
Returns:: A dictionary containing metadata.
Return type:: dict
Raises:: NotFoundError – If the file does not exist.

list(uri: str, pattern: str | None = None) → list[str][source]

List blobs in a bucket.

Parameters:

uri (str) – The URI prefix to list blobs for.
pattern (str | None) – The pattern to match blobs against.

Returns:

A list of blob URIs.

Return type:

list[str]

Raises:

NotFoundError – If the bucket or prefix does not exist.
StorageError – If the prefix is invalid.

glob(uri: str) → list[str][source]

List blobs matching a pattern.

Parameters:: uri (str) – The URI with a glob expression to match for.
Returns:: A list of blob URIs.
Return type:: list[str]

download_to_file(uri: str, dst: Path) → int[source]

Download a file from Google Cloud Storage to the local filesystem.

Parameters:

uri (str) – The URI of the file to download.
dst (Path) – The destination path to download the file to.

Returns:

The generation number of the file.

Return type:

int

Raises:

NotFoundError – If the file is not found.
StorageError – If an error occurs while downloading the file.

download_to_string(uri: str) → tuple[str, int][source]

Download a file from Google Cloud Storage and return its contents as a string.

Parameters:

uri (str) – The URI of the file to download.

Raises:

NotFoundError – If the file is not found.
StorageError – If an error occurs while downloading the file.

Returns:

A tuple containing the file contents and the generation number.

Return type:

tuple[str, int]

upload(src: Path, uri: str, revision: int | None = None) → int[source]

Upload a file to Google Cloud Storage.

Parameters:

src (Path) – The source path of the file to upload.
uri (str) – The URI to upload the file to.
revision (int | None) – The expected revision number of the file.

Returns:

The new revision number of the file.

Return type:

int

Raises:

StorageError – If an error occurs during upload.
PreconditionFailedError – If the revision number does not match.

get_session() → AuthorizedSession[source]

Get the current authenticated session.

Returns:: An authorized session.
Return type:: AuthorizedSession

storage.noop module

No-op storage class.

class otter.storage.noop.NoopStorage[source]

Bases: RemoteStorage

No-op storage helper class.

This class implements the RemoteStorage interface but does not perform any operations. It is used when the run is local only.

property name: str: The name of the storage provider.

check(uri: str) → bool[source]: Check if a file exists.

stat(uri: str) → dict[str, Any][source]: Get metadata for a file.

list(uri: str, pattern: str | None = None) → list[str][source]: List files.

glob(uri: str) → list[str][source]: List files matching a glob pattern.

download_to_file(uri: str, dst: Path) → int[source]: Download a file to the local filesystem.

download_to_string(uri: str) → tuple[str, int][source]: Download a file and return its contents as a string.

upload(src: Path, uri: str, revision: int | None = None) → int[source]: Upload a file.

get_session() → None[source]: Return a session for making requests.

Module contents

Remote storage implementation classes.

otter.storage.get_remote_storage(uri: str | None) → RemoteStorage[source]

Get a storage object for a URI.

Parameters:: uri (str) – The URI to get a storage object for.
Returns:: A remote storage class.
Return type:: RemoteStorage
Raises:: ValueError – If the URI is not supported.