API documentation¶
Core¶
-
class
girder_worker.
GirderWorkerPluginABC
(app, *args, **kwargs)[source]¶ Abstract base class for Girder Worker plugins. Plugins must descend from this class; see the Plugins section for more information.
-
class
girder_worker.task.
Task
[source]¶ Girder Worker Task object. Tasks defined by plugins must be subclasses of this class, however you will typically not need to reference it yourself, as it will be automatically instantiated by the girder_worker celery app. See Creating tasks for instructions.
-
canceled
¶ A property to indicate if a task has been canceled.
Returns: True is this task has been canceled, False otherwise. Return type: bool
-
Transforms¶
-
class
girder_worker_utils.transforms.girder_io.
GirderUploadJobArtifact
(job_id=None, name=None, **kwargs)[source]¶ This class can be used to upload a directory of files or a single file as artifacts attached to a Girder job. These files are only uploaded if they exist, so this is an optional output.
Currently, only a flat directory of files is supported; the transform does not recurse through nested directories, though that may change in the future.
Docker¶
Tasks¶
Transforms¶
-
class
girder_worker.docker.transforms.
BindMountVolume
(host_path, container_path, mode='rw')[source]¶ A volume that will be bind mounted into a docker container.
Parameters: - host_path (str) – The path on the host machine.
- container_path (str) – The path in the container this volume will be mounted at.
- mode (str) – The mounting mode
-
class
girder_worker.docker.transforms.
ChunkedTransferEncodingStream
(url, headers={}, **kwargs)[source]¶ A stream transform that allows data to be streamed using HTTP Chunked Transfer Encoding to a server.
Parameters: - url (str) – Destination URL for the stream.
- headers – HTTP headers to send.
-
class
girder_worker.docker.transforms.
Connect
(input, output)[source]¶ This utility class represents the connection between a
girder_worker.docker.transforms.NamedOutputPipe
orgirder_worker.docker.transforms.NamedInputPipe
and one of the other streaming transforms. Girder Worker will stream the data to or from the named pipe.Parameters: - input (
girder_worker.docker.transforms.NamedOutputPipe
orgirder_worker.docker.transforms.girder.GirderFileIdToStream
) – The input side of the connection - output (
girder_worker.docker.transforms.NamedInputPipe
orgirder_worker.docker.transforms.ChunkedTransferEncodingStream
orgirder_worker.docker.transforms.HostStdOut
orgirder_worker.docker.transforms.HostStdErr
) – The output side of the connection
- input (
-
class
girder_worker.docker.transforms.
ContainerStdErr
[source]¶ Represents the standard error stream of the container. Can be used with
girder_worker.docker.transforms.Connect
to redirect the containers standard error to another stream.
-
class
girder_worker.docker.transforms.
ContainerStdOut
[source]¶ Represents the standard output stream of the container. Can be used with
girder_worker.docker.transforms.Connect
to redirect the containers standard output to another stream.
-
class
girder_worker.docker.transforms.
HostStdErr
[source]¶ Represents the standard error stream on the host machine. Can be used with
girder_worker.docker.transforms.Connect
to write text to stderr.
-
class
girder_worker.docker.transforms.
HostStdOut
[source]¶ Represents the standard output stream on the host machine. Can be used with
girder_worker.docker.transforms.Connect
to write text to stdout.
-
class
girder_worker.docker.transforms.
NamedInputPipe
(name, container_path=None, host_path=None, volume=<girder_worker.docker.transforms._DefaultTemporaryVolume object>)[source]¶ A named pipe that can be open for read within a docker container. i.e. To stream data into a container.
Parameters: - name (str) – The name of the pipe.
- container_path (str) – The path in the container.
- host_path (str) – The path on the host machine.
- volume – Alternatively a
girder_worker.docker.transforms.BindMountVolume
instance can be provided. In which case the container_path and host_paths from the volume will be used when creating the pipe. The default location isgirder_worker.docker.transforms.TemporaryVolume.default
-
class
girder_worker.docker.transforms.
NamedOutputPipe
(name, container_path=None, host_path=None, volume=<girder_worker.docker.transforms._DefaultTemporaryVolume object>)[source]¶ A named pipe that can be opened for write within a docker container. i.e. To stream data out of a container.
Parameters: - name (str) – The name of the pipe.
- container_path (str) – The path in the container.
- host_path (str) – The path on the host machine.
- volume – Alternatively a
girder_worker.docker.transforms.BindMountVolume
instance can be provided. In which can the container_path and host_paths from the volume will be use when creating the pipe. The default location isgirder_worker.docker.transforms.TemporaryVolume.default
-
class
girder_worker.docker.transforms.
TemporaryVolume
(host_dir=None, mode=493)[source]¶ This is a class used to represent a temporary directory on the host that will be mounted into a docker container. girder_worker will automatically attach a default temporary volume. This can be reference using TemporaryVolume.default class attribute. A temporary volume can also be create in a particular host directory by providing the host_dir param.
Parameters: - host_dir (str) – The root directory on the host to use when creating the the temporary host path.
- mode (int) – The default mode applied to the temporary volume if it does not already exist.
-
class
girder_worker.docker.transforms.
VolumePath
(filename, volume=<girder_worker.docker.transforms._DefaultTemporaryVolume object>)[source]¶ A path on a docker volume. Must be a path relative to the root of the volume.
Parameters: - filename – The file name.
- volume (
girder_worker.docker.transforms.BindMountVolume
) – The volume this file lived on. If no volume is provided then the file will be ongirder_worker.docker.transforms.TemporaryVolume.default
-
class
girder_worker.docker.transforms.girder.
GirderFileIdToStream
(_id, **kwargs)[source]¶ This can be used to stream a Girder file into a docker container. See Streaming Girder files into a container for example usage.
Parameters: _id (str or ObjectId) – The Girder file ID.
-
class
girder_worker.docker.transforms.girder.
GirderFileIdToVolume
(_id, volume=<girder_worker.docker.transforms._DefaultTemporaryVolume object>, filename=None, **kwargs)[source]¶ This can be used to pass a Girder file into a docker container. It downloads the file to a bind mounted volume, and returns the container path of the file.
Parameters: - _id (str or ObjectId) – The Girder file ID.
- volume (
girder_worker.docker.transforms.BindMountVolume
) – The bind mount volume where the file will reside. - filename (str) – Alternate name for the file. Default is to use the name from Girder.
-
class
girder_worker.docker.transforms.girder.
GirderFolderIdToVolume
(_id, volume=<girder_worker.docker.transforms._DefaultTemporaryVolume object>, folder_name=None, **kwargs)[source]¶ This can be used to pass a Girder folder into a docker container. It downloads the folder to a bind mounted volume, and returns the container path of the directory.
Parameters: - _id (str or ObjectId) – The Girder folder ID.
- volume (
girder_worker.docker.transforms.BindMountVolume
) – The bind mount volume where the directory will reside. - folder_name (str) – Alternate name for the directory. Default is to use the name from Girder.
-
class
girder_worker.docker.transforms.girder.
GirderItemIdToVolume
(_id, volume=<girder_worker.docker.transforms._DefaultTemporaryVolume object>, **kwargs)[source]¶ This can be used to pass a Girder item into a docker container. It downloads the item to a bind mounted volume, and returns the container path of the directory.
Parameters: - _id (str or ObjectId) – The Girder item ID.
- volume (
girder_worker.docker.transforms.BindMountVolume
) – The bind mount volume where the item will reside. - item_name (str) – Alternate name for the file. Default is to use the name from Girder.
-
class
girder_worker.docker.transforms.girder.
GirderUploadVolumePathJobArtifact
(volumepath, job_id=None, name=None, upload_on_exception=False, **kwargs)[source]¶ This transform can be used to upload artifacts produced during a docker task execution and attach them to the corresponding job in Girder. This can be useful for tracing and debugging jobs, or simply collecting intermediate information during job execution. If the passed in path does not exist, this is a no-op.
Parameters: - volumepath (
girder_worker.docker.transforms.VolumePath
) – A volume path pointing to a mounted directory or file. If a directory, all files within the directory will be uploaded as artifacts to the job. If a file, just uploads the single file. If it does not exist, no action is performed. - job_id (str) – The job ID to attach the artifacts to. If calling this from Girder via
docker_run.delay
, you will not need to set this, as it will be set automatically. - name (str) – A name for the artifact. Only applies for single file paths. If not specified, will use the basename of the file.
- upload_on_exception (bool) – If True, this transform will occur even if the docker task
fails. This can be used to debug failed
docker_run
tasks.
- volumepath (
-
class
girder_worker.docker.transforms.girder.
GirderUploadVolumePathToFolder
(volumepath, folder_id, delete_file=False, **kwargs)[source]¶ This transform uploads data in a bind mount volume to a Girder folder. This should be used in
girder_result_hooks
to upload data produced by the task.Parameters: - volumepath (
girder_worker.docker.transforms.VolumePath
) – The location of the file or directory to upload. - folder_id (str or ObjectId) – The folder ID in Girder.
- delete_file (bool) – Whether to delete the data afterward.
- volumepath (
-
class
girder_worker.docker.transforms.girder.
GirderUploadVolumePathToItem
(volumepath, item_id, delete_file=False, **kwargs)[source]¶ This transform uploads data in a bind mount volume to a Girder item. This should be used in
girder_result_hooks
to upload files produced by the task.Parameters: - volumepath (
girder_worker.docker.transforms.VolumePath
) – The location of the file to upload. - item_id (str or ObjectId) – The item ID in Girder.
- delete_file (bool) – Whether to delete the file afterward.
- volumepath (
-
class
girder_worker.docker.transforms.girder.
ProgressPipe
(name='.girder_progress', volume=<girder_worker.docker.transforms._DefaultTemporaryVolume object>)[source]¶ This can be used to stream progress information out of a running docker container as part of a docker_run task. For a usage example, see Streaming progress reporting from Docker tasks to Girder jobs.
Parameters: - name (str) – The filename, which will be a named pipe open for reading from the host.
- volume (
girder_worker.docker.transforms.BindMountVolume
) – The bind mount volume where the underlying named pipe will reside.