dflow.plugins package

Submodules

dflow.plugins.bohrium module

class dflow.plugins.bohrium.BohriumContext(username: str | None = None, phone: str | None = None, password: str | None = None, bohrium_url: str | None = None, executor: str | None = None, extra: dict | None = None, authorization: str | None = None)

Bases: Context

Bohrium context

Parameters:

  • username – email for Bohrium

  • phone – phone number for Bohrium

  • password – password for Bohrium

  • bohrium_url – url for Bohrium

  • executor – executor

  • extra – extra arguments

  • authorization – JWT token

login()
render(template)

render original template and return a new template

class dflow.plugins.bohrium.BohriumDatasetsArtifact(path, sub_path=None)

Bases: DispatcherArtifact

bohrium_download(name: str, path: str)
download(name: str, path: str)
classmethod from_urn(urn: str)
get_urn() → str
modify_config(name: str, machine) → str
remote_download(name: str, path: str)
sub_path(path: str)
class dflow.plugins.bohrium.BohriumExecutor(executor: str | None = None, extra: dict | None = None)

Bases: Executor

Bohrium executor

Parameters:

  • executor – executor

  • extra – extra arguments, will override extra defined in global context

render(template)

render original template and return a new template, do not modify self in this method to make the executor reusable

class dflow.plugins.bohrium.TiefblueClient(bohrium_url: str | None = None, username: str | None = None, phone: str | None = None, password: str | None = None, authorization: str | None = None, project_id: str | None = None, token: str | None = None, prefix: str | None = None, sharePath: str | None = None, userSharePath: str | None = None, tiefblue_url: str | None = None, ticket: str | None = None)

Bases: StorageClient

copy(src, dst)
download(key, path)
get_md5(key)
get_token(retry=1)
list(prefix, recursive=False)
to_dict()
upload(key, path)
dflow.plugins.bohrium.create_job_group(job_group_name)
dflow.plugins.bohrium.login(username=None, phone=None, password=None, bohrium_url=None)
dflow.plugins.bohrium.update_headers()

dflow.plugins.datasets module

class dflow.plugins.datasets.DatasetsArtifact(element, version, type='datasets', rclone_image=None, rclone_image_pull_policy=None, rclone_type=None, ftp_host=None, webdav_host=None, user=None, password=None, rclone_kwargs=None, sub_path=None)

Bases: DispatcherArtifact

bohrium_download(name: str, path: str)
download(name: str, path: str)
classmethod from_rclone_config(config: str)
classmethod from_urn(urn: str)
get_bohrium_urn(name: str) → str
get_mount_script(path)
get_urn() → str
modify_config(name: str, machine) → str
remote_download(name: str, path: str)
render(template: PythonOPTemplate, name: str) → PythonOPTemplate
sub_path(path: str)
dflow.plugins.datasets.wait_for_mount(point, timeout=60)

dflow.plugins.dispatcher module

class dflow.plugins.dispatcher.DispatcherArtifact

Bases: CustomArtifact, ABC

abstract bohrium_download(name: str, path: str)
abstract modify_config(name: str, machine) → str
abstract remote_download(name: str, path: str)
class dflow.plugins.dispatcher.DispatcherExecutor(host: str | None = None, queue_name: str | None = None, port: int = 22, username: str = 'root', password: str | None = None, private_key_file: PathLike | None = None, image: str | None = None, image_pull_policy: str | None = None, command: str | List[str] = 'python3', remote_command: str | List[str] | None = None, map_tmp_dir: bool = True, machine_dict: dict | None = None, resources_dict: dict | None = None, task_dict: dict | None = None, json_file: PathLike | None = None, docker_executable: str | None = None, singularity_executable: str | None = None, container_args: str = '', podman_executable: str | None = None, remote_root: str | None = None, retry_on_submission_error: int | None = None, merge_sliced_step: bool = False, pre_script: str = '', post_script: str = '', clean: bool = True, remove_scheduling_strategies: bool = True, envs: Dict[str, str] | None = None, merge_bohrium_job_group: bool = False, terminate_grace_period: int | None = None)

Bases: Executor

Dispatcher executor

Parameters:

  • host – remote host

  • queue_name – queue name

  • port – SSH port

  • username – username

  • password – password

  • private_key_file – private key file for SSH

  • image – image for dispatcher

  • image_pull_policy – image pull policy for dispatcher

  • command – command for dispatcher

  • remote_command – command for running the script remotely

  • map_tmp_dir – map /tmp to ./tmp

  • machine_dict – machine config for dispatcher

  • resources_dict – resources config for dispatcher

  • task_dict – task config for dispatcher

  • json_file – JSON file containing machine and resources config

  • docker_executable – docker executable to run remotely

  • singularity_executable – singularity executable to run remotely

  • podman_executable – podman executable to run remotely

  • remote_root – remote root path for working

  • retry_on_submission_error – max retries on submission error

  • merge_sliced_step – handle multi slices in a single dispatcher job

  • terminate_grace_period – grace period in seconds after termination for collecting outputs

modify_step(step)
render(template)

render original template and return a new template, do not modify self in this method to make the executor reusable

dflow.plugins.dispatcher.update_dict(d1: dict, d2: dict) → None

dflow.plugins.launching module

dflow.plugins.lebesgue module

class dflow.plugins.lebesgue.LebesgueContext(username: str | None = None, password: str | None = None, bohrium_url: str = 'https://lbg-api.dp.tech', login_url: str = 'https://workflow.dp.tech/account_gw/login', app_name: str | None = None, org_id: str | None = None, user_id: str | None = None, tag: str | None = None, executor: str | None = None, extra: dict | None = None, authorization: str | None = None)

Bases: Context

Lebesgue context

Parameters:

  • username – user name for Lebesgue

  • password – password for Lebesgue

  • login_url – login url for Lebesgue

  • app_name – application name

  • org_id – organization ID

  • user_id – user ID

  • tag – tag

  • executor – executor

  • extra – extra arguments

  • authorization – JWT token

login()
render(template)

render original template and return a new template

class dflow.plugins.lebesgue.LebesgueExecutor(executor: str | None = None, extra: dict | None = None)

Bases: Executor

Lebesgue executor

Parameters:

  • executor – executor

  • extra – extra arguments, will override extra defined in global context

render(template)

render original template and return a new template, do not modify self in this method to make the executor reusable

dflow.plugins.metadata module

dflow.plugins.oss module

dflow.plugins.ray module

class dflow.plugins.ray.RayClusterExecutor(ray_host: str, ray_remote_path: str = '/tmp/ray', workdir: str = '~/dflow/workflows/{{workflow.name}}/{{pod.name}}', ray_install_mirror=None, ray_dependencies: List[Any] | None = None)

Bases: Executor

render(template: ScriptOPTemplate)

render original template and return a new template, do not modify self in this method to make the executor reusable

Module contents