gecko_taskgraph.transforms.job package

Submodules

gecko_taskgraph.transforms.job.common module

Common support for various job types. These functions are all named after the worker implementation they operate on, and take the same three parameters, for consistency.

gecko_taskgraph.transforms.job.common.add_artifacts(config, job, taskdesc, path)

gecko_taskgraph.transforms.job.common.add_tooltool(config, job, taskdesc, internal=False)

Give the task access to tooltool.

Enables the tooltool cache. Adds releng proxy. Configures scopes.

By default, only public tooltool access will be granted. Access to internal tooltool can be enabled via internal=True.

This can only be used with run-task tasks, as the cache name is reserved for use with run-task.

gecko_taskgraph.transforms.job.common.clone_type(config, job)

gecko_taskgraph.transforms.job.common.docker_worker_add_artifacts(config, job, taskdesc)

Adds an artifact directory to the task

gecko_taskgraph.transforms.job.common.generic_worker_add_artifacts(config, job, taskdesc)

Adds an artifact directory to the task

gecko_taskgraph.transforms.job.common.get_cache_name(config, job)

gecko_taskgraph.transforms.job.common.get_expiration(config, policy='default')

gecko_taskgraph.transforms.job.common.setup_secrets(config, job, taskdesc)

Set up access to secrets via taskcluster-proxy. The value of run[‘secrets’] should be a boolean or a list of secret names that can be accessed.

gecko_taskgraph.transforms.job.common.support_vcs_checkout(config, job, taskdesc, repo_configs)

Update a job/task with parameters to enable a VCS checkout.

This can only be used with run-task tasks, as the cache name is reserved for run-task tasks.

gecko_taskgraph.transforms.job.distro_package module

Support for running spidermonkey jobs via dedicated scripts

class gecko_taskgraph.transforms.job.distro_package.CommonSchema(*, dsc: SourceDefinition | None = None, tarball: SourceDefinition | None = None, name: str | None = None, patch: str | None = None, pre_build_command: str | None = None, arch: str | None = None, packages: list[str] | None = None, resolver: Literal['apt-get', 'aptitude'] | None = None, workdir: str)

Bases: Schema

arch: str | None

dsc: SourceDefinition | None

name: str | None

packages: list[str] | None

patch: str | None

pre_build_command: str | None

resolver: Literal['apt-get', 'aptitude'] | None

tarball: SourceDefinition | None

workdir: str

class gecko_taskgraph.transforms.job.distro_package.DebianSchema(*, dsc: SourceDefinition | None = None, tarball: SourceDefinition | None = None, name: str | None = None, patch: str | None = None, pre_build_command: str | None = None, arch: str | None = None, packages: list[str] | None = None, resolver: Literal['apt-get', 'aptitude'] | None = None, workdir: str, using: Literal['debian-package'], dist: str)

Bases: CommonSchema

dist: str

using: Literal['debian-package']

class gecko_taskgraph.transforms.job.distro_package.SourceDefinition(*, url: str, sha256: str)

Bases: Schema

sha256: str

url: str

class gecko_taskgraph.transforms.job.distro_package.UbuntuSchema(*, dsc: SourceDefinition | None = None, tarball: SourceDefinition | None = None, name: str | None = None, patch: str | None = None, pre_build_command: str | None = None, arch: str | None = None, packages: list[str] | None = None, resolver: Literal['apt-get', 'aptitude'] | None = None, workdir: str, using: Literal['ubuntu-package'], dist: str)

Bases: CommonSchema

dist: str

using: Literal['ubuntu-package']

gecko_taskgraph.transforms.job.distro_package.common_package(config, job, taskdesc, distro, version)

gecko_taskgraph.transforms.job.distro_package.docker_worker_debian_package(config, job, taskdesc)

gecko_taskgraph.transforms.job.distro_package.docker_worker_ubuntu_package(config, job, taskdesc)

gecko_taskgraph.transforms.job.hazard module

Support for running hazard jobs via dedicated scripts

class gecko_taskgraph.transforms.job.hazard.HazRunSchema(*, using: Literal['hazard'], command: str, mozconfig: str | None = None, secrets: bool | list[str] | None = None, clone_with: Literal['hg', 'git'] | None = 'git', workdir: str | None = None)

Bases: Schema

clone_with: Literal['hg', 'git'] | None

command: str

mozconfig: str | None

secrets: bool | list[str] | None

using: Literal['hazard']

workdir: str | None

gecko_taskgraph.transforms.job.hazard.docker_worker_hazard(config, job, taskdesc)

gecko_taskgraph.transforms.job.mach module

Support for running mach tasks (via run-task)

class gecko_taskgraph.transforms.job.mach.MachSchema(*, using: Literal['mach'], mach: str | TaskRefTypeSchema, sparse_profile: str | None = None, comm_checkout: bool, prepend_env: dict[str, str] | None = None, workdir: str | None = None, use_caches: bool | list[str] | None = None, clone_with: Literal['hg', 'git'] | None = 'git')

Bases: Schema

clone_with: Literal['hg', 'git'] | None

comm_checkout: bool

mach: str | TaskRefTypeSchema

prepend_env: dict[str, str] | None

sparse_profile: str | None

use_caches: bool | list[str] | None

using: Literal['mach']

workdir: str | None

gecko_taskgraph.transforms.job.mach.configure_mach(config, job, taskdesc)

gecko_taskgraph.transforms.job.mozharness module

Support for running jobs via mozharness. Ideally, most stuff gets run this way, and certainly anything using mozharness should use this approach.

class gecko_taskgraph.transforms.job.mozharness.MozharnessRunSchema(*, using: Literal['mozharness'], script: str, config_paths: list[str] | None = None, config: list[str], actions: list[str] | None = None, options: list[str] | None = None, custom_build_variant_cfg: str | None = None, extra_config: dict | None = None, tooltool_downloads: bool | Literal['public', 'internal'], secrets: bool | list[str], taskcluster_proxy: bool, keep_artifacts: bool, job_script: str | None = None, requires_signed_builds: bool, use_caches: bool | list[str] | None = None, clone_with: Literal['hg', 'git'] | None = 'git', sparse_profile: str | None = None, use_simple_package: bool, use_magic_mh_args: bool, comm_checkout: bool, workdir: str | None = None, run_as_root: bool | None = None)

Bases: Schema

actions: list[str] | None

clone_with: Literal['hg', 'git'] | None

comm_checkout: bool

config: list[str]

config_paths: list[str] | None

custom_build_variant_cfg: str | None

extra_config: dict | None

job_script: str | None

keep_artifacts: bool

options: list[str] | None

requires_signed_builds: bool

run_as_root: bool | None

script: str

secrets: bool | list[str]

sparse_profile: str | None

taskcluster_proxy: bool

tooltool_downloads: bool | Literal['public', 'internal']

use_caches: bool | list[str] | None

use_magic_mh_args: bool

use_simple_package: bool

using: Literal['mozharness']

workdir: str | None

gecko_taskgraph.transforms.job.mozharness.mozharness_on_docker_worker_setup(config, job, taskdesc)

gecko_taskgraph.transforms.job.mozharness.mozharness_on_generic_worker(config, job, taskdesc)

gecko_taskgraph.transforms.job.mozharness_test module

class gecko_taskgraph.transforms.job.mozharness_test.MozharnessTestRunSchema(*, using: Literal['mozharness-test'], test: MozharnessTestSchema, workdir: str | None = None, clone_with: Literal['hg', 'git'] | None = 'git')

Bases: Schema

clone_with: Literal['hg', 'git'] | None

test: MozharnessTestSchema

using: Literal['mozharness-test']

workdir: str | None

class gecko_taskgraph.transforms.job.mozharness_test.MozharnessTestSchema(*, test_platform: str, mozharness: ~gecko_taskgraph.transforms.test.MozharnessSchema, docker_image: ~typing.Annotated[~typing.Any, <taskgraph.util.schema.OptionallyKeyedBy object at 0x7f367f4852b0>], loopback_video: bool, loopback_audio: bool, max_run_time: ~typing.Annotated[~typing.Any, <taskgraph.util.schema.OptionallyKeyedBy object at 0x7f367f4857c0>], retry_exit_status: list[int] | None = None)

Bases: Schema

docker_image: OptionallyKeyedBy object at 0x7f367f4852b0>]

loopback_audio: bool

loopback_video: bool

max_run_time: OptionallyKeyedBy object at 0x7f367f4857c0>]

mozharness: MozharnessSchema

retry_exit_status: list[int] | None

test_platform: str

gecko_taskgraph.transforms.job.mozharness_test.get_variant(test_platform)

gecko_taskgraph.transforms.job.mozharness_test.installer_url(taskdesc)

gecko_taskgraph.transforms.job.mozharness_test.mozharness_test_on_docker(config, job, taskdesc)

gecko_taskgraph.transforms.job.mozharness_test.mozharness_test_on_generic_worker(config, job, taskdesc)

gecko_taskgraph.transforms.job.mozharness_test.test_packages_url(taskdesc)

Account for different platforms that name their test packages differently

gecko_taskgraph.transforms.job.python_test module

Support for running mach python-test tasks (via run-task)

class gecko_taskgraph.transforms.job.python_test.PythonTestSchema(*, using: Literal['python-test'], subsuite: str, workdir: str | None = None, use_caches: bool | list[str] | None = None, prepend_env: dict[str, str] | None = None, clone_with: Literal['hg', 'git'] | None = 'git')

Bases: Schema

clone_with: Literal['hg', 'git'] | None

prepend_env: dict[str, str] | None

subsuite: str

use_caches: bool | list[str] | None

using: Literal['python-test']

workdir: str | None

gecko_taskgraph.transforms.job.python_test.configure_python_test(config, job, taskdesc)

gecko_taskgraph.transforms.job.run_task module

Support for running jobs that are invoked via the run-task script.

class gecko_taskgraph.transforms.job.run_task.RunTaskSchema(*, using: Literal['run-task'], use_caches: bool | list[str] | None = None, checkout: bool = True, cwd: str | None = None, sparse_profile: str | None = None, sparse_profile_prefix: str | None = None, shallow_clone: bool | None = None, clone_with: Literal['hg', 'git'] | None = 'git', comm_checkout: bool = False, command: list[str | TaskRefTypeSchema] | str | TaskRefTypeSchema, workdir: str | None = None, tooltool_downloads: bool | Literal['public', 'internal'] = False, run_as_root: bool | None = None)

Bases: Schema

checkout: bool

clone_with: Literal['hg', 'git'] | None

comm_checkout: bool

command: list[str | TaskRefTypeSchema] | str | TaskRefTypeSchema

cwd: str | None

run_as_root: bool | None

shallow_clone: bool | None

sparse_profile: str | None

sparse_profile_prefix: str | None

tooltool_downloads: bool | Literal['public', 'internal']

use_caches: bool | list[str] | None

using: Literal['run-task']

workdir: str | None

gecko_taskgraph.transforms.job.run_task.common_setup(config, job, taskdesc, command)

gecko_taskgraph.transforms.job.run_task.docker_worker_run_task(config, job, taskdesc)

gecko_taskgraph.transforms.job.run_task.generic_worker_run_task(config, job, taskdesc)

gecko_taskgraph.transforms.job.run_task.psutil_platforms(job)

gecko_taskgraph.transforms.job.run_task.script_url(config, script)

gecko_taskgraph.transforms.job.spidermonkey module

Support for running spidermonkey jobs via dedicated scripts

class gecko_taskgraph.transforms.job.spidermonkey.SmRunSchema(*, using: Literal['spidermonkey', 'spidermonkey-package'], spidermonkey_variant: str, spidermonkey_platform: str | None = None, workdir: str | None = None, tooltool_downloads: bool | Literal['public', 'internal'], clone_with: Literal['hg', 'git'] | None = 'git')

Bases: Schema

clone_with: Literal['hg', 'git'] | None

spidermonkey_platform: str | None

spidermonkey_variant: str

tooltool_downloads: bool | Literal['public', 'internal']

using: Literal['spidermonkey', 'spidermonkey-package']

workdir: str | None

gecko_taskgraph.transforms.job.spidermonkey.docker_worker_spidermonkey(config, job, taskdesc)

gecko_taskgraph.transforms.job.spidermonkey.generic_worker_spidermonkey(config, job, taskdesc)

gecko_taskgraph.transforms.job.toolchain module

Support for running toolchain-building jobs via dedicated scripts

class gecko_taskgraph.transforms.job.toolchain.ToolchainRunSchema(*, using: ~typing.Literal['toolchain-script'], script: str, arguments: list[str] | None = None, tooltool_downloads: bool | ~typing.Literal['public', 'internal'], sparse_profile: str | None, sparse_profile_prefix: str | None = None, resources: list[str] | None = None, toolchain_artifact: str, toolchain_alias: ~typing.Annotated[~typing.Any, <taskgraph.util.schema.OptionallyKeyedBy object at 0x7f368088ef30>] | None = None, toolchain_env: dict[str, object] | None = None, toolchain_extract: bool | None = None, clone_with: ~typing.Literal['hg', 'git'] | None = 'git', workdir: str | None = None)

Bases: Schema

arguments: list[str] | None

clone_with: Literal['hg', 'git'] | None

resources: list[str] | None

script: str

sparse_profile: str | None

sparse_profile_prefix: str | None

toolchain_alias: OptionallyKeyedBy object at 0x7f368088ef30>] | None

toolchain_artifact: str

toolchain_env: dict[str, object] | None

toolchain_extract: bool | None

tooltool_downloads: bool | Literal['public', 'internal']

using: Literal['toolchain-script']

workdir: str | None

gecko_taskgraph.transforms.job.toolchain.common_toolchain(config, job, taskdesc, is_docker)

gecko_taskgraph.transforms.job.toolchain.docker_worker_toolchain(config, job, taskdesc)

gecko_taskgraph.transforms.job.toolchain.generic_worker_toolchain(config, job, taskdesc)

gecko_taskgraph.transforms.job.toolchain.get_digest_data(config, run, taskdesc, fetches)

Module contents

Convert a job description into a task description.

Jobs descriptions are similar to task descriptions, but they specify how to run the job at a higher level, using a “run” field that can be interpreted by run-using handlers in taskcluster/gecko_taskgraph/transforms/job.

class gecko_taskgraph.transforms.job.FetchArtifactSchema(*, artifact: str, dest: str | None = None, extract: bool | None = None, verify_hash: bool | None = None)

Bases: Schema

artifact: str

dest: str | None

extract: bool | None

verify_hash: bool | None

class gecko_taskgraph.transforms.job.JobDescriptionSchema(*, name: str | None = None, label: str | None = None, description: str, attributes: dict[str, object] | None=None, task_from: str | None = None, dependencies: dict[str, object] | None=None, if_dependencies: list[str] | None = None, soft_dependencies: list[str] | None = None, requires: Literal['all-completed', 'all-resolved'] | None=None, expires_after: str | None = None, expiration_policy: str | None = None, routes: list[str] | None = None, scopes: list[str] | None = None, tags: dict[str, str] | None=None, extra: dict[str, object] | None=None, treeherder: TreeherderSchema | None = None, index: IndexSchema | None = None, run_on_repo_type: list[~typing.Literal['git', 'hg']] | None=None, run_on_projects: Any, <taskgraph.util.schema.OptionallyKeyedBy object at 0x7f3679733f50>] | None=None, run_on_git_branches: list[str] | None = None, shipping_phase: Literal['build', 'promote', 'push', 'ship'] | None=None, shipping_product: str | None = None, always_target: bool | None = None, optimization: OptimizationSchema | None = None, use_sccache: bool | None = None, use_python: Literal['system', 'default'] | str | None=None, use_uv: bool | None = None, priority: Any, <taskgraph.util.schema.OptionallyKeyedBy object at 0x7f3679730260>] | None=None, when: WhenSchema | None = None, fetches: dict[str, list[str | ~gecko_taskgraph.transforms.job.FetchArtifactSchema]] | None=None, run: JobRunSchema, worker_type: str | None, worker: dict | None = None)

Bases: Schema

always_target: bool | None

attributes: dict[str, object] | None

dependencies: dict[str, object] | None

description: str

expiration_policy: str | None

expires_after: str | None

extra: dict[str, object] | None

fetches: dict[str, list[str | FetchArtifactSchema]] | None

if_dependencies: list[str] | None

index: IndexSchema | None

label: str | None

name: str | None

optimization: OptimizationSchema | None

priority: OptionallyKeyedBy object at 0x7f3679730260>] | None

requires: Literal['all-completed', 'all-resolved'] | None

routes: list[str] | None

run: JobRunSchema

run_on_git_branches: list[str] | None

run_on_projects: OptionallyKeyedBy object at 0x7f3679733f50>] | None

run_on_repo_type: list[Literal['git', 'hg']] | None

scopes: list[str] | None

shipping_phase: Literal['build', 'promote', 'push', 'ship'] | None

shipping_product: str | None

soft_dependencies: list[str] | None

tags: dict[str, str] | None

task_from: str | None

treeherder: TreeherderSchema | None

use_python: Literal['system', 'default'] | str | None

use_sccache: bool | None

use_uv: bool | None

when: WhenSchema | None

worker: dict | None

worker_type: str | None

class gecko_taskgraph.transforms.job.JobRunSchema(*, using: str, workdir: str | None = None)

Bases: Schema

using: str

workdir: str | None

class gecko_taskgraph.transforms.job.WhenSchema(*, files_changed: list[str] | None = None)

Bases: Schema

files_changed: list[str] | None

gecko_taskgraph.transforms.job.add_perfherder_fetch_content_artifact(config, jobs)

gecko_taskgraph.transforms.job.always_optimized(config, job, taskdesc)

gecko_taskgraph.transforms.job.configure_taskdesc_for_run(config, job, taskdesc, worker_implementation)

Run the appropriate function for this job against the given task description.

This will raise an appropriate error if no function exists, or if the job’s run is not valid according to the schema.

gecko_taskgraph.transforms.job.get_attribute(dict, key, attributes, attribute_name)

Get attribute_name from the given attributes dict, and if there is a corresponding value, set key in dict to that value.

gecko_taskgraph.transforms.job.get_platform(job)

gecko_taskgraph.transforms.job.make_task_description(config, jobs)

Given a build description, create a task description

gecko_taskgraph.transforms.job.run_job_using(worker_implementation, run_using, schema=None, defaults={})

Register the decorated function as able to set up a task description for jobs with the given worker implementation and run.using property. If schema is given, the job’s run field will be verified to match it.

The decorated function should have the signature using_foo(config, job, taskdesc) and should modify the task description in-place. The skeleton of the task description is already set up, but without a payload.

gecko_taskgraph.transforms.job.set_implementation(config, jobs)

gecko_taskgraph.transforms.job.set_label(config, jobs)

gecko_taskgraph.transforms.job.use_fetches(config, jobs)

gecko_taskgraph.transforms.job.use_system_python(config, jobs)

gecko_taskgraph.transforms.job.use_uv(config, jobs)