Skip to content

migrations.rst

Latest commit

 

History

History
313 lines (226 loc) · 11.6 KB

migrations.rst

File metadata and controls

313 lines (226 loc) · 11.6 KB

Migration Guide

This page can help when migrating Taskgraph across major versions.

19.x -> 20.x

  • Tasks using if-dependencies will no longer pull these dependencies into the graph during the target phase. If a desired dependency is no longer showing up in the target graph, adjust your target_tasks logic to ensure the dependency is explicitly selected.

18.x -> 19.x

  • The Schema class in taskgraph.util.schema is now based on msgspec instead of voluptuous. The old voluptuous-based class has been renamed to LegacySchema. Both will continue to be supported.

    If you have custom schemas using Schema, you have two options:

    Option 1: Switch to LegacySchema to keep using voluptuous:

    from taskgraph.util.schema import LegacySchema

    Option 2: Migrate to the new msgspec-based Schema:

    from taskgraph.util.schema import Schema

class MySchema(Schema):
    foo: str
    bar: int = 10

    The new Schema class uses kebab-case renaming and forbids unknown fields by default. See the msgspec documentation for more details.

  • validate_schema now supports both voluptuous and msgspec schemas, and optionally_keyed_by accepts a use_msgspec=True flag for msgspec output.

17.x -> 18.x

  • Stop setting the run.sparse-profile key in all tasks which perform a Mercurial clone. If sparse profiles are still required, the task must perform its own clone and not rely on the run-task script to do it.
  • run-task no longer cleans up the fetches directory. If using generic-worker's multiuser engine, this should have no impact. If you are using a worker that does not create separate task workspaces and you require the fetches dir to be cleaned up, adjust your tasks' commands to remove $MOZ_FETCHES_DIR explicitly.

16.x -> 17.x

  • Upgrade to Python 3.9 or above.

  • The base_ref parameter could now be None if it wasn't explicitly passed into the Decision task. If this was relied upon, do the following instead:

    from taskgraph.util.vcs import get_repository

repo = get_repository()
base_ref = repo.default_branch

  • The base_rev parameter is no longer being reset to the merge-base. This means in some cases, base_rev might not actually be an ancestor of head_rev. If you need the merge-base, you can use:

    from taskgraph.util.vcs import get_repository

repo = get_repository()
base_rev = repo.find_latest_common_revision(parameters["base_rev"], parameters["head_rev"])

  • The base_ref is no longer being fetched by run-task. If you were relying on it, you can run git fetch origin <base_ref> in a subprocess.

  • The run-task scripts no longer fetches all heads when head_ref isn't specified. If this behaviour was relied upon, you can run git fetch +refs/heads/*:refs/remotes/work/* in a subprocess.

15.x -> 16.x

  • Stop using the --tag flag with taskgraph build-image. Images must now be tagged manually via Docker post build.
  • Running taskgraph load-task will no longer pause task execution and instead immediately execute the task's command after loading. Now you must use the -i/--interactive flag to pause execution.
  • Utility functions in taskgraph.util.taskcluster no longer accept the use_proxy argument. Instead the presence of TASKCLUSTER_PROXY_URL in the environment will be used to determine whether or not to use the Taskcluster proxy. Remove use_proxy from all function calls, and then either set TASKCLUSTER_PROXY_URL or unset it depending on whether using the proxy is desired or not.
  • Utility functions in taskgraph.util.taskcluster now raise taskcluster.TaskclusterRestFailure instead of requests.HTTPError.

14.x -> 15.x

  • get_primary_dependency now requires a primary-dependency-label to be set on the task it is passed instead of primary-kind-dependency. Update any tasks you were calling this for to set this attribute. (If your only usage of this is indirectly through from_deps, no change is needed: from-deps generated tasks will set this for you).

13.x -> 14.x

  • The {task_workdir} string in environment variables no longer gets interpolated by run-task. This is backing out a feature introduced in version 13.x, so this release introduces no new backwards incompatibilities with version 12.x or earlier.

12.x -> 13.x

  • Remove all run.cache-dotcache keys. If it was set to true, replace it with:

    run:
  use-caches: [checkout, <caches>]

    Where caches can be any of cargo, pip, uv or npm. If the task was setting up .cache for another tool, a mount will need to be created for it manually. If use-caches was previously set to false, omit checkout in the example above. If use-caches was previously set to true, replace true with the value above (including checkout).

    In Dockerfiles, replace VOLUME /builds/worker/.cache by VOLUME /builds/worker/.task-cache/{uv, cargo, pip, npm} as necessary.

  • Invert any usage of the dict keys and values returned by get_ancestors:

    For example, if you were using:

    for label, taskid in get_ancestors(...):
  ...

    Change it to:

    for taskid, label in get_ancestors(...):
  ...

    Note that due to this change get_ancestors may return multiple tasks with the same label now, which your code may need to deal with.

11.x -> 12.x

  • Add target_tasks_method to the front of the filters parameter wherever you are also using a custom filter.

    For example, if you are passing in:

    filters: ["my_custom_filter"]

    Change it to:

    filters: ["target_tasks_method", "my_custom_filter"]

    No action is necessary if the filters parameter was empty.

  • Change all references from build-docker-image to docker-image.

10.x -> 11.x

  • A hardcoded path to a Python installation was removed for MacOS generic-workers. If Mac tasks start failing after upgrade and you are able to change the worker environment, ensure the python3 binary is available on the $PATH. If you cannot change the worker environment, add the following to the definitions of the failing tasks:

    run:
  run-task-command: ["/tools/python36/bin/python3", "run-task"]

  • When defining custom actions with taskgraph.actions.registry.register_callback_action, make the following changes:

    • If you are passing generic=True to the function, remove this argument.
    • If the argument isn't present, or you are passing generic=False, then add a new argument called permission=<cb_name>, where <cb_name> is the value of whatever you are passing to the cb_name argument.

9.x -> 10.x

  • Directories listed as VOLUME in Dockerfiles are created before any other instructions, so those instructions may need to be updated (e.g. RUN mkdir)
  • fetch-content no longer relies on file extension to detect archives, so you. may need to explicitly disable extract for some fetches.

8.x -> 9.x

  • Replace references to taskgraph.util.memoize.memoize with functools.cache. E.g, change @memoize to @cache. If using Python 3.8, use @functools.lru_cache(maxsize=None) instead.
  • Pay close attention to tasks that use task-defaults to merge configuration containing by-<attribute> keys. The :func:`taskgraph.util.templates.merge` function will no longer attempt to merge keys containing these attributes, which may result in changes to your graph. You can use the :ref:`diff feature <diffing graphs>` to help detect possible changes.

7.x -> 8.x

  • Replace all references to taskgraph.files_changed. Instead, use one of:
    • The files_changed parameter
    • The get_files_changed method on an instance of taskgraph.util.vcs.Repository
    • Mercurial repositories relying on hgmo's json-automationrelevance endpoint will need to in-line this logic into their own custom Taskgraph logic
  • In tasks using the from_deps transforms, remove from-deps.set-name if it is set to true
  • Update any references to pull request cached task indexes from {cache_prefix}.cache.head.{head_ref}... to {cache_prefix}.cache.pr... (i.e, add pr and remove the head.{head_ref})

6.x -> 7.x

  • Upgrade to Python 3.8 or higher
  • Ensure root_dir now points to taskcluster instead of taskcluster/ci. Typically this value is not passed in explicitly by consumers, but updates are likely required if you have custom code that uses any of the following objects:
    • taskgraph.config.GraphConfig
    • taskgraph.config.load_graph_config
    • taskgraph.generator.TaskGraphGenerator
    • taskgraph.generator.load_tasks_for_kinds
    • The -r/--root flag on the taskgraph binary
  • Rename the run_job_using decorator to run_task_using
  • Move config.yml from taskcluster/ci to taskcluster
  • Rename the taskcluster/ci directory to taskcluster/kinds
  • Replace references to taskgraph.transforms.job with taskgraph.transforms.run
  • Replace references to taskgraph.transforms.release_notifications with taskgraph.transforms.notify
  • Replace references to taskgraph.target_tasks._target_task with taskgraph.target_tasks.register_target_task
  • Stop using or inline taskgraph.util.decision.make_decision_task
  • Stop using the decision-mobile docker image
  • Ensure MacOS workers that need Mercurial have hg on their PATH

5.x -> 6.x

  • Replace all uses of command-context with the more generalized task-context

4.x -> 5.x

  • Upgrade to Python 3.7 or higher

3.x -> 4.x

  • Remove all uses of the disable-seccomp key in the worker section of task definitions.

2.x -> 3.x

  • Use a decision image at least as recent as this one.
  • Rename config.kind_dependencies_tasks to config.kind_dependencies_tasks.values().
  • Rename vcs.head_ref to vcs.head_rev. vcs.head_ref still exists but points to the actual reference instead of the revision.
  • Rename vcs.base_ref to vcs.base_rev. Same rationale as above.

1.x -> 2.x

  • For all kinds using the :mod:`transform loader <taskgraph.loader.transform>`, rename the following keys in both the kind.yml file and any files referenced in jobs-from:

    jobs -> tasks
jobs-from -> tasks-from
job-defaults -> task-defaults

  • Rename taskgraph.util.schema.WHITELISTED_SCHEMA_IDENTIFIERS to taskgraph.util.schema.EXCEPTED_SCHEMA_IDENTIFIERS.

  • Rename any instances of taskgraph.optimize.Either to taskgraph.optimize.Any.

  • Add a deadline parameter as the third argument to any custom optimization strategies' :func:`~taskgraph.optimize.OptimizationStrategy.should_replace_task` function. For migration purposes it doesn't need to be used.

  • Replace taskgraph.util.taskcluster.status_task with taskgraph.util.taskcluster.state_task.