External Autodiscovery plugin

[olblak]

This discussion is about getting feedback on the Updatecli external autodiscovery plugin feature.
It's still experimental, and we need as much feedback as possible to confidently move it to stable without future breaking changes

Problem

Defining update strategies in YAML is convenient, saves time, and helps teams collaborate. But when update rules grow complex, a small DSL can become limiting. Recently, we explored using WASM to let teams write manifest generators in the language they prefer. The feature is still experimental, and we welcome feedback from teams with complex update scenarios.

Solutions

Plugins can be written in any language that compiles to WebAssembly (WASM). We explored using the Extism framework.

We implemented on the Updatecli side this pull request:

• #7034

And an example of a custom plugin named demo.

A plugin must export the function _start (WASI). It is expected to receive a JSON object as input and return a JSON object containing the generated Updatecli manifests.

Input

A plugin receives a JSON object with the following fields:

{
  "scmid": "default",
  "actionid": "default",
  "rootdir"": "current directory"
  "spec": {
    "plugin_param1": "",
    "plugin_param2": ""
  }
}

The scmid field is provided by Updatecli, it's up to the plugin maintainer to decide how to use that information, but typically it's used to link resources to a scm configuration specified in the Updatecli manifest.

The actionid field is provided by Updatecli, it's up to the plugin maintainer to decide how to use that information but typically it's used to specify an action title.

The spec field is the plugin parameters detected by Updatecli.

Output

A plugin must output a JSON object such as

{
  "manifests": [
    "Updatecli manifest 1",
    "Updatecli manifest 2"
  ]
}

Where manifests contains the list of generated Updatecli manifests

Guidelines:

• Include a version field in generated manifests to indicate the minimum Updatecli version required.
• Do not set pipelineid in generated manifests (Updatecli overrides it).

Improvement

• Right now the plugin must remain local to the Updatecli manifest file, it would be nice to also support fetching plugins from an OCI registry.

[ringods]

@olblak 👋🏻

I'm trying to solve the following need: I want to generate SDKs for every version of an OpenAPI spec living an upstream repository. I have been looking at the problem from multiple angles and landed on the option of building this as custom UpdateCLI autodiscovery plugin.

My aim is to (sort of) replay the various releases from the upstream location into a git repository which will hold the generated SDK sources. The process implemented in my custom plugin would try to enumerate from the oldest version instead of the most recent one which seems to be the default in a number of plugins currently.

So here is how I would implement this:

1. fetch all the versions from the upstream location
2. fetch all the existing tags in the SDK repository
3. compare the two sets and return the oldest version possible not existing in the SDK repository
4. for the oldest version returned, process the SDK generation, commit, tag & publish

The upstream location could have a major version and some maintenance releases active in parallel. Hence, the commit and tag needs to be done on the correct branch:

• current major version on main
• maintenance version on dedicated branch, e.g. v1_1_x

Such a specific workflow is a good candidate for a custom external UpdateCLI plugin. But to implement this, I would very much like to reuse a lot of the functionality. I investigated the current architecture and found the current archicture lacking for the usecase I want to implement.

For instance, to implement step 1 of my process, fetching the existing versions, I can import Go code from the UpdateCLI repository, but I would rather not. Main reason is that I would need to import all the code for the various upstream locations I want to support, like git tags, github releases, gitlab releases and so on. It would lead to a very wide "interface" on my autodiscovery plugins to pass the required config properties for the various options. As an example, you see that already happening on the npm autodiscovery plugin to configure the registry authentication:

# updatecli.d/npm-private.yaml
autodiscovery:
  crawlers:
    npm:
      rootdir: "."
      # URL of your private npm registry
      url: "https://npm.example.com"
      # Authentication token (use environment variables for security)
      registrytoken: "${NPM_TOKEN}"
      # Optional: path to custom .npmrc file
      npmrcpath: "/path/to/.npmrc"
      versionfilter:
        kind: semver
        pattern: ">=1.0.0"

Source: https://www.updatecli.io/docs/plugins/autodiscovery/npm/#_private_registry_example

The current split in resources, scms and autodiscover plugins also lead to difficult code structuring within the UpdateCLI repository as can be seen here:

updatecli/pkg/plugins/scms/github/pullrequest.go

Lines 3 to 12 in d3152a8

	/*
	Keeping the pullrequest code under the github package to avoid cyclic dependencies between github <-> pullrequest
	We need to completely refactor the github package to split the different component into specific sub packages
	github/pullrequest
	github/target
	github/scm
	github/source
	github/condition
	*/

Proposal

While my first aim is to solve the reusability of existing functionality for custom autodiscovery plugins, the proposal here is broader but will (in my opinion) result in a much more fluent design.

If I want to reuse existing resources within the implementation of an autodiscovery plugin (bundled or external), I would like to use a preconfigured resource. If I take the npm example from before, I would use an npm resource and pass it to the autodiscovery plugin:

resources:
  example_registry: 
    name: Get latest axios version from npm registry
    kind: npm
    spec:
      # URL of your private npm registry
      url: "https://npm.example.com"
      # Authentication token (use environment variables for security)
      registrytoken: "${NPM_TOKEN}"
      # Optional: path to custom .npmrc file
      npmrcpath: "/path/to/.npmrc"      

autodiscovery:
  crawlers:
    npm:
      registry: example_registry
      rootdir: "."
      versionfilter:
        kind: semver
        pattern: ">=1.0.0"

This reuses already a lot of the npm resource interface for the authentication setup and removes this from the npm autodiscovery interface. Any changes to NPM registry auth would be limited to the npm. Notice that I'm not using the top-level sources key, but a new, to be introduced, resources key.

Now comes the biggest change, which I'm conceptually taking from Concourse CI: resource types. I propose an UpdateCLI plugin (bundled or external) to become a resource type. A resource type implementation can expose what it can be used for:

• source
• target
• condition
• action
• autodiscovery

A resource can be created under the resources key with the minimal configuration to successfully communicate with it, which means that in most cases this is the authentication setup. For example, configuring a github resource could be done as follows:

resources:
  my_ghe_codebase: 
    name: My codebase on Github Enterprise
    kind: github
    spec:
      # URL of your Github Enterprise endpoint
      # optional, defaults to https://api.github.com
      url: "https://api.mycompany.ghe.com"
      # Authentication token (use environment variables for security)
      token: "${GITHUB_TOKEN}"
      # Github owner
      owner: "me"
      # Repository to use
      repository: "mycodebase"

A resource type can expose more than one instance of each of the concepts above. For example, for the Github plugin, which works via the Github REST API, there would be various features offered for the core UpdateCLI concepts:

• source: [commits, branches, tags, releases]
• target: [tags]
• condition: [commits, branches, releases]
• action: [pullrequest]
• autodiscovery: []

The benefit towards the code structure is that you would only have a single github plugin folder bundling all the code together for all its functionality.

Using this resource in a pipeline would now be more like:

scms:
  default:
    kind: github
    spec:
      resource: my_ghe_codebase

Each of these resources should be passable to an autodiscovery plugin when their implementation allows it:

autodiscovery:
  crawlers:
    my_discovery:
      # pass a github resource allowing me to fetch github releases
      github: my_ghe_codebase
      # other attributes...

If a plugin can expose different concepts, an external WASM plugin would not only be able to implement autodiscovery functionality, but it would also be able to offer custom resource types out-of-tree to the UpdateCLI repository.

Open Issues

• I haven't given any thought so far on how this could be implemented in a backwards compatible way.

Closing remarks

• if this proposal seems a good direction, I would vote "no" for Should we move Updatecli to 1.0.0 ? while such a rearchitecting takes place.

[olblak]

Thanks for the detailed explanation, that's quite a lot of information to process

fetch all the versions from the upstream location
fetch all the existing tags in the SDK repository
compare the two sets and return the oldest version possible not existing in the SDK repository
for the oldest version returned, process the SDK generation, commit, tag & publish

Is it something you try to do from the external plugin or are you trying to generate Updatecli manifest?

I can import Go code from the UpdateCLI repository, but I would rather not.

This is also something I would avoid, Updatecli depends on a lot of library that you would end up importing as well.
I am considering moving some package to an external library as some will be usefull for https://github.com/updatecli/udash

Now comes the biggest change, which I'm conceptually taking from Concourse CI: resource types. I propose an UpdateCLI plugin (bundled or external) to become a resource type. A resource type implementation can expose what it can be used for:

I am not sure to understand, I like the concept of generic resources, but I am puzzled about how it would work technically speaking.
Anyway this change should be smooth, we heavily depend on Updatecli to maintain a lot of repositories so having to rewrite all manifests would represent a huge amount of work.

Each of these resources should be passable to an autodiscovery plugin when their implementation allows it:
I am also having a hard time how to use this because an autodiscovery can come with many different resources

if this proposal seems a good direction, I would vote "no" for Should we move Updatecli to 1.0.0 ? while such a rearchitecting takes place.

I don't think this discussion should be a blocker for 1.0.0 we can always release 2.0.0 in the future if needed. On top of this, I am not sure if I will have the bandwidth to drive a major refactoring soon. I am currently working on the milestone 1.0.0

[ringods]

@olblak I have been thinking deeper about this. Forget the manifest change for now from my previous proposal. I want to focus again on the item you started the discussion with: external autodiscovery plugin feature.

Now, why limit external WASM plugins only to the autodiscovery aspect of UpdateCLI? UpdateCLI would become much more powerful if a plugin could not only register an autodiscovery capability, but also an scm capability or resource capability.

In the current codebase, your plugin layout is based on the capability:

pkg/core/plugins
  |- autodiscovery
  |    |- ...
  |    |- githubaction
  |    |- ...
  |- changelog
  |    |- ...
  |    |- github
  |    |- ...
  |- resources
  |    |- ...
  |    |- githubrelease
  |    |- ...
  |- scms
  |    |- ...
  |    |- github
  |    |- githubsearch
  |    |- ...

(ignoring the utils folder for now)

As mentioned in my previous post, all the code related to Github is spread over multiple folders. I think it is better to think of an UpdateCLI plugin working against a specific API (e.g. Github, Gitlab, ArgoCD). This could lead to a better code structure like this:

pkg/core/plugins
  |- ...
  |- github
  |    |- autodiscovery
  |    |- resource
  |    |- scm
  |- ...

All the code for Github is now bundled together. The Github plugin would then need to register all the capabilities it has:

• github scm
• githubsearch scm
• github YAML schema
• githubrelease resource
• ...

This would require the introduction of a Registry keeping references to all this plugins. pkg/core should be clear of imports to pkg/plugins in any way. This branch in my fork already contains some of his rework:

https://github.com/ringods/updatecli/tree/rds-demo-newstyle-plugins

I introduced a plugins.Registry type:

https://github.com/ringods/updatecli/blob/rds-demo-newstyle-plugins/pkg/core/plugins/main.go

and the plugins.Plugin interface:

https://github.com/ringods/updatecli/blob/rds-demo-newstyle-plugins/pkg/core/plugins/plugin.go

The Github plugin registers its capabilities with the Registry:

https://github.com/ringods/updatecli/blob/rds-demo-newstyle-plugins/pkg/plugins/github/plugin.go#L22-L41

Once every plugin registers its capabilities, the core code must delegate to the Registry to fetch a registered capability. For example, instead of pkg/core/pipeline/scm having the SCM schemas hardcoded, we fetch the schemas from the Registry:

https://github.com/ringods/updatecli/blob/rds-demo-newstyle-plugins/pkg/core/pipeline/scm/config.go#L149-L156

This removes the imports from pkg/core to pkg/plugins, making the code more modular. The starting code in cmd is responsible for the registration of the plugins, leading to code usage like this:

                /--- cmd ---\
                |           |
            pkg/core     pkg/plugins

Quite a lot of the API of Registry, Plugin or the capabilities are far from complete in my branch. However, with this amount of actual code, I hope that I can explain better why redefining "plugin" in the UpdateCLI context and for this WASM proposal could lead to better structured codebase, and a more extensible UpdateCLI tool.

Q: do you still want the WASM interface to be stable for UpdateCLI 1.0.0? If yes, then I would very much like you to evaluate my proposal to widen the interface of what a plugin means.

NOTE: this demo branch is not intended to be or become an actual PR. It is purely for discussion purposes. If we conceptually agree the above is a good idea, all the changes should land in a series of bite-sized PRs.