[almarklein]

Intro

This PR proposes a new way to generate our API. The basic idea is that we define our API as a data structure that can be stored on disk (e.g. as json or toml). We can then manipulate that datastructure. E.g. define api differences in a similar file, that get's merged with the base API. But also derive docs from different sources and merge these into the API structure.

More info

To give a quick example, an argument to a function can look like:

  "classes": {
    "GPUSomeClass": {
      "methods": {
        "some_method": {
          "doc": "",
          "returns": {
            "doc": "",
            "type": "None"
          },
          "args": {
            "some_arg": {
              "doc": "",
              "type": "",
              "default": ""
            }
          }
        }
      }
    }
  }

So the code generation process is:

• extract: webgpu.idl -> webgpu.json
• merge: webgpu_web.json + diff_base.json + diff_docs.toml -> wgpu_api.json
• then use that to generate code.

This helps solve #845, because every API diff (for each backend) is explicitly defined.

It also solves #754: Like, every flag en enum field is documented. I took the docs of the WebGPU typscript definition, and let AI improve it and fill in the blanks. We can edit diff_docs.json as needed.

I expect this change makes the codgen easier to maintain because we have an intermediate format that can be checked before using it to generate code.

How to move forward

This is currently implemented alongside the current codegen implementation. My plan (but may change):

• Make it complete, but generate the exact same code as the current codegen.
• Have it generate the docs.
• Use it to implement (and validate via a typed JS webpu API stub!) the pyodide backend.
• Use it to implement the new wgpu backend.
• Remove the wgpu-native backend, and the associated codegen.

As an alternative, the codegen for the wgpu-native backend could also be included in the new codegen.

[Korijn]

Does this mean we get an API spec as data, and we can create multiple backends that implement it?

Maybe it's worthwhile to start by just merging the TOML pipeline as a first PR.

[almarklein]

Does this mean we get an API spec as data, and we can create multiple backends that implement it?

Yes, the spec would consist of a set of data structures. Some derived, some written by us.

The idea is that each backend has a diff data structure that defines extra functionality compared to the base API. The fact that we can have multiple backends implement the same API is not new, but it's more streamlined now. E.g. we can add extra (optional) args to existing methods. Previously we could also do that, but we'd have to mark the whole function as "apidiff" and lose all the automation for it.

Maybe it's worthwhile to start by just merging the TOML pipeline as a first PR.

Yes, I think the checkboxes in the top post are separate PRs :)

[Korijn]

each backend has a diff data structure that defines extra functionality compared to the base API

If that's the case, should we define a "pygfx core" that does not depend on "extras"? To keep it portable across backends.