Triplex / pmndrs ecosystem plugins

[itsdouges]

We have a lot of opportunities to enable plugins by third party developers for Triplex. This issue should collect all known use cases and opportunities we have to add integrations to Triplex.

[itsdouges]

Ping @krispya what would an integrated ECS view look like?

[Ctrlmonster]

Hey @itsdouges,
I'm going to copy parts of my dm here.
For context: this was about making my wip Character Animation Debug GUI work as a Triplex Plugin, so that you can configurate your animation graph in the GUI and save the changes back to a config object - the GUI.

Types of operations:

• Add new nodes to the Graph
• Add Transition between State Machine States
• Modify Transition details (i.e. duration)
• Add Synchronization/Event Markers to Timelines

What I imagined the integration could look like:

Some react component, that gets passed a SkinnedMesh ref (i.e. something like <AnimationRoom character={character} config={animGraphConfig} />). And then maybe you could right-click the SkinnedMesh in the Triplex SideView and have a "Open in AnimationRoom" option, which mounts the gui component with the SkinnedMesh as input

Thinking about how a plugin API could work:

From a plugin pov I guess the only thing I would need for writing the gui components is some "save back to code" api, that I can bind to all the different input elements.

---

RE: ECS Integration
Just off the top of my head, I would imagine you could inspect what kind of components an entity has, how many systems are picking it up right now etc. Since every Object3D in our ECS is an Entity, you should be able to select any Object3D in the Editor and then view its component and inspect their internal state. I think that would already be huge. Maybe you can edit data inside components too and save it back as the component's init state?

[krispya]

Yes, I think ideally there would be a new section on the right panel when selecting an Object3D that lets you see, edit and add components, list ids and maybe even state like if it is enabled. All of this data is written to the Objet3D under userData, or you can get it via API calls otherwise. Most components will be represented as a React component child of the Object3D, but they can be added imperatively too so we can't rely on just what is in the React tree.

So, I suppose a plugin for this would require being able to add custom sections to the panels. Also context is a must.

[isaac-mason]

Capturing the discussion on discord!

An interesting extension could be support for baking.

Essentially doing some expensive computation in the editor and saving the result for runtime. Thinking along the lines of getStaticProps in Next.js.

Some example use cases:

• creating a model with CSG and using the precomputed model at runtime
• surface sampling a geometry
• generating a navmesh

[itsdouges]

From Discord by @Ctrlmonster

We have a lot of different projects in r3f that do ui -> code / object generation, i.e. navmesh generator, animation system, material/shader graph editors.
Ideally all of these could eventually be integrated as Triplex plugins. For example Alma generates shader code, my Anim editor could generate a config file, why not write these directly into some file via a Triplex mechanism. Then the question is, how can they be intergrated ui wise? I imagine Triplex would need to allow a Plugin to add new windows / options / widgets etc.

Edit:
Also an ecs plugin can only really give insight at simulation time (or when you pause the simulation at a certain point)

[itsdouges]

Another use case: https://github.com/try-triplex/triplex/issues/68 (Export to GLB)

[itsdouges]

Another use case: https://github.com/try-triplex/triplex/issues/6 (create component via gltfjsx)

[itsdouges]

Would you say the current categories of plugins we're talking about originate from a component prop?

E.g: for navmesh @isaac-mason, given a specific component with a specific prop, show a different prop UI that you can enter and then do things, which you can then save and it then gets persisted to the prop (either inline or saved to a file, depending on configuration).

And then for anim @Ctrlmonster same thing?

Trying to flesh out what this would fundamentally look like.

[isaac-mason]

Sorry, I missed this!

I imagine a simple react navmesh api could look something like this:

const Example = () => {
  const { navMesh } = useNavigation()

  // ...
}

const App = () => (
  <Navigation cellSize={0.2} cellHeight={0.2}>
    <Walkable>
      <SomeMeshToWalkOn />
    </Walkable>

    <Example />
  </Navigation>
)

In this imaginary api, navmesh generation logic lives in <Navigation>. I'm guessing if this were to integrate with triplex, logic for conditonally generating & persisting in "editor mode" vs loading persisted in "app mode" would also have to be inside <Navigation>?

Maybe if triplex provided some api for checking if we're editing, and a means of persisting data, we could build triplex-friendly components that look something like this:

// very simplified example
const Navigation = () => {
// ...

  useEffect(() => {
    if (triplex.editing) {
      // generate
      const navMesh = generate()
      // persist Uint8Array
      triplex.save('navMesh', exportNavMesh(navMesh))

      setNavMesh(navMesh)
    } else {
      // load persisted navmesh 
      const navMesh = importNavMesh(triplex.load('navMesh'))

      setNavMesh(navMesh)
    }
  }, [])

  // ...
}

[itsdouges]

I want to explore plugins soon. If anyone has more comments on how things should work let me know.