schema: native imports for composing .pg files

Problem

A .pg schema must live in one file. As an ontology grows, there is no way to split it into modules (e.g. core.pg, product.pg, finance.pg) with explicit dependencies, or to reuse a shared core vocabulary across graphs without copying its definitions.

Proposed change

Add native import syntax to .pg, resolved by the compiler into one schema catalog:

// product.pg
import { Goal, Serves } from "./core.pg"

node Requirement { statement: String }
edge MotivatedBy: Requirement -> Goal

// humanitas.pg
import "./core.pg"
import "./product.pg"

Named imports bring in only the listed definitions; importing a node does not pull in edges that touch it.
Importing an edge resolves its endpoint types without pulling unrelated edges.
A definition reached through several paths is included once; conflicting same-named definitions are a located compile error.
First scope is local relative paths only. No registries, remote fetching, or versioning.

Affected areas

Schema grammar (no import syntax today), module loading before catalog validation, and the source-loading contract for CLI, server, and embedded entry points. A server must not treat client-supplied import paths as filesystem access, so a submitted source bundle or caller-supplied resolver is likely needed. Open questions: export visibility, aliases, import cycles, and whether persisted type identity survives moving a definition between files.

Alternatives considered

An external manifest that composes files (adds a second authoring format), plain concatenation (no import scope), or staying single-file (fine for small models). We drafted a fuller write-up and can turn it into an RFC if maintainers want one.

rfc: answer review of schema namespaces and multi-file schemas

Summary

A .pg schema is one source text. The compiler has no import. You cannot split core.pg, product.pg, and finance.pg and name the dependency in the file. The workaround is to copy definitions or to concatenate files. Copying drifts. Concatenation pulls every definition, and a second copy of the same name is a catalog error with no file location.

This is a feature request. On main 0a62bb7b2aa5b99b012f413807c216277f7f9596 the missing feature is still missing. I reproduced that absence. I did not find an implementation.

Status check

Checked again immediately before this comment.

State: open.
Comments: none.
Labels: accepted, feature, P-low. The body is unchanged since 2026-10-01.
No linked fix pull request.
Timeline cross-references are draft RFC pull requests, not a reproduction and not a fix. See Related work.

I am not closing this issue.

Reproduction

Verdict: reproduced.

The report says a .pg schema must live in one file, and it proposes import. On this main, the grammar rejects both example imports. Each loader I ran accepts one schema source. Concatenation works and has the costs below.

What differed from the report:

The report links the grammar at e16955b1. That grammar and current schema.pest differ only in the range_bound rules (RFC 0040 range spelling). schema_decl is still interface_decl | node_decl | edge_decl. There is still no import rule.
The report does not include command output. The outputs below are from this run.
schema plan opens the graph before it parses. On a missing graph the import error is hidden by "dataset was not found". lint and init show the parse error directly.
I also measured identity. The report leaves that as an open question. A reorder of the same declarations publishes a new contract and keeps the same Lance dataset directories.

Environment

Commit: 0a62bb7b2aa5b99b012f413807c216277f7f9596 (origin/main, 2026-10-18). Clean tree.
CLI: target/debug/omnigraph 0.14.0, internal schema 16. The binary is newer than the schema compiler and cluster loader sources in this tree. I did not rebuild it in this session.
Host: Linux, rustc 1.97.1.
Lance on the path: 11.0.0 (lance-table-11.0.0 in the missing-dataset error).

Example files are the report's text.

product.pg:

import { Goal, Serves } from "./core.pg"

node Requirement { statement: String }
edge MotivatedBy: Requirement -> Goal

humanitas.pg:

import "./core.pg"
import "./product.pg"

core.pg declares node Goal and edge Serves: Goal -> Goal.

Grammar and parse_schema

omnigraph lint reads one file and calls parse_schema(&str).

$ omnigraph lint --schema product.pg --query q.gq
Error:
   0: parse error:  --> 1:1
        |
      1 | import { Goal, Serves } from "./core.pg"
        | ^---
        |
        = expected schema_file

Exit code 1. The same command on humanitas.pg fails at import "./core.pg" with expected schema_file.

omnigraph init --schema product.pg <new-uri> fails with that same parse error. The CLI prints the diagnostic twice in the error chain.

CLI schema plan and apply

$ omnigraph schema plan --schema product.pg <initialized-graph>

Same parse error, exit 1.

$ omnigraph schema plan --schema <directory> <graph>
Error: Is a directory (os error 21)

$ omnigraph schema apply --schema <directory> <graph>
Is a directory (os error 21)
command_outcome: {"execution":"not_started","effects":"none","action":"refresh"}

schema plan against a URI with no graph fails earlier, before parse:

storage: Dataset at path .../__manifest was not found

Cluster config (GraphConfig.schema)

A single file validates. cluster validate --json reports one schema resource, schema.knowledge, with a path and a sha256 digest.

A list fails:

ERROR invalid_cluster_yaml cluster.yaml: could not parse cluster.yaml: graphs.knowledge.schema: invalid type: sequence, expected path string at line 6 column 13

A directory fails:

ERROR schema_file_missing graphs.knowledge.schema: could not read schema file '.../schema/': Is a directory (os error 21)

Queries already allow a file, a directory, or a list. Schema does not.

Embedded API

I did not link a separate Rust program. The local CLI is that API:

init reads the file and calls Omnigraph::init_with_options(&uri, &schema_source, ...).
schema plan calls plan_schema(&schema_source).
schema apply calls apply_schema_as_with_catalog_check(schema_source, ...).

Each function takes one &str. There is no path list and no resolver argument.

Server

The HTTP surface has GET /schema. It returns one schema_source string. There is no schema-apply route on the graph. Served schema change goes through POST /cluster/plan and POST /cluster/deployments.

The body is a CapturedDeployment. Its sources map is sha256 -> file text. The comment on that type says the server never opens caller-supplied paths. bundle_from_capture fills that map from bytes the CLI already read. validate_bundle parses config_semantics back into RawClusterConfig, where schema is one PathBuf.

I did not boot a server and I did not POST a bundle. The list-form refusal above is the same serde type the server reparses. A hand-built bundle with a schema array would fail that parse with deployment_input_invalid. I did not send that request.

Workaround

Concatenation of core.pg plus the Requirement / MotivatedBy declarations:

lint accepts the file. Query goals checks as a read.
init creates the graph.
The stored source contains Goal, Serves, Requirement, and MotivatedBy. There is no way to take Goal and leave Serves behind, except by editing a copy.

A second copy of core.pg concatenated again:

catalog error: duplicate node type: Goal

Two edge Serves declarations:

catalog error: duplicate edge type: Serves

Serves and SERVES in one file:

catalog error: edge name collision after case folding: 'SERVES' conflicts with 'Serves'

A lowercase edge serves fails earlier, at parse (expected type_name). The grammar only admits a type name that starts with an uppercase letter.

MotivatedBy in a file that does not also declare Goal:

catalog error: edge MotivatedBy references unknown target type: Goal

None of these catalog errors give a second location. The parser has already accepted the file as one string.

Root cause

schema_file is SOI ~ schema_decl* ~ EOI, and schema_decl is an interface, a node, or an edge.

schema.pest

parse_schema parses that one string. Catalog construction then rejects a repeated node name, a repeated edge name, and an edge name that collides after to_lowercase().

catalog/mod.rs

Loaders that turn a path into that string:

CLI init, schema plan, schema apply, and lint use fs::read_to_string on one PathBuf.
GraphConfig.schema is one PathBuf. The cluster loader reads that one path. types.rs, config.rs.

There is no module table, no export list, and no cycle check, because nothing imports.

Identity, hashes, and the ledger

Two hashes exist, and they answer different questions.

source_hash is sha256 of the schema source bytes. The handle digest and the schema-apply intent both use it. The cluster resource schema.<graph> is the same hash of the one file. For the concatenated example, that digest is 8fef634d27796994c94a0b872f049d578ea07e701ccfd129363cc500e69c785a.
schema_ir_hash is sha256 of the canonical shape JSON. The shape sorts interfaces, nodes, and edges by name before the hash. Declaration order is not part of that hash.

Stable type identity is not either hash. resolve in schema_ir.rs matches an accepted declaration by kind and name. @rename_from is used only when the name did not match. A hint that matches the current name is ignored. The dataset path is nodes/{type_id}-{incarnation} or edges/{type_id}-{incarnation}, from the stable id, not from the file and not from the type name.

table_path_for_identity

What a move between files does today

The engine never sees files. A move between files is a different source string, same as a reorder inside one file.

I initialized from the concatenated file, then applied the same declarations in the other order.

schema plan --json: supported: true, step_count: 0.
schema apply --json: applied: true, step_count: 0, graph_manifest_version: 2.
schema show returned the reordered source.
sha256 of that stored source is a1eee46a3699185d33bcf5fd8f15f0b7fbdcb0b57f6166b89b69e629923e2006.
Dataset directories did not change: nodes/0000000000000001-0000000000000007, nodes/0000000000000002-0000000000000008, edges/0000000000000003-0000000000000009, edges/0000000000000004-000000000000000a.

So a move that keeps the name keeps the stable id and the Lance dataset. It still publishes a new schema contract, because the source bytes changed. No @rename_from is required. The same publication already exists for a comment-only edit (apply_schema_noop_returns_not_applied in tests/schema_apply.rs). I did not re-run that Rust test.

What a rename does

I then applied node Objective @rename_from("Goal") and pointed the edges at Objective.

Plan and apply reported one step: rename_type of node Goal to Objective.
Apply set graph_manifest_version to 3.
The same four dataset directories remained.
RenameTable keeps the source entry's identity and table_path. It changes the table-key alias.

A name change without @rename_from is not this path. It plans as drop plus add. Populated drops are accepted today. That is issue #984. Nothing infers that a new name in a new file is the old type.

How a source bundle fits the wire

The server path already receives bytes, not paths. A safe design has to keep that.

Two shapes fit:

The client composes files into one source, then sends today's bundle. sources stays digest -> text. config_semantics can stay a single path. No server filesystem access. This is the shape draft PR rfc: schema namespaces and multi-file schemas #1015 chooses. The list form in cluster.yaml is a different config_semantics JSON value. An older server refuses it when it reparses RawClusterConfig.
The client sends every file in the bundle and a resolver reads only that map. Today's map keys are content digests, not relative paths. A path-keyed map, or a set of schema parts, is a new bundle field. The resolver must reject .., absolute paths, and symlinks. check_config_path already rejects .. and symlinks for relative cluster paths. It still allows an absolute path, and that path is read on the CLI before the bytes enter the bundle.

Do not resolve a client path on the server. The current server does not.

Recommended fix

The user need has two parts.

Split one large schema across files, and share a whole core file inside one cluster.
Import a named subset. Importing a node does not import edges that touch it. Importing an edge imports its endpoint types.

For the first part, compose files before the hash. Do not add import.

That is the design in draft PR #1015 at 7565f5ff (pull request). The loader joins the declared files in a fixed order, with a banner line when there is more than one file, and every existing consumer hashes that one text. parse_schema, the schema contract, schema_ir_hash, and the server bundle stay as they are for a single file.

Invariants:

Invariant 6. Identity stays the stable id. A file path is not an identity. A move that keeps the name is a new source and zero migration steps, as measured above. A move that changes the name, including a move into a namespace, is an explicit @rename_from on the existing RenameType / RenameTable path.
Invariant 12. One stored schema text. The repository keeps the files. The graph keeps the accepted text.
Invariant 2 and invariant 4. One schema apply still publishes one manifest.
Invariant 8. A repeated name and a file named twice are errors. The error names both files. Today it does not.
Invariant 9. Composition is loader structure. It is not a string the query engine interprets.
Trust boundary. The server still receives bytes.

This deprioritizes a self-contained .pg file. product.pg would not say that it needs core.pg. An editor or a one-file lint needs the composing declaration. That cost is real. It is the right cost while the goal is one catalog and one hash. A second resolution language would add cycles, export rules, and a path resolver, which is more liability than file size requires.

Affected class for this fix: CLI init, schema plan, schema apply, and lint. Cluster GraphConfig and source capture. Error spans across files. Schema-contract source bytes and the cluster schema digest. Not Lance field ids. Not the single-path server bundle. The list form changes config_semantics and older servers refuse it.

Named, scoped import stays a separate need.

#1015 does not cover it. The RFC says so. It drops selective imports, a file that can be checked alone, and sharing a core file across clusters (.. stays refused). It treats a file named twice as an error, not as one inclusion.

Build selective import only if a schema must take Goal and not Serves. Correct shape:

The compiler resolves a relative import graph into one catalog, then hashes one canonical text.
A named import includes only the listed definitions.
An edge import also includes its endpoint types. It does not include other edges.
A definition reached by several paths is included once.
Two different definitions with the same name are a compile error that names both files.
Cycles are an error.
Decide export and aliases before the grammar ships. An alias is a second name. It must not mint a second stable id.
The server receives a bundle. The resolver reads the bundle. It does not open a client path.

Do not key the stable id on the file path. The measurement above is the behavior to keep. A move between files that keeps the name keeps the id. Authors will see a no-step contract publication when the composed bytes change. Document that next to the banner rule, because the banner and the file order are part of source_hash.

Invariants at extra risk if import is added badly:

Invariant 6, if a path or an alias becomes identity.
Invariant 10, if the server opens a client path.
Invariant 8, if a diamond merge or a cycle fails open.
Invariant 12, if each file stays a stored contract and the catalog is rebuilt from several hashes.

Alternatives

Stay on one file. Correct for a small model. It does not scale to the 76-node, 426-edge schema cited in rfc: schema namespaces and multi-file schemas #1015, and copied templates collide on Event or OwnedBy.
Prefix names (Crm_Contact). Works today. It gives no file boundary and no query scope. Edge names still fold into one graph-wide namespace.
A new manifest format only for composition. rfc: schema namespaces and multi-file schemas #1015 avoids a new format by extending cluster.yaml, which already lists query files.
Persist file boundaries in the schema contract. That needs a storage stamp, an upgrade, and a new history archive. The gain is display. Defer it.

Liability

Compose-before-hash removes copy drift and unlocated duplicate errors for whole files. It adds a hashed banner. A later banner edit gives every multi-file graph a new contract and no migration steps. The list form is a version skew. Ship it only when every server that reads those bundles understands it.

Selective import removes the need to paste a subset by hand. It adds cycle, export, alias, and path-escape liability. Protobuf's public import is the caution: file structure leaks to callers. Do not add that until the subset rule is a real requirement.

Leaving #838 open while #1015 is only a draft keeps the selective-import request visible. Closing it as a duplicate of a pull request would hide that request.

Lance

Lance does not fix this, and a Lance upgrade does not change the answer.

Pinned crates.io Lance is 11.0.0. The latest stable release is v13.0.0 (2026-10-07). v14.0.0-beta.19 is a pre-release (2026-10-16). Draft PR #1072 upgrades OmniGraph to 13.0.0 and says it adds no graph or CLI capability.

Lance's schema format assigns field ids inside one dataset. A new field does not keep a caller-chosen id when non-reusable ids are on. OmniGraph table paths use the stable type id. Property names become Lance field names. The .pg file path is not an input. Moving a definition between files does not change a table's Lance schema when the properties stay the same. The reorder run kept the same dataset directories.

lance#9547, merged and released in 13.0.0, takes merge_columns field ids from the dataset manifest. That removes a caller-supplied id. It does not add a schema module system. lance#8915 in v12 is a namespace-API change for merge-insert keys, not for source files. lance#5864 and lance#5817 discuss logical types. They affect a property type in any schema, one file or many. I read the v12 and v13 breaking-change lists and the schema format page. I did not read every open Lance pull request. None of the items above compose .pg files.

Related work

rfc: schema namespaces and multi-file schemas #1015, open, head 7565f5ff. RFC "Schema namespaces and multi-file schemas". It now links this issue and records a disposition. It does not implement the loader. implementation: not-started. It would cover whole-file composition and namespaces. It would not cover named, scoped imports. It is a pull request. It does not supersede this issue.
rfc: polymorphic types with interfaces, unions and polymorphic edges #884, open. RFC for interfaces, unions, and polymorphic edges. It shares qualified type spelling with rfc: schema namespaces and multi-file schemas #1015. It does not define file imports. The cross-reference on this issue comes from that relationship. Interface rename still needs @rename_from from rfc: polymorphic types with interfaces, unions and polymorphic edges #884 before an existing interface can move.
rfc: schema apply with live branches #963, open. RFC for schema apply while other branches are live. Apply stays one contract publication. A multi-file composer still runs before that publication. A byte-level file move would still be a new contract. rfc: schema apply with live branches #963 orders contracts with a schema sequence because a rename and a rename back can repeat a content hash.
docs(schema): OG-DS-102/103/104 are listed as refusals, but populated drops plan as supported #984, open. Populated drops plan as supported, while the docs list them as refusals. A file move that also changes a name, without @rename_from, hits this. rfc: schema namespaces and multi-file schemas #1015 proposes a narrow refusal for a probable namespace move. It does not decide docs(schema): OG-DS-102/103/104 are listed as refusals, but populated drops plan as supported #984.
feat(query): expose engine catalogue as built-in types #1008, merged. Engine catalogue @ types are derived from the binary. They are not part of schema_source. An import design should not try to import them. rfc: schema namespaces and multi-file schemas #1015 leaves them outside namespaces.
Editor support for .pg: tree-sitter grammar + Zed extension #508, open. Editor tree-sitter grammar for .pg. Any import syntax or namespace block has to land there too. It is not a schema-composition design.
build(storage): upgrade lance to 13.0.0 #1072, open draft. Lance 13 upgrade. No schema composer.

No other open issue I found asks for .pg imports or multi-file schemas. There is no roadmap document in docs/dev/ for this. The direction in flight is the #1015 draft.

Supersession

No open issue supersedes #838. #1015 is the closest design, and it is a pull request. Its own text says which parts of #838 it keeps and which it drops. The dropped part, named selective import, is still untracked except here. I am not closing this issue.

Limitations

I used the prebuilt debug CLI. I did not rebuild it, and I did not add a Rust test.
I did not boot the HTTP server and I did not POST /cluster/plan or /cluster/deployments.
I did not read schema_ir_hash out of __manifest. Empty plan steps plus the sorted shape hash are the evidence that the shape did not change.
I did not decode table-key strings. The dataset directory names are the stable ids, and they did not change.
The rename example also rewrote edge endpoint names to Objective. The plan's only step was the node rename.
Lance: I did not review every open upstream pull request.

Navigation Menu

Repository navigation

schema: native imports for composing .pg files #838

Problem

Proposed change

Affected areas

Alternatives considered

Summary

Status check

Reproduction

Root cause

Identity, hashes, and the ledger

How a source bundle fits the wire

Recommended fix

Lance

Related work

Supersession

Limitations

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

schema: native imports for composing .pg files #838

Description

Problem

Proposed change

Affected areas

Alternatives considered

Activity

ragnorc commented on Oct 18, 2026

Summary

Status check

Reproduction

Root cause

Identity, hashes, and the ledger

How a source bundle fits the wire

Recommended fix

Lance

Related work

Supersession

Limitations

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions