diff --git a/.changeset/27747-odata-guide-readme-link.md b/.changeset/27747-odata-guide-readme-link.md new file mode 100644 index 00000000000..13f95803f0c --- /dev/null +++ b/.changeset/27747-odata-guide-readme-link.md @@ -0,0 +1,10 @@ +--- +'@objectstack/driver-odata': patch +--- + +docs(driver-odata): the README links the OData datasources guide + +Clause-②: no +Metadata-protocol: none + +The package README now points to the OData datasources guide (`/docs/data-modeling/odata-datasources`), which covers checking a service, mounting the plugin, binding objects to entity sets, credentials through `external.credentialsRef`, what the service is asked behind the engine's floor, writes, and how failures reach the caller. No code changes. diff --git a/content/docs/data-modeling/external-datasources.mdx b/content/docs/data-modeling/external-datasources.mdx index 6ac3c8c601d..8d28e60472a 100644 --- a/content/docs/data-modeling/external-datasources.mdx +++ b/content/docs/data-modeling/external-datasources.mdx @@ -17,9 +17,11 @@ auto-connected, validated at boot, and queryable — with no application code.** This guide is about reading/writing the *live* remote tables (federation). To *copy* rows into ObjectStack-owned tables instead, see seed data / data sync. For the plain multi-datasource routing of **managed** databases ObjectStack owns, see -[Database Drivers](/docs/data-modeling/drivers). When the remote is not a database a -built-in driver reaches — a REST API, a planning engine, a device gateway — serve the -object from your own plugin driver instead: [Virtual objects](/docs/data-modeling/virtual-objects). +[Database Drivers](/docs/data-modeling/drivers). When the remote is an OData v4 service +(SAP S/4HANA, Dynamics 365), bind it through the shipped OData driver: +[OData datasources](/docs/data-modeling/odata-datasources). When it is neither — a REST +API, a planning engine, a device gateway — serve the object from your own plugin driver +instead: [Virtual objects](/docs/data-modeling/virtual-objects). ## 1. Declare the datasource @@ -312,6 +314,8 @@ default allows all connects (subject to the gating above). - [Database Drivers](/docs/data-modeling/drivers) — managed multi-datasource routing, [read-only via database privileges](/docs/data-modeling/drivers#read-only-grant-it-at-the-database-not-in-metadata), and [why the platform does not route read replicas](/docs/data-modeling/drivers#read-replicas-the-platform-does-not-route). +- [OData datasources](/docs/data-modeling/odata-datasources) — the same binding over an + OData v4 service, through the shipped OData driver. - [Virtual objects](/docs/data-modeling/virtual-objects) — the same binding served by a plugin driver you write, for a remote that is not a database. - [Datasource reference](/docs/references/data/datasource) — every `defineDatasource` field. diff --git a/content/docs/data-modeling/index.mdx b/content/docs/data-modeling/index.mdx index e5025e0e6e8..b0b80795c3f 100644 --- a/content/docs/data-modeling/index.mdx +++ b/content/docs/data-modeling/index.mdx @@ -60,6 +60,7 @@ That one definition is enough to get a persisted table, CRUD + query endpoints, + diff --git a/content/docs/data-modeling/meta.json b/content/docs/data-modeling/meta.json index 75c75e57905..0d68d3a7fbd 100644 --- a/content/docs/data-modeling/meta.json +++ b/content/docs/data-modeling/meta.json @@ -17,6 +17,7 @@ "seed-data", "import-mappings", "external-datasources", + "odata-datasources", "drivers", "virtual-objects", "analytics" diff --git a/content/docs/data-modeling/odata-datasources.mdx b/content/docs/data-modeling/odata-datasources.mdx new file mode 100644 index 00000000000..da5122eb2d4 --- /dev/null +++ b/content/docs/data-modeling/odata-datasources.mdx @@ -0,0 +1,284 @@ +--- +title: "OData datasources — ERP records, live, with no driver of your own" +navTitle: OData Datasources +description: Declare a datasource over an OData v4 service such as SAP S/4HANA or Dynamics 365, bind objects to its entity sets, and read and write them live, with no code per entity set. +--- + +Many business systems already publish their records over **OData v4**: SAP S/4HANA, +Microsoft Dynamics 365 and Dataverse, and any service that serves a CSDL 4.0 +`$metadata` document. For those you need no driver of your own. Mount +`@objectstack/driver-odata` once per service, declare a datasource, and bind an object +to each entity set you want. Every view, form, permission set, formula and hook then +works on those records, and no row is stored on the platform side. + +The OData driver is a **plugin driver**, so everything [Virtual objects](/docs/data-modeling/virtual-objects) +says about plugin drivers holds here too: the engine re-applies every query, a read +matches at most 1000 rows, and the sharing model needs a deliberate choice. The one +difference is that this driver is already written. +[Which way in](/docs/data-modeling/virtual-objects#which-way-in-odata-a-plugin-driver-or-a-real-object) +says when to choose it over a driver of your own or a synced copy. + +Everything on this page is what the driver's own tests assert, against an in-process +OData service and behind the real engine +(`packages/drivers/driver-odata/src/odata-driver.floor.test.ts` and +`odata-driver.conformance.test.ts`). Read them when this page and the runtime seem to +disagree. + +## Check the service first + +- **It speaks OData v4.** Open `$metadata`. The root element must say + `Version="4.0"`. Only the v4 dialect ships. A service that speaks only OData v2 needs a + v2 dialect, which this package does not include. Many older SAP Gateway services are + v2 only, so check each service you plan to bind. +- **Each entity set you bind has a key.** The key becomes the record's `id`. An entity + type with no usable key is refused at the first request. +- **You know how it authenticates.** The driver sends no credential, a bearer token, or + HTTP basic. It does not run an OAuth2 flow ([Credentials](#3-credentials)). + +## 1. Mount the plugin + +The plugin builds the driver from its options and publishes it as the service +`driver.`. Mount it in the stack's `plugins`, one plugin per service: + +```typescript +import { defineStack } from '@objectstack/spec'; +import { createODataDriverPlugin } from '@objectstack/driver-odata'; +import { erpSecretStore } from './secrets'; // your host's secret store; see Credentials + +export default defineStack({ + // …the datasource and objects of step 2… + plugins: [ + createODataDriverPlugin({ + name: 'erp', // = the datasource name + serviceRoot: 'https://erp.example.com/odata/v4/sales/', + auth: { scheme: 'bearer' }, + secrets: erpSecretStore, + }), + ], +}); +``` + +| Option | Default | Meaning | +| :--- | :--- | :--- | +| `name` | `odata` | The datasource it serves. It is also the driver name and the `driver.` service suffix. | +| `serviceRoot` | (required) | The service root URL, http or https, with no credentials and no query string. | +| `version` | `4.0` | The protocol version. `4.0` is the only value. | +| `auth` | `{ scheme: 'none' }` | `none`, `bearer`, or `basic` with a `username`. The secret is the datasource's `external.credentialsRef`. | +| `secrets` | (none) | Where `credentialsRef` is resolved: an object with `resolve(credentialsRef)`. Required for `bearer` and `basic`. | +| `pageSize` | `500` | Rows asked for per server page (`Prefer: odata.maxpagesize`). | +| `requestTimeoutMs` | `30000` | The timeout of each request. | + +The options are checked when the plugin is built. An unknown key, a `serviceRoot` with +a user name or a query in it, or a version other than `4.0` refuses the boot with every +problem named. + +## 2. Declare the datasource and bind objects + +{/* os:check */} +```typescript +import { defineStack } from '@objectstack/spec'; + +export default defineStack({ + manifest: { id: 'com.example.sales', name: 'Sales', version: '1.0.0', type: 'app' }, + datasources: [ + { + name: 'erp', + label: 'ERP sales service (OData v4)', + driver: 'erp', // the plugin's name, i.e. the `driver.erp` service + schemaMode: 'external', + config: {}, // the plugin's options configure the driver, not this + external: { + allowWrites: true, + credentialsRef: 'sys_secret:7c1e…', // the bearer token, as a secret-store handle + validation: { onMismatch: 'warn', checkOnBoot: false }, + }, + }, + ], + objects: [ + { + name: 'sales_order', + label: 'Sales order', + datasource: 'erp', + sharingModel: 'public_read_write', // see "Choose the sharing model" below + external: { + remoteName: 'SalesOrders', // the entity set + writable: true, + // remote property → local field; a property not named keeps its own name + columnMap: { + SalesOrderID: 'order_number', + CustomerName: 'customer_name', + NetAmount: 'net_amount', + OverallStatus: 'status', + DeliveryNote: 'delivery_note', + }, + }, + fields: { + order_number: { type: 'text', label: 'Order' }, + customer_name: { type: 'text', label: 'Customer' }, + net_amount: { type: 'number', label: 'Net amount' }, + status: { type: 'text', label: 'Status' }, + delivery_note: { type: 'text', label: 'Delivery note' }, + }, + }, + ], +}); +``` + +What each piece does on this driver: + +- **Three names stay equal:** the plugin's `name`, the datasource `name`, and the + datasource `driver`. The connect step finds `driver.erp` already registered and uses + it as it is. `config` is left empty, because the plugin's options already configured + the driver. +- **`remoteName` names the entity set.** Without it the driver looks for an entity set + named like the object. +- **This driver applies `external.columnMap` itself**, in both directions: rows arrive + under your field names, and filters, sorts and writes go out under the service's + property names. That is a property of this driver, not of plugin drivers in general. +- **`id` is the entity key.** A single-property key is read as text (`'4711'`). A + composite key is the body of its key predicate (`'OrderID=1,ProductID=7'`), which is + also what `GET /data//` takes. +- **Declare each field by hand.** Nothing generates fields from `$metadata` today. + Navigation properties are never followed, so a related entity set is an object of its + own. +- **The service document is read lazily.** `connect()` asks the service nothing, and + `$metadata` is read once, at the first request. A wrong `serviceRoot` or an unknown + entity set shows up on the first read of the object, not at boot. +- **`checkOnBoot: false`** keeps the boot from depending on the service being up. The + driver's tests do not cover the boot-time drift check over an OData service, so test + it against your service before you turn it on. + +## 3. Credentials + +⛔ Never put a credential in the stack, in the plugin options, or in the `serviceRoot` +URL. The secret is the datasource's `external.credentialsRef`, and the plugin resolves it +through the `secrets` store you pass: + +- **`bearer`** sends `Authorization: Bearer `. +- **`basic`** takes `username` from the options and the password from the secret. +- **`none`** sends nothing. A `credentialsRef` declared beside `none` is refused, so a + credential you meant to send is never left out by mistake. + +The plugin resolves the secret at the **first request**, not at boot, and again after a +`401`. So a token rotated in the secret store is picked up by the next request that the +old token fails. The driver does not obtain or refresh tokens itself. A service that +issues short-lived OAuth2 access tokens needs something that keeps the stored secret +current. + +`secrets` is any object with `resolve(credentialsRef)`, answering the secret or +`undefined`: + +- **A host that composes its own kernel** passes the same binder it gives + `DatasourceAdminServicePlugin`. A `sys_secret:` handle that the Add Datasource + wizard minted then resolves, as on any federated datasource + ([Credentials](/docs/data-modeling/external-datasources#4-credentials)). +- **`os serve`** builds its secret store internally and does not hand it to a stack's + plugins. Pass a `resolve` backed by your host's own secret handling instead, and + declare `credentialsRef` as a handle that it resolves. + +Resolution fails closed. Each of these refuses the request rather than send it without +the credential: an auth scheme with no `credentialsRef`, a `credentialsRef` with no +`secrets`, and a handle that does not resolve. The caller is answered +`503 ERR_DATASOURCE_UNAVAILABLE`, and the cause goes to the server log. + +## 4. What the service is asked + +Behind the floor, a list read is one `GET` on the entity set: + +```text +GET SalesOrders?$filter=OverallStatus eq 'open'&$orderby=NetAmount desc,SalesOrderID&$top=1001 +Prefer: odata.maxpagesize=500 +``` + +- **`$filter`** carries every part of the `where` the driver can translate: `$eq`, + `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$between`, `$in`, `$nin`, `$null`, `$contains`, + `$startsWith`, `$endsWith` and `$icontains` (ASCII comparands), under `$and` and `$or`. +- **The rest is left to the engine**, which re-applies the whole query to the rows the + service returns: `$not`, `$like`, `$exists`, field references, and ordering or + negation on text. The driver only ever asks a **wider** question. It drops a member of + an `$and` it cannot translate, and drops a whole `$or` when one branch cannot be + translated, so the service never returns fewer rows than match. This is also why a + service that compares text case-insensitively still answers correctly. +- **`$orderby`** carries the sort, then the entity key, so pages partition the result. +- **Paging** follows `@odata.nextLink` until 1001 rows are read. A link to another + origin is refused, so the credential never goes to a host you did not configure. +- **The page size and the total** are the engine's, applied to the rows read. A count + is computed the same way. The engine aggregates in memory. + +**The 1000-row cap applies.** A read that matches more than 1000 rows answers +`422 EXTERNAL_ROW_CAP_EXCEEDED` rather than a partial set +([the cap](/docs/data-modeling/virtual-objects#the-engine-re-applies-every-query)). Every +`$filter` the driver can push down makes the service return only matching rows, so the cap +counts matching rows, not the whole entity set. Give list views a filter the service +can run. For an entity set users must browse whole, mirror it into a real object +instead. + +## 5. Writes + +Writes are the same double opt-in as federation: `external.allowWrites` on the +datasource and `external.writable` on the object +([Writes](/docs/data-modeling/external-datasources#5-writes-double-opt-in)). + +| The platform does | The service is sent | +| :--- | :--- | +| `POST /data/sales_order` | `POST SalesOrders` | +| `PATCH /data/sales_order/:id` | `PATCH SalesOrders()` | +| `DELETE /data/sales_order/:id` | `DELETE SalesOrders()` | + +- **`If-Match`** carries the entity's `@odata.etag` when the driver's last read of that + record returned one, so a write over someone else's change is refused by the service. +- **Only properties the entity type declares are sent.** The fields the engine stamps + itself (`updated_at`, `updated_by`) are not the service's and are left out. +- **The service's refusal reaches the caller.** A `400`, `409`, `412` or `422` + from the service answers `422 EXTERNAL_WRITE_REFUSED` with the service's own message. + Let the ERP's validation be the rule, and do not retry around it. +- **Refused by name:** `upsert`, the bulk writes, and transactions. Without `$batch` + there is no atomic multi-row write. A write that selects its rows by a filter is + refused too. Read the ids, then write each one. + +### Choose the sharing model + +A record from an OData service has no `owner_id` and no `created_by`, so the default +sharing model admits no row for a member, and an owner-keyed write is refused. The rule +and the ways out are the same as on any plugin driver +([Choose the sharing model on purpose](/docs/data-modeling/virtual-objects#3-the-datasource-the-object-the-hook-the-views)): + +- **Read-only data:** leave `allowWrites` off and declare `sharingModel: 'public_read'`. + Who may read is the permission sets' decision. +- **Editable data:** declare `sharingModel: 'public_read_write'`, and decide who may edit + in a `beforeUpdate` hook. A member's delete is still refused on this model. + +## When the service fails + +| What happened | The caller is answered | +| :--- | :--- | +| A read the service refuses for its own sake (`400`, `404`, `422`) | `400 INVALID_QUERY`, with a message naming the entity set, the status and the service's own text | +| A write the service refuses | `422 EXTERNAL_WRITE_REFUSED`, with the service's message | +| A `5xx`, a timeout, an unreachable service, a `401` after the retry, a `403` or a `429` | `503 ERR_DATASOURCE_UNAVAILABLE`; the cause goes to the server log | +| A read that matches more than 1000 rows | `422 EXTERNAL_ROW_CAP_EXCEEDED` | + +An outage is never answered as an empty list. A read refusal's message is withheld, and +goes to the server log, when the read carries a predicate the caller did not write, such +as a sharing scope. The full mapping is in +[Virtual objects](/docs/data-modeling/virtual-objects#1-the-driver). + +## What it does not do + +- **OData v2.** Only the v4 dialect ships. +- **`$expand` and navigation properties.** Bind each related entity set as an object of + its own. +- **Actions and functions** of the service. `execute` is refused. +- **Transactions and `$batch`.** Each write is one request. +- **More than 1000 matching rows** in one read (the cap above). +- **A read scoped by a parent's access** (an object `controlled_by_parent`): it is + refused, because `$filter` has no form of that join. + +## See also + +- [Virtual objects](/docs/data-modeling/virtual-objects): which way in, the engine's + floor, the sharing-model callout, and writing a driver for a system with no OData. +- [External datasources](/docs/data-modeling/external-datasources): the datasource and + `external` binding, federation over SQL, the write gates and credentials. +- [Pulling a mapping](/docs/automation/jobs#pulling-a-mapping): a real object fed from a + remote API on a schedule. +- `packages/drivers/driver-odata/README.md`: the package and its exports. diff --git a/content/docs/data-modeling/virtual-objects.mdx b/content/docs/data-modeling/virtual-objects.mdx index 4dcfd873454..6a1ad5e9c43 100644 --- a/content/docs/data-modeling/virtual-objects.mdx +++ b/content/docs/data-modeling/virtual-objects.mdx @@ -19,10 +19,22 @@ Everything on this page is what the platform's reference test boots and asserts in `ui/packages/plugin-gantt/src/ObjectGantt.pluginDriverTree.test.tsx`). Read them when this page and the runtime seem to disagree. -## Virtual, or a real object? +## Which way in: OData, a plugin driver, or a real object -Make it virtual when a copy would be **wrong**, and real when a copy is merely -**convenient**: +Three rules, checked in order: + +1. **Slow-changing reference data: a real object, fed by a sync.** Codes, catalogues and + registries that change monthly are better copied. Declare a real object and fill it + with a [scheduled pull](/docs/automation/jobs#pulling-a-mapping). +2. **The remote exposes OData v4: an OData datasource.** SAP S/4HANA, Dynamics 365, + Dataverse and other CSDL 4.0 services are served by the shipped OData driver. You + declare a datasource and one object per entity set, and write no driver code + ([OData datasources](/docs/data-modeling/odata-datasources)). +3. **No OData, rows computed on request, or a private API: a plugin driver of your + own**, which is what the rest of this page builds. + +The first rule is the question of whether to copy at all. Make the object virtual when a +copy would be **wrong**, and real when a copy is merely **convenient**: | The data is… | Choose | Why | | :--- | :--- | :--- | @@ -34,6 +46,11 @@ Make it virtual when a copy would be **wrong**, and real when a copy is merely If the remote is a SQL database you can reach, you need no driver of your own — use a built-in one through [External datasources](/docs/data-modeling/external-datasources). +**Salesforce takes a driver of its own.** Salesforce can read other systems' OData +services (Salesforce Connect), but it does not publish its own records as an OData +service, so the OData datasource cannot read it. Serve Salesforce objects from a plugin +driver over its REST API, built to the contract on this page. + A virtual object pays for its freshness on every request: each list read is a round trip to the remote, and a read can match at most 1000 rows ([the cap](#the-engine-re-applies-every-query)). That is the trade the table above @@ -595,7 +612,9 @@ plan's view and its edit door. A gantt over the tree, a per-caller lock formula, `beforeUpdate` guard, writes through `update` only. **SAP sales orders — must not land.** The ERP is the system of record, and orders may not -be copied out of it. The driver reads orders through the ERP's own query API and writes +be copied out of it. When the ERP publishes the orders as an OData v4 service, bind them +through an [OData datasource](/docs/data-modeling/odata-datasources) and write no driver. +Otherwise the driver reads orders through the ERP's own query API and writes the few fields users may change through its update API; the ERP's own validation answers a write it rejects, so let that error reach the caller rather than retrying around it. Refuse `create` and `delete` until the integration deliberately offers them, and keep @@ -622,6 +641,8 @@ separate data-sync metadata type: a sync is defined on its target, as that mappi ## See also +- [OData datasources](/docs/data-modeling/odata-datasources) — the same binding over an + OData v4 service, with the driver already written. - [External datasources](/docs/data-modeling/external-datasources) — the datasource and `external` binding this page builds on, federation over SQL, the write gates and credentials. diff --git a/packages/drivers/driver-odata/README.md b/packages/drivers/driver-odata/README.md index ffaa362984a..a9559a29489 100644 --- a/packages/drivers/driver-odata/README.md +++ b/packages/drivers/driver-odata/README.md @@ -10,6 +10,8 @@ The driver is a **plugin driver**: a server plugin publishes it as the service external-driver floor and re-applies every query to the rows the service returns. The driver pushes down what it can translate and only ever asks the service a wider question; see [Virtual objects](https://objectstack.ai/docs/data-modeling/virtual-objects). +The guide is [OData datasources](https://objectstack.ai/docs/data-modeling/odata-datasources): +checking a service, credentials, what the service is asked, writes and failures. ## Usage diff --git a/scripts/docs-audit/handwritten-docs.json b/scripts/docs-audit/handwritten-docs.json index 9ce1b9ca99d..05b00f4b3f9 100644 --- a/scripts/docs-audit/handwritten-docs.json +++ b/scripts/docs-audit/handwritten-docs.json @@ -87,6 +87,7 @@ "content/docs/data-modeling/indexing.mdx", "content/docs/data-modeling/object-extensions.mdx", "content/docs/data-modeling/objects.mdx", + "content/docs/data-modeling/odata-datasources.mdx", "content/docs/data-modeling/queries.mdx", "content/docs/data-modeling/relationships.mdx", "content/docs/data-modeling/schema-design.mdx",