Skip to content

Commit 4265099

Browse files
committed
feat(datasource): show the field reference the service generates
`--config` takes a JSON object whose shape depends on the family, and the only way to learn that shape was to read the catalog's schema -- which described an older request format and would have had you build a body the API rejects. `hotdata datasource fields [FAMILY]` renders the reference the service now generates from its own request models: config, credentials and selector fields with types, requiredness and defaults, plus what the family supports. Raw JSON Schema under -o json for a UI or a script. `--config`'s help points at it, as the old help pointed at the catalog. A sibling verb rather than an overload of `types`, whose positional filters catalog ENTRY names -- and an entry is not a family (`postgres` is an entry, `sql` is its family), so one positional meaning both would be ambiguous exactly where people are already confused. Response structs carry unknown keys through to -o json rather than dropping them: a client that silently discarded a field would hand a user a reference less complete than the wire, which is the failure this endpoint exists to end.
1 parent 08b91a2 commit 4265099

6 files changed

Lines changed: 880 additions & 17 deletions

File tree

‎README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,10 @@ Three nouns, and each one has a stable id:
9292
```sh
9393
hotdata datasource types # browse: SQL dialects, ~150 API services,
9494
# buckets, iceberg, kafka, api
95+
hotdata datasource fields sql # the fields --config, --credentials and
96+
# --selector take for one family, and
97+
# what that family can do (-o json for
98+
# the JSON Schema)
9599

96100
# 1. Check credentials without creating anything — nothing is persisted:
97101
hotdata datasource validate --family sql --config @source.json

‎skills/hotdata/SKILL.md‎

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -242,13 +242,22 @@ Pull data from external sources (SQL databases, APIs, S3/GCS/Azure buckets, Iceb
242242
- **ingest** (`ing_…`) — a saved load definition: `datasource + selector + destination + type/schedule`. One datasource can back many ingests.
243243
- **run** (`run_…`) — one execution attempt, with snapshots of the config version, selector, and destination it used.
244244

245-
Read commands (`datasource list|show|types`, `ingest list|show|runs`, `run show`) work with a login session JWT. Commands that persist a credential (`datasource create`, `datasource update-config`, `ingest create`) **require a workspace API key** (`HOTDATA_API_KEY` / `--api-key`, `hd_...`) — the run outlives the 5-minute JWT.
245+
Read commands (`datasource list|show|types|fields`, `ingest list|show|runs`, `run show`) work with a login session JWT. Commands that persist a credential (`datasource create`, `datasource update-config`, `ingest create`) **require a workspace API key** (`HOTDATA_API_KEY` / `--api-key`, `hd_...`) — the run outlives the 5-minute JWT.
246246

247247
```bash
248248
hotdata datasource types [filter] # browse available source types. The FAMILY
249-
# column is what --family takes. -o json
250-
# includes each entry's config_schema —
251-
# the exact fields --config takes.
249+
# column is what --family takes.
250+
251+
hotdata datasource fields [family] # THE FIELD REFERENCE. With a family: the
252+
# fields --config, --credentials and
253+
# --selector take, with types, which are
254+
# required, and what the family supports
255+
# (write modes, continuous, row filter).
256+
# With none: every family, one row each.
257+
# -o json returns the JSON Schema itself.
258+
# The service generates it from the models that validate the request, so it
259+
# cannot name a field the API rejects. Read it before writing source.json or a
260+
# selector — do not guess field names, and do not carry your own list.
252261

253262
# --- datasources -------------------------------------------------------------
254263
hotdata datasource validate --family sql --config @source.json
@@ -262,6 +271,7 @@ hotdata datasource create --family sql --config @source.json --display-name "pro
262271
# --config also accepts a bare config object, @- (stdin), or inline JSON.
263272
# --credentials takes the secret half separately. Keep secrets out of argv.
264273
# Families: sql, filesystem (buckets), kafka, iceberg, delta, ducklake, rest.
274+
# The fields each half takes: hotdata datasource fields <family>.
265275

266276
hotdata datasource list [--family sql] [--state active] # ids, families, states
267277
hotdata datasource show <datasource-id> # state, config version, discovery
@@ -276,7 +286,8 @@ hotdata datasource delete <datasource-id> # 409 while any ingest references it
276286
# --- ingests -----------------------------------------------------------------
277287
hotdata ingest create --datasource-id ds_01J --type one-time \
278288
--selector @selector.json --destination @destination.json
279-
# selector.json is family-specific (what subset to read).
289+
# selector.json is family-specific (what subset to read) — its fields, and the
290+
# write modes this family accepts: hotdata datasource fields <family>.
280291
# destination.json is {"database_id", "schema", "table", "write_mode"} —
281292
# write_mode: replace | append | upsert. Both are IMMUTABLE after creation.
282293
# A one-time ingest runs immediately and reports its initial run id.

0 commit comments

Comments
 (0)