Runbook

Local development

Install dependencies:
```
uv sync
```
Configure the required environment:
```
export HOTDATA_API_KEY=<api_key>
```
.env.example documents optional variables such as HOTDATA_DATABASE_ID, HOTDATA_SCHEMA, HOTDATA_WRITE_DISPOSITION, HOTDATA_DECLARED_TABLES, HOTDATA_API_BASE_URL, and retry knobs. Copy it to .env if you use a shell env loader.
Pass the workspace id to examples with --workspace-id, or configure it in your own pipeline with hotdata(workspace_id=...). The workspace id is a routing parameter, not a secret.
Validate tooling:
```
uv run ruff check .
uv run pytest
```

Demo pipeline (downloads FRED macro-economic data and loads into Hotdata):
```
uv run hotdata-dlt-demo --workspace-id <workspace_id>
```
Requires HOTDATA_API_KEY to be set; pass your workspace id as the argument. On the first run the demo creates an instant database and prints its id; pass that id on later runs (--database-id <id>) to load into the same database instead of creating a new one. Instant databases are addressed by id, not by name.
Merge demo:
```
uv run hotdata-dlt-merge-demo --workspace-id <workspace_id>
```
Demonstrates replace followed by keyed merge, which maps to Hotdata's native upsert load mode when the table has a primary key.
ibis demo:
```
uv sync --extra ibis
uv run hotdata-dlt-ibis-demo --workspace-id <workspace_id>
```
Demonstrates pipeline.dataset().ibis() against the loaded instant database.

Create a new module in src/hotdata_dlt_destination/pipelines/.
Define a dlt.resource with explicit name and write_disposition.
Build the pipeline with destination=hotdata(workspace_id=..., database_name=..., declared_tables=[...]).
Add script entrypoint in pyproject.toml.
Add tests covering schema shape, write-disposition behavior, and retry/error handling.

Retryable failures auto-retry based on HOTDATA_MAX_RETRIES and HOTDATA_RETRY_BACKOFF_SECONDS.
If a load fails after max retries, rerun the same pipeline. Deterministic row keys keep upsert behavior stable.
For append mode recovery, rerun from the same source checkpoint to avoid gaps or duplicate records. Append is not deduplicated by the destination.
For merge mode, declare a stable primary_key before the first table creation. Keyed merge uses Hotdata's native upsert mode; a merge without a resolvable key falls back to a client-side combine.
If a table was created without the desired primary key or storage layout, recreate it deliberately and reload the data. Existing managed tables do not gain new keys or physical layout changes in place.

401 / 403: verify HOTDATA_API_KEY and the workspace id you passed.
404 on destination paths: verify HOTDATA_API_BASE_URL is the API host (for example https://api.hotdata.dev).
429 / 5xx: increase retry/backoff values.
table not declared: list every expected target table in declared_tables. Missing tables can be added to the existing database in place on a later run, but declaring them up front keeps table keys and layouts predictable.
merge duplicates rows: declare primary_key on the resource before the table is created. Keyless merge falls back to _dlt_id, which is generated by dlt for each extract.
hard_delete does not remove existing rows: verify the resource uses write_disposition="merge" with a resolvable primary_key. Other modes may filter flagged rows from the incoming batch without deleting already-loaded rows.
Slow non-keyed merge or insert-only: those fallback paths read existing target rows, combine client-side, and replace the table. Prefer keyed merge where possible.
Dataset readback fails: pass the pinned database_id to hotdata(...). Database names are labels and cannot identify an instant database.