Skip to content

Commit 6227a9b

Browse files
authored
docs(skills): improve accuracy, structure, and consistency across CLI skills (#172)
1 parent 01b72da commit 6227a9b

10 files changed

Lines changed: 188 additions & 278 deletions

File tree

‎skills/hotdata-analytics/SKILL.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ version: 0.5.0
1818

1919
```bash
2020
hotdata query "<sql>" [--workspace-id <workspace_id>] [--database <database>] [--output table|json|csv]
21-
hotdata query status <query_run_id> [--output table|json|csv]
21+
hotdata query status <query_run_id>
2222
```
2323

2424
- **PostgreSQL dialect.** Quote mixed-case identifiers: `"CustomerName"`.
@@ -65,6 +65,7 @@ hotdata results <result_id> [--workspace-id <workspace_id>] [--output table|json
6565

6666
- Prefer **`results <id>`** over re-running identical heavy queries.
6767
- Query footers may include `[result-id: rslt...]`; also available from `queries <query_run_id>`.
68+
- `results list --limit` defaults to **100** (max **1000**) — unlike `queries list`, which defaults to **20**.
6869

6970
---
7071

@@ -103,9 +104,9 @@ Full procedure: [references/WORKFLOWS.md](references/WORKFLOWS.md).
103104
For equality, range, and sort-heavy OLAP — not full-text or vector (see **`hotdata-search`**):
104105

105106
```bash
106-
hotdata indexes create --catalog <connection-name-or-id> --schema <schema> --table <table> \
107+
hotdata indexes create --catalog <catalog-alias> --schema <schema> --table <table> \
107108
--name idx_orders_created --column created_at --type sorted [--async]
108109
```
109110

110-
List and delete use the same `hotdata indexes` commands as in the search skill; only **`--type sorted`** is the analytics focus here.
111+
List and delete use the same `hotdata indexes` commands as in the search skill; only **`--type sorted`** is the analytics focus here. With `--async`, track the build via **`hotdata jobs list`** (see **`hotdata`** skill → Jobs).
111112

‎skills/hotdata-analytics/references/WORKFLOWS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,7 @@ hotdata query "SELECT * FROM chain_db.public.revenue_slice WHERE ..."
8484

8585
### Naming and documentation
8686

87-
- Prefer predictable `--name` values: `chain_<topic>_<YYYYMMDD>`.
87+
- Prefer predictable `--catalog` / `--table` values, e.g. catalog `chain_db` + table `chain_<topic>_<YYYYMMDD>` (the chain table is named by `databases load --table`, not a `--name` flag).
8888
- Record long-lived chains in **context:DATAMODEL → Derived tables (Chain)** with the **full** SQL name you use (`database.schema.table`).
8989
- Promote join/grain findings to **context:DATAMODEL** when they should be shared or persisted (**`hotdata`** skill).
9090

‎skills/hotdata-geospatial/SKILL.md‎

Lines changed: 56 additions & 174 deletions
Original file line numberDiff line numberDiff line change
@@ -6,22 +6,34 @@ version: 0.5.0
66

77
# Hotdata Geospatial Skill
88

9-
Use this skill when working with geospatial data in Hotdata. Hotdata supports a subset of PostGIS-style functions using **PostgreSQL dialect SQL**. This reference is dataset-agnostic — apply it to any table with geometry columns.
9+
Hotdata supports a subset of PostGIS-style functions in **PostgreSQL-dialect SQL**. This skill is dataset-agnostic — apply it to any table with geometry columns.
1010

11-
**Related skills:** **`hotdata`** (core CLI), **`hotdata-search`** (BM25/vector), **`hotdata-analytics`** (OLAP SQL).
11+
**Requires the core `hotdata` skill** for auth, workspace, and table discovery. **Related:** **`hotdata-analytics`** (OLAP SQL), **`hotdata-search`** (BM25/vector).
12+
13+
## Running these queries
14+
15+
All SQL below runs through the core CLI:
16+
17+
```bash
18+
hotdata query "<sql>" [--workspace-id <id>] [--database <db>] [--output table|json|csv]
19+
```
20+
21+
- **Fully qualify tables** as `<connection>.<schema>.<table>` (or `<catalog>.<schema>.<table>` for a managed database) — every `<table>` placeholder below means a qualified name.
22+
- **PostgreSQL dialect:** double-quote any non-lowercase identifier (e.g. `"GeoID"`).
23+
- Discover candidate tables/columns with **`hotdata tables list --connection-id <id>`** (see core skill).
1224

1325
---
1426

15-
## Geometry Columns
27+
## Geometry columns
1628

17-
Most geospatial datasets in Hotdata carry one or both of:
29+
Most geospatial tables in Hotdata carry one or both of:
1830

1931
| Column | Type | Description |
2032
|---|---|---|
2133
| `wkb_geometry` | `Binary` | WKB-encoded geometry (polygon, point, multipolygon, etc.) |
22-
| `wkb_geometry_bbox` | `Struct` | Precomputed bounding box with fields `xmin`, `ymin`, `xmax`, `ymax` (Float32) |
34+
| `wkb_geometry_bbox` | `Struct` | Precomputed bounding box: `xmin`, `ymin`, `xmax`, `ymax` (Float32) |
2335

24-
**Always parse `wkb_geometry` with `ST_GeomFromWKB()` before using it in any spatial function:**
36+
**Parse `wkb_geometry` with `ST_GeomFromWKB()`** before any spatial function:
2537

2638
```sql
2739
ST_GeomFromWKB(wkb_geometry)
@@ -34,98 +46,27 @@ wkb_geometry_bbox['xmin'] -- ✓ works
3446
(wkb_geometry_bbox).xmin -- ✗ not supported
3547
```
3648

37-
Discover geometry columns with:
38-
39-
```sql
40-
hotdata tables list --connection-id <id>
41-
```
49+
Find these columns by their `Binary` / `Struct` types in `hotdata tables list --connection-id <id>`.
4250

4351
---
4452

45-
## Supported Functions
46-
47-
### Input / Construction
48-
49-
| Function | Example |
50-
|---|---|
51-
| `ST_GeomFromWKB(col)` | `ST_GeomFromWKB(wkb_geometry)` |
52-
| `ST_GeomFromText(wkt)` | `ST_GeomFromText('POLYGON((...))')` |
53-
| `ST_MakePoint(lon, lat)` | `ST_MakePoint(-122.27, 37.80)` |
54-
55-
### Output
56-
57-
| Function | Example |
58-
|---|---|
59-
| `ST_AsText(geom)` | `ST_AsText(ST_GeomFromWKB(wkb_geometry))` → WKT string |
60-
| `ST_AsBinary(geom)` | `ST_AsBinary(ST_GeomFromWKB(wkb_geometry))` → WKB binary |
61-
62-
### Accessors / Inspection
63-
64-
| Function | Returns |
65-
|---|---|
66-
| `ST_GeometryType(geom)` | e.g. `ST_Polygon`, `ST_MultiPolygon`, `ST_Point` |
67-
| `ST_IsValid(geom)` | boolean |
68-
| `ST_NumPoints(geom)` | integer |
69-
| `ST_NPoints(geom)` | integer (alias for ST_NumPoints) |
70-
| `ST_X(point)` | longitude (float) |
71-
| `ST_Y(point)` | latitude (float) |
72-
| `ST_Centroid(geom)` | point geometry |
53+
## Functions
7354

74-
### Measurement
55+
Common building blocks (full catalog, the unsupported set + workarounds, and unit conversions: **[references/functions.md](references/functions.md)**):
7556

76-
| Function | Unit | Notes |
77-
|---|---|---|
78-
| `ST_Area(geom)` | degrees² | Multiply by `111000 * 111000` for m², then `* 10.7639` for ft² |
79-
| `ST_Length(geom)` | degrees | Multiply by `111000` for approximate meters |
80-
| `ST_Distance(geom_a, geom_b)` | degrees | Multiply by `111000` for approximate meters |
81-
82-
> **No meter-native measurements:** `::geography` cast is not supported. All measurements are in decimal degrees. The conversion factor ~111,000 m/degree is accurate at mid-latitudes (~30–50°N/S) and degrades toward the poles.
83-
84-
### Spatial Relationships
85-
86-
All return `boolean`:
87-
88-
| Function | Meaning |
89-
|---|---|
90-
| `ST_Within(a, b)` | `a` is completely inside `b` |
91-
| `ST_Contains(a, b)` | `a` contains `b` |
92-
| `ST_Covers(a, b)` | `a` covers `b` (includes boundary) |
93-
| `ST_CoveredBy(a, b)` | `a` is covered by `b` |
94-
| `ST_Intersects(a, b)` | geometries share any space |
95-
| `ST_Overlaps(a, b)` | geometries overlap (same dimension) |
96-
| `ST_Touches(a, b)` | share boundary only, no interior overlap |
97-
| `ST_Crosses(a, b)` | geometries cross (different dimensions) |
98-
| `ST_Disjoint(a, b)` | geometries share no space |
99-
| `ST_Equals(a, b)` | geometries are spatially identical |
100-
101-
### Processing / Geometry Operations
102-
103-
| Function | Notes |
104-
|---|---|
105-
| `ST_ConvexHull(geom)` | Returns convex hull polygon |
106-
| `ST_Simplify(geom, tolerance)` | Douglas-Peucker simplification; tolerance in degrees |
107-
| `ST_OrientedEnvelope(geom)` | Minimum oriented bounding box |
57+
- **Construct:** `ST_GeomFromWKB`, `ST_GeomFromText('POLYGON((...))')`, `ST_MakePoint(lon, lat)`
58+
- **Inspect:** `ST_GeometryType`, `ST_IsValid`, `ST_X` / `ST_Y`, `ST_Centroid`, `ST_AsText`
59+
- **Relate (boolean):** `ST_Within`, `ST_Contains`, `ST_Covers`, `ST_Intersects`, `ST_Overlaps`, `ST_Touches`, `ST_Disjoint`, `ST_Equals`
60+
- **Measure (in degrees):** `ST_Distance`, `ST_Length`, `ST_Area`
61+
- **Process:** `ST_ConvexHull`, `ST_Simplify(geom, tol)`, `ST_OrientedEnvelope`
10862

109-
---
110-
111-
## Not Supported
112-
113-
| Category | Not Supported | Workaround |
114-
|---|---|---|
115-
| Output | `ST_AsGeoJSON`, `ST_AsEWKT` | Use `ST_AsText`; parse WKT client-side |
116-
| Cast | `::geography` | Multiply degrees by ~111,000 for meters |
117-
| Input | `ST_MakeEnvelope`, `ST_GeomFromGeoJSON`, `ST_MakeLine` | Use `ST_GeomFromText('POLYGON(...)')` for envelopes |
118-
| Accessors | `ST_SRID`, `ST_IsEmpty`, `ST_NumGeometries`, `ST_GeometryN`, `ST_ExteriorRing`, `ST_PointN`, `ST_StartPoint`, `ST_EndPoint` | — |
119-
| Measurement | `ST_Perimeter`, `ST_MaxDistance` | — |
120-
| Relationships | `ST_DWithin` | Use `ST_Within` + `ST_GeomFromText('POLYGON(...)')` |
121-
| Processing | `ST_Buffer`, `ST_Envelope`, `ST_Boundary`, `ST_Union`, `ST_Intersection`, `ST_Difference`, `ST_SymDifference`, `ST_Collect`, `ST_ClosestPoint`, `ST_Snap`, `ST_BoundingDiagonal`, `ST_Expand` | Use `ST_OrientedEnvelope` instead of `ST_Envelope` |
122-
| Projection | `ST_Transform`, `ST_SetSRID`, `ST_FlipCoordinates` | — |
63+
**Key limits** (see reference for the full list + workarounds): no `::geography` cast — **measurements are in decimal degrees**, convert with the factors in the reference (distance `× 111000` ≈ meters at mid-latitudes; **area is only order-of-magnitude**). No `ST_Buffer`, `ST_DWithin`, `ST_MakeEnvelope`, `ST_Union`, `ST_Transform`, or GeoJSON I/O — use the documented substitutes.
12364

12465
---
12566

126-
## Common Patterns
67+
## Common patterns
12768

128-
### Check geometry types in a table
69+
### Geometry types in a table
12970

13071
```sql
13172
SELECT ST_GeometryType(ST_GeomFromWKB(wkb_geometry)) AS geom_type, COUNT(*)
@@ -134,9 +75,9 @@ WHERE wkb_geometry IS NOT NULL
13475
GROUP BY 1
13576
```
13677

137-
### Bounding box filter (replaces ST_MakeEnvelope / ST_DWithin)
78+
### Bounding-box filter (replaces ST_MakeEnvelope / ST_DWithin)
13879

139-
Use `ST_GeomFromText` with a closed WKT polygon ring:
80+
Use `ST_GeomFromText` with a **closed** WKT polygon ring (repeat the first vertex):
14081

14182
```sql
14283
WHERE ST_Within(
@@ -145,9 +86,7 @@ WHERE ST_Within(
14586
)
14687
```
14788

148-
**Vertex order:** `(minLon minLat, maxLon minLat, maxLon maxLat, minLon maxLat, minLon minLat)` — close the ring by repeating the first point.
149-
150-
**Faster alternative** using the precomputed bbox struct (no WKB parsing):
89+
**Faster** on large tables — filter the precomputed bbox struct (no WKB parsing); use the `ST_Within` form when you need centroid-in-polygon precision:
15190

15291
```sql
15392
WHERE wkb_geometry_bbox['xmin'] >= <minLon>
@@ -156,129 +95,72 @@ WHERE wkb_geometry_bbox['xmin'] >= <minLon>
15695
AND wkb_geometry_bbox['ymax'] <= <maxLat>
15796
```
15897

159-
Use the bbox approach for large tables where WKB parsing is expensive; use `ST_Within` when you need centroid-in-polygon precision.
160-
161-
### Point-in-polygon test
98+
### Point-in-polygon
16299

163100
```sql
164101
SELECT *
165102
FROM <table>
166-
WHERE ST_Contains(
167-
ST_GeomFromWKB(wkb_geometry),
168-
ST_MakePoint(<lon>, <lat>)
169-
)
103+
WHERE ST_Contains(ST_GeomFromWKB(wkb_geometry), ST_MakePoint(<lon>, <lat>))
170104
```
171105

172-
### Nearest neighbors (closest N features to a point)
106+
### Nearest neighbors (closest N to a point)
173107

174108
```sql
175-
SELECT
176-
<id_col>,
177-
ST_Distance(
178-
ST_Centroid(ST_GeomFromWKB(wkb_geometry)),
179-
ST_MakePoint(<lon>, <lat>)
180-
) * 111000 AS dist_meters
109+
SELECT <id_col>,
110+
ST_Distance(ST_Centroid(ST_GeomFromWKB(wkb_geometry)), ST_MakePoint(<lon>, <lat>)) * 111000 AS dist_meters
181111
FROM <table>
182112
WHERE wkb_geometry IS NOT NULL
183113
ORDER BY dist_meters
184114
LIMIT 10
185115
```
186116

187-
### Distance between two known points
117+
### Distance between two points
188118

189119
```sql
190120
SELECT
191121
ST_Distance(ST_MakePoint(<lon1>, <lat1>), ST_MakePoint(<lon2>, <lat2>)) * 111000 AS dist_meters,
192122
ST_Distance(ST_MakePoint(<lon1>, <lat1>), ST_MakePoint(<lon2>, <lat2>)) * 69.0 AS dist_miles
193123
```
194124

195-
### Area of polygon features
125+
### Area of polygons (order-of-magnitude — see reference caveat)
196126

197127
```sql
198-
SELECT
199-
<id_col>,
200-
ST_Area(ST_GeomFromWKB(wkb_geometry)) * 111000 * 111000 AS area_sqm,
201-
ST_Area(ST_GeomFromWKB(wkb_geometry)) * 111000 * 111000 * 10.7639 AS area_sqft,
202-
ST_Area(ST_GeomFromWKB(wkb_geometry)) * 111000 * 111000 / 4047 AS area_acres
128+
SELECT <id_col>,
129+
ST_Area(ST_GeomFromWKB(wkb_geometry)) * 111000 * 111000 AS area_sqm,
130+
ST_Area(ST_GeomFromWKB(wkb_geometry)) * 111000 * 111000 / 4047 AS area_acres
203131
FROM <table>
204132
WHERE wkb_geometry IS NOT NULL
205133
```
206134

207135
### Centroid coordinates
208136

209137
```sql
210-
SELECT
211-
<id_col>,
138+
SELECT <id_col>,
212139
ST_X(ST_Centroid(ST_GeomFromWKB(wkb_geometry))) AS lon,
213140
ST_Y(ST_Centroid(ST_GeomFromWKB(wkb_geometry))) AS lat
214141
FROM <table>
215142
WHERE wkb_geometry IS NOT NULL
216143
```
217144

218-
### Convert to WKT for export or inspection
219-
220-
```sql
221-
SELECT <id_col>, ST_AsText(ST_GeomFromWKB(wkb_geometry)) AS wkt
222-
FROM <table>
223-
WHERE wkb_geometry IS NOT NULL
224-
LIMIT 10
225-
```
226-
227-
### Simplify geometry for faster rendering
145+
### Export / simplify as WKT
228146

229147
```sql
230-
SELECT <id_col>, ST_AsText(ST_Simplify(ST_GeomFromWKB(wkb_geometry), 0.0001)) AS simplified_wkt
231-
FROM <table>
232-
WHERE wkb_geometry IS NOT NULL
148+
-- raw WKT
149+
SELECT <id_col>, ST_AsText(ST_GeomFromWKB(wkb_geometry)) AS wkt FROM <table> WHERE wkb_geometry IS NOT NULL LIMIT 10
150+
-- simplified (tolerance in degrees, ~11 m at mid-latitudes)
151+
SELECT <id_col>, ST_AsText(ST_Simplify(ST_GeomFromWKB(wkb_geometry), 0.0001)) AS wkt FROM <table> WHERE wkb_geometry IS NOT NULL
233152
```
234153

235-
Tolerance is in degrees (~11 m at mid-latitudes). Increase for coarser simplification, decrease for finer.
236-
237154
---
238155

239-
## Unit Conversion Reference
240-
241-
| To get | Multiply degrees by |
242-
|---|---|
243-
| Meters (distance) | × 111,000 |
244-
| Kilometers (distance) | × 111 |
245-
| Miles (distance) | × 69.0 |
246-
| Feet (distance) | × 364,173 |
247-
| m² (area) | × 111,000² = × 12,321,000,000 |
248-
| ft² (area) | × 111,000² × 10.7639 |
249-
| Acres (area) | × 111,000² ÷ 4,047 |
250-
251-
> These conversions assume ~37°N latitude. They are approximations — accuracy decreases significantly above 60°N or below 60°S.
252-
253-
---
254-
255-
## Workflow: Exploring a New Geospatial Dataset
256-
257-
1. **Check for geometry columns:**
258-
```
259-
hotdata tables list --connection-id <id>
260-
```
261-
Look for `Binary` (WKB) or `Struct` (bbox) typed columns.
262-
263-
2. **Verify geometry types:**
264-
```sql
265-
SELECT ST_GeometryType(ST_GeomFromWKB(wkb_geometry)) AS type, COUNT(*)
266-
FROM <table> WHERE wkb_geometry IS NOT NULL GROUP BY 1
267-
```
268-
269-
3. **Check coverage (bounding box of entire dataset):**
270-
```sql
271-
SELECT
272-
MIN(wkb_geometry_bbox['xmin']) AS min_lon,
273-
MIN(wkb_geometry_bbox['ymin']) AS min_lat,
274-
MAX(wkb_geometry_bbox['xmax']) AS max_lon,
275-
MAX(wkb_geometry_bbox['ymax']) AS max_lat
276-
FROM <table>
277-
WHERE wkb_geometry_bbox IS NOT NULL
278-
```
156+
## Workflow: explore a new geospatial table
279157

280-
4. **Sample WKT to understand geometry structure:**
158+
1. **Find geometry columns** — `hotdata tables list --connection-id <id>`; look for `Binary` (WKB) / `Struct` (bbox) types.
159+
2. **Geometry types** — run the "Geometry types in a table" pattern above.
160+
3. **Coverage / extent** — aggregate the bbox struct:
281161
```sql
282-
SELECT ST_AsText(ST_GeomFromWKB(wkb_geometry)) FROM <table>
283-
WHERE wkb_geometry IS NOT NULL LIMIT 3
162+
SELECT MIN(wkb_geometry_bbox['xmin']) AS min_lon, MIN(wkb_geometry_bbox['ymin']) AS min_lat,
163+
MAX(wkb_geometry_bbox['xmax']) AS max_lon, MAX(wkb_geometry_bbox['ymax']) AS max_lat
164+
FROM <table> WHERE wkb_geometry_bbox IS NOT NULL
284165
```
166+
4. **Sample WKT** — run the "Export as WKT" pattern with `LIMIT 3` to see geometry structure.

0 commit comments

Comments
 (0)