Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Remote tests (`TestConnectRemote`) self-skip unless `SQLITE_REMOTE_URL` + `SQLIT

The module is a **two-driver, one-surface bridge**: any connection — local or remote — becomes the same Starlark object with the same ~24 methods, because both drivers yield a `*sql.DB`.

- **`sqlite.go`** — the module entry. `Module` holds a `base.ConfigurableModule`; `NewModule()` / `NewModuleWithFileAccess(allowed)` construct it. `LoadModule()` exposes three builtins: **`connect`** (local), **`connect_remote`** (libSQL), **`register_function`** (custom scalar funcs). Config keys + defaults + the `restrictFileAccess` flag + `isInMemoryDSN` live here.
- **`sqlite.go`** — the module entry. `Module` holds a `base.ConfigurableModule`; `NewModule()` / `NewModuleWithFileAccess(allowed)` / `NewModuleWithHostPolicy(policy)` construct it. `LoadModule()` exposes three builtins: **`connect`** (local), **`connect_remote`** (libSQL, with `file:` routed to the local path), **`register_function`** (custom scalar funcs, rejected when disabled by the host). Config keys, policy flags and `isInMemoryDSN` live here.
- **`database.go`** — `newDatabaseInstance(db, maxRows, restrictFileAccess)` wraps a `*sql.DB` into the Starlark object and registers every method (`query`, `query_one`, `execute`, `select`, `insert`, `insert_many`, `update`, `delete`, `upsert`, `count`, `batch`, `begin`, `prepare`, `prepare_query`, `attach`/`detach`, `create_table`/`drop_table`/`truncate_table`, `tables`/`table_exists`/`table_info`/`indices`, `close`).
- **`operations.go`** — the largest file: the actual SQL behind those methods (statement building, arg binding, result shaping).
- **`transaction.go`** — `begin` returns a transaction object mirroring the connection's methods plus `commit`/`rollback`.
Expand All @@ -51,10 +51,11 @@ Tier-2 hardening landed in v0.1.0; the iron rule is **opt-in / default-off so ol

1. **No host panics from script input.** `register_function` wraps the user callback in a `recover()` (`function.go`): a panicking custom function becomes a script-level error, never a host crash. Don't remove the deferred recover.
2. **Bounded results.** Every query helper routes materialization through `processQueryRows(rows, maxRows)` (`utils.go`). `max_rows` (config / `SQLITE_MAX_ROWS`, default `0` = unlimited) caps rows so a hostile `SELECT *` can't exhaust host memory. New query paths must go through this function, not raw `rows.Scan` loops.
3. **Opt-in file-access gate.** `NewModuleWithFileAccess(false)` sets `restrictFileAccess`; `connect` and `attach` then reject any on-disk path except the configured `database`, allowing only in-memory DSNs. `isInMemoryDSN` must parse `mode=memory` as a real query parameter (via `net/url`), **not** as a substring — a loose `strings.Contains(s, ":memory:")` was a real bypass (`file:/etc/passwd?x=:memory:`). When touching DSN classification, keep it parse-based.
3. **Opt-in file-access gate.** `NewModuleWithFileAccess(false)` sets `restrictFileAccess` and freezes the configured `database` at construction through `base.SetHostOnly`; no `set_database` is exported. `connect` (including `connect_remote` with a `file:` URL) rejects other disk paths; `attach` allows only memory. `isInMemoryDSN` trusts only the parsed empty/`:memory:` path, not query parameters such as `mode=memory`, because SQLite and Go parse malformed URI queries differently. Keep that strict classification.
4. **Backward compatibility.** `NewModule()` == `NewModuleWithFileAccess(true)`, `max_rows` defaults to unlimited. Any new safety lever must default to the historical behavior.
5. **PRAGMA config.** `journal_mode` / `synchronous` / `foreign_keys` / `busy_timeout` / `cache_size` are standard SQLite PRAGMAs applied at connect — they are not custom invented defaults.
6. **UDF register/connect serialization (PKG-29).** modernc's `Driver.Open` reads its process-global UDF map (`d.udfs`) with no lock, while `RegisterScalarFunction` (called by `register_function`) writes it. `connect` opens through `localDriverName` — a `funcMutex`-serialized wrapper (`ensureLocalDriver` / `udfSerializedDriver`) whose `Open` holds `funcMutex.RLock` while modernc reads the map, mutually exclusive with `register_function`'s `Lock`/write. This is required because a connection's `Open` is called lazily by `database/sql` (and from parallel host machines), so the "register before connect" convention doesn't hold. **Keep `connect` on `localDriverName`, never raw `"sqlite"`.** Scope/limits (documented in `ensureLocalDriver`): it protects this module's own register vs connect, not a different process-wide user of raw `"sqlite"`, nor collation/connection-hook registration (this module registers neither).
7. **Opt-in UDF exclusion (STAR-102).** `DisableCustomFunctions` uses `noFunctionsDriverName`, backed by an independent zero-value modernc driver with no registered functions, collations or hooks. It also rejects `register_function` before touching the global registry. Every local path, including the libSQL `file:` fallback and pool replacement, must preserve this choice. The legacy path still uses `localDriverName`; network server functions remain server-controlled. This is exclusion, not isolated/budgeted execution of enabled UDFs.

## Test organization

Expand All @@ -70,6 +71,6 @@ Three layers must stay in sync (enforced by the doc standard, `plan/starpkg文
## Release discipline

- **Floor = go 1.20**, registered under **ENG-09 SEP** (`plan/ENG-09-SEP…`): the libSQL remote client requires go 1.20; `modernc.org/sqlite` is capped at **v1.31.x** because v1.32+ uses `runtime.Pinner` (a go 1.21 API → declares go 1.20 but won't build on it). Don't bump modernc past v1.31.x without raising the floor.
- **CI matrix** = `[1.20.x, 1.25.x]` via the centralized reusable workflow in `1set/meta`.
- **CI matrix** = `[1.20.x, 1.27.x]` via the centralized reusable workflow in `1set/meta`.
- **Releases so far**: `v0.1.0` (local modernc, Tier-2 hardened) → `v0.2.0` (added remote libSQL `connect_remote`). The libSQL dependency adds **~6 MB / +70%** to a built binary vs modernc-only — a candidate for build-tag isolation in a future version.
- **Bumping the version, the go floor, or tagging are user-confirmed actions** — never tag autonomously; default to patch bumps; published tags are immutable.
33 changes: 29 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ SQL functions — pure Go, no cgo, all platforms.
- **Prepared statements** — `prepare` / `prepare_query` for repeated execution.
- **Custom SQL functions** — `register_function` runs Starlark logic inside SQL queries.
- **Local & remote** — local files / in-memory via `connect`; remote libSQL (self-hosted `sqld` or Turso Cloud) via `connect_remote`, exposing the same object surface.
- **Safe by default** — SQL-injection-resistant parameter binding, automatic SQLite ⇄ Starlark type conversion, plus opt-in host hardening (`max_rows`, `NewModuleWithFileAccess`).
- **Parameter binding and host controls** — SQLite ⇄ Starlark type conversion, configurable row limits, and opt-in file/function restrictions (`NewModuleWithFileAccess`, `NewModuleWithHostPolicy`).

For the complete per-builtin reference — signatures, parameters, returns,
errors, examples — and the configuration accessors, see
Expand Down Expand Up @@ -135,9 +135,11 @@ errors, and examples of every builtin and method above.
The module's options (`database`, `timeout`, `busy_timeout`, `foreign_keys`,
`journal_mode`, `synchronous`, `cache_size`, `max_rows`) are configured via
environment variables (`SQLITE_*`) or per-option `get_<key>` / `set_<key>`
accessor builtins, and serve as defaults for `connect` / `connect_remote`. Two
opt-in host levers (`max_rows` and `NewModuleWithFileAccess`) bound what an
untrusted script can reach; both default to off. `timeout` bounds each database
accessor builtins, and serve as defaults for `connect` / `connect_remote`.
`NewModuleWithFileAccess(false)` fixes the allowed `database` at construction
and omits `set_database`; later environment changes cannot widen that path.
`max_rows` remains a configurable row limit, with `0` meaning unlimited.
`timeout` bounds each database
operation (a per-query deadline, and a cancellation point tied to the script
thread) — this is what stops an unreachable remote from hanging the host. The
connection PRAGMAs (`foreign_keys`, `journal_mode`, …) are applied to every
Expand All @@ -146,6 +148,29 @@ so its schema and data persist across queries. See the
[Configuration section of docs/API.md](docs/API.md#configuration) for the full
option table, defaults, accessors, and host-hardening details.

Hosts can opt out of custom SQLite functions while retaining ordinary local SQL:

```go
module := sqlite.NewModuleWithHostPolicy(sqlite.HostPolicy{
RestrictFileAccess: true,
DisableCustomFunctions: true,
})
```

`DisableCustomFunctions` rejects `register_function` and opens local connections
on a driver without process-wide custom functions, including functions registered
by another module. SQLite built-ins, transactions and prepared queries remain
available. A `file:` URL passed to `connect_remote` follows this same local path,
file restrictions and connection PRAGMAs. Real network libSQL endpoints retain
their existing behavior; server functions and network access are controlled by
the host/server separately. No connection is opened until a script requests one.

`NewModule()` and the zero `HostPolicy` retain custom functions and mutable
configuration. Enabled script UDFs still use the global registry and a separate
Starlark thread: they do not inherit the caller's execution budget or cancellation,
and registrations are not released on connection close. The opt-out is suitable
for hosts that do not need UDFs; it is not a process or filesystem sandbox.

## License

MIT
51 changes: 36 additions & 15 deletions docs/API.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,9 +95,10 @@ rows = db.query("SELECT * FROM notes")
db.close()
```

> Note: `connect_remote` is unaffected by the file-access restriction
> (`NewModuleWithFileAccess`) — it opens a network libSQL endpoint, not a local
> file. Gate remote access at the network/credential layer.
> A `file:` URL is a local connection and follows `connect`'s file/function
> restrictions, module defaults and connection PRAGMAs. `auth_token` is unused
> for local files. HTTP(S), libSQL and WebSocket endpoints remain remote; gate
> their access at the network/credential layer.

### `close()`

Expand Down Expand Up @@ -997,7 +998,7 @@ accessor — never a getter — but this module has none.)

| Option | Getter | Setter | Type | Env var | Default | Description |
|--------|--------|--------|------|---------|---------|-------------|
| `database` | `get_database` | `set_database` | string | `SQLITE_DATABASE` | `:memory:` | Path to the SQLite database (use `:memory:` for in-memory) |
| `database` | `get_database` | `set_database` (unrestricted modules only) | string | `SQLITE_DATABASE` | `:memory:` | Path to the SQLite database (use `:memory:` for in-memory) |
| `timeout` | `get_timeout` | `set_timeout` | float | `SQLITE_TIMEOUT` | `30.0` | Per-operation deadline in seconds (per-query; cancels with the script thread; 0 = none) |
| `busy_timeout` | `get_busy_timeout` | `set_busy_timeout` | float | `SQLITE_BUSY_TIMEOUT` | `5.0` | Busy timeout in seconds |
| `foreign_keys` | `get_foreign_keys` | `set_foreign_keys` | bool | `SQLITE_FOREIGN_KEYS` | `true` | Enable foreign key constraints |
Expand Down Expand Up @@ -1028,23 +1029,43 @@ db = connect("app.db") # opens with journal_mode=WAL

### Host hardening (opt-in, host side)

Two **opt-in** levers let the host bound what a script can reach. Both default to
**off**, so existing scripts keep working unchanged. They are configured by the
host in Go, not from a script.
File and function restrictions are **opt-in**, fixed by the host in Go. The
default constructors continue to allow custom functions and local files.

- **Bound result size — `max_rows`.** A query helper materializes every returned
row into memory. Set `max_rows` (the config option above, or `SQLITE_MAX_ROWS`)
to cap the number of rows any single query returns; exceeding the cap raises a
script error instead of allocating without bound. `0` (the default) means
unlimited.
unlimited. This configuration is script-settable; it is not an immutable
host policy in the default module.
- **Restrict file access — `NewModuleWithFileAccess`.** By default a script may
`connect(path)` to any file path (or `attach` any database).
`NewModuleWithFileAccess(false)` locks this down: scripts may only open
**in-memory** databases (`:memory:`, `file::memory:…`, `mode=memory`) or the
**one** `database` configured on the module — any other path, and any `attach`
**in-memory** databases (`:memory:`, `file::memory:…`) or the
**exact** `database` configured on the module — any other path, and any `attach`
of an on-disk database, is rejected. `NewModule()` equals
`NewModuleWithFileAccess(true)`.

> Note: `connect_remote` is unaffected by `NewModuleWithFileAccess` — it opens a
> network libSQL endpoint, not a local file. Gate remote access at the
> network/credential layer.
`NewModuleWithFileAccess(true)`. In restricted mode, `database` is captured
at construction, has no `set_database`, and ignores subsequent environment
changes. This also applies to `connect_remote` with a `file:` URL. Paths are
compared as configured DSNs; this is not a filesystem or symlink sandbox.

- **Disable custom functions — `NewModuleWithHostPolicy`.** Set
`HostPolicy{DisableCustomFunctions: true}` to reject `register_function` and
exclude process-wide functions from every local connection, including pooled
replacements and local `file:` URLs passed to `connect_remote`. Functions
registered before or after construction by another module are also excluded.
The independent driver does not inherit custom collations or connection hooks
installed on the default driver. Ordinary SQLite built-ins, queries, writes,
transactions, attached databases and prepared statements remain available.
Combine with `RestrictFileAccess: true` for the file restriction above.
Policy flags have no script getters/setters or environment-variable controls.

A zero `HostPolicy` preserves `NewModule()` behavior. When custom functions
remain enabled, registrations are process-wide and permanent, and script UDF
callbacks use a separate thread without the caller's step budget/cancellation.
Disabling UDFs avoids that callback path; it does not provide isolated or
budgeted execution for UDFs that remain enabled.

> Network libSQL endpoints retain their existing behavior. Server-side functions
> and network authorization are controlled separately; the local UDF policy does
> not claim to restrict a remote server. All connections are explicit script calls.
Loading
Loading