From 45e7cdf4c190b7a62ec0967b5c3d4cf0895d01c4 Mon Sep 17 00:00:00 2001 From: Philip Z Date: Sat, 10 Oct 2026 15:26:38 +0800 Subject: [PATCH] docs: lead runtime README with core application value Show composable Q, safe E, governed Save, audit evidence and Agent Kit entry. Keep advanced details in RUNTIME_GUIDE.md and correct stale versions and claims. Signed-off-by: Philip Z --- README.md | 392 ++++++++++++++++++++--------------------------- RUNTIME_GUIDE.md | 264 +++++++++++++++++++++++++++++++ 2 files changed, 433 insertions(+), 223 deletions(-) create mode 100644 RUNTIME_GUIDE.md diff --git a/README.md b/README.md index 2f65adb..4603df3 100644 --- a/README.md +++ b/README.md @@ -1,256 +1,202 @@ -# TeaQL Python SDK +# TeaQL Python -## Sensitive log data +Review the business operation, not pages of persistence plumbing. -Runtime diagnostic logs redact payload values by default, before delivery to -file, console, buffers, or custom logging sinks. Selecting a diagnostic sink -alone does not authorize plaintext. For controlled troubleshooting only: +TeaQL Python is a reusable, domain-independent runtime for model-aware applications. +A generated domain library gives your application language-native **Q** queries, +**E** expressions and governed graph mutations. The runtime executes them +through metadata, Context, Checker, Policy and provider boundaries. -```bash -export TEAQL_ALLOW_SENSITIVE_PLAINTEXT_LOGS=I_UNDERSTAND_SENSITIVE_DATA_MAY_BE_WRITTEN_TO_DISK -``` +The goal is to let people and coding agents express—and review—business intent +in a small amount of application code. -Only this exact value enables plaintext permission; empty values, `true`, and -whitespace variants do not. Enabling it emits a warning. Credential-classified -fields remain redacted. The flag does not force every sink to expose values. -SQL without reliable field/literal provenance may be suppressed and marked -`NOT REPLAYABLE`. Execution parameters and persisted business data are unchanged. - -Do not put sensitive data in free-text comments or purpose declarations. -TeaQL cannot govern arbitrary application prints or independent driver loggers; -configure those separately. This setting does not erase older plaintext files. -Restrict access and retention when using plaintext diagnostics, then unset the -variable and restart processes when troubleshooting is complete. - -TeaQL Python SDK is a runtime engine and toolkit for building data-driven business applications. It uses `teaql-rs` as the cross-language design baseline; verified coverage and remaining gaps are tracked in [conformance](https://github.com/teaql/teaql-conformance). - -## Recommended Agent Harness - -When building database-backed applications with the TeaQL Python runtime, we -recommend using it together with the [TeaQL Agent Kit](https://github.com/teaql/teaql-agent-kit). -The Agent Kit is TeaQL's continuously evolving **Harness Engineering** method. -It gives coding agents a model-mediated, executable workflow for domain -modeling, deterministic evaluation and repair, code generation, implementation, -and evidence-based verification as the generator and runtimes evolve. - -## 1. Minimum Version Requirements - -* **Python**: 3.10+ (Recommended 3.12+) -* **Testing**: `pytest` 7.4+ -* **Dependencies**: `pydantic` >= 2.0, `aiosqlite`, `aiomysql`, `asyncpg`, `cryptography` >= 42 - -## 2. Tests Performed - -The SDK includes tests in the following areas. These suites do not establish complete Rust parity or acceptance of every external provider: -* **Cross-language contracts**: Shared fixtures and integration tests verify individual portable behaviors; unresolved coverage stays explicit in conformance. -* ✅ **`teaql.core` Core Tests**: Extensively tested attribute extraction, safe nullability checks, and relationship building for `Value`, `GraphNode`, `Entity`, `Mutation`, `Query`, `Expr`, and `SafeExpression`. -* ✅ **`teaql.runtime` Runtime Tests**: Verified the context propagation of `UserContext` and the complete lifecycle hooking for `record_sql_log` and `record_metadata_log`. -* ✅ **`teaql.sql` / `teaql.data_service` Tests**: Tested the SQL AST compilation engine and the underlying command dispatch mechanism. -* ✅ **Provider & External Integrations**: Includes integration tests for the SQLite driver and FastAPI web endpoints. - -## 3. Available Modules - -The SDK's organizational architecture strictly mirrors the Rust version: -* `teaql.core`: Provides core underlying data structures (e.g., `Value`, `GraphNode`, `Entity`, `SelectQuery`, `MutationRequest`). -* `teaql.data_service`: Defines universal data service abstractions, handling structured inputs and outputs. -* `teaql.sql`: Provides a cross-dialect SQL compilation executor, translating ASTs into physical queries for various databases. -* `teaql.runtime`: Contains the pipeline and application context mechanisms (e.g., `UserContext`, environment mounts). -* `teaql.provider`: Packages for physical database drivers and the canonical TeaQL Federal Protocol client. -* `teaql.web`: Web framework integration middleware (e.g., `FastAPI` / `Starlette`). - -## 4. Features - -* **Entity & Value Mapping**: Provides type-safe mapping between native Python types and TeaQL core primitives (I64, Text, F64, Null, etc.). -* **SQL Compilation & AST Building**: A dynamic, secure SQL query builder that generates standardized `INSERT`, `UPDATE`, `DELETE`, and `SELECT` statements while abstracting away dialect differences. -* **Facet Aggregation & Grouping**: Out-of-the-box support for multi-dimensional facet aggregations, group-bys, and hierarchical data processing. -* **Provider Support**: Highly extensible asynchronous database connectivity (integrating third-party async drivers like `aiosqlite` through a unified Transport layer). -* **Context & Logging Management**: Built-in support for lifecycle context passing, end-to-end tracing, and SQL execution log interception and dispatch. -* **Governed Mutation Policy**: An application-owned policy can review an - immutable whole-graph plan after Checker/Fix and before the first provider - mutation. Exact policy identity and approval state are retained with audit - evidence. Missing customer policy or approval emits stable warnings without - changing persistence semantics; an explicit denial fails closed. -* **Governed Business ID Lifecycle**: Core model contracts, a context-owned - profile/key/service boundary, retry-safe assignment, an in-memory allocator, - and explicit-schema durable SQLite allocation extend the portable - `daily-permuted-v1` encoder without exposing its internal sequence. -* **TeaQL Federal Protocol Client**: `TeaQLFederalClient` and `TfpHttpProvider` - execute governed canonical TFP v1 queries and audited mutations against a - remote TeaQL endpoint such as Rust. Direct query execution returns - `SmartList`; Python intentionally does not expose a TFP server endpoint. +## What you gain -```python -from teaql.provider.tfp_client import FederalQuery, TeaQLFederalClient - -client = TeaQLFederalClient("https://orders.example.com/tfp") -orders = await client.execute_query(FederalQuery( - entity="CustomerOrder", - filter_condition={"status": {"$eq": "NEW"}}, - comment="List new orders", - purpose="Render operations queue", -)) -await client.aclose() +| Capability | Benefit to application developers | +| --- | --- | +| Composable Q API | Combine filters, projections and relation requests without duplicating relationship-loading code; review what is being requested. | +| Safe E API | Distinguish an unloaded field from a loaded NULL; do not mistake an incomplete result for a business fact. | +| Governed graph Save | Stage creates, updates and deletions, then validate and persist through one Save boundary rather than hand-written SQL loops. | +| Execution and audit evidence | Follow declared intent, SQL routes and mutation responsibility when debugging or explaining a change. | + +## Try it with your coding agent + +You can start with an agent you already use. Paste this prompt: + +```text +Follow the current instructions at https://github.com/teaql/teaql-agent-kit. +Build a small school-management application in Python using SQLite. +Evaluate and repair the model, generate the library and workspace, then verify +Q, E, Checker and audited Save. Report commands, results and remaining frictions. ``` -## Trace Chain Local Source Status - -Query and Mutation Requests own their required non-blank intent independently -of logging. Physical SQL paths use the Rust canonical algorithm, with twelve -byte-identical frozen vectors, owned query origins and qualified relation frames. -Native SQLite tests observe three real relation levels without inserting expected -frames, including a deepest-query failure. A successful write and a failed -authoritative readback keep separate canonical paths and statement outcomes; -the failed transaction rolls back. - -Parent query diagnostics include declared descendant binding provenance before -safe projection. The existing mask algorithm, expanded SQL and execution values -are unchanged; provenance is not stored on Context or exposed in log entries. -This additional compilation has not been performance benchmarked. - -Graph saves now use an explicit invocation-owned `GraphMutationSession` and an -immutable persistent parent scope. The original Context's data service is never -swapped; child saves receive the session and their parent scope explicitly. -Unannotated children inherit, local reasons append once, and deletion creates its -scope before execution. The typed ledger can store an owned complete replacement -chain. Five shared graph fixtures verify fifteen per-entity expectations; these -helper tests are separate from generated traversal evidence. - -[The generated Trace Chain example](examples/trace-chain/) drives six mutations -through actual Q/E/save APIs and SQLite, observing request lineage, SQL metadata -and safe audits after commit. It also verifies three query relation levels, -concurrent independent saves in one Context, write/readback failure rollback, -and missing root intent before transaction access. Its verifier executes twice -on one SQLite file without cleanup and checks all generated-library hashes. -All ten example groups and the source suite pass twice locally (424 tests and -eight explicit external-provider skips per source run). - -SQL transactions queue owned audit snapshots until commit, including explicit -transactions and automatic batches. Rollback discards them. A completion failure -is reported with `GraphCommittedError.committed == True`; all remaining audit -deliveries and graph cleanup callbacks are still attempted, with no postcommit -rollback. Reentrant independent root saves fail instead of silently joining. -The private generated save implementation has changed; old generated libraries -must be regenerated, while public `audit_as(...).save(context)` stays unchanged. - -The aggregate follow-up adds runtime-owned query projection snapshots and generated -`query_projection(alias)` / `has_query_projection(alias)` accessors. Missing aliases -raise `KeyError`; zero and null are present values. Projections are owned snapshots, -not writable model fields, live calculations or part of a mutation payload. The -generated example covers root/nested counts, original Trace Chain ancestry, -masked descendant intent, logging on/off, provider failure recovery and saving -only a modeled field. An alias named `save` cannot replace the save method. -Regenerate libraries to acquire this projection API; related aggregate streaming -remains unsupported and rejects before provider I/O. - -Relation attachment retains original scalar keys in invocation-local runtime -state. It does not infer membership from hydrated objects: a forward reference -filtered to `None` must not remove its child from a separately loaded parent -list, and a later sibling can still load using the same FK. Keys are not added -to returned records or mutation payloads. Native SQLite cases exercise these -boundaries with related counts, nested ancestry and logging both on and off. - -The reusable [generated Facet example](examples/facet-trace/) exercises 20 -root/nested/loaded-relation scenarios twice on one retained SQLite database. -It checks actual counts and returned metadata, original root/ancestor routes, -future-binding masking, logging on/off, and independent next-query state. The -all-examples gate now includes it and fingerprints the generated library. - -This is local-source/generated-consumer evidence, not complete Trace Chain. -Prepared-batch grouping, generated ledger overrides, complete entry-point and -inherited mutation privacy coverage, successful readback diagnostic alignment, -live PostgreSQL/MySQL graph acceptance, and immutable internal Registry replay -remain open. No released-package parity is claimed. +The [Agent Kit](https://github.com/teaql/teaql-agent-kit) guides modeling, +evaluation/repair, generation and application customization. It may install +toolchains and download dependencies; review installation permissions and the +final execution report. It is a workflow, not a guarantee of unattended correctness. + +## Run the local example + +Python 3.10+; Python 3.12+ is recommended. Async SQL execution uses the selected provider driver. From a checkout of this repository: ```bash -PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=src python -m pytest -q -PYTHONDONTWRITEBYTECODE=1 bash scripts/verify-examples.sh -# Repeat both commands without source changes. +python -m venv .venv +. .venv/bin/activate +python -m pip install -e . +export PYTHONPATH="$PWD/src" +(cd examples/school-management && python -m app.main) ``` -## Security Foundation Status +The [School example](examples/school-management/README.md) +exercises SQLite schema/bootstrap and generated application APIs against this +repository's local runtime. No external database server or generator is needed +to run the retained generated example. Use only the example's test database; +reset/repeated-run behavior is documented in its README. -Python currently provides governed local SQL execution and a **TFP client**; -it does not claim a public TFP server endpoint. Application queries retain -non-empty comment/purpose and audited mutations retain their audit reason. -Runtime logging should keep parameterized SQL and intent separate from any -restricted value-bearing diagnostic output. +## Query and read the result -Python implements the portable `tqr1` tuple codec shared with Go and .NET. It -encrypts and authenticates entity type, internal ID, optimistic version, -issued/expiry time, and purpose with AES-256-GCM. Key rings permit rotation; -encoding always uses the active key while decoding can accept retained keys. +These application-code excerpts assume the example's generated imports, trusted +Context and an existing object identity. `Q` and `E` belong to the generated +domain library, not a generic runtime object with arbitrary field strings. +See the [application source](examples/school-management/app/main.py) for complete setup. ```python -import os -from datetime import timedelta -from teaql.runtime import AeadEntityReferenceCodec, UserContext - -codec = AeadEntityReferenceCodec(2, { - 1: bytes.fromhex(os.environ["TEAQL_ENTITY_REFERENCE_KEY_V1_HEX"]), - 2: bytes.fromhex(os.environ["TEAQL_ENTITY_REFERENCE_KEY_V2_HEX"]), -}) -context = UserContext().with_entity_reference_codec(codec) -token = context.encode_entity_reference( - "OrderItem", 42, 7, "edit-order", timedelta(minutes=30) +school = await ( + Q.schools().with_id_is(school_id) + .select_platform_with(Q.platforms_minimal().select_name().select_base_url()) + .select_school_type_with(Q.school_types_minimal().select_name().select_code()) + .comment("what: load the school and its reference data") + .purpose("why: review a school before editing") + .execute_for_one(context) ) -claims = context.decode_entity_reference(token, "OrderItem", "edit-order") +if school is None: + raise ValueError("School not found") ``` -Without a configured codec the runtime fails closed. Local debugging can use -the canonical long `TEAQL_UNSAFE_RAW_ENTITY_REFERENCES` acknowledgement, which -emits visibly distinct `tqr0` references. It must not be enabled in production. +The selected relation request is another typed query: it can be composed and +reused independently. `comment` describes **what**; `purpose` describes **why**. +Both must be non-empty, even when logging is disabled. -The wire format, fail-closed behavior, shared golden vector, and exact -development-only acknowledgement are maintained in the canonical -[opaque entity reference contract](https://github.com/teaql/teaql-conformance/blob/main/design/opaque-entity-references.md). -Opaque tokens never replace the backend's authorization, tenant, ownership, -role, or optimistic-version checks. +```python +type_code = E.school(school).school_type().code().eval() +``` -The repeatable [`examples/opaque-entity-reference`](examples/opaque-entity-reference) -example proves the exact cross-language golden vector and purpose-substitution -rejection. +E reads the loaded graph without implicitly querying the database. Unloaded +modeled fields remain guarded, rather than silently becoming NULL. The exact +error/result representation follows the language's API. A NULL fallback is +not permission to ignore an unloaded property. -## Business ID V1 Foundation +## Change the object and save -The pure runtime encoder maps a durable internal sequence into the canonical -six-character, scope-specific Base36 code: +Load the complete scalar fields of every persisted object you actually modify. +A display-only partial projection is not a complete mutation input. Unmodified +reference objects do not have to become fully loaded just to save their parent. ```python -from teaql.core import BusinessIdEncodingKey, BusinessIdScope -from teaql.runtime import encode_business_id_permutation_v1 +school.update_name("Riverside Academy") +await school.audit_as("Rename the reviewed school").save(context) +``` -scope = BusinessIdScope( - "tenant-a", "commerce_order", "order_number", "20260925" -) -key = BusinessIdEncodingKey(1, key_from_secret_manager) -code = encode_business_id_permutation_v1(0, scope, key) +Save traverses the reached mutation graph, runs Checker/Fix and the configured +customer Mutation Policy, then uses the provider transaction boundary. +An explicit policy denial blocks persistence. Missing customer policy or exact +approval emits a warning; it is not automatic rejection. Marking for deletion +stages a change—the audited Save performs it. Atomicity is provider/route scoped, +not a distributed transaction across unrelated databases. + +## See why an operation happened + +Query and Mutation diagnostic logs are enabled by default and can be switched +off independently. Logging off does not disable intent validation or Policy. + +Intent, physical SQL paths and mutation responsibility are separate facts. +For example, a safe diagnostic can tell an operator which query populated a +screen, and an audit can explain why a saved object changed. + +Illustrative, shortened view—not a literal logger format or a benchmark result: + +```text +query entity=School comment="load the reviewed object" purpose="edit form" outcome=success +SQL: SELECT id, version FROM school_data WHERE id = 42; +trace: School -> request -> sqlite -> select +mutation entity=School auditReason="Rename the reviewed school" outcome=committed +lineage: School#42("Rename the reviewed school") ``` -The retained [`examples/business-id`](examples/business-id) flow proves durable -concurrent allocation infrastructure and aggregate retry reuse. Generated -strongly typed fields and external lookup remain a separate generator -capability. Secret key material is application-owned and must not be placed in -KSML or generated source. +SQL execution stays parameterized. Diagnostic SQL has parameters substituted +after field-aware masking; it is not a question-mark template you must manually +reconstruct. Ordinary bindings stay visible, sensitive bindings use the shared +mask algorithm, and credentials/unknown bindings stay hidden. A masked statement +is labeled `MASKED; NOT REPLAYABLE`; SQL that cannot be safely rendered is omitted. +Application mutation audits are delivered after commit and discarded on rollback. +This is not a durable audit outbox. Do not put secrets in comment or purpose. -### Mutation Policy installation +## Use the published runtime -Policy implementations are installed from trusted application startup through -`UserContext`; request JSON cannot select or replace them. Built-in SQL and TFP -providers enter the same governed boundary. +Current documented runtime: **0.3.0**. Keep domain-library and runtime +versions compatible; regenerate older libraries when their provider or graph +adapter SPI has changed. -```python -context = ( - UserContext.new() - .with_mutation_policy_registry(policy_registry) - .with_mutation_policy_approval_provider(approval_provider) - .with_mutation_governance_sink(warning_sink) -) +```bash +python -m pip install teaql==0.3.0 +``` + +Generation is a separate service; see the [Agent Kit](https://github.com/teaql/teaql-agent-kit) +for producing the application-specific library and workspace. Repository examples +use local source; isolated released-package regression is a separate gate. + +## Supported scope + +Python provides asynchronous local SQL execution and a TFP client, not a public TFP server endpoint. Generated classes and expressions expose model-aware APIs; Python annotations are not a Rust-style compile-time guarantee. Provider and advanced-feature coverage is tracked explicitly in conformance. + +One semantic model can produce language-native applications across +[Java](https://github.com/teaql/teaql-java), +[Rust](https://github.com/teaql/teaql-rs), +[TypeScript](https://github.com/teaql/teaql-ts), +[Go](https://github.com/teaql/teaql-golang), +[Swift](https://github.com/teaql/teaql-swift), +[.NET](https://github.com/teaql/teaql-dotnet) and +[Python](https://github.com/teaql/teaql-python). +That does not imply identical APIs, providers or complete cross-language feature parity. +The semantic model defines business objects, fields, relationships and states; +it is not the language model used by a coding agent. +The [conformance matrices](https://github.com/teaql/teaql-conformance) +distinguish implemented source, executed examples, artifact verification and release. + +## Verification and customization + +```bash +python -m pip install -e '.[test]' +PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=src python -m pytest -q +PYTHONDONTWRITEBYTECODE=1 bash scripts/verify-examples.sh ``` -Generated graph saves call `preflight_mutation(...)` for every operation before -the first provider write. See the repeatable -[`examples/mutation-policy`](examples/mutation-policy) example for allow, -approval, audit propagation, and zero-write denial evidence. +The full example gate is the acceptance entry point for local runtime changes. +External-provider tests need their configured services; skips are not live-provider +evidence. A committed-operation error after audit delivery failure must not be +retried as if the business transaction rolled back. + +Install application-owned policies, ID/business-ID services, logging sinks, +transports and other supported SPIs through trusted runtime/Context setup. +Request JSON is not allowed to replace those trusted capabilities. + +- [Runtime guide: modules, SPI and advanced verification](RUNTIME_GUIDE.md) +- [Modules and providers](RUNTIME_GUIDE.md#3-available-modules) +- [Mutation Policy example](examples/mutation-policy) +- [Facet example](examples/facet-trace/README.md) +- [Opaque reference example](examples/opaque-entity-reference) + +## Safe troubleshooting + +Default diagnostics do not authorize plaintext sensitive data. Controlled +debugging requires the exact acknowledgement below and any sensitive sink +required by the runtime: + +```bash +export TEAQL_ALLOW_SENSITIVE_PLAINTEXT_LOGS=I_UNDERSTAND_SENSITIVE_DATA_MAY_BE_WRITTEN_TO_DISK +``` ---- -To run test validations and business logic simulations locally, simply run `pytest` in the project root. +Debug output is explicitly labeled. Credentials and unknown bindings remain +protected; the flag does not erase old files or govern application/driver +prints outside TeaQL. Restrict access and retention, then unset it and restart. +See the [runtime guide](RUNTIME_GUIDE.md) for log switches and extension details. diff --git a/RUNTIME_GUIDE.md b/RUNTIME_GUIDE.md new file mode 100644 index 0000000..ce58c21 --- /dev/null +++ b/RUNTIME_GUIDE.md @@ -0,0 +1,264 @@ +# TeaQL Python Runtime Guide + +Reference details for the runtime and its extension points. Start with the +[README](README.md) for the application-facing APIs and runnable examples. + +Current published versions and acceptance status are tracked in +[TeaQL Conformance](https://github.com/teaql/teaql-conformance). Individual +verification notes below describe their own scope, not all-provider parity. + +## Sensitive log data + +Runtime diagnostic logs redact payload values by default, before delivery to +file, console, buffers, or custom logging sinks. Selecting a diagnostic sink +alone does not authorize plaintext. For controlled troubleshooting only: + +```bash +export TEAQL_ALLOW_SENSITIVE_PLAINTEXT_LOGS=I_UNDERSTAND_SENSITIVE_DATA_MAY_BE_WRITTEN_TO_DISK +``` + +Only this exact value enables plaintext permission; empty values, `true`, and +whitespace variants do not. Enabling it emits a warning. Credential-classified +fields remain redacted. The flag does not force every sink to expose values. +SQL without reliable field/literal provenance may be suppressed and marked +`NOT REPLAYABLE`. Execution parameters and persisted business data are unchanged. + +Do not put sensitive data in free-text comments or purpose declarations. +TeaQL cannot govern arbitrary application prints or independent driver loggers; +configure those separately. This setting does not erase older plaintext files. +Restrict access and retention when using plaintext diagnostics, then unset the +variable and restart processes when troubleshooting is complete. + +TeaQL Python SDK is a runtime engine and toolkit for building data-driven business applications. It uses `teaql-rs` as the cross-language design baseline; verified coverage and remaining gaps are tracked in [conformance](https://github.com/teaql/teaql-conformance). + +## Recommended Agent Harness + +When building database-backed applications with the TeaQL Python runtime, we +recommend using it together with the [TeaQL Agent Kit](https://github.com/teaql/teaql-agent-kit). +The Agent Kit is TeaQL's continuously evolving **Harness Engineering** method. +It gives coding agents a model-mediated, executable workflow for domain +modeling, deterministic evaluation and repair, code generation, implementation, +and evidence-based verification as the generator and runtimes evolve. + +## 1. Minimum Version Requirements + +* **Python**: 3.10+ (Recommended 3.12+) +* **Testing**: `pytest` 7.4+ +* **Dependencies**: `pydantic` >= 2.0, `aiosqlite`, `aiomysql`, `asyncpg`, `cryptography` >= 42 + +## 2. Tests Performed + +The SDK includes tests in the following areas. These suites do not establish complete Rust parity or acceptance of every external provider: +* **Cross-language contracts**: Shared fixtures and integration tests verify individual portable behaviors; unresolved coverage stays explicit in conformance. +* ✅ **`teaql.core` Core Tests**: Extensively tested attribute extraction, safe nullability checks, and relationship building for `Value`, `GraphNode`, `Entity`, `Mutation`, `Query`, `Expr`, and `SafeExpression`. +* ✅ **`teaql.runtime` Runtime Tests**: Verified the context propagation of `UserContext` and the complete lifecycle hooking for `record_sql_log` and `record_metadata_log`. +* ✅ **`teaql.sql` / `teaql.data_service` Tests**: Tested the SQL AST compilation engine and the underlying command dispatch mechanism. +* ✅ **Provider & External Integrations**: Includes integration tests for the SQLite driver and FastAPI web endpoints. + +## 3. Available Modules + +The SDK's organizational architecture strictly mirrors the Rust version: +* `teaql.core`: Provides core underlying data structures (e.g., `Value`, `GraphNode`, `Entity`, `SelectQuery`, `MutationRequest`). +* `teaql.data_service`: Defines universal data service abstractions, handling structured inputs and outputs. +* `teaql.sql`: Provides a cross-dialect SQL compilation executor, translating ASTs into physical queries for various databases. +* `teaql.runtime`: Contains the pipeline and application context mechanisms (e.g., `UserContext`, environment mounts). +* `teaql.provider`: Packages for physical database drivers and the canonical TeaQL Federal Protocol client. +* `teaql.web`: Web framework integration middleware (e.g., `FastAPI` / `Starlette`). + +## 4. Features + +* **Entity & Value Mapping**: Provides type-safe mapping between native Python types and TeaQL core primitives (I64, Text, F64, Null, etc.). +* **SQL Compilation & AST Building**: A dynamic, secure SQL query builder that generates standardized `INSERT`, `UPDATE`, `DELETE`, and `SELECT` statements while abstracting away dialect differences. +* **Facet Aggregation & Grouping**: Out-of-the-box support for multi-dimensional facet aggregations, group-bys, and hierarchical data processing. +* **Provider Support**: Highly extensible asynchronous database connectivity (integrating third-party async drivers like `aiosqlite` through a unified Transport layer). +* **Context & Logging Management**: Built-in support for lifecycle context passing, end-to-end tracing, and SQL execution log interception and dispatch. +* **Governed Mutation Policy**: An application-owned policy can review an + immutable whole-graph plan after Checker/Fix and before the first provider + mutation. Exact policy identity and approval state are retained with audit + evidence. Missing customer policy or approval emits stable warnings without + changing persistence semantics; an explicit denial fails closed. +* **Governed Business ID Lifecycle**: Core model contracts, a context-owned + profile/key/service boundary, retry-safe assignment, an in-memory allocator, + and explicit-schema durable SQLite allocation extend the portable + `daily-permuted-v1` encoder without exposing its internal sequence. +* **TeaQL Federal Protocol Client**: `TeaQLFederalClient` and `TfpHttpProvider` + execute governed canonical TFP v1 queries and audited mutations against a + remote TeaQL endpoint such as Rust. Direct query execution returns + `SmartList`; Python intentionally does not expose a TFP server endpoint. + +```python +from teaql.provider.tfp_client import FederalQuery, TeaQLFederalClient + +client = TeaQLFederalClient("https://orders.example.com/tfp") +orders = await client.execute_query(FederalQuery( + entity="CustomerOrder", + filter_condition={"status": {"$eq": "NEW"}}, + comment="List new orders", + purpose="Render operations queue", +)) +await client.aclose() +``` + +## Trace Chain Local Source Status + +Query and Mutation Requests own their required non-blank intent independently +of logging. Physical SQL paths use the Rust canonical algorithm, with twelve +byte-identical frozen vectors, owned query origins and qualified relation frames. +Native SQLite tests observe three real relation levels without inserting expected +frames, including a deepest-query failure. A successful write and a failed +authoritative readback keep separate canonical paths and statement outcomes; +the failed transaction rolls back. + +Parent query diagnostics include declared descendant binding provenance before +safe projection. The existing mask algorithm, expanded SQL and execution values +are unchanged; provenance is not stored on Context or exposed in log entries. +This additional compilation has not been performance benchmarked. + +Graph saves now use an explicit invocation-owned `GraphMutationSession` and an +immutable persistent parent scope. The original Context's data service is never +swapped; child saves receive the session and their parent scope explicitly. +Unannotated children inherit, local reasons append once, and deletion creates its +scope before execution. The typed ledger can store an owned complete replacement +chain. Five shared graph fixtures verify fifteen per-entity expectations; these +helper tests are separate from generated traversal evidence. + +[The generated Trace Chain example](examples/trace-chain/) drives six mutations +through actual Q/E/save APIs and SQLite, observing request lineage, SQL metadata +and safe audits after commit. It also verifies three query relation levels, +concurrent independent saves in one Context, write/readback failure rollback, +and missing root intent before transaction access. Its verifier executes twice +on one SQLite file without cleanup and checks all generated-library hashes. +An earlier source checkpoint ran ten example groups and 424 source tests twice, +with eight explicit external-provider skips per source run. For current release +qualification, use the linked conformance receipt and the current example inventory. + +SQL transactions queue owned audit snapshots until commit, including explicit +transactions and automatic batches. Rollback discards them. A completion failure +is reported with `GraphCommittedError.committed == True`; all remaining audit +deliveries and graph cleanup callbacks are still attempted, with no postcommit +rollback. Reentrant independent root saves fail instead of silently joining. +The private generated save implementation has changed; old generated libraries +must be regenerated, while public `audit_as(...).save(context)` stays unchanged. + +The aggregate follow-up adds runtime-owned query projection snapshots and generated +`query_projection(alias)` / `has_query_projection(alias)` accessors. Missing aliases +raise `KeyError`; zero and null are present values. Projections are owned snapshots, +not writable model fields, live calculations or part of a mutation payload. The +generated example covers root/nested counts, original Trace Chain ancestry, +masked descendant intent, logging on/off, provider failure recovery and saving +only a modeled field. An alias named `save` cannot replace the save method. +Regenerate libraries to acquire this projection API; related aggregate streaming +remains unsupported and rejects before provider I/O. + +Relation attachment retains original scalar keys in invocation-local runtime +state. It does not infer membership from hydrated objects: a forward reference +filtered to `None` must not remove its child from a separately loaded parent +list, and a later sibling can still load using the same FK. Keys are not added +to returned records or mutation payloads. Native SQLite cases exercise these +boundaries with related counts, nested ancestry and logging both on and off. + +The reusable [generated Facet example](examples/facet-trace/) exercises 20 +root/nested/loaded-relation scenarios twice on one retained SQLite database. +It checks actual counts and returned metadata, original root/ancestor routes, +future-binding masking, logging on/off, and independent next-query state. The +all-examples gate now includes it and fingerprints the generated library. + +These are scoped source/generated-consumer cases, not complete Trace Chain. +Prepared-batch grouping, generated ledger overrides, complete entry-point and +inherited mutation privacy coverage, successful readback diagnostic alignment, +live PostgreSQL/MySQL graph acceptance, and immutable internal Registry replay +remain open. No released-package parity is claimed. + +```bash +PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=src python -m pytest -q +PYTHONDONTWRITEBYTECODE=1 bash scripts/verify-examples.sh +# Repeat both commands without source changes. +``` + +## Security Foundation Status + +Python currently provides governed local SQL execution and a **TFP client**; +it does not claim a public TFP server endpoint. Application queries retain +non-empty comment/purpose and audited mutations retain their audit reason. +Safe diagnostics retain expanded SQL and intent with field-aware parameter +masking; plaintext sensitive output requires the explicit debug acknowledgement. + +Python implements the portable `tqr1` tuple codec shared with Go and .NET. It +encrypts and authenticates entity type, internal ID, optimistic version, +issued/expiry time, and purpose with AES-256-GCM. Key rings permit rotation; +encoding always uses the active key while decoding can accept retained keys. + +```python +import os +from datetime import timedelta +from teaql.runtime import AeadEntityReferenceCodec, UserContext + +codec = AeadEntityReferenceCodec(2, { + 1: bytes.fromhex(os.environ["TEAQL_ENTITY_REFERENCE_KEY_V1_HEX"]), + 2: bytes.fromhex(os.environ["TEAQL_ENTITY_REFERENCE_KEY_V2_HEX"]), +}) +context = UserContext().with_entity_reference_codec(codec) +token = context.encode_entity_reference( + "OrderItem", 42, 7, "edit-order", timedelta(minutes=30) +) +claims = context.decode_entity_reference(token, "OrderItem", "edit-order") +``` + +Without a configured codec the runtime fails closed. Local debugging can use +the canonical long `TEAQL_UNSAFE_RAW_ENTITY_REFERENCES` acknowledgement, which +emits visibly distinct `tqr0` references. It must not be enabled in production. + +The wire format, fail-closed behavior, shared golden vector, and exact +development-only acknowledgement are maintained in the canonical +[opaque entity reference contract](https://github.com/teaql/teaql-conformance/blob/main/design/opaque-entity-references.md). +Opaque tokens never replace the backend's authorization, tenant, ownership, +role, or optimistic-version checks. + +The repeatable [`examples/opaque-entity-reference`](examples/opaque-entity-reference) +example proves the exact cross-language golden vector and purpose-substitution +rejection. + +## Business ID V1 Foundation + +The pure runtime encoder maps a durable internal sequence into the canonical +six-character, scope-specific Base36 code: + +```python +from teaql.core import BusinessIdEncodingKey, BusinessIdScope +from teaql.runtime import encode_business_id_permutation_v1 + +scope = BusinessIdScope( + "tenant-a", "commerce_order", "order_number", "20260925" +) +key = BusinessIdEncodingKey(1, key_from_secret_manager) +code = encode_business_id_permutation_v1(0, scope, key) +``` + +The retained [`examples/business-id`](examples/business-id) flow proves durable +concurrent allocation infrastructure and aggregate retry reuse. Generated +strongly typed fields and external lookup remain a separate generator +capability. Secret key material is application-owned and must not be placed in +KSML or generated source. + +### Mutation Policy installation + +Policy implementations are installed from trusted application startup through +`UserContext`; request JSON cannot select or replace them. Built-in SQL and TFP +providers enter the same governed boundary. + +```python +context = ( + UserContext.new() + .with_mutation_policy_registry(policy_registry) + .with_mutation_policy_approval_provider(approval_provider) + .with_mutation_governance_sink(warning_sink) +) +``` + +Generated graph saves call `preflight_mutation(...)` for every operation before +the first provider write. See the repeatable +[`examples/mutation-policy`](examples/mutation-policy) example for allow, +approval, audit propagation, and zero-write denial evidence. + +--- +To run test validations and business logic simulations locally, simply run `pytest` in the project root.