diff --git a/content/agenticdata/README.md b/content/agenticdata/README.md index 2cc840d..3ca9b92 100644 --- a/content/agenticdata/README.md +++ b/content/agenticdata/README.md @@ -10,7 +10,7 @@ Your role: you are a data platform engineer at **Cymbal**, a commerce company wh 1. **A live replica instead of exports.** Datastream captures every insert, update, and delete from Cloud SQL for PostgreSQL and merges it into BigQuery within moments — over Private Service Connect, with no public IPs anywhere. 2. **Numbers people can trust.** A medallion architecture (bronze → silver → gold) built with Dataform, where the SQL is authored by the Antigravity CLI under your direction, complete with data tests. The raw data contains real-world-style flaws (duplicates, typos, orphaned records) — cleaning them is the point. -3. **Governance before agents.** Knowledge Catalog labels the tiers, scans quality continuously, locks down PII with enforced column-level security, and gives business terms a home — the grounding layer that makes AI answers trustworthy. +3. **Governance before agents.** Knowledge Catalog makes every table findable, holds the raw data to a quality standard, and locks down PII with enforced column-level security — the grounding layer that makes AI answers trustworthy. 4. **Agents on top.** A BigQuery conversational data agent over the gold layer, then an ADK multi-agent system: a concierge agent that answers live operational questions straight from Postgres and delegates analytical questions to the data agent over the **A2A protocol** — *agent-to-agent*, the open standard (governed by the Linux Foundation) that lets AI agents from any framework or vendor discover and call each other, the way HTTP lets services talk to each other. ## Architecture @@ -47,7 +47,7 @@ Your role: you are a data platform engineer at **Cymbal**, a commerce company wh ▲
Knowledge Catalog
Knowledge Catalog
- tier aspects · data quality
PII policy tags · glossary · lineage + search · data quality
PII policy tags ▲
A2A diff --git a/content/agenticdata/labs/3_dataform.md b/content/agenticdata/labs/3_dataform.md index fc9efc1..a2ee917 100644 --- a/content/agenticdata/labs/3_dataform.md +++ b/content/agenticdata/labs/3_dataform.md @@ -31,7 +31,7 @@ The Dataform CLI is already on your machine, and the project is pre-wired — th cd ~/bootkon/content/agenticdata/src/dataform ``` -Two files are worth a look before you brief anyone. workflow_settings.yaml (generated for you from the template by the bootstrap) — `defaultProject` should read `{{ PROJECT_ID }}`, and the git-ignored `.df-credentials.json` next to it tells the CLI to use your logged-in identity (Application Default Credentials) against BigQuery — no keys involved. And sources.js: it declares the six bronze tables so every model can reference them with `${ref(...)}`, which is what builds the dependency graph — and, later, your lineage. +Two files are worth a look before you brief anyone. workflow_settings.yaml (generated for you from the template by the bootstrap) — `defaultProject` should read `{{ PROJECT_ID }}`, and the git-ignored `.df-credentials.json` next to it tells the CLI to use your logged-in identity (Application Default Credentials) against BigQuery — no keys involved. And sources.js: it declares the six bronze tables so every model can reference them with `${ref(...)}`, which is what builds the dependency graph. ### Brief your co-engineer diff --git a/content/agenticdata/labs/4_governance.md b/content/agenticdata/labs/4_governance.md index 39c5b88..d3fcc72 100644 --- a/content/agenticdata/labs/4_governance.md +++ b/content/agenticdata/labs/4_governance.md @@ -5,49 +5,36 @@ -You have a working medallion — now make it *trustworthy*. In this lab you govern it with **Knowledge Catalog** (formerly Dataplex): label the tiers so everyone can find the right data, measure quality automatically, lock down PII, and give business terms a home. In Lab 5 an AI agent becomes a consumer of exactly this governed layer — governance is what keeps agents grounded. - -This lab is console-first: you built everything with code so far; now see how the platform describes itself. +You have a working medallion — now make it *trustworthy*. Before an AI agent touches your data in Lab 5, you need three answers: **Can I find it? Can I trust it? Who may see it?** You answer each one with **Knowledge Catalog** (formerly Dataplex), and you'll find that the catalog has already done most of the work. ### About Knowledge Catalog -Knowledge Catalog (formerly Dataplex Universal Catalog) is Google Cloud's metadata and governance layer. Every dataset, table and column is an **entry** in the catalog; you enrich entries with **aspects** (structured, typed metadata — like the data tiers you are about to define), attach **business glossary** terms, and let **data profiling** and **auto data quality** scans measure the actual content on a schedule or on demand. Because BigQuery reports **lineage** automatically, the catalog also knows where every table came from. All of this metadata is searchable — by humans and, increasingly, by Gemini and agents. That is the deeper point of this lab: a well-curated catalog is the grounding layer that keeps AI answers trustworthy. +Knowledge Catalog is Google Cloud's metadata and governance layer, and it **fills itself**: every BigQuery dataset, table and column you created in Labs 2–3 is already in it, together with the descriptions from your Dataform code. You add the two things the platform can't guess: **data quality rules**, which measure what's actually in a table, and **policy tags**, which decide who may read a sensitive column. Humans search this metadata, and agents rely on it to understand your data. Learn more: -- [Knowledge Catalog overview](https://docs.cloud.google.com/dataplex/docs/introduction) -- [Aspects and aspect types](https://docs.cloud.google.com/dataplex/docs/enrich-entries-metadata) -- [Auto data quality](https://docs.cloud.google.com/dataplex/docs/auto-data-quality-overview) and [data profiling](https://docs.cloud.google.com/dataplex/docs/data-profiling-overview) +- [Knowledge Catalog overview](https://docs.cloud.google.com/knowledge-catalog/docs/introduction) +- [Auto data quality](https://docs.cloud.google.com/knowledge-catalog/docs/auto-data-quality-overview) - [Column-level security with policy tags](https://docs.cloud.google.com/bigquery/docs/column-level-security-intro) -- [Business glossary](https://docs.cloud.google.com/dataplex/docs/manage-glossaries) -- [Data lineage](https://docs.cloud.google.com/dataplex/docs/about-data-lineage) -### Label the medallion tiers with aspects +### Find your data -Aspects are structured metadata attached to catalog entries. We'll create a **Data tier** aspect type and stamp bronze/silver/gold onto the datasets. +Nobody has registered a single table, so search anyway. Open [Knowledge Catalog search](https://console.cloud.google.com/dataplex/dp-search) and ask in plain English: -1. Open [Knowledge Catalog](https://console.cloud.google.com/dataplex) and go to Metadata types → Aspect types. -2. Click Create (or *Create aspect type*) and use: - - Aspect type ID: `data-tier` - - Display name: `Data tier` - - Location: `global` — this matters: an aspect type can only be attached to entries in the same location or in `global`, and your BigQuery datasets live in the multi-region `us`. A regional aspect type (e.g. `us-central1`) will *not* show up when you try to attach it below. -3. Add a field (under *Template*, click Add Field): - - Type: **Enum** (Type), Name: `tier`, Display name: `Tier` - - Enum values (click Add an Enum Value three times): `bronze`, `silver`, `gold` - - Check Is Required. -4. Click Create. +``` +Which tables contain revenue by day? +``` -Now attach it. Go to Search, search for `cymbal_gold`, and open the dataset entry: +`fct_daily_revenue` comes up (if the results look off, a plain `revenue` finds it too). The catalog picked it up from BigQuery automatically, with no crawler and no registration step. -5. In the entry's details, find `Aspects` → under *Optional aspects* click Add, filter for **Data tier**, set Tier to `gold` → save. -6. Repeat for `cymbal_silver` (`silver`) and `cymbal_bronze` (`bronze`). +Now see what the catalog knows about it. In [BigQuery](https://console.cloud.google.com/bigquery), open `cymbal_gold` → `fct_daily_revenue` → Details. If agy gave the model a `description` in its Dataform config, it shows up here, so documentation written in code lands in the catalog. (No description? An agent would have to guess what this table means. Lab 5 fixes that with agent instructions.) -Verify the point of the exercise: in the catalog search bar, filter by your new aspect (e.g. search for `cymbal` and use the aspect filter for `Data tier = gold`) — anyone in the company can now find the *consumable* data without asking around. +### Hold bronze to a standard -### Measure quality automatically +In Lab 3 your assertions tested what Dataform *built*, at build time. A **data quality scan** checks a table independently of any pipeline, on demand or on a schedule, and publishes a score to the catalog. Point one at bronze, where you know the flaws are. -In Lab 3 your assertions tested what Dataform *built*. Knowledge Catalog's **auto data quality** watches tables *continuously* — no pipeline required. Let's point it at the flaws you know are in bronze. +The rules live in a short spec file: orders_quality.yaml. It has two rules, the same ones your silver assertions enforce: `status` must be one of six known values, and `order_ts` must not lie in the future. -First, allow the Dataplex service agent to use your scan service account: +The scan runs as `dataquality-service-account`, which is pre-provisioned in your project with BigQuery read access. First, allow the Knowledge Catalog service agent to act as it: ```bash gcloud iam service-accounts add-iam-policy-binding dataquality-service-account@{{ PROJECT_ID }}.iam.gserviceaccount.com \ @@ -55,51 +42,42 @@ gcloud iam service-accounts add-iam-policy-binding dataquality-service-account@{ --role=roles/iam.serviceAccountTokenCreator ``` -Now profile the customer data: - -1. In [Knowledge Catalog](https://console.cloud.google.com/dataplex), open Data profiling & quality. -2. Click Create data profile scan: - - Display name: `cymbal-profile-bronze-customers` - - Table: browse to `cymbal_bronze` → `cymbal_customers` - - Scope *Entire data*, sampling *All data*, Publish *results to Knowledge Catalog* - - Credential type: **Service account** → `dataquality-service-account` - - Schedule: *On-demand* -3. Create it, open it, and click Run now. -4. When the job finishes (a few minutes), explore the results — look at the `country` column: there's your planted ~1.5% NULL rate, and the `email` column's distinct count hints at the duplicates. - -Then hold bronze orders to a standard: - -5. Back on the same page, click Create data quality scan: - - Display name: `cymbal-dq-bronze-orders` - - Table: `cymbal_bronze` → `cymbal_orders` - - Scope *Entire data*, sampling *All data*, Publish *results to Knowledge Catalog* - - Credential type: **Service account** → `dataquality-service-account` - - Schedule: *On-demand* -6. Add two rules (rule type *Row check* / validity): - - Dimension Validity: - ``` - status IN ('pending','paid','shipped','delivered','cancelled','returned') - ``` - - Dimension Accuracy: - ``` - order_ts <= CURRENT_TIMESTAMP() - ``` - -7. Run the scan. **It fails — on purpose.** The `shiped` typo and the future-dated orders you saw in Lab 3 are now caught by governance, not just by your pipeline. Discuss with your table: the same rules would pass on `cymbal_silver.stg_orders` — why keep both layers scanned? (If you have time, clone the scan onto silver and prove it passes.) +IAM changes take a moment to propagate, so give the binding about a minute before you create the scan. If the create command below **fails** with a permission error on the service account, that's propagation: wait a minute and run it again. + +Then create the scan: + +```bash +cd ~/bootkon +gcloud dataplex datascans create data-quality bronzeorders \ + --location={{ REGION }} \ + --data-source-resource=//bigquery.googleapis.com/projects/{{ PROJECT_ID }}/datasets/cymbal_bronze/tables/cymbal_orders \ + --data-quality-spec-file=content/agenticdata/src/governance/orders_quality.yaml \ + --service-account=dataquality-service-account@{{ PROJECT_ID }}.iam.gserviceaccount.com +``` + +A word on the flags: your datasets sit in the `us` multi-region, and a scan may live in any region inside it, so `{{ REGION }}` it is. With no `--schedule`, the scan runs on demand. The spec's `catalogPublishingEnabled: true` publishes every result to the catalog and to BigQuery. + +Now run it: + +```bash +gcloud dataplex datascans run bronzeorders --location={{ REGION }} +``` + +The job takes two to three minutes. Then, in [BigQuery](https://console.cloud.google.com/bigquery), open `cymbal_bronze` → `cymbal_orders` and click Data quality (refresh the page if the tab is still empty). + +**The scan fails, on purpose.** A few percent of rows break the `status` rule (the `shiped` typo), and a handful break the `order_ts` rule. Your pipeline already cleans these flaws away, but now the catalog records them too, so anyone who opens this table sees that bronze is raw and shouldn't be consumed directly. ### Lock down PII `stg_customers.email` is personal data. Enforce column-level security with a policy tag: 1. Open [BigQuery policy tags](https://console.cloud.google.com/bigquery/policy-tags) and click Create Taxonomy: - - Taxonomy name: `cymbal-governance`, location `us` - - Policy tag: `PII`, description: `Personal data — restricted` + - Taxonomy name: `cymbal`, location `us` + - Policy tag: `pii`, description: `Personal data — restricted` 2. Create it, then toggle `Enforce access control` on. -3. In [BigQuery](https://console.cloud.google.com/bigquery), open `cymbal_silver` → `stg_customers` → Edit schema, select the `email` column, click *Add policy tag*, and pick `cymbal-governance > PII`. Save. +3. In [BigQuery](https://console.cloud.google.com/bigquery), open `cymbal_silver` → `stg_customers` → Edit schema, select the `email` column, click *Add policy tag*, and pick `cymbal > pii`. Save. -Now prove it works — this query **must fail** with an access-denied error on the tagged column: +Now prove it works. This query **must fail** with an access-denied error on the tagged column: ```sql SELECT email FROM `{{ PROJECT_ID }}.cymbal_silver.stg_customers` LIMIT 5 @@ -111,26 +89,17 @@ And this one works fine: SELECT * EXCEPT (email) FROM `{{ PROJECT_ID }}.cymbal_silver.stg_customers` LIMIT 5 ``` -You are the project owner and *still* can't read that column — fine-grained access is a separate grant (Fine-Grained Reader). That's exactly the guarantee you want before letting AI agents loose on the warehouse. (At scale you wouldn't tag by hand: [Sensitive Data Protection discovery](https://docs.cloud.google.com/sensitive-data-protection/docs/data-profiles) profiles your tables continuously and pushes its findings into the catalog as aspects.) - -### Give the business a vocabulary +Even with admin rights on BigQuery, you can't read that column: reading tagged data needs a separate grant (**Fine-Grained Reader**). That's the guarantee you want before you let AI agents loose on the warehouse. -1. In [Knowledge Catalog](https://console.cloud.google.com/dataplex), open Glossaries and create glossary `Cymbal Business Glossary` (location `us-central1`). -2. Add a term: **Lifetime value** — Description: *"Gross revenue of a customer's non-cancelled orders, in the order currency. Source of truth: cymbal_gold.dim_customer_360.lifetime_value."* -3. Open the `dim_customer_360` entry via catalog Search, go to its schema, select the `lifetime_value` column and attach the term. - -Then try the agentic side of governance — in the catalog Search, ask in natural language: - -``` -Which tables contain revenue by day? -``` +❗ If the first query suddenly works again later, a `dataform run` has rebuilt `stg_customers` and dropped the hand-attached tag. Attach it again (step 3). -Gemini-powered search reads the same metadata you just curated — every aspect, term, and description you add makes both humans *and* agents smarter. +### Challenge: prove silver passes -### Admire the Lineage +**\[TASK\]** Take up to 5 minutes. -One more look: open `cymbal_gold.fct_daily_revenue` in [BigQuery](https://console.cloud.google.com/bigquery) and its Lineage tab. Bronze→silver→gold, captured automatically from the Dataform runs. (Datastream's Postgres→bronze hop publishes its metadata to the catalog in Preview, but doesn't draw lineage edges yet — watch that space.) +1. Clone the scan onto `cymbal_silver.stg_orders`: use the same spec file and the new scan ID `silverorders`. Only the scan ID and `--data-source-resource` change. Run it and compare both scores in the Data quality tab. +2. Discuss with your table: if silver passes, why keep scanning bronze at all? ### Success -🎉 Splendid{% if MY_NAME %}, {{ MY_NAME }}{% endif %}! Your platform now explains itself: tiers are labeled, quality is measured continuously (and honestly — bronze fails, as it should), PII is locked down even against project owners, business terms live next to the data, and lineage draws itself. Governance done — the agents can come. 🛡️ +🎉 Splendid{% if MY_NAME %}, {{ MY_NAME }}{% endif %}! Your platform now explains itself: every table is findable without anyone registering it, bronze carries an honest (failing) quality score, and customer emails are locked down even against admins. Governance done — the agents can come. 🛡️ diff --git a/content/agenticdata/labs/6_a2a.md b/content/agenticdata/labs/6_a2a.md index bb2b663..def2f20 100644 --- a/content/agenticdata/labs/6_a2a.md +++ b/content/agenticdata/labs/6_a2a.md @@ -14,7 +14,7 @@ One conversation, two planes, and the entire platform you built today underneath ### About ADK and the A2A protocol -The **Agent Development Kit (ADK)** is Google's open-source framework for building agents in code: an `Agent` combines a model, an instruction, **tools** (plain Python functions the model may call — their docstrings become the tool documentation the model reasons over) and optionally **sub-agents** it can delegate whole conversations to0. The **A2A protocol** (agent-to-agent, governed by the Linux Foundation) standardizes how agents call *each other* across process, vendor and framework boundaries: every A2A server publishes an **agent card** — a JSON discovery document describing its skills and endpoint — and clients like ADK's `RemoteA2aAgent` read that card to route work to it. That is the point of this finale: once exposed over A2A, your analyst could be consumed by *any* A2A-speaking agent in the company, written in any framework — not just by your concierge. +The **Agent Development Kit (ADK)** is Google's open-source framework for building agents in code: an `Agent` combines a model, an instruction, **tools** (plain Python functions the model may call — their docstrings become the tool documentation the model reasons over) and optionally **sub-agents** it can delegate whole conversations to. The **A2A protocol** (agent-to-agent, governed by the Linux Foundation) standardizes how agents call *each other* across process, vendor and framework boundaries: every A2A server publishes an **agent card** — a JSON discovery document describing its skills and endpoint — and clients like ADK's `RemoteA2aAgent` read that card to route work to it. That is the point of this finale: once exposed over A2A, your analyst could be consumed by *any* A2A-speaking agent in the company, written in any framework — not just by your concierge. Learn more: - [Agent Development Kit](https://adk.dev/) @@ -51,7 +51,7 @@ cd ~/bootkon/content/agenticdata/src/adk && agy And paste the brief: -```bash +``` /goal Read AGENTS.md, then create the files of the cymbal_analyst package (cymbal_analyst/__init__.py, agent.py, a2a_server.py) exactly as it describes. ``` @@ -98,13 +98,37 @@ adk web --port 8000 --allow_origins "*" Open it via **Web Preview** (the Web Preview button in the Cloud Shell toolbar → *Preview on port 8000*), and select **cymbal_concierge**. -Now run the two-plane demo: +Start with the analytical plane. Ask: ``What was our total revenue in the last 30 days?`` + +Watch the event trace in the UI: concierge → *A2A call to cymbal_analyst on :8001* → Conversational Analytics API → SQL on `cymbal_gold`. That's four systems for one question, and four places where things can break. Got a revenue figure? Skip ahead to **Run the two-plane demo**. Got an error instead? Then debug it in the next section. + +### Debug the analyst with agy + +Read the error first. The concierge only knows that its A2A call failed; the full story is in the analyst's terminal, the one running `uvicorn`. Scroll up to the Python traceback. + +Don't fix it by hand: diagnosing errors is where a co-engineer earns its keep. In the analyst's terminal, stop the server with `Ctrl+C`, copy the traceback, and start agy in the project folder: + +```bash +cd ~/bootkon/content/agenticdata/src/adk && agy +``` + +Paste this brief, followed by the traceback: + +``` +/goal The cymbal_analyst A2A server fails when its tool calls the BigQuery data agent. The traceback is below. First explain the root cause to me and how you verified it, then fix it with the smallest possible change. +``` + +Let agy investigate, and watch *how* it narrows the problem down. Review the fix with `/diff`, and make sure you can explain the root cause yourself before you move on. + +Then quit agy with `/quit` and restart the analyst. Back in the ADK UI, start a new session (so the old error doesn't confuse the conversation) and ask the revenue question again. This time you get a figure. Reading the error, delegating the diagnosis, reviewing the fix and verifying the result is the agentic workflow in miniature. + +### Run the two-plane demo + +With the analyst answering, try the other plane, then both at once: -1. **Analytical** — ask: ``What was our total revenue in the last 30 days?`` - Watch the event trace in the UI: concierge → *A2A call to cymbal_analyst on :8001* → Conversational Analytics API → SQL on `cymbal_gold`. Four systems, one question. -2. **Operational** — ask: ``What's the status of order 421337?`` +1. **Operational**: ask ``What's the status of order 421337?`` This time: concierge → `check_order_status` → IAP tunnel → live Postgres. If the simulator deleted it, the concierge says so — it's reading production truth, not a copy. (Grab a fresh existing id from the simulator terminal and try again.) -3. **Both at once** — ask: ``Is order 421337 delivered, and how does its product category perform overall?`` and watch it route each half correctly. +2. **Both at once**: ask ``Is order 421337 delivered, and how does its product category perform overall?`` and watch it route each half correctly. The point to savor: the analyst answers from data that started this afternoon as Postgres rows, flowed through Datastream into bronze, was cleaned by agy's Dataform code into gold, got governed in Knowledge Catalog, and is served by a published data agent — now consumed by another agent, over an open protocol, on behalf of a user. **That is agentic data.** diff --git a/content/agenticdata/provisioning/bk-init b/content/agenticdata/provisioning/bk-init index b46285e..a581a41 100755 --- a/content/agenticdata/provisioning/bk-init +++ b/content/agenticdata/provisioning/bk-init @@ -78,6 +78,6 @@ fi "$SET_VAR" --comment "Route google-genai through Vertex AI" "$LOCAL" GOOGLE_GENAI_USE_VERTEXAI TRUE "$SET_VAR" --comment "Location for the GenAI/Vertex client" "$LOCAL" GOOGLE_CLOUD_LOCATION global "$SET_VAR" --comment "Gemini model id used by the Cymbal agents" "$LOCAL" BK_CYMBAL_MODEL gemini-2.5-flash -"$SET_VAR" --comment "Conversational Analytics data agent id" "$LOCAL" BK_DATA_AGENT_ID cymbal-data-agent +"$SET_VAR" --if-empty --comment "Conversational Analytics data agent id" "$LOCAL" BK_DATA_AGENT_ID cymbal-data-agent "$SET_VAR" --comment "Cymbal DB host (reached via the bk-tunnel)" "$LOCAL" BK_CYMBAL_DB_HOST localhost "$SET_VAR" --comment "Cymbal DB port (reached via the bk-tunnel)" "$LOCAL" BK_CYMBAL_DB_PORT 5432 diff --git a/content/agenticdata/src/governance/orders_quality.yaml b/content/agenticdata/src/governance/orders_quality.yaml new file mode 100644 index 0000000..5b8b14e --- /dev/null +++ b/content/agenticdata/src/governance/orders_quality.yaml @@ -0,0 +1,14 @@ +# Data quality rules for Cymbal orders (Lab 4). The same checks as the +# rowConditions in stg_orders.sqlx: bronze fails them (the planted 'shiped' +# typo and the future-dated orders), silver passes. +rules: + - column: status + dimension: VALIDITY + setExpectation: + values: [pending, paid, shipped, delivered, cancelled, returned] + - column: order_ts + dimension: ACCURACY + rowConditionExpectation: + sqlExpression: order_ts <= CURRENT_TIMESTAMP() +samplingPercent: 100 +catalogPublishingEnabled: true diff --git a/content/agenticdata/src/prompts.md b/content/agenticdata/src/prompts.md index f33fbbd..568723c 100644 --- a/content/agenticdata/src/prompts.md +++ b/content/agenticdata/src/prompts.md @@ -63,3 +63,13 @@ describes. ``` Fallback: `git restore content/agenticdata/src/adk/cymbal_analyst/` + +## Lab 6 — debug the analyst + +Run inside `content/agenticdata/src/adk` after stopping the A2A server with +`Ctrl+C`; participants paste the traceback from the analyst's terminal right +after the brief: + +``` +/goal The cymbal_analyst A2A server fails when its tool calls the BigQuery data agent. The traceback is below. First explain the root cause to me and how you verified it, then fix it with the smallest possible change. +```