diff --git a/README.md b/README.md index b813a6f7..de544986 100644 --- a/README.md +++ b/README.md @@ -121,9 +121,9 @@ Documentation is available at https://kiro.dev/docs/powers/ --- ### databricks -**Databricks AI Dev Kit** - Comprehensive Databricks development toolkit with expert guidance for building data pipelines, ML workflows, dashboards, jobs, and applications on the Databricks platform, plus an optional MCP server adding 44 live tools (180+ operations). +**Databricks** - Official Databricks agent skills for building data pipelines, jobs, dashboards, apps, ML and AI agents on the Databricks Platform, installed and updated with the Databricks CLI (`databricks aitools install --agents kiro`). -**MCP Servers:** databricks (optional, ai-dev-kit local MCP server) +**MCP Servers:** None (Knowledge Base Power) --- diff --git a/databricks/POWER.md b/databricks/POWER.md deleted file mode 100644 index 0299f9df..00000000 --- a/databricks/POWER.md +++ /dev/null @@ -1,1610 +0,0 @@ ---- -name: "databricks" -displayName: "Databricks AI Dev Kit" -description: "Comprehensive Databricks development toolkit: expert guidance for building data pipelines, ML workflows, dashboards, jobs and applications on the Databricks platform, plus an optional MCP server adding 44 live tools (180+ operations)." -keywords: ["databricks", "spark", "delta", "mlflow", "unity catalog", "pipelines", "jobs", "sql", "data engineering", "machine learning"] -author: "Databricks" ---- - -# Databricks AI Dev Kit Power - -## Overview - -The Databricks AI Dev Kit Power provides comprehensive access to the Databricks Lakehouse platform through expert steering files for every Databricks capability — pipelines, jobs, dashboards, ML, governance, and apps — plus an optional MCP server adding 44 live tools (180+ operations). - -The steering files are the default and work on their own. The MCP server is optional and -is installed separately (Step 1b); upstream marks it deprecated. Until it is installed and -enabled, the tools documented under [Available MCP Servers](#available-mcp-servers) are -not present, and the Power ships with its `mcp.json` entry `disabled: true`. - -**Key capabilities:** -- **SQL & Compute**: Execute SQL on warehouses, run Python/Scala on clusters, manage compute lifecycle -- **Pipelines & Jobs**: Build Spark Declarative Pipelines (streaming tables, CDC, SCD Type 2, Auto Loader) and orchestrate multi-task DAGs -- **Unity Catalog**: Manage tables, volumes, grants, tags, storage credentials, system tables, metric views, and External Iceberg Reads (cross-engine interop via UC IRC) -- **AI/BI Dashboards**: Create interactive visualizations, KPIs, and analytics dashboards -- **Genie Spaces**: Natural-language data exploration over governed datasets -- **Agent Bricks**: Build Knowledge Assistants (RAG) and Multi-Agent Supervisors -- **Vector Search**: Semantic search and RAG with managed indexes -- **Model Serving**: Deploy ML models, AI agents, and pay-per-token Foundation Model APIs (FMAPI) to scalable endpoints; route through Unity AI Gateway for guardrails, fallbacks, and rate limiting -- **MLflow**: Track experiments, evaluate models, instrument tracing, query metrics -- **Lakebase**: Provisioned and autoscale managed PostgreSQL for OLTP workloads -- **Databricks Apps**: Full-stack web applications on the Lakehouse -- **Asset Bundles**: Infrastructure-as-code for Databricks resources - -**Authentication**: Requires a Databricks workspace and either an OAuth-authenticated CLI profile, an existing `~/.databrickscfg` profile, or a Personal Access Token (PAT). - -## Agent First-Action Protocol (CRITICAL) - -> **⚠️ MANDATORY pre-flight gate — read before any tool call:** -> -> On the **first turn** of any new conversation in this Power — and **before invoking any Databricks MCP tool** — the agent MUST run the credential-detection flow defined in [Step 3: Configure Authentication](#step-3-configure-authentication). Do not enumerate tools, call `get_current_user`, or attempt any MCP operation until detection has completed and the user has confirmed which credentials to use. -> -> **In-session auth-failure recovery.** If at any point during the session a Databricks MCP tool returns `Invalid access token`, `401 Unauthorized`, `403 token expired`, or any equivalent auth-failure status: -> -> 1. **Stop** further tool calls immediately. Do not retry the failed call. -> 2. **Re-enter** the credential-detection flow — surface the current credential summary to the user and ask which option (A / B / C / D) they want to repair or switch to. -> 3. **Recommend OAuth U2M (Option A)** as the first fix for interactive use. It auto-refreshes hourly and avoids the long-lived-secret expiry that typically causes these errors. -> -> **No-credentials default.** When no Databricks credentials are detected on first run, the agent's default recommendation is **Option A — OAuth U2M via the Databricks CLI**, since it's the safest interactive flow and covers the most common use case (a single human developer on a workstation). -> -> **Never report the MCP server as working without checking that it can start.** A -> populated `mcp.json` entry proves only that the file was written, not that the server -> runs. Reading the config and inferring "configured, therefore working" is wrong, and it -> leads the user to enable an entry that cannot spawn. Before describing the server as -> available, confirm the interpreter exists: -> -> ```bash -> ls "$HOME/.ai-dev-kit/.venv/bin/python" 2>/dev/null || echo "MCP_SERVER_NOT_INSTALLED" -> ``` -> -> If that prints `MCP_SERVER_NOT_INSTALLED`, say so plainly and point the user at -> [Step 1b](#step-1b-install-the-mcp-server-optional). Do not offer to pick a profile and -> flip `disabled` to `false` — the failure is at `spawn`, before any profile is read, so -> enabling it only adds an `ENOENT` to the log. The skills remain fully usable meanwhile. - -## Available Steering Files - -This Power's steering files are downloaded from the [ai-dev-kit repository](https://github.com/databricks-solutions/ai-dev-kit/tree/main/databricks-skills) during onboarding and copied into the Power's `steering/` directory. Skills load on-demand based on your task. - -### Skill Catalog (34+ skills) - -| Skill | Domain | Description | -|-------|--------|-------------| -| databricks-spark-declarative-pipelines | Data Engineering | Streaming tables, CDC, SCD Type 2, Auto Loader | -| databricks-spark-structured-streaming | Data Engineering | Kafka, stateful ops, stream joins | -| databricks-jobs | Data Engineering | Scheduled workflows, multi-task DAGs, serverless | -| databricks-zerobus-ingest | Data Engineering | Zero-copy ingestion | -| databricks-execution-compute | Data Engineering | Code execution + cluster / warehouse lifecycle | -| databricks-iceberg | Data Engineering | Iceberg tables, UniForm, interop | -| databricks-aibi-dashboards | Analytics | Interactive visualizations, KPIs, analytics | -| databricks-dbsql | Analytics | SQL best practices, materialized views | -| databricks-genie | Analytics | Natural language data exploration | -| databricks-metric-views | Analytics | Reusable business metrics | -| databricks-unity-catalog | Governance | Tables, volumes, governance, system tables | -| databricks-agent-bricks | AI/ML | Knowledge Assistants, Supervisor Agents | -| databricks-vector-search | AI/ML | Semantic search and RAG applications | -| databricks-model-serving | AI/ML | Deploy ML models and AI agents | -| databricks-mlflow-evaluation | AI/ML | Model evaluation, scoring, traces | -| databricks-ai-functions | AI/ML | AI functions, forecasting, document processing | -| databricks-synthetic-data-gen | AI/ML | Synthetic data generation | -| databricks-apps-python | App Development | Full-stack web applications | -| databricks-app-apx | App Development | App frontend/backend patterns | -| databricks-lakebase-provisioned | App Development | Managed PostgreSQL (provisioned) | -| databricks-lakebase-autoscale | App Development | Managed PostgreSQL (autoscale) | -| databricks-bundles | Platform | Databricks Asset Bundles | -| databricks-config | Platform | Workspace configuration | -| databricks-python-sdk | Platform | Python SDK patterns and examples | -| databricks-unstructured-pdf-generation | Platform | PDF generation and upload | -| databricks-docs | Platform | Documentation search | -| mlflow-onboarding | MLflow | MLflow getting started | -| instrumenting-with-mlflow-tracing | MLflow | MLflow tracing instrumentation | -| querying-mlflow-metrics | MLflow | Querying MLflow metrics | -| retrieving-mlflow-traces | MLflow | Retrieving MLflow traces | -| searching-mlflow-docs | MLflow | Searching MLflow documentation | -| analyze-mlflow-trace | MLflow | Analyzing MLflow traces | -| analyze-mlflow-chat-session | MLflow | Analyzing MLflow chat sessions | -| agent-evaluation | MLflow | Agent evaluation patterns | -| spark-python-data-source | Data Engineering | Custom Spark data sources | - -### Skill Loading - -This Power installs **all 34+ skills** (`--skills-profile all`). Skills are markdown files that load on demand based on the task at hand — having the full set present incurs no runtime cost, and it avoids re-running the installer when work crosses domains (e.g., an ML engineer who occasionally builds a dashboard). Persona-specific subsets are not used. - -## Available MCP Servers - -### databricks -**Package:** `databricks-solutions/ai-dev-kit` (`databricks-mcp-server`) -**Connection:** Local Python process (`uv` venv) configured via `~/.kiro/settings/mcp.json` - -44 tools spanning 180+ operations. The most frequently used are itemized below; see the full category index further down. - -#### SQL & Compute - -1. **execute_sql** — Run a SQL statement on a SQL warehouse - - Required: `statement` (string), `warehouse_id` (string) - - Optional: `catalog`, `schema`, `parameters`, `wait_timeout` - - Returns: Result rows, schema, statement state - -2. **execute_sql_multi** — Run multiple SQL statements in sequence - - Required: `statements` (string[]), `warehouse_id` (string) - - Returns: Results per statement - -3. **execute_code** — Execute Python/Scala/R/SQL on an interactive cluster - - Required: `cluster_id` (string), `language` (string), `code` (string) - - Returns: Execution output - -4. **manage_cluster** — Create / get / list / start / restart / terminate / edit clusters - - Required: `action` (string) - - Optional: cluster spec fields per action - - Returns: Cluster object or list - -5. **manage_sql_warehouse** — Create / get / list / start / stop / edit / delete SQL warehouses - - Required: `action` (string) - - Optional: warehouse spec fields per action - - Returns: Warehouse object or list - -6. **manage_warehouse** — Companion warehouse helper for SQL workflows - - Required: `action` (string) - - Returns: Warehouse state - -7. **list_compute** — List available compute resources (clusters + warehouses) - - Optional: `filter_type` - - Returns: Compute summary - -8. **get_table_stats_and_schema** — Inspect table schema, row counts, partitioning - - Required: `table_name` (fully-qualified, e.g. `catalog.schema.table`) - - Returns: Schema, statistics, partition info - -#### Jobs & Pipelines - -9. **manage_jobs** — Create / get / list / update / delete jobs - - Required: `action` (string) - - Optional: job spec (`tasks`, `schedule`, `parameters`, `email_notifications`) - - Returns: Job object or list - -10. **manage_job_runs** — Run / wait / cancel / get-output for job runs - - Required: `action` (string), `job_id` or `run_id` - - Returns: Run state, output, logs - -11. **manage_pipeline** — Create / get / list / update / delete Spark Declarative Pipelines (DLT) - - Required: `action` (string) - - Optional: pipeline spec, libraries, target schema - - Returns: Pipeline object - -12. **manage_pipeline_run** — Start / stop / get-update for pipeline runs - - Required: `action` (string), `pipeline_id` - - Returns: Run/update state - -#### Unity Catalog - -13. **manage_uc_objects** — Create / get / list / update / delete catalogs, schemas, tables, volumes, functions - - Required: `action` (string), `object_type` (string) - - Optional: object spec - - Returns: UC object or list - -14. **manage_uc_grants** — Grant / revoke / list privileges on UC objects - - Required: `action` (string), `securable_type`, `full_name` - - Returns: Permission assignments - -15. **manage_uc_storage** — Manage external locations and storage credentials - - Required: `action` (string) - - Returns: Storage credential / external location - -16. **manage_uc_tags** — Apply / remove tags on UC objects - - Required: `action` (string), `object`, `tags` - - Returns: Tag assignments - -17. **manage_uc_connections** — Manage federated catalog connections - - Required: `action` (string) - - Returns: Connection object - -18. **manage_uc_sharing** — Manage Delta Sharing shares and recipients - - Required: `action` (string) - - Returns: Share / recipient - -19. **manage_uc_monitors** — Manage Lakehouse Monitoring profiles - - Required: `action` (string), `table_name` - - Returns: Monitor state - -20. **manage_uc_security_policies** — Manage row-level / column-level security policies - - Required: `action` (string) - - Returns: Policy object - -21. **manage_metric_views** — Create / list / update reusable business metrics - - Required: `action` (string) - - Returns: Metric view definition - -#### Dashboards, Genie & Agent Bricks - -22. **manage_dashboard** — Create / get / list / delete / publish AI/BI dashboards - - Required: `action` (string) - - Optional: dashboard JSON, draft/publish flags - - Returns: Dashboard object - -23. **manage_genie** — Create / list / configure Genie Spaces - - Required: `action` (string) - - Returns: Genie Space object - -24. **ask_genie** — Ask a question against a Genie Space - - Required: `space_id`, `question` - - Returns: Generated SQL, result rows, explanation - -25. **manage_ka** — Create / list / update Knowledge Assistants (RAG agents) - - Required: `action` (string) - - Returns: Knowledge Assistant object - -26. **manage_mas** — Create / list / update Multi-Agent Supervisors - - Required: `action` (string) - - Returns: Multi-Agent Supervisor object - -#### Model Serving & Vector Search - -27. **manage_serving_endpoint** — Get / list / query serving endpoints - - Required: `action` (string) - - Optional: `endpoint_name`, `inputs` (for query) - - Returns: Endpoint object or model output - -28. **manage_vs_endpoint** — Create / list / delete vector search endpoints - - Required: `action` (string) - - Returns: VS endpoint object - -29. **manage_vs_index** — Create / list / sync / delete vector search indexes - - Required: `action` (string) - - Returns: VS index object - -30. **manage_vs_data** — Upsert / delete documents in a direct-access VS index - - Required: `action` (string), `index_name`, `data` - - Returns: Operation status - -31. **query_vs_index** — Similarity search against a VS index - - Required: `index_name`, `query_text` or `query_vector` - - Optional: `num_results`, `filters` - - Returns: Top-k matched documents - -#### Lakebase, Apps & Files - -32. **manage_lakebase_database** — Create / get / list / delete Lakebase databases - - Required: `action` (string) - - Returns: Lakebase database object - -33. **manage_lakebase_branch** — Manage Lakebase branches - - Required: `action` (string) - - Returns: Branch state - -34. **manage_lakebase_sync** — Configure Lakebase ↔ UC sync - - Required: `action` (string) - - Returns: Sync configuration - -35. **generate_lakebase_credential** — Issue short-lived Lakebase credentials - - Required: `database_name` - - Returns: Credential - -36. **manage_app** — Create / get / list / delete / deploy Databricks Apps - - Required: `action` (string) - - Returns: App object - -37. **manage_workspace_files** — Read / write / list / delete workspace files - - Required: `action` (string), `path` - - Returns: File contents or status - -38. **manage_volume_files** — Read / write / list / delete volume files - - Required: `action` (string), `path` - - Returns: File contents or status - -39. **get_volume_folder_details** — Inspect a UC volume folder - - Required: `volume_path` - - Returns: Folder listing with metadata - -#### Workspace, Tracking & Utilities - -40. **manage_workspace** — Workspace-level operations (notebooks, folders, permissions) - - Required: `action` (string) - - Returns: Workspace object - -41. **get_current_user** — Identify the authenticated principal - - No parameters required - - Returns: User profile, workspace, entitlements - -42. **generate_and_upload_pdf** — Render HTML to PDF and upload to a volume - - Required: `html`, `volume_path` - - Returns: Uploaded file path - -43. **list_tracked_resources** — List resources created/tracked by this MCP session - - Optional: `resource_type` - - Returns: Tracked resources - -44. **delete_tracked_resource** — Delete a previously tracked resource - - Required: `resource_type`, `resource_id` - - Returns: Deletion status - -### Full Tool Index - -| Category | Tool count | Operations | -|----------|-----------|------------| -| SQL & Compute | 8 | execute_sql, execute_sql_multi, execute_code, manage_cluster, manage_sql_warehouse, manage_warehouse, list_compute, get_table_stats_and_schema | -| Jobs & Pipelines | 4 | manage_jobs, manage_job_runs, manage_pipeline, manage_pipeline_run | -| Unity Catalog | 9 | manage_uc_objects, manage_uc_grants, manage_uc_storage, manage_uc_tags, manage_uc_connections, manage_uc_sharing, manage_uc_monitors, manage_uc_security_policies, manage_metric_views | -| Dashboards / Genie / Agents | 5 | manage_dashboard, manage_genie, ask_genie, manage_ka, manage_mas | -| Model Serving / Vector Search | 5 | manage_serving_endpoint, manage_vs_endpoint, manage_vs_index, manage_vs_data, query_vs_index | -| Lakebase / Apps / Files | 8 | manage_lakebase_database, manage_lakebase_branch, manage_lakebase_sync, generate_lakebase_credential, manage_app, manage_workspace_files, manage_volume_files, get_volume_folder_details | -| Workspace / Utilities | 5 | manage_workspace, get_current_user, generate_and_upload_pdf, list_tracked_resources, delete_tracked_resource | - -## Tool Usage Examples - -### Executing SQL - -**Run a query on a warehouse:** -```javascript -usePower("databricks", "databricks", "execute_sql", { - "statement": "SELECT current_catalog(), current_schema()", - "warehouse_id": "abc123def456" -}) -// Returns: Single-row result with current catalog and schema -``` - -**Inspect a table:** -```javascript -usePower("databricks", "databricks", "get_table_stats_and_schema", { - "table_name": "main.sales.orders" -}) -// Returns: Columns, types, row count, partition info -``` - -### Managing Jobs - -**Create a job:** -```javascript -usePower("databricks", "databricks", "manage_jobs", { - "action": "create", - "name": "nightly_orders_etl", - "tasks": [{ - "task_key": "transform", - "notebook_task": { "notebook_path": "/Workspace/etl/transform_orders" }, - "job_cluster_key": "main" - }], - "schedule": { "quartz_cron_expression": "0 0 2 * * ?", "timezone_id": "UTC" } -}) -// Returns: job_id of the created job -``` - -**Run and wait:** -```javascript -usePower("databricks", "databricks", "manage_job_runs", { - "action": "run_now_and_wait", - "job_id": 123456789 -}) -// Returns: Run state and task outputs -``` - -### Spark Declarative Pipelines - -**Create a streaming pipeline:** -```javascript -usePower("databricks", "databricks", "manage_pipeline", { - "action": "create", - "name": "bronze_to_silver_orders", - "target": "main.silver", - "libraries": [{ "notebook": { "path": "/Workspace/dlt/orders_silver" } }], - "continuous": true -}) -// Returns: pipeline_id -``` - -### Unity Catalog - -**Create a managed table:** -```javascript -usePower("databricks", "databricks", "manage_uc_objects", { - "action": "create", - "object_type": "table", - "full_name": "main.sandbox.customer", - "columns": [ - { "name": "id", "type_text": "BIGINT" }, - { "name": "email", "type_text": "STRING" } - ] -}) -// Returns: Table object with metadata -``` - -**Grant SELECT to a group:** -```javascript -usePower("databricks", "databricks", "manage_uc_grants", { - "action": "grant", - "securable_type": "table", - "full_name": "main.sales.orders", - "principal": "analysts", - "privileges": ["SELECT"] -}) -// Returns: Updated permission assignments -``` - -### Genie & Knowledge Assistants - -**Ask Genie a natural-language question:** -```javascript -usePower("databricks", "databricks", "ask_genie", { - "space_id": "01ef...", - "question": "What were the top 5 products by revenue last month?" -}) -// Returns: Generated SQL, result rows, narrative explanation -``` - -### Vector Search - -**Similarity search over a managed index:** -```javascript -usePower("databricks", "databricks", "query_vs_index", { - "index_name": "main.search.docs_idx", - "query_text": "How do I configure Auto Loader for Parquet?", - "num_results": 5 -}) -// Returns: Top-5 matched documents with scores -``` - -### Model Serving - -**Query an endpoint:** -```javascript -usePower("databricks", "databricks", "manage_serving_endpoint", { - "action": "query", - "endpoint_name": "claims-classifier", - "inputs": { "dataframe_records": [{ "claim_text": "..." }] } -}) -// Returns: Model predictions -``` - -## Combining Tools (Workflows) - -### Workflow 1: Build a Streaming ETL with Job Orchestration - -```javascript -// Step 1: Create the bronze→silver pipeline -const pipeline = usePower("databricks", "databricks", "manage_pipeline", { - "action": "create", - "name": "orders_silver", - "target": "main.silver", - "libraries": [{ "notebook": { "path": "/Workspace/dlt/orders_silver" } }] -}) - -// Step 2: Wrap the pipeline in a scheduled job -const job = usePower("databricks", "databricks", "manage_jobs", { - "action": "create", - "name": "orders_silver_orchestrator", - "tasks": [{ - "task_key": "run_dlt", - "pipeline_task": { "pipeline_id": pipeline.pipeline_id } - }], - "schedule": { "quartz_cron_expression": "0 0 * * * ?", "timezone_id": "UTC" } -}) - -// Step 3: Trigger an initial run and wait -const run = usePower("databricks", "databricks", "manage_job_runs", { - "action": "run_now_and_wait", - "job_id": job.job_id -}) - -// Step 4: Verify rows landed in silver -const stats = usePower("databricks", "databricks", "get_table_stats_and_schema", { - "table_name": "main.silver.orders" -}) -``` - -### Workflow 2: Govern a New Table and Publish a Dashboard - -```javascript -// Step 1: Create the table -usePower("databricks", "databricks", "manage_uc_objects", { - "action": "create", - "object_type": "table", - "full_name": "main.analytics.daily_revenue", - "columns": [ - { "name": "day", "type_text": "DATE" }, - { "name": "revenue", "type_text": "DECIMAL(18,2)" } - ] -}) - -// Step 2: Grant access -usePower("databricks", "databricks", "manage_uc_grants", { - "action": "grant", - "securable_type": "table", - "full_name": "main.analytics.daily_revenue", - "principal": "analysts", - "privileges": ["SELECT"] -}) - -// Step 3: Tag for discovery -usePower("databricks", "databricks", "manage_uc_tags", { - "action": "set", - "object": { "type": "table", "full_name": "main.analytics.daily_revenue" }, - "tags": { "domain": "finance", "tier": "gold" } -}) - -// Step 4: Publish a dashboard -usePower("databricks", "databricks", "manage_dashboard", { - "action": "create", - "name": "Daily Revenue", - "warehouse_id": "abc123def456", - "dataset_query": "SELECT * FROM main.analytics.daily_revenue ORDER BY day DESC" -}) -``` - -### Workflow 3: Build a RAG Knowledge Assistant - -```javascript -// Step 1: Create a vector search endpoint -usePower("databricks", "databricks", "manage_vs_endpoint", { - "action": "create", - "name": "kb-endpoint" -}) - -// Step 2: Create an index over governed docs -usePower("databricks", "databricks", "manage_vs_index", { - "action": "create", - "endpoint_name": "kb-endpoint", - "index_name": "main.kb.docs_idx", - "source_table": "main.kb.docs", - "primary_key": "doc_id", - "embedding_source_column": "content" -}) - -// Step 3: Wire up the Knowledge Assistant -const ka = usePower("databricks", "databricks", "manage_ka", { - "action": "create", - "name": "internal-kb-assistant", - "vector_index": "main.kb.docs_idx" -}) - -// Step 4: Query Genie or the assistant directly -usePower("databricks", "databricks", "query_vs_index", { - "index_name": "main.kb.docs_idx", - "query_text": "What is our incident escalation policy?", - "num_results": 5 -}) -``` - -## SQL & DBSQL Syntax Guide - -The MCP server runs SQL through Databricks SQL warehouses (Photon-accelerated ANSI SQL with Delta and Unity Catalog extensions). - -### Core Patterns - -**Three-part naming (always use it):** -```sql -SELECT * FROM .. -``` - -**Time travel:** -```sql -SELECT * FROM main.sales.orders VERSION AS OF 42; -SELECT * FROM main.sales.orders TIMESTAMP AS OF '2026-05-01'; -``` - -**MERGE for upserts:** -```sql -MERGE INTO target t USING source s ON t.id = s.id -WHEN MATCHED THEN UPDATE SET * -WHEN NOT MATCHED THEN INSERT *; -``` - -**Materialized views (Delta Live / DBSQL):** -```sql -CREATE MATERIALIZED VIEW main.gold.orders_daily AS -SELECT order_date, COUNT(*) AS n FROM main.silver.orders GROUP BY order_date; -``` - -**Streaming tables (DLT):** -```sql -CREATE OR REFRESH STREAMING TABLE bronze_orders -AS SELECT * FROM cloud_files('/Volumes/main/raw/orders', 'json'); -``` - -### Critical Rules - -1. **Always fully qualify** with `catalog.schema.table` — UC enforces three-part naming -2. **Use parameter markers** for user input: `:param_name` (passed in `parameters` arg) -3. **Set a `wait_timeout`** for long queries (e.g., `"30s"`); poll with subsequent calls if needed -4. **Prefer warehouses over clusters** for SQL — cheaper, faster cold start, Photon-enabled -5. **Use `OPTIMIZE` and `VACUUM`** on Delta tables for performance/storage hygiene -6. **Check warehouse state** before executing — start with `list_compute` if unsure - -## Best Practices - -### ✅ Do - -- **Start with `get_current_user`** to confirm workspace, profile, and entitlements before doing real work -- **Use SQL warehouses for SQL** — they're cheaper, auto-suspend, and Photon-accelerated -- **Use Unity Catalog three-part names** everywhere (`catalog.schema.object`) -- **Tag and grant on creation** — wire up `manage_uc_tags` and `manage_uc_grants` immediately after `manage_uc_objects` -- **Track resources** — use `list_tracked_resources` to audit what this session created; clean up with `delete_tracked_resource` -- **Pin pipelines and jobs** to specific cluster policies for cost control -- **Validate schemas** with `get_table_stats_and_schema` before writing transformations -- **Prefer DLT (manage_pipeline)** over hand-rolled jobs for streaming and CDC -- **Wait on runs** explicitly (`run_now_and_wait`) when downstream steps depend on output -- **Load skill files on demand** — don't pre-load all 33 skills; the on-demand pattern is faster - -### ❌ Don't - -- **Don't run SQL on interactive clusters** when a warehouse will do — cost spikes and cold-start delays -- **Don't omit catalog/schema** — relying on session defaults breaks portability -- **Don't hardcode warehouse IDs or job IDs** — fetch them with `list_compute` and `manage_jobs` (action: list) -- **Don't grant `ALL_PRIVILEGES`** when `SELECT` or `MODIFY` is enough -- **Don't create resources without tagging** — UC tags are the only durable discovery mechanism -- **Don't skip `wait_timeout`** on `execute_sql` — the default may not suit long-running statements -- **Don't disable workspace safeguards** (deletion protection, cluster policies) without explicit need -- **Don't store PATs in source code** — use `~/.databrickscfg` profiles or environment variables - -## Troubleshooting - -### Error: "The MCP server is disabled" -**Cause:** `mcp.json` has `"disabled": true` (default state ships disabled for safety). -**Solution:** -1. Complete the onboarding steps (see Configuration) -2. Set `"disabled": false` in `~/.kiro/settings/mcp.json` under the Power's `mcpServers.databricks` entry -3. Reload Kiro to reconnect the server - -### Error: "Authentication failed" / "401 Unauthorized" -**Cause:** Profile in `~/.databrickscfg` is missing, expired, or doesn't match `DATABRICKS_CONFIG_PROFILE`. -**Solution:** -1. Run `databricks auth login --profile ` to refresh OAuth -2. Or update `mcp.json` to use explicit `DATABRICKS_HOST` + `DATABRICKS_TOKEN` env vars -3. Verify with `get_current_user` — it should return a valid user - -### Error: "Warehouse not running" / "Cluster not running" -**Cause:** Compute is stopped; `execute_sql` / `execute_code` requires a running compute. -**Solution:** -1. Use `manage_sql_warehouse` (action: start) or `manage_cluster` (action: start) -2. Or rely on serverless warehouses — they auto-start on first query -3. Re-run the original tool call - -### Error: "Permission denied on UC object" -**Cause:** The authenticated user lacks the required UC privilege. -**Solution:** -1. Use `manage_uc_grants` (action: list) to inspect current grants -2. Have a metastore admin grant the missing privilege -3. Confirm `USE CATALOG` and `USE SCHEMA` are also granted (often forgotten) - -### Error: "Pipeline failed with EXPECTATION_VIOLATED" -**Cause:** A DLT expectation rejected rows. -**Solution:** -1. Inspect pipeline events with `manage_pipeline_run` (action: get_update) -2. Loosen `EXPECT` constraints or move to `EXPECT ... ON VIOLATION DROP ROW` if the data is genuinely dirty -3. Check upstream sources for schema drift - -### Error: "Installer ran but `~/.ai-dev-kit/.venv/bin/python` is missing" -**Cause:** In almost every case this is not a failed install. `install.sh` is now -skills-only and never builds that venv — only `databricks-mcp-server/mcp_install.sh` -does. If you ran Step 1 and skipped Step 1b, the venv is absent because nothing was -asked to create it. - -**Solution:** Run Step 1b. Note two details that make this fail silently otherwise: -1. `mcp_install.sh` cannot be piped from `curl`. It resolves its own location from - `BASH_SOURCE[0]`, so under `bash <(curl …)` it computes `PARENT_DIR=/dev` and exits - with `MCP setup script not found at /dev/fd/setup.sh`. Run it from a clone. -2. Its default venv location is `/.venv`, which is **not** the path this Power's - `mcp.json` references. Pass `--venv-dir "$HOME/.ai-dev-kit/.venv"` explicitly. - -Only if Step 1b itself fails is a prerequisite likely at fault — check that `uv` and -`git` are on `PATH` and that `$HOME` has disk space. - -### Warning: "SKILL.md not for skill: …databricks-app-apx/SKILL.md" -**Cause:** The upstream `databricks-solutions/apx` repository ships `SKILL.md` with a frontmatter `name:` field (`apx`) that doesn't match the directory name (`databricks-app-apx`) the installer creates. Kiro flags the mismatch but loads the skill anyway. -**Impact:** None — the skill is functional, just registered under a different internal name. The warning is cosmetic. -**Solution:** No action needed. Patching the local file isn't durable (the next skill update overwrites it). The proper fix is upstream: file an issue at https://github.com/databricks-solutions/apx requesting the frontmatter `name:` field be aligned with the distributed directory name. - -## Configuration - -### Prerequisites - -- **uv** — Python package manager ([install](https://docs.astral.sh/uv/getting-started/installation/)) -- **git** — Version control -- **Databricks workspace** — AWS, Azure, or GCP -- **Databricks CLI v1.0.0+** — [install](https://docs.databricks.com/en/dev-tools/cli/install.html). No longer optional: `install.sh` delegates skill installation to `databricks aitools`, so the installer fails without it. -- **jq** (recommended) — Steps 2.5 / 2.6 / 2.7 below use `jq` for idempotent `mcp.json` edits. Each step also includes a non-`jq` fallback, but installing `jq` (`brew install jq` on macOS, `apt install jq` on Linux) makes the onboarding flow fully scriptable. - -### Install at a glance - -The flow has three install-bug workarounds (Steps 2.5 / 2.6 / 2.7) that look optional but aren't — every fresh Kiro install of this Power hits at least one of them today. Skim this checklist before starting so the ordering is clear: - -| Step | What it does | Why it's there | -|------|--------------|----------------| -| 1 | Snapshot `~/.kiro/skills`, then run the official installer | Snapshot lets Step 2 know exactly which skills the installer added | -| 1b | *(optional)* Install the MCP server via its own installer | `install.sh` is skills-only now; nothing else creates `~/.ai-dev-kit/.venv` | -| 2 | Diff the post-install state, write a `.skill-manifest.txt`, move skills into the Power's `steering/` directory | Manifest-driven move beats glob-based cleanup — safe with upstream additions and other tools | -| 2.5 | Add the top-level `mcpServers` key to `~/.kiro/settings/mcp.json` if missing | Kiro's schema requires it; fresh installs error out without this | -| 2.6 | Remove the standalone `mcpServers.databricks` entry the installer writes | Otherwise Kiro launches two copies of the same MCP server | -| 2.7 | Expand `~` to `$HOME` in the Power's `command` and `args` paths | Node's `spawn()` doesn't expand `~` — server fails with `ENOENT` if you skip this | -| 3 | Configure authentication (Options A → D; Option A = OAuth U2M recommended for interactive) | Power ships `disabled: true` until you pick a method | -| 4 | Smoke-test the connection with `get_current_user` | Verifies auth + env-var resolution + server enablement in one call | - -### Step 1: Run the Official Installer (with pre-snapshot) - -The installer pulls skills from **four upstream sources** (`databricks-solutions/ai-dev-kit/databricks-skills/`, `mlflow/skills/`, `databricks-solutions/apx/`, `databricks/databricks-agent-skills/`) and writes them into `~/.kiro/skills/`. To track exactly which skills this Power is responsible for — so we can move them precisely in Step 2 and clean them up precisely on uninstall — snapshot the directory **before** the installer runs. - -```bash -# Snapshot the pre-install state (empty list is fine if the directory doesn't exist yet) -mkdir -p "$HOME/.kiro/skills" -ls -1 "$HOME/.kiro/skills" 2>/dev/null | sort > /tmp/kiro-skills-before.txt - -# Run the installer -bash <(curl -sL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/install.sh) --tools kiro --global --silent -``` - -> The installer may exit non-zero even on success. Verify the skills landed with: -> ```bash -> ls ~/.kiro/skills/ | head -5 -> ``` - -> **`install.sh` no longer installs the MCP server.** It is skills-only and delegates -> to `databricks aitools`; its own header now reads *"The (deprecated, optional) MCP -> server has its own installer."* Do not check for `~/.ai-dev-kit/.venv/bin/python` -> here — that path is built only by Step 1b below, and expecting it at this point is -> what produces the "installer ran but the venv is missing" symptom in -> [Troubleshooting](#error-installer-ran-but-ai-dev-kitvenvbinpython-is-missing). - -**Installer flags:** -- `--tools kiro` — Configure for Kiro only -- `--global` — Install skills to `~/.kiro/skills/` and MCP config to `~/.kiro/settings/mcp.json` -- `--silent` — Non-interactive (defaults: all skills, DEFAULT profile) -- `--profile PROFILE_NAME` — Use a specific Databricks CLI profile -- `--force` — Reinstall even if up to date -- `--skills-only` — No longer meaningful for MCP: this installer is skills-only regardless. Retained for compatibility. - -### Step 1b: Install the MCP Server (optional) - -Skip this if you only want the skills — they work on their own, and every capability in -the [Skill Catalog](#skill-catalog-34-skills) is available without the MCP server. Do -this if you want the live tools listed under [Available MCP Servers](#available-mcp-servers). - -The MCP server ships its own installer, and it must be run from a checkout rather than -piped from `curl`: - -```bash -# Clone into the layout this Power's mcp.json references -git clone --depth 1 https://github.com/databricks-solutions/ai-dev-kit.git "$HOME/.ai-dev-kit/repo" - -# --venv-dir is required: the default is /.venv, which is not the path referenced -# by mcp.json. Without it the server still fails with ENOENT. -bash "$HOME/.ai-dev-kit/repo/databricks-mcp-server/mcp_install.sh" \ - --tools kiro --global --venv-dir "$HOME/.ai-dev-kit/.venv" -``` - -Verify: - -```bash -ls "$HOME/.ai-dev-kit/.venv/bin/python" -``` - -The MCP server connects when Kiro launches, not when a chat session starts, so quit Kiro -fully (`Cmd+Q` on macOS) and reopen. Confirm in the **Kiro - MCP Logs** output channel: - -``` -[databricks] Connected with transport -[databricks] mcp.connect.ok -``` - -An `ENOENT` line naming `.venv/bin/python` instead means the venv is missing or its path -does not match `mcp.json`; see [Troubleshooting](#error-installer-ran-but-ai-dev-kitvenvbinpython-is-missing). - -### Step 2: Build a Manifest, Then Copy Skills to the Power's Steering Directory - -Use the pre-install snapshot from Step 1 to compute exactly which skill directories the installer added. Persist that list as a manifest inside the Power's installed directory so future updates and the uninstall flow operate on the exact same set of files (no glob guessing, no leftover skills, no accidental deletion of unrelated files). - -**Agent execution note:** Run the bash block below as a single inline `bash -c '...'` invocation. Do **not** write it to a scratch file like `/tmp/kiro-step2.sh` first — Kiro's sandbox blocks `Write`-tool writes to `/tmp/`, even though shell-internal redirects (e.g., `> /tmp/kiro-skills-after.txt` *from inside* `bash -c`) are allowed. Skipping the file-write attempt avoids a noisy "Access denied" error and lets execution succeed on the first try. - -```bash -POWER_DIR="$HOME/.kiro/powers/installed/databricks" -POWER_STEERING="$POWER_DIR/steering" -MANIFEST="$POWER_DIR/.skill-manifest.txt" - -# Snapshot post-install state -ls -1 "$HOME/.kiro/skills" 2>/dev/null | sort > /tmp/kiro-skills-after.txt - -# Compute the diff: skills that exist now but didn't before. This is what the installer added. -comm -13 /tmp/kiro-skills-before.txt /tmp/kiro-skills-after.txt > "$MANIFEST.tmp" - -# Reset the steering directory and write the manifest -rm -rf "$POWER_STEERING" -mkdir -p "$POWER_STEERING" -mv "$MANIFEST.tmp" "$MANIFEST" - -# Copy each manifested skill into the Power's steering directory, then delete the original -while IFS= read -r skill; do - [ -z "$skill" ] && continue - src="$HOME/.kiro/skills/$skill" - if [ -e "$src" ]; then - cp -r "$src" "$POWER_STEERING/$skill" - rm -rf "$src" - fi -done < "$MANIFEST" - -# Clean up scratch files -rm -f /tmp/kiro-skills-before.txt /tmp/kiro-skills-after.txt - -echo "Installed $(wc -l < "$MANIFEST" | tr -d ' ') skills to $POWER_STEERING" -echo "Manifest saved to $MANIFEST" -``` - -**Why this is more robust than glob-based cleanup:** - -- The manifest captures the **exact** set of skills the installer wrote, regardless of upstream naming conventions or new skill sources added later -- Uninstall and update flows can replay the manifest to remove or refresh only what the Power owns — no risk of deleting skills installed by other tools -- The manifest is human-readable; you can `cat ~/.kiro/powers/installed/databricks/.skill-manifest.txt` to audit what the Power is responsible for - -**Edge case — incremental reinstall:** if `comm -13` returns an empty manifest (e.g., you re-ran Step 1 without first removing `~/.kiro/skills/`), the installer detected the skills were already installed and skipped them. In that case, regenerate the manifest from the existing steering directory: - -```bash -ls -1 "$POWER_STEERING" 2>/dev/null | sort > "$MANIFEST" -``` - -This scopes skills to the Power (loaded only when the Power is active) and removes duplicates from the global skills namespace. Click **Update** on the Databricks Power in Kiro afterward to refresh the steering files list. - -### Step 2.5: Ensure `~/.kiro/settings/mcp.json` Schema is Valid - -> **⚠️ Agent behavior (REQUIRED):** Kiro's user-level `mcp.json` schema requires a **top-level `mcpServers` key** to exist (even if empty) alongside the `powers` block. After a fresh install of just this Power, that top-level key may be missing, and Kiro will surface this error: -> -> ``` -> Invalid mcp.json format in user directory: -> [{ "code": "invalid_type", "expected": "object", "received": "undefined", -> "path": ["mcpServers"], "message": "Required" }] -> ``` -> -> Before proceeding to Step 3, the agent MUST check `~/.kiro/settings/mcp.json` and fix it if needed. - -**Three cases to handle:** - -**Case 1 — File does not exist.** Create it with the minimum valid shape: -```json -{ - "mcpServers": {}, - "powers": { - "mcpServers": {} - } -} -``` - -**Case 2 — File exists but the top-level `mcpServers` key is missing.** This is the usual post-install state. Add the key without disturbing anything else: - -*Before:* -```json -{ - "powers": { - "mcpServers": { - "power-databricks-databricks": { ... } - } - } -} -``` - -*After:* -```json -{ - "mcpServers": {}, - "powers": { - "mcpServers": { - "power-databricks-databricks": { ... } - } - } -} -``` - -**Case 3 — Top-level `mcpServers` already exists** (e.g., the ai-dev-kit installer wrote its own `databricks` entry there, or another Power did). Leave it alone — schema is already valid. Proceed to Step 2.6. - -**Idempotent fix command** (Unix; uses `jq` if available, otherwise prints instructions): - -```bash -MCP_CFG="$HOME/.kiro/settings/mcp.json" - -if [ ! -f "$MCP_CFG" ]; then - mkdir -p "$(dirname "$MCP_CFG")" - printf '%s\n' '{' ' "mcpServers": {},' ' "powers": { "mcpServers": {} }' '}' > "$MCP_CFG" - echo "Created $MCP_CFG with minimum valid shape." -elif command -v jq >/dev/null 2>&1; then - if ! jq -e '.mcpServers' "$MCP_CFG" >/dev/null 2>&1; then - tmp=$(mktemp) - jq '. + {mcpServers: (.mcpServers // {})}' "$MCP_CFG" > "$tmp" && mv "$tmp" "$MCP_CFG" - echo "Added top-level mcpServers: {} to $MCP_CFG." - else - echo "Top-level mcpServers already present — no changes needed." - fi -else - echo "jq not installed. Open $MCP_CFG and ensure the top-level object contains a 'mcpServers' key (use {} if you have no top-level MCP servers)." -fi -``` - -After fixing, reload Kiro to clear the validation error before continuing to Step 2.6. - -### Step 2.6: Remove the Standalone `databricks` Entry the Installer Wrote - -> **⚠️ Agent behavior (REQUIRED):** Applies only if you ran the optional [Step 1b](#step-1b-install-the-mcp-server-optional). `mcp_install.sh` writes a top-level `mcpServers.databricks` entry to `~/.kiro/settings/mcp.json`. That entry was designed for users running ai-dev-kit *without* a Kiro Power — for our flow it is a duplicate registration of the same MCP server the Power owns. (Step 1's `install.sh` no longer writes any MCP entry; it is skills-only. If you skipped Step 1b there is nothing to remove and this step is a no-op.) Leaving both entries in place causes Kiro to launch two copies of the server (same binary, two different env blocks), which can confuse tool routing. -> -> The agent MUST delete the top-level `mcpServers.databricks` entry after the installer runs, leaving only the Power's entry under `powers.mcpServers.power-databricks-databricks`. - -**Idempotent fix command:** - -```bash -MCP_CFG="$HOME/.kiro/settings/mcp.json" - -if command -v jq >/dev/null 2>&1; then - if jq -e '.mcpServers.databricks' "$MCP_CFG" >/dev/null 2>&1; then - tmp=$(mktemp) - jq 'del(.mcpServers.databricks)' "$MCP_CFG" > "$tmp" && mv "$tmp" "$MCP_CFG" - echo "Removed standalone mcpServers.databricks entry." - else - echo "No standalone entry to remove — already clean." - fi -else - echo "jq not installed. Open $MCP_CFG and delete the 'mcpServers.databricks' object manually, leaving 'mcpServers' as {} or with other entries you intentionally configured." -fi -``` - -**Before:** -```json -{ - "mcpServers": { - "databricks": { - "command": "/Users//.ai-dev-kit/.venv/bin/python", - "args": ["/Users//.ai-dev-kit/repo/databricks-mcp-server/run_server.py"], - "defer_loading": true, - "env": { "DATABRICKS_CONFIG_PROFILE": "DEFAULT" } - } - }, - "powers": { - "mcpServers": { - "power-databricks-databricks": { ... } - } - } -} -``` - -**After:** -```json -{ - "mcpServers": {}, - "powers": { - "mcpServers": { - "power-databricks-databricks": { ... } - } - } -} -``` - -Reload Kiro to drop the now-removed standalone server connection. The Power's entry under `powers.mcpServers` is the one that runs from here on. - -> **If you have OTHER MCP servers** (not `databricks`) under top-level `mcpServers` that you intentionally configured for non-Power use, the `jq` snippet above leaves them alone — it only removes the `databricks` key. - -### Step 2.7: Expand `~` in the Power's `command` and `args` Paths - -> **⚠️ Agent behavior (REQUIRED):** The Power's bundled `mcp.json` ships with `~/.ai-dev-kit/...` paths (tilde for the user's home directory). Kiro copies these paths verbatim into `~/.kiro/settings/mcp.json` under `powers.mcpServers.power-databricks-databricks`. **Node's `child_process.spawn()` does not expand `~`** — it looks for a literal directory named `~`, which doesn't exist, and the MCP server fails to launch with: -> -> ``` -> Error connecting to MCP server: spawn ~/.ai-dev-kit/.venv/bin/python ENOENT -> ``` -> -> **`${HOME}` is not an alternative.** Kiro does expand `${VAR}` in MCP config, -> including in `powers.mcpServers`, but the expansion is gated on the -> `kiroAgent.mcpApprovedEnvVars` setting: any variable not on that list is left -> as a literal string and only recorded for an unapproved-variable warning. That -> setting is per-user, so a Power published in the catalog cannot add `HOME` to -> it, and `${HOME}` would ship broken for anyone who has not approved it by hand. -> This is also why the `env` block's `${DATABRICKS_CONFIG_PROFILE}` does resolve: -> the installer's onboarding gets that variable onto the approved list. Expanding -> the tilde to an absolute path at install time, as below, is the reliable fix. -> -> The agent MUST replace `~` with the absolute home path in both `command` and every entry in `args` before the server can launch. - -**Idempotent fix command:** - -```bash -MCP_CFG="$HOME/.kiro/settings/mcp.json" -HOME_PATH="$HOME" - -if command -v jq >/dev/null 2>&1; then - tmp=$(mktemp) - jq --arg h "$HOME_PATH" ' - if .powers.mcpServers["power-databricks-databricks"] then - .powers.mcpServers["power-databricks-databricks"].command |= ($h + (.|ltrimstr("~"))) | - .powers.mcpServers["power-databricks-databricks"].args |= map($h + (.|ltrimstr("~"))) - else . end - ' "$MCP_CFG" > "$tmp" && mv "$tmp" "$MCP_CFG" - echo "Expanded ~ to $HOME_PATH in command and args." -else - echo "jq not installed. Open $MCP_CFG and replace ~/.ai-dev-kit/... with $HOME/.ai-dev-kit/... in both 'command' and every entry of 'args'." -fi -``` - -**Before:** -```json -"command": "~/.ai-dev-kit/.venv/bin/python", -"args": ["~/.ai-dev-kit/repo/databricks-mcp-server/run_server.py"] -``` - -**After** (assuming `$HOME = /Users/`): -```json -"command": "/Users//.ai-dev-kit/.venv/bin/python", -"args": ["/Users//.ai-dev-kit/repo/databricks-mcp-server/run_server.py"] -``` - -The transformation is per-user (your `$HOME` ends up baked in), so this fix has to run on each developer's machine — it can't be pre-applied in the bundled file. - -Verify the binaries exist at the expanded paths: - -```bash -ls -la "$HOME/.ai-dev-kit/.venv/bin/python" -ls -la "$HOME/.ai-dev-kit/repo/databricks-mcp-server/run_server.py" -``` - -If either is missing, Step 1's installer didn't complete — re-run it. Otherwise, reload Kiro; the `ENOENT` error should clear. - -### Step 3: Configure Authentication - -The Power ships with a baseline `mcp.json` that uses an env-var reference for the profile name and is **disabled by default** for safety. Pick one of the four options below and apply the matching "after" configuration. - -> **⚠️ Agent behavior during onboarding (REQUIRED):** -> -> Before configuring or modifying any credentials, the agent MUST: -> -> 1. **Check for existing credentials** in two locations: -> - The top-level `mcpServers` block in `~/.kiro/settings/mcp.json` (e.g., from a prior installation or another Databricks-using Power) -> - Profiles in `~/.databrickscfg` -> -> **Read these with a command that cannot emit a secret. Do not `cat` either file.** -> A bare `cat ~/.databrickscfg` prints every `token` and `client_secret` in the -> file straight into the chat transcript, which defeats rule 2 below no matter how -> carefully rule 2 is worded. Enumerate the fields you need instead: -> -> ```bash -> # Profile names, hosts, auth types and client_ids only. -> # `token` and `client_secret` are never selected, so they cannot be echoed. -> awk '/^\[/{p=$0} /^[[:space:]]*(host|auth_type|client_id)[[:space:]]*=/{print p, $0}' \ -> ~/.databrickscfg 2>/dev/null || echo "NO_DATABRICKSCFG" -> ``` -> -> ```bash -> # MCP config without env values, which can hold a PAT -> python3 -c 'import json,os;d=json.load(open(os.path.expanduser("~/.kiro/settings/mcp.json")));\ -> print([(k,v.get("command") or v.get("url"),v.get("disabled")) for k,v in (d.get("mcpServers") or {}).items()])' \ -> 2>/dev/null || echo "NO_MCP_JSON" -> ``` -> -> When a fingerprint is genuinely needed for rule 2, extract just that field and -> slice it — never print the whole file and trim afterwards. -> 2. **Show the user enough to identify the credentials, but never the full secret.** The user needs to recognize *which* credential they are looking at — show identifiers and short prefix/suffix fingerprints — but the bulk of the secret must not be echoed. -> -> | Display this | Hide this | -> |---|---| -> | Profile name (e.g., `[DEFAULT]`, `[prod-aws]`) | — | -> | Workspace host URL | — | -> | `auth_type` field value | — | -> | `client_id` (full — it's a public identifier) | `client_secret` (only first 4 + last 4 chars; never the middle) | -> | PAT in the form `dapi` + 4 chars + `***` + last 4 chars (e.g., `dapi5f2a***4f2c`), plus token comment if present | The middle of the PAT | -> | OAuth secret first 4 + `***` + last 4 chars (e.g., `abcd***9z8y`), plus last-modified date if known | The middle of the secret | -> | OAuth U2M `auth_type = databricks-cli` label only | Cached access/refresh tokens from `~/.databricks/token-cache.json` | -> -> Preserving the `dapi` prefix on PATs makes the token type unmistakable; the 4-char fingerprints on either side are enough for the user to recognize a credential they generated themselves (matching it against their notes or a credential manager) without exposing usable token material. -> -> 3. **Ask for explicit confirmation** before reusing detected credentials. Offer three choices: -> - Reuse the detected credentials for this Power -> - Configure a different authentication option (A / B / C / D below) — **default recommendation is Option A (OAuth U2M)** for interactive use; auto-refreshes hourly and avoids the long-lived-secret expiry that PATs and OAuth M2M secrets eventually hit -> - Skip — the user will edit `mcp.json` themselves -> -> If **no credentials at all** are detected, skip the "reuse" choice and walk the user directly through Option A (OAuth U2M) unless they explicitly ask for a different option. -> 4. **Never copy credentials between configurations without explicit user approval.** Silent reuse is not acceptable, even when the credentials appear identical or compatible. -> 5. **Present the choices neutrally.** Do not label any option as "recommended", "quickest", or "easiest" — credential reuse is the user's decision on its merits, not the agent's recommendation. -> 6. **Always present all four options in A → B → C → D order**, even when no credentials are detected for a given option. Do **not** skip an option because detection found nothing for it — instead, show it with the "no credentials found, here's how to set up" path. Skipping options (e.g., showing only B/C/D, or A/C/D) makes the list look broken and confuses the user. The list must always read A, B, C, D — never B, C, D or A, B, D. -> 7. **On auth failure during a session, loop back to detection.** If a Databricks MCP tool returns `Invalid access token`, `401 Unauthorized`, `403 token expired`, or any equivalent auth-failure status, immediately pause tool execution and re-enter this detection flow. Do not retry the failed tool, attempt a silent token refresh, or guess at a fix. Surface the current credential summary and recommend Option A (OAuth U2M) as the first repair path. -> -> Example of an acceptable summary the agent shows the user: -> ``` -> Found existing Databricks configuration: -> -> ~/.kiro/settings/mcp.json (top-level mcpServers.databricks): -> host = https://.cloud.databricks.com -> auth method = OAuth U2M (via DATABRICKS_CONFIG_PROFILE=DEFAULT) -> -> ~/.databrickscfg: -> [DEFAULT] -> host = https://.cloud.databricks.com -> auth_type = databricks-cli -> [prod-aws] -> host = https://.cloud.databricks.com -> auth_type = pat -> token = dapi5f2a***4f2c (dapi prefix + 4 + last 4) -> comment = "Power onboarding 2026-05-19" -> [ci-sp] -> host = https://.cloud.databricks.com -> client_id = 1a2b3c4d-5e6f-7890-abcd-ef1234567890 -> client_secret = abcd***9z8y (first 4 + last 4 only) -> -> Would you like to: -> 1. Reuse the [DEFAULT] profile for this Power -> 2. Configure a different auth option (A / B / C / D below) -> 3. Skip — I'll edit mcp.json myself -> ``` - -#### Baseline (as shipped — before any configuration) - -This is the file you'll find at `~/.kiro/powers/installed/databricks/mcp.json` immediately after installation. The `${DATABRICKS_CONFIG_PROFILE}` reference is resolved from your shell environment when Kiro launches the server. - -```json -{ - "mcpServers": { - "databricks": { - "command": "~/.ai-dev-kit/.venv/bin/python", - "args": ["~/.ai-dev-kit/repo/databricks-mcp-server/run_server.py"], - "env": { - "DATABRICKS_CONFIG_PROFILE": "${DATABRICKS_CONFIG_PROFILE}" - }, - "disabled": true - } - } -} -``` - -> All four options below modify the **runtime** MCP config at `~/.kiro/settings/mcp.json` (under `powers.mcpServers.power-databricks-databricks`), not the bundled file above. Kiro mirrors the bundled config there during Power activation. -> -> **Where credentials are read from:** the MCP server reads its environment from the `env` block of `mcp.json` at the moment Kiro launches it. Pick the option whose env block matches the credentials you have, then edit `mcp.json` directly. - -#### Recommended setup before you pick an option: a multi-environment `~/.databrickscfg` - -The cleanest long-term setup is to keep **all** your Databricks credentials in one place: `~/.databrickscfg`. Every Databricks tool on your machine — the CLI, Terraform provider, VS Code extension, local notebooks, this Power — reads the same file, so you set it up once and get a consistent identity everywhere. Switching environments becomes a one-line shell change (`export DATABRICKS_CONFIG_PROFILE=u2m-prod`) instead of editing `mcp.json` every time. - -Use a naming convention that encodes both the environment **and** the auth method, so it's obvious what each profile does at a glance: `-` (e.g., `u2m-dev`, `m2m-dev`, `pat-dev`, then the same for `qa` and `prod`). - -**Step-by-step:** - -1. **Create your U2M profiles first** with `databricks auth login` — one per environment. The CLI writes the `[u2m-*]` blocks for you: - ```bash - databricks auth login --host https://.cloud.databricks.com --profile u2m-dev - databricks auth login --host https://.cloud.databricks.com --profile u2m-qa - databricks auth login --host https://.cloud.databricks.com --profile u2m-prod - ``` - -2. **Add M2M and PAT profiles by hand** if you need them. Open `~/.databrickscfg` in an editor and append blocks. The CLI does not write these for you. - -3. **Lock the file down** so secrets aren't world-readable: - ```bash - chmod 600 ~/.databrickscfg - ``` - -**Reference template** — fill in only the environments and auth methods you actually use: - -```ini -# ~/.databrickscfg -# Naming: - -# Auth methods: u2m (interactive), m2m (service principal), pat (legacy) - -[u2m-dev] -host = https://.cloud.databricks.com -auth_type = databricks-cli - -[u2m-qa] -host = https://.cloud.databricks.com -auth_type = databricks-cli - -[u2m-prod] -host = https://.cloud.databricks.com -auth_type = databricks-cli - -[m2m-dev] -host = https://.cloud.databricks.com -client_id = -client_secret = - -[m2m-qa] -host = https://.cloud.databricks.com -client_id = -client_secret = - -[m2m-prod] -host = https://.cloud.databricks.com -client_id = -client_secret = - -[pat-dev] -host = https://.cloud.databricks.com -token = dapi... - -[pat-qa] -host = https://.cloud.databricks.com -token = dapi... - -[pat-prod] -host = https://.cloud.databricks.com -token = dapi... -``` - -**Switching environments at runtime:** - -```bash -export DATABRICKS_CONFIG_PROFILE=u2m-dev # work in dev -# Kiro restart / MCP server reconnect - -export DATABRICKS_CONFIG_PROFILE=u2m-prod # switch to prod -# Kiro restart / MCP server reconnect -``` - -The Power's `mcp.json` env block stays exactly what ships by default — `{"DATABRICKS_CONFIG_PROFILE": "${DATABRICKS_CONFIG_PROFILE}"}` — and Option C (Existing profile) handles all three auth methods because the Databricks SDK reads the profile and picks the matching flow. Once `~/.databrickscfg` is set up, you rarely need to switch options. - -**When you don't need this:** -- You only ever work against one Databricks workspace with one auth method → a single `[DEFAULT]` profile (or the env-var options below) is fine. -- You don't have permission to install/run `databricks auth login` and don't want to hand-write profiles → skip to Options B or D and use shell env vars directly. - -**Quick reference (Databricks-recommended order):** - -| Option | Method | Best for | Token lifetime | Status | -|--------|--------|----------|----------------|--------| -| A | OAuth U2M (CLI login) | Interactive use by a human | 1 h, auto-refreshed | Recommended | -| B | OAuth M2M (service principal) | Headless / CI/CD / production agents | 1 h access token; secret up to 730 days | Recommended | -| C | Existing `~/.databrickscfg` profile | You already have a working profile | Depends on the underlying auth in the profile | Recommended | -| D | Personal Access Token (PAT) | Tools that don't support OAuth | Up to 730 days; auto-revoked after 90 days unused | **Legacy** — Databricks recommends OAuth instead | - -Reference: [PAT (legacy)](https://docs.databricks.com/aws/en/dev-tools/auth/pat) · [OAuth U2M](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-u2m) · [OAuth M2M](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m) - ---- - -#### Option A — OAuth U2M via Databricks CLI (recommended for interactive use) - -OAuth user-to-machine: the Databricks CLI opens a browser, you authenticate as yourself, and credentials land in `~/.databrickscfg`. Each access token is valid for one hour and is auto-refreshed by the Databricks SDK. This is the safest interactive flow because there's no long-lived secret to leak. - -> **Agent reuse-detection for Option A:** If `~/.databrickscfg` contains profiles with `auth_type = databricks-cli`, the agent MUST list each one with its host so the user can pick which to use, e.g.: -> ``` -> Found OAuth U2M profiles in ~/.databrickscfg: -> 1. [DEFAULT] host = https://.cloud.databricks.com -> 2. [prod-aws] host = https://.cloud.databricks.com -> 3. Run a fresh databricks auth login (creates a new profile) -> ``` -> If no U2M profiles exist, say so explicitly and walk the user through `databricks auth login` below. - -```bash -databricks auth login --host https://.cloud.databricks.com --profile DEFAULT -``` - -This writes a profile to `~/.databrickscfg` like: -```ini -[DEFAULT] -host = https://.cloud.databricks.com -auth_type = databricks-cli -``` - -Then export the profile name in your shell so the `${DATABRICKS_CONFIG_PROFILE}` reference in `mcp.json` resolves correctly when Kiro launches the MCP server: - -```bash -export DATABRICKS_CONFIG_PROFILE=DEFAULT -``` - -**Before** (`~/.kiro/settings/mcp.json` — Power entry as shipped): -```json -{ - "mcpServers": { - "databricks": { - "command": "~/.ai-dev-kit/.venv/bin/python", - "args": ["~/.ai-dev-kit/repo/databricks-mcp-server/run_server.py"], - "env": { - "DATABRICKS_CONFIG_PROFILE": "${DATABRICKS_CONFIG_PROFILE}" - }, - "disabled": true - } - } -} -``` - -**After** (Option A applied): -```json -{ - "mcpServers": { - "databricks": { - "command": "~/.ai-dev-kit/.venv/bin/python", - "args": ["~/.ai-dev-kit/repo/databricks-mcp-server/run_server.py"], - "env": { - "DATABRICKS_CONFIG_PROFILE": "${DATABRICKS_CONFIG_PROFILE}" - }, - "disabled": false - } - } -} -``` - -The only edit is `"disabled": true` → `"disabled": false`. The env reference and shell variable do the rest. - ---- - -#### Option B — OAuth M2M with Service Principal (recommended for headless / CI/CD) - -OAuth machine-to-machine: a Databricks service principal authenticates with `client_id` + `client_secret` and the SDK auto-issues 1-hour access tokens with no browser flow. Use this for production agents, scheduled jobs, and shared environments. Per Databricks docs, this is intended for *"unattended processes, such as automated CLI commands or REST API calls made from scripts or applications."* - -> **Agent reuse-detection for Option B:** If `~/.databrickscfg` contains profiles with `client_id` set, or the shell environment has `DATABRICKS_CLIENT_ID` / `DATABRICKS_CLIENT_SECRET`, the agent MUST list each candidate with its `client_id` (full) and secret fingerprint so the user can recognize it, e.g.: -> ``` -> Found OAuth M2M credentials: -> 1. ~/.databrickscfg [ci-sp] -> host = https://.cloud.databricks.com -> client_id = 1a2b3c4d-5e6f-7890-abcd-ef1234567890 -> client_secret = abcd***9z8y (first 4 + last 4 only) -> 2. Shell env DATABRICKS_CLIENT_ID -> host = https://.cloud.databricks.com -> client_id = 9z8y7x6w-...-fedcba0987654321 -> client_secret = wxyz***1234 (first 4 + last 4 only) -> 3. Generate a new service principal + OAuth secret (steps below) -> ``` -> If no M2M credentials are detected, say so explicitly and walk the user through the **Prerequisites** below. - -**Prerequisites:** -1. **Create a service principal** in the Databricks account console (Identity and access → Service principals) -2. **Grant workspace entitlements and permissions** so the SP can do what you need -3. **Generate an OAuth secret** — Settings → Identity and access → Service principals → select SP → Secrets tab → Generate secret. Set lifetime up to 730 days (max). Copy `client_id` and `client_secret` immediately — the secret is shown only once. A service principal can have up to 5 active OAuth secrets. - -Export the three environment variables in your shell so the `${VAR}` references in `mcp.json` resolve when Kiro launches the MCP server: - -```bash -export DATABRICKS_HOST="https://.cloud.databricks.com" -export DATABRICKS_CLIENT_ID="" -export DATABRICKS_CLIENT_SECRET="" -``` - -**Before** (as shipped): -```json -{ - "mcpServers": { - "databricks": { - "command": "~/.ai-dev-kit/.venv/bin/python", - "args": ["~/.ai-dev-kit/repo/databricks-mcp-server/run_server.py"], - "env": { - "DATABRICKS_CONFIG_PROFILE": "${DATABRICKS_CONFIG_PROFILE}" - }, - "disabled": true - } - } -} -``` - -**After** (Option B applied — replace profile env with host + client credentials): -```json -{ - "mcpServers": { - "databricks": { - "command": "~/.ai-dev-kit/.venv/bin/python", - "args": ["~/.ai-dev-kit/repo/databricks-mcp-server/run_server.py"], - "env": { - "DATABRICKS_HOST": "${DATABRICKS_HOST}", - "DATABRICKS_CLIENT_ID": "${DATABRICKS_CLIENT_ID}", - "DATABRICKS_CLIENT_SECRET": "${DATABRICKS_CLIENT_SECRET}" - }, - "disabled": false - } - } -} -``` - -> **Never hardcode `client_secret`** in the JSON file. Reference it through `${DATABRICKS_CLIENT_SECRET}` and source it from a secret manager (HashiCorp Vault, AWS Secrets Manager, OS keychain) or environment-injection mechanism appropriate to your runtime. -> -> ai-dev-kit's MCP server README only documents PAT and profile auth, but the underlying Databricks SDK's unified-auth chain detects M2M credentials automatically — no MCP-server-side change required beyond the env block above. - ---- - -#### Option C — Existing `~/.databrickscfg` Profile - -If you already have a working profile (e.g., `prod-aws`) in `~/.databrickscfg` — set up earlier via `databricks auth login`, manual editing, or a configuration-management tool — just point the env reference at it. The underlying auth method (OAuth U2M, OAuth M2M, or PAT) is whatever the profile already uses. - -> **Agent reuse-detection for Option C:** The agent MUST enumerate every profile in `~/.databrickscfg`, showing host, `auth_type`, and a credential fingerprint appropriate to the profile's auth method, so the user can pick one explicitly. Example: -> ``` -> Profiles in ~/.databrickscfg: -> 1. [DEFAULT] -> host = https://.cloud.databricks.com -> auth_type = databricks-cli (OAuth U2M — no static credential) -> 2. [prod-aws] -> host = https://.cloud.databricks.com -> auth_type = pat -> token = dapi5f2a***4f2c (dapi prefix + 4 + last 4) -> 3. [ci-sp] -> host = https://.cloud.databricks.com -> client_id = 1a2b3c4d-5e6f-7890-abcd-ef1234567890 -> client_secret = abcd***9z8y (first 4 + last 4 only) -> ``` -> If `~/.databrickscfg` does not exist or has no profiles, the agent MUST say so explicitly and offer to switch the user to Option A. - -```bash -export DATABRICKS_CONFIG_PROFILE=prod-aws -``` - -**Before** (as shipped): -```json -{ - "mcpServers": { - "databricks": { - "command": "~/.ai-dev-kit/.venv/bin/python", - "args": ["~/.ai-dev-kit/repo/databricks-mcp-server/run_server.py"], - "env": { - "DATABRICKS_CONFIG_PROFILE": "${DATABRICKS_CONFIG_PROFILE}" - }, - "disabled": true - } - } -} -``` - -**After** (Option C applied — same env reference, just enable the server): -```json -{ - "mcpServers": { - "databricks": { - "command": "~/.ai-dev-kit/.venv/bin/python", - "args": ["~/.ai-dev-kit/repo/databricks-mcp-server/run_server.py"], - "env": { - "DATABRICKS_CONFIG_PROFILE": "${DATABRICKS_CONFIG_PROFILE}" - }, - "disabled": false - } - } -} -``` - -> Per Databricks docs, *"Values in `.databrickscfg` take precedence over environment variables"* — so if your profile has a `host` set, that takes precedence over `DATABRICKS_HOST`. - ---- - -#### Option D — Personal Access Token (legacy) - -> **Databricks officially marks PAT as legacy.** From the docs: *"Where possible, Databricks recommends using OAuth instead of PATs for user account authentication because OAuth provides stronger security."* Use this only if Options A–C aren't available — for example, an automated tool that doesn't support OAuth and where you can't use a service principal. - -> **Agent reuse-detection for Option D:** If `~/.databrickscfg` contains profiles with `token` set (or `auth_type = pat`), or the shell environment has `DATABRICKS_TOKEN`, or the top-level `mcpServers` block in `~/.kiro/settings/mcp.json` already has a PAT configured, the agent MUST list each candidate with its host and PAT fingerprint, e.g.: -> ``` -> Found existing PATs: -> 1. ~/.kiro/settings/mcp.json (top-level mcpServers.databricks) -> host = https://.cloud.databricks.com -> token = dapi5f2a***4f2c (dapi prefix + 4 + last 4) -> 2. ~/.databrickscfg [prod-aws] -> host = https://.cloud.databricks.com -> token = dapie8cf***84fc (dapi prefix + 4 + last 4) -> comment = "Power onboarding 2026-05-19" -> 3. Generate a new PAT (workspace UI steps below) -> ``` -> If no PAT is detected, say so explicitly and walk the user through the **Generate a PAT in the workspace UI** steps below. - -PATs are simple bearer tokens. **Constraints worth knowing:** -- Lifetime up to ~730 days (set at creation) -- *"Databricks automatically revokes PATs that haven't been used for 90 days"* -- Up to 600 PATs per workspace -- *"You can't use personal access tokens to automate Databricks account-level functionality"* - -**Generate a PAT in the workspace UI:** -1. Top bar → username → Settings → Developer -2. Next to **Access tokens**, click **Manage** → **Generate new token** -3. Add a comment, set lifetime in days, optionally restrict scopes -4. Click **Generate** and copy the token immediately (it is shown only once) - -Export the host and token in your shell so the `${VAR}` references in `mcp.json` resolve when Kiro launches the MCP server: - -```bash -export DATABRICKS_HOST="https://.cloud.databricks.com" -export DATABRICKS_TOKEN="dapi..." -``` - -**Before** (as shipped): -```json -{ - "mcpServers": { - "databricks": { - "command": "~/.ai-dev-kit/.venv/bin/python", - "args": ["~/.ai-dev-kit/repo/databricks-mcp-server/run_server.py"], - "env": { - "DATABRICKS_CONFIG_PROFILE": "${DATABRICKS_CONFIG_PROFILE}" - }, - "disabled": true - } - } -} -``` - -**After** (Option D applied — replace profile env with host + token): -```json -{ - "mcpServers": { - "databricks": { - "command": "~/.ai-dev-kit/.venv/bin/python", - "args": ["~/.ai-dev-kit/repo/databricks-mcp-server/run_server.py"], - "env": { - "DATABRICKS_HOST": "${DATABRICKS_HOST}", - "DATABRICKS_TOKEN": "${DATABRICKS_TOKEN}" - }, - "disabled": false - } - } -} -``` - -> **Never hardcode a PAT in the JSON file.** Always reference it through `${DATABRICKS_TOKEN}` and keep the secret in your shell environment, a secret manager, or your OS keychain. - -### Step 4: Verify the Connection - -After saving, Kiro auto-reconnects the MCP server. Verify by calling: - -```javascript -usePower("databricks", "databricks", "get_current_user", {}) -// Returns: { user_name: "...", workspace_url: "...", entitlements: [...] } -``` - -A successful response confirms auth, env-var resolution, and server enablement are all wired correctly. - -### Updating Skills - -Re-run the snapshot/installer/diff pattern from Steps 1 and 2 so the manifest gets regenerated with any newly-added or removed skills: - -```bash -# 1. Snapshot the current Power-owned manifest as the "before" baseline -POWER_DIR="$HOME/.kiro/powers/installed/databricks" -ls -1 "$POWER_DIR/steering" 2>/dev/null | sort > /tmp/kiro-skills-before.txt - -# 2. Re-run the installer in skills-only force-refresh mode -bash <(curl -sL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/install.sh) --tools kiro --global --skills-only --force --silent - -# 3. Re-run Step 2's diff-and-move snippet to refresh the manifest and steering directory -``` - -After updating, click **Update** on the Power in Kiro to pick up any new skills. - -### Uninstalling Skills - -The manifest makes uninstall surgical — only the skills this Power owns get removed: - -```bash -POWER_DIR="$HOME/.kiro/powers/installed/databricks" -MANIFEST="$POWER_DIR/.skill-manifest.txt" - -if [ -f "$MANIFEST" ]; then - while IFS= read -r skill; do - [ -z "$skill" ] && continue - rm -rf "$HOME/.kiro/skills/$skill" - done < "$MANIFEST" -fi - -rm -rf "$POWER_DIR" -``` - -This will not touch skills installed by other tools or other Powers, even if they happen to share a name prefix. - -## Tips - -1. **Start with `get_current_user`** to verify your identity, workspace, and entitlements -2. **Use `list_compute` first** when you don't know warehouse/cluster IDs — never hardcode them -3. **Prefer `ask_genie` for ad-hoc questions** — it generates SQL and explanations in one shot -4. **Lean on the steering files** — each skill encodes idiomatic patterns and gotchas for its domain -5. **Use Asset Bundles (`databricks-bundles` skill)** for production-grade resource definitions over imperative tool calls -6. **Combine `manage_uc_objects` + `manage_uc_grants` + `manage_uc_tags`** in every table-creation flow -7. **Wait explicitly on long-running runs** with `run_now_and_wait` rather than polling -8. **Track and clean up** sandbox resources with `list_tracked_resources` / `delete_tracked_resource` -9. **Use `get_table_stats_and_schema`** before authoring transformations — schema-on-read surprises are common -10. **Keep `disabled: true` until onboarding is complete** — flip it only after auth is verified - -## License, Trademarks, and Support - -### Power packaging - -This Power's manifest (`POWER.md`, `mcp.json`, `steering/`) is published under the same license terms as the [kirodotdev/powers](https://github.com/kirodotdev/powers) catalog. See Contributors below. - -### Upstream Databricks AI Dev Kit - -The MCP server and skills installed by this Power are sourced from the [databricks-solutions/ai-dev-kit](https://github.com/databricks-solutions/ai-dev-kit) repository and are subject to the [Databricks License](https://databricks.com/db-license-source). See ai-dev-kit's [LICENSE.md](https://github.com/databricks-solutions/ai-dev-kit/blob/main/LICENSE.md) and [NOTICE.txt](https://github.com/databricks-solutions/ai-dev-kit/blob/main/NOTICE.txt) for full terms and third-party attribution. Users accept those terms when they run the installer. - -### Trademarks - -"Databricks", "Databricks AI Dev Kit", "Unity Catalog", "Unity AI Gateway", "Delta Lake", "Agent Bricks", "Genie", "Lakebase", "MLflow", "Databricks Apps", and other Databricks product names referenced in this Power are trademarks of Databricks, Inc. - -"Amazon Web Services", "AWS", and "Kiro" are trademarks of Amazon.com, Inc. or its affiliates. - -### Contributors - -- **Antony Prasad Thevaraj** ([@antonyprasad-db](https://github.com/antonyprasad-db)) — Senior Specialist Solutions Architect, Databricks (AWS Partnership). Built the underlying **Databricks AI Dev Kit** (the MCP server, skills, and unified installer) at [databricks-solutions/ai-dev-kit](https://github.com/databricks-solutions/ai-dev-kit), including the Kiro IDE installer support in [PR #511](https://github.com/databricks-solutions/ai-dev-kit/pull/511) that this Power depends on. Authored the **v2 of this Power** for submission to the [kirodotdev/powers](https://github.com/kirodotdev/powers) catalog — adding the Agent First-Action Protocol (credential pre-flight + 401 recovery), the manifest-driven install/uninstall flow, the install-blocking `mcp.json` fixes (Steps 2.5 / 2.6 / 2.7), the SKILL.md naming-warning troubleshooting entry, and the Trademarks subsection. -- **Venkatavaradhan Viswanathan** ([@venkatavaradhanv](https://github.com/venkatavaradhanv)) — Senior Solutions Architect, AWS. Created the original wrapper Power around the Databricks AI Dev Kit at [venkatavaradhanv/databricks](https://github.com/venkatavaradhanv/databricks), packaging the catalog-ready `POWER.md`, the parametrized `mcp.json`, the four-option auth documentation (OAuth U2M / OAuth M2M / existing profile / PAT), and the security-conscious credential-detection onboarding flow (8 use-cases across the 4 auth options with safe fingerprinting rules). - -### Support - -- **Power packaging / installation issues** — [github.com/kirodotdev/powers/issues](https://github.com/kirodotdev/powers/issues) -- **MCP server, skills, or installer issues** — [github.com/databricks-solutions/ai-dev-kit/issues](https://github.com/databricks-solutions/ai-dev-kit/issues) -- **Databricks platform support** — [help.databricks.com](https://help.databricks.com) -- **Privacy** — [databricks.com/legal/privacynotice](https://www.databricks.com/legal/privacynotice) - ---- - -**Package:** `databricks-solutions/ai-dev-kit` (`databricks-mcp-server`) -**Source:** [github.com/databricks-solutions/ai-dev-kit](https://github.com/databricks-solutions/ai-dev-kit) -**License:** [Databricks License](https://databricks.com/db-license-source) -**Connection:** Local Python process via `~/.kiro/settings/mcp.json` diff --git a/databricks/mcp.json b/databricks/mcp.json deleted file mode 100644 index f2d60d54..00000000 --- a/databricks/mcp.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "mcpServers": { - "databricks": { - "command": "~/.ai-dev-kit/.venv/bin/python", - "args": ["~/.ai-dev-kit/repo/databricks-mcp-server/run_server.py"], - "env": { - "DATABRICKS_CONFIG_PROFILE": "${DATABRICKS_CONFIG_PROFILE}" - }, - "disabled": true - } - } -} diff --git a/databricks/plugin.json b/databricks/plugin.json new file mode 100644 index 00000000..7889ef30 --- /dev/null +++ b/databricks/plugin.json @@ -0,0 +1,13 @@ +{ + "$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json", + "name": "databricks", + "displayName": "Databricks", + "version": "1.0.0", + "description": "Official Databricks agent skills for Kiro: expert guidance for building data pipelines, jobs, dashboards, apps, ML and AI agents on the Databricks Platform, installed and updated with the Databricks CLI.", + "author": { + "name": "Databricks", + "url": "https://www.databricks.com" + }, + "keywords": ["databricks", "unity catalog", "lakeflow", "databricks sql", "lakebase", "databricks apps"], + "repository": "https://github.com/kirodotdev/powers" +} diff --git a/databricks/skills/databricks-kiro-setup/SKILL.md b/databricks/skills/databricks-kiro-setup/SKILL.md new file mode 100644 index 00000000..56468d79 --- /dev/null +++ b/databricks/skills/databricks-kiro-setup/SKILL.md @@ -0,0 +1,244 @@ +--- +name: "databricks-kiro-setup" +description: "Sets up the official Databricks agent skills in Kiro with the Databricks CLI, and shows how to add a governed Databricks MCP such as Genie One. Use when starting Databricks work in Kiro before the Databricks skills (such as databricks-core) are installed, when installing, updating, or removing them, when signing in the Databricks CLI, when adding a Databricks MCP to Kiro, when migrating from the Databricks AI Dev Kit Power, or when troubleshooting any of these." +metadata: + author: "Databricks" +--- + +# Databricks in Kiro + +## Overview + +This Power sets up the official [Databricks agent skills](https://github.com/databricks/databricks-agent-skills) in Kiro. The skills teach Kiro how to build on the Databricks Platform: Lakeflow pipelines, jobs, AI/BI dashboards, Databricks Apps, Unity Catalog governance, Lakebase, Model Serving, MLflow and Agent Bricks. + +The Databricks CLI installs the skills (`databricks aitools install --agents kiro`, CLI v1.20.0 and above) into `~/.kiro/skills/` and records the install, so `databricks aitools update` can refresh them. The Power also adds the MLflow skills from [mlflow/skills](https://github.com/mlflow/skills), which the CLI does not carry. + +This Power ships no MCP server. Databricks-provided MCPs run on your own workspace and need your credentials, so you add them to your Kiro MCP config yourself; see [Optional: governed data access](#optional-governed-data-access). + +> **Upgrading from the "Databricks AI Dev Kit" Power?** That version ran the ai-dev-kit installer, copied skills into the Power's `steering/` directory, and shipped a local MCP server that upstream has deprecated. See [Migrating from the AI Dev Kit Power](#migrating-from-the-ai-dev-kit-power) before you install. + +## On activation + +> **Run these checks on the first turn of a new conversation that uses this skill, before giving Databricks guidance.** The commands assume a POSIX shell (macOS, Linux, or WSL or Git Bash on Windows). Run each command on its own line, without trailing `#` comments: Kiro's terminal can be an interactive zsh, which passes `#` through as an argument. +> +> 1. **CLI version.** Run `databricks --version`. The skills need **v1.20.0 or later**, the first release with Kiro support in `databricks aitools`. If the CLI is missing or older, walk the user through [Step 1](#step-1-install-or-upgrade-the-databricks-cli). +> 2. **Skills installed.** Check locally, which works offline: `ls "$HOME/.databricks/aitools/skills/.state.json" "$HOME/.kiro/skills/databricks-core/SKILL.md"` (for a project install, `.databricks/aitools/skills/.state.json` and `.kiro/skills/databricks-core/SKILL.md`). If either file is missing, walk the user through [Step 3](#step-3-install-the-databricks-skills). +> 3. **Authentication.** The skills run Databricks CLI commands, which need a configured profile. List existing profiles with `databricks auth profiles --skip-validate`, which shows names and hosts and never a secret. Ignore its `Valid` column: with `--skip-validate` it always reads `NO`. If there are no profiles, walk the user through [Step 2](#step-2-authenticate-the-databricks-cli). +> 4. **Leftovers from the old Power.** If `grep -l 'databricks-mcp-server/run_server.py' "$HOME/.kiro/settings/mcp.json" .kiro/settings/mcp.json 2>/dev/null` prints a file name, the old ai-dev-kit MCP server is still configured. Mention it once and point to [Migrating from the AI Dev Kit Power](#migrating-from-the-ai-dev-kit-power); if the user declines, don't raise it again in this conversation. Never delete anything without explicit approval. +> 5. **No MCP tools by default.** This Power ships no MCP server. Do not describe Databricks MCP tools as available unless the user has added a Databricks MCP and Kiro shows it connected. + +## Setup + +### Prerequisites + +- **Kiro IDE 1.0.288 or later**, the first release that installs Powers in the Agent Plugins format +- **Databricks workspace** on AWS, Azure, or GCP +- **Databricks CLI v1.20.0 or later** ([install](https://docs.databricks.com/aws/en/dev-tools/cli/install)) +- **git**, for the MLflow skills in Step 4 + +### Step 1: Install or upgrade the Databricks CLI + +Check the version. It needs to be v1.20.0 or later: + +```bash +databricks --version +``` + +If the command is missing or reports an older version, install or upgrade it with the method for your platform from the [CLI install guide](https://docs.databricks.com/aws/en/dev-tools/cli/install). On macOS with Homebrew, install with: + +```bash +brew tap databricks/tap +brew install databricks +``` + +or upgrade an existing install with `brew upgrade databricks`. Older CLIs reject the Kiro target with `Error: unknown agent "kiro"`. + +### Step 2: Authenticate the Databricks CLI + +> **⚠️ Agent behavior (REQUIRED):** before configuring credentials, check for existing profiles with a command that cannot print a secret. **Do not `cat ~/.databrickscfg`**: that prints every `token` and `client_secret` into the chat transcript. This lists profile names and hosts only, and honors `DATABRICKS_CONFIG_FILE`: +> +> ```bash +> databricks auth profiles --skip-validate +> ``` +> +> Show the user the profile names and hosts (ignore the `Valid` column), then ask whether to reuse one or create a new one. Never reuse or copy credentials without explicit approval. + +For interactive use, sign in with OAuth (user-to-machine). The CLI opens a browser, and the access token refreshes automatically: + +```bash +databricks auth login --host https:// --profile +``` + +For other methods (service principal OAuth for automation, personal access tokens), see [Databricks CLI authentication](https://docs.databricks.com/aws/en/dev-tools/cli/authentication). + +### Step 3: Install the Databricks skills + +```bash +databricks aitools install --agents kiro --scope global +``` + +This installs the Databricks agent skills into `~/.kiro/skills/`. `--agents kiro` works even when the `kiro` command is not on your `PATH`. If `~/.kiro/skills/` already has a folder with the same name as a Databricks skill, for example a copy from the ai-dev-kit installer, the CLI moves it to a backup in the system temp directory, which the OS may clear; copy out any local edits first. Add `--experimental` to also install the experimental skills, such as `spark-python-data-source` (custom Spark data sources). + +To install for one workspace only, use `--scope project`. The CLI then stores the skills under `.databricks/aitools/skills/` in that workspace and links them from `.kiro/skills/`. Bundle projects usually gitignore `.databricks/`, so don't commit `.kiro/skills/`; each teammate runs the install in their own checkout. + +Check the result with `databricks aitools list`, which shows the installed version of each skill. + +### Step 4: Add the MLflow skills + +The CLI does not carry the MLflow skills (tracing, evaluation, trace analysis, metrics), so install them from [mlflow/skills](https://github.com/mlflow/skills). The block replaces any existing folders with these eight names; if any exist and the user didn't install them from `mlflow/skills`, this Power, or ai-dev-kit, ask before running it. + +```bash +tmp="$(mktemp -d)" +ok=0 +if git clone --depth 1 --quiet https://github.com/mlflow/skills.git "$tmp/mlflow-skills"; then + mkdir -p "$HOME/.kiro/skills" + for s in agent-evaluation analyze-mlflow-chat-session analyze-mlflow-trace \ + instrumenting-with-mlflow-tracing mlflow-onboarding querying-mlflow-metrics \ + retrieving-mlflow-traces searching-mlflow-docs; do + dest="$HOME/.kiro/skills/$s" + if [ -e "$dest.old" ] && [ ! -e "$dest" ]; then mv "$dest.old" "$dest"; fi + rm -rf "$dest.new" "$dest.old" + if [ -d "$tmp/mlflow-skills/$s" ] && cp -R "$tmp/mlflow-skills/$s" "$dest.new"; then + if [ -e "$dest" ]; then mv "$dest" "$dest.old"; fi + if mv "$dest.new" "$dest"; then + rm -rf "$dest.old"; ok=$((ok + 1)) + else + if [ -e "$dest.old" ]; then mv "$dest.old" "$dest"; fi + rm -rf "$dest.new"; echo "Could not replace $s; kept the existing copy" + fi + else + rm -rf "$dest.new"; echo "Skipped $s; any existing copy was left in place" + fi + done +else + echo "Could not clone mlflow/skills; existing MLflow skills were left unchanged" +fi +rm -rf "$tmp" +echo "Installed $ok of 8 MLflow skills" +[ "$ok" -eq 8 ] +``` + +These are the same eight MLflow skills the ai-dev-kit installer used; `mlflow/skills` has more, which you can add to the list. The loop names the skills directly so it behaves the same in zsh (the macOS default) and bash. Each skill is copied to a temporary folder first and swapped in only after the copy succeeds; if the swap fails, the previous copy is put back, and a copy left over from an interrupted run is restored before the next attempt. The block exits non-zero unless all eight were installed. For a project install, replace `$HOME/.kiro/skills` with `.kiro/skills`. The CLI doesn't track these skills; re-run this step to refresh them from the `main` branch of `mlflow/skills`. + +### Step 5: Start a new chat session + +Kiro loads skills when a chat session starts, so open a new session to pick them up. Ask something like "Using the databricks-core skill, which workspace am I connected to?" to confirm. + +## Optional: governed data access + +To query your data from Kiro, add a [Databricks-provided MCP](https://docs.databricks.com/aws/en/agents/mcp-tools/built-in-mcp-services) through Unity Gateway. For analytics, start with `system.ai.genie_one_mcp`, which answers natural-language questions with Genie One. `system.ai.dbsql` runs specific SQL on a SQL warehouse (Unity Gateway beta, which an account admin enables). Each is a Unity Catalog securable, so grants and policies govern who can call it and what each call can do. + +For local use, these MCPs accept a Databricks personal access token in an `Authorization: Bearer` header. Open your MCP config with **Open User MCP Config (JSON)** (all workspaces) or **Open Workspace MCP Config (JSON)** (this workspace) in the command palette, add the server inside the existing `mcpServers` object, and reference the token through an environment variable rather than pasting it into the file: + +```json +{ + "mcpServers": { + "databricks-genie-one": { + "url": "https:///ai-gateway/mcp-services/system.ai.genie_one_mcp", + "headers": { "Authorization": "Bearer ${DATABRICKS_MCP_TOKEN}" } + } + } +} +``` + +Kiro expands `${DATABRICKS_MCP_TOKEN}` from the environment it was launched with, so export the variable in the shell profile that starts Kiro, then quit Kiro fully (Cmd+Q on macOS) and reopen it. Kiro picks up `mcp.json` edits without a restart, but it reads environment variables only at launch. Treat that profile as a credential file: don't commit or sync it, and give the token a short lifetime. + +Kiro expands the variable in workspace MCP configs too, so add the entry to the user config and review a workspace's `.kiro/settings/mcp.json` before trusting the workspace. + +Use a name other than `DATABRICKS_TOKEN`: the Databricks CLI reads that variable for its own sign-in, so exporting it would change how the skills' CLI commands authenticate. For other sign-in options, see [MCP authentication and networking](https://docs.databricks.com/aws/en/agents/mcp-tools/authentication-networking). + +> **Agent behavior:** never write a token value into `mcp.json`, a shell profile, or the chat; the user sets the variable themselves. After the user adds the entry, check that Kiro's MCP panel shows the server connected before describing its tools as available. + +## Keeping skills current + +To refresh the Databricks skills, run: + +```bash +databricks aitools update --scope global +``` + +It updates the skills for every coding agent the CLI detects on this machine, not only Kiro, and it can switch Claude Code, Codex and Copilot from raw skill files to the Databricks plugin. If the user has other coding agents set up, tell them before running it. Use `--scope project` from the workspace folder for a project install. + +If Kiro's skills stay on an older version after another agent's skills were updated, `--force` re-downloads them, but it does so for every detected agent and replaces any local edits to Databricks skills in all of them. Get the user's approval before using it. + +To remove the Databricks skills from Kiro, get the user's approval first, then run: + +```bash +databricks aitools uninstall --agents kiro --scope global +``` + +Use `--scope project` for a project install. In an interactive terminal the command asks for confirmation; when run without a terminal it proceeds without asking. The CLI only removes skill folders that still match what it installed, so edited copies and skills it didn't install stay in place; leave them unless the user asks to remove them. + +Re-run [Step 4](#step-4-add-the-mlflow-skills) to refresh the MLflow skills, or delete their eight directories from `~/.kiro/skills/` to remove them. + +## Migrating from the AI Dev Kit Power + +Update this Power in Kiro (or uninstall and reinstall it). Kiro deletes a Power's installed folder and its `powers.mcpServers` entries when it reinstalls it, so the skill copies the old version placed in its `steering/` directory, including the MLflow skills, and its disabled MCP server entry are removed. Then run [Step 3](#step-3-install-the-databricks-skills) and [Step 4](#step-4-add-the-mlflow-skills). + +A few things live outside the Power, so updating it doesn't remove them: + +- **A top-level `databricks` MCP server.** If you installed the old optional MCP server, Kiro's `mcp.json` may still have an entry that runs `databricks-mcp-server/run_server.py`. Kiro has no command to delete a server, so open the file as described in [Optional: governed data access](#optional-governed-data-access) and delete the `databricks` entry. +- **Credential and profile exports in your shell profile.** The old Power's setup options had you export `DATABRICKS_CONFIG_PROFILE`, `DATABRICKS_HOST` with `DATABRICKS_TOKEN`, or `DATABRICKS_CLIENT_ID` and `DATABRICKS_CLIENT_SECRET`. These override the profile you choose for the skills, so commands can reach the wrong workspace or fail with "more than one authorization method configured". Remove those exports and use a profile from [Step 2](#step-2-authenticate-the-databricks-cli). If a token or client secret sat in the profile in plain text, revoke it. +- **The ai-dev-kit clone** (`~/.ai-dev-kit`, or the directory in `AIDEVKIT_HOME`). Kiro no longer needs it, but ai-dev-kit may still serve other coding agents or projects. Remove it only if you don't use ai-dev-kit anywhere else, and delete the directory rather than running ai-dev-kit's `install.sh --uninstall`: the uninstaller removes skill folders by name from `~/.kiro/skills/`, including ones the Databricks CLI installed in Step 3. + +## Troubleshooting + +### `Error: unknown agent "kiro"` + +The Databricks CLI is older than v1.20.0. Upgrade it ([Step 1](#step-1-install-or-upgrade-the-databricks-cli)) and re-run Step 3. + +### Kiro doesn't use the skills + +Skills load when a chat session starts. Open a new session, and check that `databricks-core/SKILL.md` exists in `~/.kiro/skills/`, or in `.kiro/skills/` for a project install. + +### A command fails with stray `#` arguments + +Kiro's terminal can be an interactive zsh, which doesn't treat `#` as a comment. Run commands without trailing comments. + +### A `${...}` reference in `mcp.json` stays literal + +Kiro expands variables from the environment it was launched with. Export the variable in the shell profile that starts Kiro, then quit and reopen Kiro. Some earlier Kiro versions also required listing the name in the `kiroAgent.mcpApprovedEnvVars` setting. + +### A Databricks MCP server doesn't connect + +Check the server's status and error in Kiro's MCP panel. If the token reference stayed literal, see the previous entry. An authentication error (HTTP 401) usually means the token is missing, invalid, or expired: create a new one, update the variable, and relaunch Kiro. A permission error (HTTP 403) usually means the identity behind the token isn't allowed to use that MCP service; ask a workspace or account admin, using [Databricks-provided MCP services](https://docs.databricks.com/aws/en/agents/mcp-tools/built-in-mcp-services). `system.ai.dbsql` also needs the Unity Gateway beta enabled on the account. + +### A warning that a skill name doesn't match its directory + +Two MLflow skills (`analyze-mlflow-chat-session`, `analyze-mlflow-trace`) declare a `name` that differs from their directory name. Kiro warns and still loads them. + +### Authentication errors when a skill runs a CLI command + +Run `databricks auth profiles` to check which profiles are valid, then sign in again with `databricks auth login --host https:// --profile `. + +### Duplicate Databricks skills + +Overlapping skills with older names (for example `databricks-genie` next to `databricks-genie-agents`) usually come from the ai-dev-kit installer, which copied its own skill set into `~/.kiro/skills/`. A skill can also appear twice if it's installed at both global and project scope. Show the user the overlapping folders and remove only the ones they choose. + +## License, Trademarks, and Support + +### Power packaging + +This Power's packaging (`plugin.json` and this skill) is published under the same license terms as the [kirodotdev/powers](https://github.com/kirodotdev/powers) catalog. + +### Skills + +The Databricks agent skills come from [databricks/databricks-agent-skills](https://github.com/databricks/databricks-agent-skills) and are installed by the Databricks CLI, which is subject to the [Databricks License](https://github.com/databricks/cli?tab=License-1-ov-file#readme) and [Databricks Privacy Notice](https://www.databricks.com/legal/privacynotice). The MLflow skills come from [mlflow/skills](https://github.com/mlflow/skills) under that repository's license. + +### Trademarks + +"Databricks", "Unity Catalog", "Delta Lake", "Agent Bricks", "Genie", "Lakebase", "MLflow", "Databricks Apps", and other Databricks product names referenced in this Power are trademarks of Databricks, Inc. + +"Amazon Web Services", "AWS", and "Kiro" are trademarks of Amazon.com, Inc. or its affiliates. + +### Contributors + +- **Antony Prasad Thevaraj** ([@antonyprasad-db](https://github.com/antonyprasad-db)), Senior Specialist Solutions Architect, Databricks (AWS Partnership). Co-authored Kiro support in `databricks aitools` ([databricks/cli#6908](https://github.com/databricks/cli/pull/6908)) and maintains this Power. +- **Venkatavaradhan Viswanathan** ([@venkatavaradhanv](https://github.com/venkatavaradhanv)), Senior Solutions Architect, AWS. Created the original wrapper Power at [venkatavaradhanv/databricks](https://github.com/venkatavaradhanv/databricks). + +### Support + +- **Power packaging or installation** — [github.com/kirodotdev/powers/issues](https://github.com/kirodotdev/powers/issues) +- **Databricks agent skills** — [github.com/databricks/databricks-agent-skills/issues](https://github.com/databricks/databricks-agent-skills/issues) +- **Databricks CLI and `databricks aitools`** — [github.com/databricks/cli/issues](https://github.com/databricks/cli/issues) +- **Databricks platform support** — [help.databricks.com](https://help.databricks.com) diff --git a/databricks/steering/README.md b/databricks/steering/README.md deleted file mode 100644 index c390c4ae..00000000 --- a/databricks/steering/README.md +++ /dev/null @@ -1,63 +0,0 @@ -# Databricks Power — Steering Files - -This directory holds expert skill files that provide guidance for Databricks capabilities. Skills are loaded on-demand when the Power is activated. - -## How Skills Get Here - -Skills are installed by the official Databricks AI Dev Kit installer during onboarding: - -```bash -bash <(curl -sL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/install.sh) --tools kiro --global --silent -``` - -The installer downloads skills from the [ai-dev-kit repository](https://github.com/databricks-solutions/ai-dev-kit/tree/main/databricks-skills) and places them in this directory. - -## Available Skill Categories - -| Skill | Description | -|-------|-------------| -| databricks-spark-declarative-pipelines | Streaming tables, CDC, SCD Type 2, Auto Loader | -| databricks-jobs | Scheduled workflows, multi-task DAGs, serverless | -| databricks-aibi-dashboards | Interactive visualizations, KPIs, analytics | -| databricks-unity-catalog | Tables, volumes, governance, system tables | -| databricks-genie | Natural language data exploration | -| databricks-agent-bricks | Knowledge Assistants, Supervisor Agents | -| databricks-vector-search | Semantic search and RAG applications | -| databricks-model-serving | Deploy ML models and AI agents | -| databricks-mlflow-evaluation | Model evaluation, scoring, traces | -| databricks-apps-python | Full-stack web applications | -| databricks-lakebase-provisioned | Managed PostgreSQL (provisioned) | -| databricks-lakebase-autoscale | Managed PostgreSQL (autoscale) | -| databricks-metric-views | Reusable business metrics | -| databricks-spark-structured-streaming | Kafka, stateful ops, stream joins | -| databricks-dbsql | SQL best practices, materialized views | -| databricks-iceberg | Iceberg tables, UniForm, interop | -| databricks-bundles | Databricks Asset Bundles | -| databricks-config | Workspace configuration | -| databricks-python-sdk | Python SDK patterns and examples | -| databricks-synthetic-data-gen | Synthetic data generation | -| databricks-zerobus-ingest | Zero-copy ingestion | -| databricks-execution-compute | Code execution + cluster / warehouse lifecycle | -| databricks-ai-functions | AI functions, forecasting, document processing | -| databricks-app-apx | App frontend/backend patterns | -| databricks-unstructured-pdf-generation | PDF generation and upload | -| databricks-docs | Documentation search | -| mlflow-onboarding | MLflow getting started | -| instrumenting-with-mlflow-tracing | MLflow tracing instrumentation | -| querying-mlflow-metrics | Querying MLflow metrics | -| retrieving-mlflow-traces | Retrieving MLflow traces | -| searching-mlflow-docs | Searching MLflow documentation | -| analyze-mlflow-trace | Analyzing MLflow traces | -| analyze-mlflow-chat-session | Analyzing MLflow chat sessions | -| agent-evaluation | Agent evaluation patterns | -| spark-python-data-source | Custom Spark data sources | - -## Skill Loading - -This Power installs **all 34+ skills** (`--skills-profile all`). Skills load on demand based on the task at hand — persona-specific subsets are not used. - -## Updating Skills - -```bash -bash <(curl -sL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/install.sh) --tools kiro --global --skills-only --force --silent -```