From 55b568f829545e508a404e1ee9daa3be642ff4a7 Mon Sep 17 00:00:00 2001 From: Ethan Jackson Date: Wed, 26 Aug 2026 14:48:50 -0400 Subject: [PATCH 1/2] concierge context updates --- .../boc_rate_decisions/99_starter_agent.ipynb | 96 ++++- .../context/artifacts/README.md.md | 6 +- ...aieng__forecasting__langfuse_tracing.py.md | 55 +++ ..._aieng__forecasting__methods__README.md.md | 2 +- ...ing__methods__agentic__agent_factory.py.md | 280 +++++++++---- .../artifacts/implementations__README.md.md | 4 +- ..._rate_decisions__99_starter_agent.ipynb.md | 2 +- ...nt__skills__research-playbook__SKILL.md.md | 6 +- ...casting__03_one_agent_three_tasks.ipynb.md | 391 ++++++++++++++++-- ...sting__05_adaptive_agent_training.ipynb.md | 60 +-- ...il_forecasting__06_protected_eval.ipynb.md | 2 +- ...ions__energy_oil_forecasting__README.md.md | 14 +- ...il_forecasting__analyst_agent__agent.py.md | 27 +- ...nt__skills__research-playbook__SKILL.md.md | 6 +- ...tions__energy_oil_forecasting__tasks.py.md | 82 ++-- ...nt__skills__research-playbook__SKILL.md.md | 6 +- ...etting_started__99_repo_concierge.ipynb.md | 4 +- ...ementations__getting_started__README.md.md | 4 +- ...nt__skills__research-playbook__SKILL.md.md | 6 +- .../concierge_agent/context/catalog.yaml | 44 +- .../references/catalog-summary.yaml | 4 +- 21 files changed, 863 insertions(+), 238 deletions(-) diff --git a/implementations/boc_rate_decisions/99_starter_agent.ipynb b/implementations/boc_rate_decisions/99_starter_agent.ipynb index 82f9ceb5..0332d2b3 100644 --- a/implementations/boc_rate_decisions/99_starter_agent.ipynb +++ b/implementations/boc_rate_decisions/99_starter_agent.ipynb @@ -20,10 +20,18 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 5, "id": "cell-01", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "RUN_AGENT = True | model = gemini-3.1-flash-lite-preview\n" + ] + } + ], "source": [ "import warnings\n", "from pathlib import Path\n", @@ -48,7 +56,7 @@ "# ── Run guard ──────────────────────────────────────\n", "# Live agent calls cost tokens and need PROXY_* in the repo-root .env, plus warm\n", "# data caches. Default False so `Run All` is safe; set True to call the model.\n", - "RUN_AGENT = False\n", + "RUN_AGENT = True\n", "\n", "from boc_rate_decisions.starter_agent import (\n", " build_starter_agent_config,\n", @@ -72,10 +80,32 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 6, "id": "cell-03", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Agent: boc_starter_agent\n", + "Search enabled: True\n", + "Code-exec enabled: False\n", + "Skills loaded: ['forecasting', 'research-playbook']\n", + "\n", + "── System instruction (edit this in starter_agent/agent.py) ──\n", + "\n", + "## Role\n", + "\n", + "You are a Bank of Canada monetary-policy analyst — fluent in the policy-rate path, the 2% CPI inflation target, labour-market and bond-market conditions, and the Bank's institutional behaviour (gradualism, data dependence, reluctance to surprise markets). This is a starter agent: keep your reasoning transparent and your claims honest.\n", + "\n", + "## How to respond\n", + "\n", + "- For open-ended questions, scenario analysis, or anything conversational, answer directly and concisely — do NOT ask for a JSON payload.\n", + "- When you are handed a task that asks for a structured probability distribution over the next decision, produce a calibrated one. ...\n" + ] + } + ], "source": [ "config = build_starter_agent_config(\n", " model=AGENT_MODEL,\n", @@ -104,10 +134,40 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 7, "id": "cell-05", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Following the Bank of Canada’s (BoC) decision on October 23, 2024, to cut the policy rate by 50 basis points to 3.75%, the Governing Council has shifted toward a more balanced, data-dependent approach. The next decision is scheduled for December 11, 2024.\n", + "\n", + "### The Case for a Cut\n", + "* **Preventing Undershooting:** With inflation at 1.6% (September data), it has dipped below the 2% target. A further cut helps pull inflation back toward the midpoint, ensuring it doesn't drift persistently lower in an economy with excess supply.\n", + "* **Supporting Growth:** The economy remains in excess supply, and the labour market is soft, particularly for youth and newcomers. Continued normalization is viewed as necessary to provide support to aggregate demand.\n", + "\n", + "### The Case for a Hold\n", + "* **Assessment of Transmission:** After an \"oversized\" 50bp move, the Bank may prefer to pause to evaluate the impact of recent easing on household spending and business investment before committing to further aggressive moves.\n", + "* **Gradualism:** The Bank typically avoids back-to-back oversized cuts unless economic data signals a rapid deterioration. A 25bp cut or a hold is more consistent with the Bank’s historical preference for predictable, incremental policy adjustments.\n", + "\n", + "### Conclusion: What is more likely?\n", + "**A 25-basis-point cut is the most likely outcome.**\n", + "\n", + "While the 50bp move in October was a strong signal of the Bank’s intent to normalize policy quickly, a 25bp move in December allows the Bank to maintain the momentum of easing while exercising the caution typical of their \"data-dependent\" framework. A hold is less likely unless incoming data (particularly jobs and CPI) shows a surprising and sudden stabilization that negates the need for further stimulus. \n", + "\n", + "**Summary:** The Governing Council's focus has transitioned from fighting high inflation to supporting the economy toward a soft landing; they are firmly in an \"easing cycle,\" making a cut more probable than a hold.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Root node boc_starter_agent was cancelled.\n" + ] + } + ], "source": [ "from aieng.forecasting.methods.agentic import build_adk_agent\n", "from aieng.forecasting.methods.agentic.adk_runner import AdkTextRunner, AdkTextRunnerConfig\n", @@ -140,10 +200,26 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 8, "id": "cell-07", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Decision 2026-04-29 forecast from as_of=2026-04-01 (T-28)\n", + "Actual outcome: HOLD\n", + "\n", + " outcome agent prob climatology\n", + " cut 25.00% 10.87%\n", + " hold 70.00% 76.09% <- ACTUAL\n", + " hike 5.00% 13.04%\n", + "\n", + "Agent put 70% on what happened (its top pick ✓).\n" + ] + } + ], "source": [ "from datetime import datetime, timezone\n", "\n", @@ -238,7 +314,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.12.3" + "version": "3.12.12" } }, "nbformat": 4, diff --git a/implementations/getting_started/concierge_agent/context/artifacts/README.md.md b/implementations/getting_started/concierge_agent/context/artifacts/README.md.md index 213fb6e7..2357dd5c 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/README.md.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/README.md.md @@ -27,7 +27,7 @@ Also in this README: [Setup](#setup) · [Core concepts](#core-concepts) · [Repo - **Core library** — `aieng-forecasting` (`aieng.forecasting`): data services, cutoff enforcement, forecasting tasks, prediction payloads, backtesting, evaluation, and artifacts. - **Reusable methods** — `aieng.forecasting.methods`: `Predictor` implementations including naive baselines (continuous, binary, and categorical), Darts numerical predictors, LLM-process predictors (continuous, binary-probability, and categorical-probability), and ADK-based agentic infrastructure (`build_adk_agent`, `AdkTextRunner`, `AgentPredictor`). - **Reference implementations** — `implementations//`: notebooks, helper modules, task-specific configuration, and co-located YAML specs. -- **Tracing** — Langfuse / OpenTelemetry bootstrap (`aieng.forecasting.langfuse_tracing`) for LiteLLM and Google ADK. +- **Tracing** — Langfuse / OpenTelemetry bootstrap (`aieng.forecasting.langfuse_tracing`) for LiteLLM and Google ADK. Agent `search_web` calls nest inner search and leakage-verifier generations in the same trace. - **Data scripts** — `scripts/`: one fetch script per data source, plus `build_e2b_template.py` for the agentic code-execution sandbox. ## Two ways to use a forecaster @@ -111,6 +111,8 @@ When you open a **Coder workspace**, startup runs automatically in the backgroun **Your next step:** run [`00_environment_check.ipynb`](implementations/getting_started/00_environment_check.ipynb) top to bottom. That notebook will confirm that startup succeeded. +**ADK web UI.** [Guide 5](guides/05-access-adk-web-via-ssh-tunnel.md) is how you serve the concierge (or any other bootcamp agent) in the browser. On Coder, `adk web` binds to `localhost` *inside* the workspace — the same guide tunnels that port to your laptop (macOS, Windows, and Linux). Skip the tunnel half if you are running the repo locally. + On first boot, keys are verified against live services and your onboarding status is recorded. Workspace restarts reload keys without re-running the full test suite. **Local machine or troubleshooting** — fetch and verify keys manually: @@ -207,7 +209,7 @@ uv run pre-commit run --all-files ## Documentation - Per-implementation READMEs under [`implementations/`](implementations/) — the primary user surface. -- [`guides/`](guides/) — self-contained, step-by-step strategy guides for the most common build-phase tasks: onboarding a dataset, creating an experiment, customizing an agent's strategy, and auditing a result before you believe it. +- [`guides/`](guides/) — self-contained, step-by-step strategy guides for the most common build-phase tasks: onboarding a dataset, creating an experiment, customizing an agent's strategy, and auditing a result before you believe it. [Guide 5](guides/05-access-adk-web-via-ssh-tunnel.md) is how you serve the concierge or any bootcamp agent under `adk web` (and tunnel that UI from Coder to your laptop). - [Architecture atlas](https://vectorinstitute.github.io/agentic-forecasting/architecture-atlas.html) ([source](docs/architecture-atlas.html)) — a self-contained visual atlas of the system architecture: the loop, the temporal fence, predictor families, the harness, agent anatomy, and how each reference implementation instantiates them. - [`aieng-forecasting/README.md`](aieng-forecasting/README.md) and [`aieng-forecasting/aieng/forecasting/methods/README.md`](aieng-forecasting/aieng/forecasting/methods/README.md) — the library and the method catalog. - [`planning-docs/roadmap.md`](planning-docs/roadmap.md) — architecture principles and extension ideas. diff --git a/implementations/getting_started/concierge_agent/context/artifacts/aieng-forecasting__aieng__forecasting__langfuse_tracing.py.md b/implementations/getting_started/concierge_agent/context/artifacts/aieng-forecasting__aieng__forecasting__langfuse_tracing.py.md index 3aa109fd..66c511d7 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/aieng-forecasting__aieng__forecasting__langfuse_tracing.py.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/aieng-forecasting__aieng__forecasting__langfuse_tracing.py.md @@ -9,6 +9,10 @@ Call :func:`init_langfuse_tracing` once at process startup when using the ``llm`` or ``agentic`` extras and Langfuse credentials are set in the environment. +Inner LiteLLM calls that ADK OpenInference does not see (``search_web``'s +grounded search and leakage verifier) should wrap with +:func:`langfuse_generation` so they nest under the active agent trace. + Call :func:`print_langfuse_trace_url` after a ``predict()`` call to flush pending spans and print a clickable Langfuse UI link. """ @@ -17,11 +21,20 @@ from __future__ import annotations import logging import os +from contextlib import contextmanager +from typing import Any, Iterator logger = logging.getLogger(__name__) +class _NoOpObservation: + """Stand-in when Langfuse is unavailable so callers can always ``.update()``.""" + + def update(self, **_kwargs: Any) -> None: + return None + + def _langfuse_credentials_present() -> bool: pub = os.environ.get("LANGFUSE_PUBLIC_KEY", "").strip() sec = os.environ.get("LANGFUSE_SECRET_KEY", "").strip() @@ -93,6 +106,48 @@ class _LangfuseTracingBootstrap: _bootstrap = _LangfuseTracingBootstrap() +@contextmanager +def langfuse_generation( + name: str, + *, + model: str | None = None, + input: Any = None, # noqa: A002 — matches Langfuse observation field name + metadata: dict[str, Any] | None = None, +) -> Iterator[Any]: + """Open a Langfuse generation nested under the current observation. + + Used for inner LiteLLM calls that ADK OpenInference does not see (the + ``search_web`` googleSearch completion and the independent leakage + verifier). When an ADK tool span is already active, the new generation + becomes its child, so verifier traces show up inside the agent tree + rather than as a separate root. + + No-op when credentials are absent or the SDK raises, so tool code can + wrap completions without a tracing extra. + """ + if not _langfuse_credentials_present(): + yield _NoOpObservation() + return + try: + from langfuse import get_client # noqa: PLC0415 + + kwargs: dict[str, Any] = {"name": name, "as_type": "generation"} + if model is not None: + kwargs["model"] = model + if input is not None: + kwargs["input"] = input + if metadata: + kwargs["metadata"] = metadata + observation_cm = get_client().start_as_current_observation(**kwargs) + except Exception: + logger.debug("langfuse_generation(%s) failed; continuing without a span.", name, exc_info=True) + yield _NoOpObservation() + return + + with observation_cm as generation: + yield generation + + def init_langfuse_tracing() -> None: """Wire LiteLLM and Google ADK to Langfuse. diff --git a/implementations/getting_started/concierge_agent/context/artifacts/aieng-forecasting__aieng__forecasting__methods__README.md.md b/implementations/getting_started/concierge_agent/context/artifacts/aieng-forecasting__aieng__forecasting__methods__README.md.md index e7beeec3..d6b555d2 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/aieng-forecasting__aieng__forecasting__methods__README.md.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/aieng-forecasting__aieng__forecasting__methods__README.md.md @@ -108,7 +108,7 @@ from aieng.forecasting.methods.agentic import ( |---|---|---| | `agentic/adk_runner.py` | `AdkTextRunner` | Async text-in / text-out wrapper around ADK `InMemoryRunner`. Manages ADK sessions (fresh-per-message or sticky) and optionally traces each turn to Langfuse via `propagate_attributes`. | | `agentic/adk_runner.py` | `AdkTextRunnerConfig` | Pydantic configuration for `AdkTextRunner` (session mode, Langfuse fields). | -| `agentic/agent_factory.py` | `build_adk_agent` | Generic ADK `LlmAgent` factory with optional code execution, context retrieval, skills, generation controls, and structured output schema. | +| `agentic/agent_factory.py` | `build_adk_agent` | Generic ADK `LlmAgent` factory with optional code execution, context retrieval, skills, generation controls, and structured output schema. `search_web` runs a cutoff-aware googleSearch sub-call plus an independent leakage verifier on historical origins (skipped when `as_of` is today or later); both inner LLM calls emit nested Langfuse generations (`search_web.google_search`, `search_web.leakage_verifier`) under the agent trace. | | `agentic/agent_factory.py` | `AgentConfig` | Pydantic configuration for reusable ADK agents. `output_schema=None` supports interactive/free-form agents; a structured `AgentForecastOutput` schema supports Track 1 predictors. The `function_tools` field attaches conventional ADK tools (e.g. `ForecastTool`). Use-case-specific prompts and presets should live in `implementations//`. | | `agentic/forecast_tool.py` | `ForecastTool` | Conventional ADK `FunctionTool` that runs a pre-specified `Predictor` (AutoARIMA by default) on any registered series at a given cutoff/horizon, returning a structured JSON forecast. A controlled, reproducible alternative to open-ended code execution; series data never enters the LLM context. | | `agentic/outputs.py` | `AgentForecastOutput` | Abstract output adapter interface for converting structured agent JSON into evaluation `Prediction` objects. | diff --git a/implementations/getting_started/concierge_agent/context/artifacts/aieng-forecasting__aieng__forecasting__methods__agentic__agent_factory.py.md b/implementations/getting_started/concierge_agent/context/artifacts/aieng-forecasting__aieng__forecasting__methods__agentic__agent_factory.py.md index be9c2a65..bbb4ec3a 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/aieng-forecasting__aieng__forecasting__methods__agentic__agent_factory.py.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/aieng-forecasting__aieng__forecasting__methods__agentic__agent_factory.py.md @@ -21,6 +21,7 @@ import json import logging import os import warnings +from datetime import date, datetime, timezone from pathlib import Path from typing import Any, Callable, Sequence @@ -129,19 +130,27 @@ class ContextRetrievalConfig(BaseModel): the calling agent can retrieve grounded, sourced web context without a direct Gemini API key. - Temporal cutoff enforcement has two layers. The first is soft - (LLM-judgment-based): when ``enforce_cutoff`` is ``True`` and the calling - agent passes a ``cutoff_date`` to the tool, the inner proxy prompt - explicitly asks the model to exclude post-cutoff sources. This alone is - not a hard guarantee — backtests have shown it leak real post-cutoff - information despite the instruction. The second, hard layer is an - independent verifier call (see ``verifier_model`` etc. below): a separate - LLM call extracts and judges each factual claim in the search result - against the cutoff, strips violations, and retries the search with - feedback when it cannot produce a sufficiently confident result — + Temporal cutoff enforcement has two layers, and both run only when the + effective cutoff is a *retrospective* date (strictly before UTC today). + Live origins (cutoff on or after today) skip the fence so current news + is not stripped; set ``enforce_cutoff=False`` to skip it even on + historical dates. The first layer is soft (LLM-judgment-based): the + inner proxy prompt asks the model to exclude post-cutoff sources. This + alone is not a hard guarantee — backtests have shown it leak real + post-cutoff information despite the instruction. The second, hard layer + is an independent verifier call (see ``verifier_model`` etc. below): a + separate LLM call extracts and judges each factual claim in the search + result against the cutoff, strips violations, and retries the search + with feedback when it cannot produce a sufficiently confident result — returning an explicit failure sentinel rather than silently risky content if verification never succeeds within the attempt budget. + Both inner LLM calls (grounded search and verifier) emit Langfuse + generations named ``search_web.google_search`` and + ``search_web.leakage_verifier`` when tracing is configured, nested + under the ADK ``search_web`` tool span so they appear in the same + agent trace. + Attributes ---------- enabled : bool, default=False @@ -155,11 +164,12 @@ class ContextRetrievalConfig(BaseModel): when ``enabled`` is ``True``. enforce_cutoff : bool, default=True When ``True``, the ``search_web`` tool appends a cutoff-date - constraint to the user prompt whenever ``cutoff_date`` is supplied by - the calling agent, and runs the independent leakage verifier - described above. Set to ``False`` for live (non-backtest) agents - where no temporal fence is needed — the verifier is skipped entirely - in that case, at zero extra cost. + constraint and runs the independent leakage verifier whenever the + effective cutoff (harness ``as_of``, else the LLM-supplied + ``cutoff_date``) is strictly before UTC today. Cutoffs on or after + today are treated as live and skip both layers automatically. Set + to ``False`` to skip the fence even on historical dates, at zero + extra verifier cost. temperature : float | None, default=None Sampling temperature for the inner search call. max_output_tokens : int | None, default=None @@ -275,6 +285,69 @@ def _build_leakage_verification_schema() -> dict[str, Any]: } +def _utc_today() -> date: + """Calendar date in UTC; used to decide whether a cutoff is retrospective.""" + return datetime.now(timezone.utc).date() + + +def _is_retrospective_cutoff(cutoff: str) -> bool: + """Return True when *cutoff* is a calendar date strictly before UTC today. + + Live origins (today or a future date) should not run the leakage fence: + there is nothing post-cutoff to leak, and the verifier would strip + current news. Unparseable values are treated as retrospective so a + malformed date cannot silently disable the guard. + """ + raw = cutoff.strip()[:10] + try: + cutoff_d = date.fromisoformat(raw) + except ValueError: + return True + return cutoff_d < _utc_today() + + +def _verification_skip_reason(effective_cutoff: str | None, *, enforce_cutoff: bool) -> str | None: + """Why cutoff enforcement is skipped, or ``None`` when the verifier should run.""" + if not effective_cutoff: + return "no_cutoff" + if not enforce_cutoff: + return "enforce_cutoff_disabled" + if not _is_retrospective_cutoff(effective_cutoff): + return "live_as_of" + return None + + +def _usage_from_litellm(resp: Any) -> dict[str, int]: + """Map LiteLLM token counts into Langfuse ``usage_details``. + + Keys must be ``input`` / ``output`` (optionally ``input_cached_tokens``). + Those are the usage types on Langfuse's model price table. Sending + ``input_tokens`` / ``output_tokens`` still shows counts in the UI but + matches no price, so ``totalCost`` stays null — which is what happened + on ``search_web.google_search`` and ``search_web.leakage_verifier`` + until this mapping was aligned with the LLM-process path and ADK + OpenInference. + """ + usage = getattr(resp, "usage", None) + if usage is None: + return {} + try: + in_tok = int(getattr(usage, "prompt_tokens", 0) or 0) + out_tok = int(getattr(usage, "completion_tokens", 0) or 0) + except (TypeError, ValueError): + return {} + details: dict[str, int] = {"input": in_tok, "output": out_tok} + prompt_details = getattr(usage, "prompt_tokens_details", None) + if prompt_details is not None: + try: + cached = int(getattr(prompt_details, "cached_tokens", 0) or 0) + except (TypeError, ValueError): + cached = 0 + if cached: + details["input_cached_tokens"] = cached + return details + + _LEAKAGE_VERIFIER_INSTRUCTION = """\ You are an independent fact-checker verifying that a web search result contains \ no information published on or after a given cutoff date. @@ -302,6 +375,7 @@ async def _verify_no_leakage( verifier_model: str, openai_base_url: str, openai_api_key: str | None, + trace_metadata: dict[str, Any] | None = None, ) -> _LeakageVerification: """Judge a search result for post-cutoff claims via an independent verifier call. @@ -312,39 +386,62 @@ async def _verify_no_leakage( it consumes a retry attempt like any other rejection. """ import litellm # noqa: PLC0415 + from aieng.forecasting.langfuse_tracing import langfuse_generation # noqa: PLC0415 from aieng.forecasting.methods.llm_processes._client import ( # noqa: PLC0415 make_json_schema_response_format, strip_markdown_fence, ) model = verifier_model if verifier_model.startswith("openai/") else f"openai/{verifier_model}" - resp = await litellm.acompletion( - model=model, - api_base=openai_base_url, - api_key=openai_api_key, - messages=[ - {"role": "system", "content": _LEAKAGE_VERIFIER_INSTRUCTION}, - { - "role": "user", - "content": f"Original query: {query}\nCutoff date: {cutoff_date}\n\nSearch result to verify:\n{text}", - }, - ], - response_format=make_json_schema_response_format("LeakageVerification", _build_leakage_verification_schema()), - temperature=0.0, - max_tokens=2048, - timeout=60.0, - ) - raw = resp.choices[0].message.content or "{}" - try: - return _LeakageVerification.model_validate(json.loads(strip_markdown_fence(raw))) - except (json.JSONDecodeError, ValidationError): - logger.warning("Leakage verifier returned unparseable output; treating as non-clean: %r", raw[:200]) - return _LeakageVerification( - flagged_claims=["verifier response could not be parsed"], - filtered_text=text, - confidence=1, - clean=False, + messages = [ + {"role": "system", "content": _LEAKAGE_VERIFIER_INSTRUCTION}, + { + "role": "user", + "content": f"Original query: {query}\nCutoff date: {cutoff_date}\n\nSearch result to verify:\n{text}", + }, + ] + with langfuse_generation( + name="search_web.leakage_verifier", + model=verifier_model, + input={"messages": messages}, + metadata=trace_metadata, + ) as generation: + resp = await litellm.acompletion( + model=model, + api_base=openai_base_url, + api_key=openai_api_key, + messages=messages, + response_format=make_json_schema_response_format( + "LeakageVerification", _build_leakage_verification_schema() + ), + temperature=0.0, + max_tokens=2048, + timeout=60.0, ) + raw = resp.choices[0].message.content or "{}" + try: + verdict = _LeakageVerification.model_validate(json.loads(strip_markdown_fence(raw))) + except (json.JSONDecodeError, ValidationError): + logger.warning("Leakage verifier returned unparseable output; treating as non-clean: %r", raw[:200]) + verdict = _LeakageVerification( + flagged_claims=["verifier response could not be parsed"], + filtered_text=text, + confidence=1, + clean=False, + ) + update: dict[str, Any] = { + "output": { + "clean": verdict.clean, + "confidence": verdict.confidence, + "flagged_claims": verdict.flagged_claims, + "filtered_text": verdict.filtered_text, + }, + } + usage = _usage_from_litellm(resp) + if usage: + update["usage_details"] = usage + generation.update(**update) + return verdict def _build_search_tool( @@ -360,14 +457,15 @@ def _build_search_tool( server-side grounding and returns a synthesised answer plus source URLs extracted from ``choices[0].provider_specific_fields["grounding_metadata"]``. - When a ``cutoff_date`` is supplied and ``config.enforce_cutoff`` is - ``True``, the raw result is passed through an independent leakage - verifier (:func:`_verify_no_leakage`) before being returned. On a flagged - result, the search is retried (up to ``config.verifier_max_attempts`` - times) with the previously flagged claims injected as explicit negative - feedback. If no attempt is verified clean, an explicit - ``[SEARCH_VERIFICATION_FAILED]`` sentinel is returned instead of - potentially-leaky content. + When a retrospective ``cutoff_date`` (or harness ``as_of``) is present + and ``config.enforce_cutoff`` is ``True``, the raw result is passed + through an independent leakage verifier (:func:`_verify_no_leakage`) + before being returned. Cutoffs on or after UTC today skip the fence + (live search). On a flagged result, the search is retried (up to + ``config.verifier_max_attempts`` times) with the previously flagged + claims injected as explicit negative feedback. If no attempt is verified + clean, an explicit ``[SEARCH_VERIFICATION_FAILED]`` sentinel is returned + instead of potentially-leaky content. """ def _format_result(content: str, sources: list[str]) -> str: @@ -375,32 +473,47 @@ def _build_search_tool( content += "\n\nSources:\n" + "\n".join(sources[:5]) return content - async def _do_search(user_content: str) -> tuple[str, list[str]]: + async def _do_search(user_content: str, *, trace_metadata: dict[str, Any] | None = None) -> tuple[str, list[str]]: import litellm # noqa: PLC0415 + from aieng.forecasting.langfuse_tracing import langfuse_generation # noqa: PLC0415 search_model = config.search_model if not search_model.startswith("openai/"): search_model = f"openai/{search_model}" - resp = await litellm.acompletion( - model=search_model, - api_base=openai_base_url, - api_key=openai_api_key, - messages=[ - {"role": "system", "content": config.instruction}, - {"role": "user", "content": user_content}, - ], - tools=[{"googleSearch": {}}], - max_tokens=config.max_output_tokens or 4096, - temperature=config.temperature or 0.0, - timeout=60.0, - ) - content = resp.choices[0].message.content or "" - psf = getattr(resp.choices[0], "provider_specific_fields", {}) or {} - gm = psf.get("grounding_metadata") or {} - sources: list[str] = [ - uri for c in gm.get("groundingChunks", []) if (uri := (c.get("web") or {}).get("uri")) is not None + messages = [ + {"role": "system", "content": config.instruction}, + {"role": "user", "content": user_content}, ] - return content, sources + with langfuse_generation( + name="search_web.google_search", + model=config.search_model, + input={"messages": messages}, + metadata=trace_metadata, + ) as generation: + resp = await litellm.acompletion( + model=search_model, + api_base=openai_base_url, + api_key=openai_api_key, + messages=messages, + tools=[{"googleSearch": {}}], + max_tokens=config.max_output_tokens or 4096, + temperature=config.temperature or 0.0, + timeout=60.0, + ) + content = resp.choices[0].message.content or "" + psf = getattr(resp.choices[0], "provider_specific_fields", {}) or {} + gm = psf.get("grounding_metadata") or {} + sources: list[str] = [ + uri for c in gm.get("groundingChunks", []) if (uri := (c.get("web") or {}).get("uri")) is not None + ] + update: dict[str, Any] = {"output": content} + usage = _usage_from_litellm(resp) + if usage: + update["usage_details"] = usage + extra_meta = {**(trace_metadata or {}), "source_count": len(sources)} + update["metadata"] = extra_meta + generation.update(**update) + return content, sources async def search_web(query: str, cutoff_date: str | None = None, tool_context: ToolContext | None = None) -> str: """Search the web and return a grounded summary with source URLs. @@ -428,6 +541,11 @@ def _build_search_tool( the calling LLM cannot see, omit, or alter it. This closes a bypass where the model simply didn't pass ``cutoff_date`` and both the soft cutoff instruction and the verifier below were silently skipped. + + Cutoff enforcement (soft prompt + verifier) runs only when that + effective date is strictly before UTC today. An ``as_of`` of today + or later is treated as a live origin and searched without a + temporal fence, matching ``enforce_cutoff=False``. """ harness_as_of = tool_context.state.get(AS_OF_STATE_KEY) if tool_context is not None else None if harness_as_of and cutoff_date and harness_as_of != cutoff_date: @@ -437,20 +555,33 @@ def _build_search_tool( harness_as_of, ) effective_cutoff = harness_as_of or cutoff_date - - needs_verification = bool(effective_cutoff and config.enforce_cutoff) - if not needs_verification: - content, sources = await _do_search(query) + skip_reason = _verification_skip_reason(effective_cutoff, enforce_cutoff=config.enforce_cutoff) + base_meta: dict[str, Any] = { + "effective_cutoff": effective_cutoff, + "harness_as_of": harness_as_of, + "llm_cutoff_date": cutoff_date, + } + if skip_reason is not None: + logger.info( + "search_web: skipping cutoff enforcement (%s); cutoff=%s", + skip_reason, + effective_cutoff, + ) + content, sources = await _do_search( + query, + trace_metadata={**base_meta, "verification_skipped": skip_reason}, + ) return _format_result(content, sources) negative_guidance = "" for attempt in range(1, config.verifier_max_attempts + 1): + attempt_meta = {**base_meta, "attempt": attempt, "verifier_max_attempts": config.verifier_max_attempts} user_content = ( query + f"\n\nOnly include and cite information published strictly before {effective_cutoff}." ) if negative_guidance: user_content += f"\n\n{negative_guidance}" - content, sources = await _do_search(user_content) + content, sources = await _do_search(user_content, trace_metadata=attempt_meta) verdict = await _verify_no_leakage( text=content, query=query, @@ -458,6 +589,7 @@ def _build_search_tool( verifier_model=config.verifier_model, openai_base_url=openai_base_url, openai_api_key=openai_api_key, + trace_metadata=attempt_meta, ) logger.info( "search_web verification attempt %d/%d: clean=%s confidence=%d flagged=%d", diff --git a/implementations/getting_started/concierge_agent/context/artifacts/implementations__README.md.md b/implementations/getting_started/concierge_agent/context/artifacts/implementations__README.md.md index 52ecf07e..c4cdb28c 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/implementations__README.md.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/implementations__README.md.md @@ -32,11 +32,11 @@ implementations/ `-- pyproject.toml # local workspace packaging ``` -YAML backtest and eval specs live under each use case in `specs/`. Each directory is independent; see its `README.md` for the walkthrough. For the build-phase moves — onboarding data, standing up an experiment, customizing an agent, auditing a result — see [`guides/`](../guides/). +YAML backtest and eval specs live under each use case in `specs/`. Each directory is independent; see its `README.md` for the walkthrough. For the build-phase moves — onboarding data, standing up an experiment, customizing an agent, auditing a result — see [`guides/`](../guides/). To chat with the concierge or a domain starter in the ADK browser UI, see [`guides/05-access-adk-web-via-ssh-tunnel.md`](../guides/05-access-adk-web-via-ssh-tunnel.md) (includes the Coder SSH tunnel). Every domain use case (all except `getting_started`) also ships a `starter_agent/` module and a `99_starter_agent.ipynb` — a fresh, hackable **starter agent** that is the consistent "build your own" entry point for that use case (toggleable news search + code execution, two lightweight tool-usage skills, an interactive cell, and one scored forecast). -`getting_started/` additionally ships a **`concierge_agent/`** module and **`99_repo_concierge.ipynb`** — a repo onboarding helper (not a forecaster) that answers questions about how the codebase works using a committed public-`main` knowledge digest. From the repository root: `uv run adk run implementations/getting_started/concierge_agent`. See [`getting_started/README.md`](getting_started/README.md) and the notebook for full usage. +`getting_started/` additionally ships a **`concierge_agent/`** module and **`99_repo_concierge.ipynb`** — a repo onboarding helper (not a forecaster) that answers questions about how the codebase works using a committed public-`main` knowledge digest. From the repository root: `uv run adk run implementations/getting_started/concierge_agent` (or `uv run adk web implementations/getting_started/concierge_agent` for the browser UI — [guide 5](../guides/05-access-adk-web-via-ssh-tunnel.md)). See [`getting_started/README.md`](getting_started/README.md) and the notebook for full usage. --- diff --git a/implementations/getting_started/concierge_agent/context/artifacts/implementations__boc_rate_decisions__99_starter_agent.ipynb.md b/implementations/getting_started/concierge_agent/context/artifacts/implementations__boc_rate_decisions__99_starter_agent.ipynb.md index b314bfe2..0b34803a 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/implementations__boc_rate_decisions__99_starter_agent.ipynb.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/implementations__boc_rate_decisions__99_starter_agent.ipynb.md @@ -42,7 +42,7 @@ AGENT_MODEL = "gemini-3.1-flash-lite-preview" # ── Run guard ────────────────────────────────────── # Live agent calls cost tokens and need PROXY_* in the repo-root .env, plus warm # data caches. Default False so `Run All` is safe; set True to call the model. -RUN_AGENT = False +RUN_AGENT = True from boc_rate_decisions.starter_agent import ( build_starter_agent_config, diff --git a/implementations/getting_started/concierge_agent/context/artifacts/implementations__boc_rate_decisions__starter_agent__skills__research-playbook__SKILL.md.md b/implementations/getting_started/concierge_agent/context/artifacts/implementations__boc_rate_decisions__starter_agent__skills__research-playbook__SKILL.md.md index d824d247..7151df28 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/implementations__boc_rate_decisions__starter_agent__skills__research-playbook__SKILL.md.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/implementations__boc_rate_decisions__starter_agent__skills__research-playbook__SKILL.md.md @@ -21,11 +21,13 @@ Always pass `cutoff_date` equal to the `as_of` date in your payload. It is the temporal fence that keeps post-origin information out of a historical forecast. A forecast that "knew" what happened after `as_of` is not a forecast. -`search_web` runs an independent verifier on every result and returns +On historical origins (strictly before today's date), `search_web` runs an +independent verifier on every result and returns `[SEARCH_VERIFICATION_FAILED]` instead of content it couldn't confirm as pre-cutoff. Treat that as no verified news for the query — proceed on your other signals and say so, never filling the gap from your own background -knowledge. +knowledge. On a live origin (`as_of` is today), the verifier is skipped so +current news is not stripped. ## How to search diff --git a/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__03_one_agent_three_tasks.ipynb.md b/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__03_one_agent_three_tasks.ipynb.md index bfd18f5d..a2b0c5a0 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__03_one_agent_three_tasks.ipynb.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__03_one_agent_three_tasks.ipynb.md @@ -9,14 +9,19 @@ kind: notebook > **Part 3 of 7.** This notebook builds on the agentic predictor introduced in > [`02_intro_agentic_predictor.ipynb`](02_intro_agentic_predictor.ipynb). -A single Analyst Agent — backed by bounded Google Search — answers three tasks -using **one system prompt** and **task-specific user payloads**: +**Identity vs role.** One Analyst Agent (system prompt + toolbelt) answers three +different questions. The identity is fixed; only the **task spec** in the user +payload changes: | Stream | Task | Output | |--------|------|--------| -| A | Trajectory | 5/10/21-day price forecasts | -| B | Binary shock | P(WTI +$5 in 5 days) | -| C | Scenario analysis | Top 3 expert scenarios for 60 days | +| 1 | Trajectory | 5/10/21-day price forecasts | +| 2 | Binary shock | P(WTI +$5 in 5 days) | +| 3 | Scenario analysis | Top 3 expert scenarios for 60 days | + +A **task spec** is the ask: the question, the rules, and the required JSON shape. +It is *not* the system prompt. Edit the identity strings once, then edit each +stream's task spec and re-run (keep `USE_CACHE = False` after edits). ## Cell 2 (code) @@ -39,16 +44,22 @@ AGENT_MODEL = "gemini-3.1-flash-lite-preview" # ── Cache control ───────────────────────────────────────────────────────────── # Set to False to force a full end-to-end agent run (ignores all cached results). +# Keep False if you edit the identity or any stream's task spec. USE_CACHE = False from aieng.forecasting.evaluation.task import ForecastingTask +from aieng.forecasting.methods.agentic import ( + AgentPredictor, + ContinuousAgentForecastOutput, + DiscreteAgentForecastOutput, +) from energy_oil_forecasting.analysis import compute_brier_score, trajectory_mae_table +from energy_oil_forecasting.analyst_agent import build_wti_multitask_news_config from energy_oil_forecasting.data import WTI_SERIES_ID, build_wti_service, naive_utc_now from energy_oil_forecasting.paths import ( PROPHET_SHOCK_TRAJ_CACHE, PROPHET_TRAJ_CACHE, SCENARIO_CACHE, - SCENARIO_ORIGIN, SHOCK_ANALYST_CACHE, SHOCK_HORIZON, SHOCK_ORIGINS, @@ -62,7 +73,10 @@ from energy_oil_forecasting.prophet_baseline import ( prophet_prob_shock, wti_series_to_price_df, ) -from energy_oil_forecasting.tasks import TASK_SPECS, build_wti_news_predictor +from energy_oil_forecasting.tasks import ( + ScenarioAgentForecastOutput, + WtiMultitaskPromptBuilder, +) from energy_oil_forecasting.viz import ( conf_bar, make_shock_comparison_chart, @@ -79,28 +93,193 @@ price_df = wti_series_to_price_df(ctx.get_series(WTI_SERIES_ID)) prophet_traj_df = load_prophet_trajectories(price_df, TRAJECTORY_ORIGINS, PROPHET_TRAJ_CACHE) prophet_shock_df = load_prophet_trajectories(price_df, SHOCK_ORIGINS, PROPHET_SHOCK_TRAJ_CACHE) print(f"Price history through {price_df.index[-1].date()}") + + +def preview_user_payload(builder: WtiMultitaskPromptBuilder, task: ForecastingTask, origin: pd.Timestamp) -> None: + """Show the JSON user payload the agent would receive (no model call).""" + as_of = origin - pd.Timedelta(days=1) + origin_ctx = data_service.context(as_of=as_of) + payload = json.loads(builder(task=task, context=origin_ctx)) + hist_lines = payload["target_history_csv"].splitlines() + ask_prose, _, ask_schema = payload["task_spec"].partition("Required JSON format:") + display( + Markdown( + f"### User payload preview " + f"*(as_of {payload['as_of']}, WTI ${payload['origin_price_usd_bbl']:.2f}/bbl)*\n\n" + "This is how we assign the task: the ask rides in `task_spec`; " + "horizons and quantiles come from the `ForecastingTask`.\n\n" + f"**Price history** — last 10 of {len(hist_lines) - 1} rows:\n\n" + "```\n" + "\n".join(hist_lines[-10:]) + "\n```\n\n" + f"**horizons:** `{payload['horizons']}` · " + f"**standard_quantiles:** {len(payload['standard_quantiles'])} levels\n\n" + f"**task_spec** ({len(payload['task_spec'])} chars) — prose:\n\n" + + ask_prose.strip() + + "\n\n**Required JSON format:**\n\n```json\n" + + ask_schema.strip() + + "\n```" + ) + ) ``` ## Cell 3 (markdown) --- -## Stream 1 — Trajectory Forecast +## Shared identity — system prompt + toolbelt -Compare Prophet fan charts to the news-grounded agent at three origins. +This is what the agent *is*. The same `analyst_config` is reused by all three streams. +Edit the strings below to change persona or search behaviour; do **not** put the +trajectory / shock / scenario ask here — that belongs in each stream's task spec. ## Cell 4 (code) ```python +# ── Editable identity (shared by Streams 1–3) ───────────────────────────────── +# Task-agnostic: persona + how to read the payload. The ask is NOT here. + +SYSTEM_INSTRUCTION = """ +## Role + +You are an expert WTI crude oil market analyst. + +## Input + +You will receive a JSON payload containing: +- `task_spec`: the exact question and required JSON output schema +- `as_of`: the forecast origin date (temporal cutoff) +- `horizons`: integer horizon steps (business days ahead) +- `standard_quantiles`: quantile levels for continuous forecasts (when applicable) +- `origin_price_usd_bbl`: WTI close on the origin date +- `target_history_csv`: compressed WTI daily close history + +When context retrieval is enabled, call ``search_web`` BEFORE answering. + +## Output contract + +Read the data (and briefing, if retrieved) carefully, then execute the task in `task_spec` precisely. + +If a `set_model_response` tool is available, call it with your complete JSON as `json_response` — the exact schema is described in `task_spec`. Otherwise return the JSON directly as plain text with no preamble. +""".strip() + +SEARCH_INSTRUCTION = """ +You are an oil market intelligence specialist with access to web search. + +Search for information relevant to the query and return a concise structured markdown summary (3-5 paragraphs) covering relevant aspects of: +- WTI/Brent crude price level and recent trend +- OPEC+ production decisions and supply outlook +- Geopolitical risks in the Persian Gulf, Middle East, key shipping lanes +- US Strategic Petroleum Reserve and energy policy signals +- Notable tanker/shipping incidents or supply disruption signals +- Published analyst forecasts or unusual price-target revisions + +Ground your summary in the search results you actually retrieve. When a cutoff date is specified, do not report or speculate about events that occurred after that date. + +Before finalizing your summary, reason step by step: (1) for each candidate fact, judge its actual recency from the substance of the result itself, never from a source's claimed publish date or byline timestamp — those are frequently stale or updated after original publication; (2) discard anything you cannot confidently place before the cutoff date; (3) only then write your summary. Do not supplement the search results with your own background/training knowledge — if the results are insufficient, say so explicitly rather than filling gaps from memory. +""".strip() + +_base = build_wti_multitask_news_config(model=AGENT_MODEL) +analyst_config = _base.model_copy( + update={ + "instruction": SYSTEM_INSTRUCTION, + "context_retrieval": _base.context_retrieval.model_copy(update={"instruction": SEARCH_INSTRUCTION}), + } +) + +cr = analyst_config.context_retrieval +display( + Markdown( + f"### Toolbelt inventory *(agent `{analyst_config.name}`, model `{analyst_config.model}`)*\n\n" + "| Capability | Status |\n|---|---|\n" + f"| `search_web` (context-retrieval sub-agent) | " + f"{'**on**' if cr.enabled else 'off'} — cutoff = payload `as_of` |\n" + f"| Search model | `{cr.search_model}` |\n" + f"| Temporal-leakage verifier | `{cr.verifier_model}` " + f"(max {cr.verifier_max_attempts} attempts, confidence ≥ {cr.verifier_confidence_threshold}) |\n" + f"| Skills | none (`skills_dirs` empty) |\n" + f"| Code execution | {'on' if analyst_config.code_execution.enabled else '**off**'} |\n" + f"| `run_forecast` / function tools | " + f"{'yes' if analyst_config.function_tools else '**none**'} |\n" + "| `set_model_response` | attached **per stream** via `output_schema`, not by identity |\n" + ) +) +display(Markdown("### System instruction\n\n```\n" + SYSTEM_INSTRUCTION + "\n```")) +display(Markdown("### Search sub-agent instruction\n\n```\n" + SEARCH_INSTRUCTION + "\n```")) +``` + +## Cell 5 (markdown) + +--- +## Stream 1 — Trajectory Forecast + +**Question:** Where will WTI be in 5, 10, and 21 business days? + +Same identity as above. The task spec below is the ask — edit horizons or rules, +then re-run (keep `USE_CACHE = False`). Compare Prophet fan charts to the +news-grounded agent at three origins. + +**Try this:** set `TRAJECTORY_HORIZONS = [5, 21]` and update the spec wording to match. + +## Cell 6 (code) + +```python +# ── Stream 1 task spec (edit this) ──────────────────────────────────────────── +TRAJECTORY_HORIZONS = [5, 10, 21] # feeds ForecastingTask; listed again in the ask + +_TRAJ_SCHEMA = ContinuousAgentForecastOutput.prompt_schema_json() +TRAJECTORY_TASK_SPEC = f"""Forecast the WTI crude oil price at each horizon listed in the payload +(`horizons`, business days ahead). Default horizons for this demo: {TRAJECTORY_HORIZONS}. + +Rules: + - Produce one forecast for each horizon in `horizons`. + - Use exactly the quantile levels from `standard_quantiles` — no additions, no omissions. + - `point_forecast` must exactly equal the 0.50 quantile value. + - Quantile values must be strictly non-decreasing as quantile levels increase. + - Document your reasoning in the `rationale` fields. + +If a `set_model_response` tool is available, call it with your complete JSON as `json_response`. Otherwise return the JSON directly as plain text. + +Required JSON format: +{_TRAJ_SCHEMA} +""" + +_prose, _, _schema = TRAJECTORY_TASK_SPEC.partition("Required JSON format:") +display( + Markdown( + "### Task spec — Stream 1\n\n" + + _prose.strip() + + "\n\n**Required JSON format** (`ContinuousAgentForecastOutput`):\n\n```json\n" + + _schema.strip() + + "\n```" + ) +) +``` + +## Cell 7 (code) + +```python +# ── Assign role: wire identity + task spec (no model call) ──────────────────── trajectory_task = ForecastingTask( task_id="wti_trajectory_demo", target_series_id=WTI_SERIES_ID, - horizons=[5, 10, 21], + horizons=list(TRAJECTORY_HORIZONS), frequency="B", description="Trajectory demo for NB3", ) +traj_prompt_builder = WtiMultitaskPromptBuilder(task_spec=TRAJECTORY_TASK_SPEC) +traj_predictor = AgentPredictor( + agent_config=analyst_config, + prompt_builder=traj_prompt_builder, + output_schema=ContinuousAgentForecastOutput, +) -traj_predictor = build_wti_news_predictor("trajectory", model=AGENT_MODEL) +print(f"Predictor schema: {traj_predictor.output_schema.__name__}") +preview_user_payload(traj_prompt_builder, trajectory_task, TRAJECTORY_ORIGINS[-1]) +``` +## Cell 8 (code) + +```python +# ── Run trajectory agent at three origins ───────────────────────────────────── +# Uses analyst_config + TRAJECTORY_TASK_SPEC. Keep USE_CACHE = False after edits. if USE_CACHE and TRAJ_AGENT_CACHE.exists(): with open(TRAJ_AGENT_CACHE) as f: traj_agent_results = json.load(f) @@ -121,17 +300,17 @@ else: json.dump(traj_agent_results, f, indent=2) print(f"Saved {len(traj_agent_results)} agent trajectory runs.") -# Summary: agent point forecasts at each origin print("\nAgent trajectory summary:") for r in traj_agent_results: preds = r["predictions"] - pts = [f"h{[5, 10, 21][i]}=${preds[i]['payload']['point_forecast']:.1f}" for i in range(len(preds))] + hs = TRAJECTORY_HORIZONS + pts = [f"h{hs[i]}=${preds[i]['payload']['point_forecast']:.1f}" for i in range(len(preds))] origin_price_rows = price_df[price_df.index >= pd.Timestamp(r["origin"])] origin_price = f"WTI=${origin_price_rows.iloc[0]['price']:.2f}" if not origin_price_rows.empty else "" print(f" {r['origin']} {origin_price} {' | '.join(pts)}") ``` -## Cell 5 (code) +## Cell 9 (code) ```python # ── I/O inspection: 2026-03-02 — conflict onset, most informative ──────────── @@ -140,7 +319,7 @@ inspect_rec = next((r for r in traj_agent_results if r["origin"] == INSPECT_ORIG if inspect_rec: origin_ts = pd.Timestamp(INSPECT_ORIGIN) - bday_dates = pd.bdate_range(start=origin_ts + pd.offsets.BDay(1), periods=21) + bday_dates = pd.bdate_range(start=origin_ts + pd.offsets.BDay(1), periods=max(TRAJECTORY_HORIZONS)) origin_price_row = price_df[price_df.index >= origin_ts] origin_price = float(origin_price_row.iloc[0]["price"]) if not origin_price_row.empty else float("nan") @@ -148,7 +327,7 @@ if inspect_rec: rationale = preds[0].get("metadata", {}).get("rationale", "") if preds else "" table_rows = "| Horizon | Agent ($) | 80% CI | Actual ($) | Agent err | Prophet err |\n|---|---|---|---|---|---|\n" - for i, h in enumerate([5, 10, 21]): + for i, h in enumerate(TRAJECTORY_HORIZONS): actual_rows = price_df[price_df.index >= bday_dates[h - 1]] actual = float(actual_rows.iloc[0]["price"]) if not actual_rows.empty else float("nan") pt = preds[i]["payload"]["point_forecast"] @@ -175,7 +354,7 @@ if inspect_rec: ) ``` -## Cell 6 (code) +## Cell 10 (code) ```python # ── Trajectory fan chart: Prophet fan vs agent error bars at 3 origins ─────── @@ -190,14 +369,58 @@ if not mae_df.empty: print(f"\nMean MAE Prophet: ${mean_mae['Prophet MAE']:.2f} Agent: ${mean_mae['Agent MAE']:.2f}") ``` -## Cell 7 (markdown) +## Cell 11 (markdown) --- ## Stream 2 — Binary Shock Prediction -## Cell 8 (code) +**Question:** What is P(WTI closes more than $5/bbl higher in 5 trading days)? + +Same identity. A different task spec. Edit the threshold or horizon wording below — +if you change the scored definition, also update `SHOCK_THRESHOLD` / `SHOCK_HORIZON` +so the scorer stays aligned. + +**Try this:** raise the bar to +$10 and compare probabilities. + +## Cell 12 (code) + +```python +# ── Stream 2 task spec (edit this) ──────────────────────────────────────────── +# Scorer uses SHOCK_THRESHOLD / SHOCK_HORIZON from paths.py — keep them in sync. +_SHOCK_SCHEMA = DiscreteAgentForecastOutput.prompt_schema_json() +SHOCK_TASK_SPEC = f"""Estimate P(up) — the probability that WTI will close MORE THAN +${int(SHOCK_THRESHOLD)}/bbl HIGHER than today's price at the end of +{SHOCK_HORIZON} trading days. + +This is a directional upside question only. + +Calibration guidance: + - No unusual upside catalyst -> base rate ~10-15% + - Escalating unconfirmed risk -> 20-40% + - Confirmed supply disruption -> 60-85% + +If a `set_model_response` tool is available, call it with your complete JSON as `json_response`. Otherwise return the JSON directly as plain text. + +Required JSON format: +{_SHOCK_SCHEMA} +""" + +_prose, _, _schema = SHOCK_TASK_SPEC.partition("Required JSON format:") +display( + Markdown( + "### Task spec — Stream 2\n\n" + + _prose.strip() + + "\n\n**Required JSON format** (`DiscreteAgentForecastOutput`):\n\n```json\n" + + _schema.strip() + + "\n```" + ) +) +``` + +## Cell 13 (code) ```python +# ── Assign role: wire identity + task spec (no model call) ──────────────────── shock_task = ForecastingTask( task_id="wti_upshock_demo", target_series_id=WTI_SERIES_ID, @@ -205,9 +428,22 @@ shock_task = ForecastingTask( frequency="B", description="Binary upshock demo", ) +shock_prompt_builder = WtiMultitaskPromptBuilder(task_spec=SHOCK_TASK_SPEC) +shock_predictor = AgentPredictor( + agent_config=analyst_config, + prompt_builder=shock_prompt_builder, + output_schema=DiscreteAgentForecastOutput, +) + +print(f"Predictor schema: {shock_predictor.output_schema.__name__}") +preview_user_payload(shock_prompt_builder, shock_task, SHOCK_ORIGINS[-1]) +``` -shock_predictor = build_wti_news_predictor("shock", model=AGENT_MODEL) +## Cell 14 (code) +```python +# ── Run shock agent across origins ──────────────────────────────────────────── +# Uses analyst_config + SHOCK_TASK_SPEC. Keep USE_CACHE = False after edits. if USE_CACHE and SHOCK_ANALYST_CACHE.exists(): with open(SHOCK_ANALYST_CACHE) as f: shock_results = json.load(f) @@ -230,14 +466,14 @@ else: ) with open(SHOCK_ANALYST_CACHE, "w") as f: json.dump(shock_results, f, indent=2) + print(f"Saved {len(shock_results)} shock forecasts.") agent_probs = [r["probability"] for r in shock_results] outcomes = [r["outcome"] for r in shock_results] print(f"Agent Brier score: {compute_brier_score(agent_probs, outcomes):.4f}") -print(f"Task spec preview:\n{TASK_SPECS['shock'][:200]}...") ``` -## Cell 9 (code) +## Cell 15 (code) ```python # ── Per-origin forecast cards ───────────────────────────────────────────────── @@ -272,7 +508,7 @@ for r in shock_results: ) ``` -## Cell 10 (code) +## Cell 16 (code) ```python # ── Prophet probabilities for the shock origins ─────────────────────────────── @@ -302,46 +538,116 @@ print("Mean Brier score (lower = better, 0.25 = random ceiling):") display(brier_df) ``` -## Cell 11 (markdown) +## Cell 17 (markdown) --- ## Stream 3 — Scenario Analysis -## Cell 12 (code) +**Question:** What three scenarios are oil-market analysts debating for WTI over the next 60 days? + +Same identity. Track 2 structured qualitative analysis — no ground truth to score. +Edit the task spec (number of scenarios, framing) or the origin, then re-run. + +**Try this:** change "three scenarios" to "two bullish and one bearish", or set +`SCENARIO_AS_OF = pd.Timestamp("2026-02-02")` (pre-shock) and compare. + +## Cell 18 (code) ```python +# ── Stream 3 task spec (edit this) ──────────────────────────────────────────── +# SCENARIO_AS_OF = SCENARIO_ORIGIN # 2026-03-02 — conflict onset +SCENARIO_AS_OF = pd.Timestamp("2026-02-02") # pre-shock, quieter market +# SCENARIO_AS_OF = pd.Timestamp.today() # live — no deep historical fence + +_SCENARIO_SCHEMA = ScenarioAgentForecastOutput.prompt_schema_json() +SCENARIO_TASK_SPEC = f"""Identify the three scenarios that oil market analysts and experts are most +actively debating for WTI crude over the next 60 days, given the current +market context and price history. + +For each scenario: + - Give it a concise name (3-6 words) + - Describe it in 1-2 sentences + - Assign a probability (all three must sum to <= 1.0) + - Provide an expected WTI price range at the 60-day horizon as [low, high] + - Give your point estimate for WTI at 60 days under this scenario + - List 1-2 key drivers that would cause this scenario to materialise + +Also identify which scenario is the base case and provide an overall +one-paragraph reasoning summary. + +If a `set_model_response` tool is available, call it with your complete JSON as `json_response`. Otherwise return the JSON directly as plain text. + +Required JSON format: +{_SCENARIO_SCHEMA} +""" + +_origin_price_row = price_df[price_df.index >= SCENARIO_AS_OF] +_origin_price = float(_origin_price_row.iloc[0]["price"]) if not _origin_price_row.empty else float("nan") +_prose, _, _schema = SCENARIO_TASK_SPEC.partition("Required JSON format:") +display( + Markdown( + f"### Task spec — Stream 3 *(origin {SCENARIO_AS_OF.date()}, WTI ${_origin_price:.2f}/bbl)*\n\n" + + _prose.strip() + + "\n\n**Required JSON format** (`ScenarioAgentForecastOutput`):\n\n```json\n" + + _schema.strip() + + "\n```" + ) +) +``` + +## Cell 19 (code) + +```python +# ── Assign role: wire identity + task spec (no model call) ──────────────────── scenario_task = ForecastingTask( task_id="wti_scenario_demo", target_series_id=WTI_SERIES_ID, - horizons=[21], + horizons=[21], # ForecastingTask requires a horizon; the 60-day ask lives in the spec frequency="B", description="Scenario analysis demo", ) +scenario_prompt_builder = WtiMultitaskPromptBuilder(task_spec=SCENARIO_TASK_SPEC) +scenario_predictor = AgentPredictor( + agent_config=analyst_config, + prompt_builder=scenario_prompt_builder, + output_schema=ScenarioAgentForecastOutput, +) -scenario_predictor = build_wti_news_predictor("scenario", model=AGENT_MODEL) +print(f"Predictor schema: {scenario_predictor.output_schema.__name__}") +preview_user_payload(scenario_prompt_builder, scenario_task, SCENARIO_AS_OF) +``` + +## Cell 20 (code) + +```python +# ── Run the scenario agent ──────────────────────────────────────────────────── +# Uses analyst_config + SCENARIO_TASK_SPEC. Keep USE_CACHE = False after edits. +if USE_CACHE: + print("USE_CACHE is True — cached cards ignore edits to the task spec.") if USE_CACHE and SCENARIO_CACHE.exists(): with open(SCENARIO_CACHE) as f: scenario_payload = json.load(f) print("Loaded cached scenario analysis.") else: - as_of = SCENARIO_ORIGIN - pd.Timedelta(days=1) + as_of = SCENARIO_AS_OF - pd.Timedelta(days=1) origin_ctx = data_service.context(as_of=as_of) preds = scenario_predictor.predict(scenario_task, origin_ctx) scenario_payload = preds[0].metadata with open(SCENARIO_CACHE, "w") as f: json.dump(scenario_payload, f, indent=2) + print("Saved scenario analysis.") -# ── Rich scenario cards ─────────────────────────────────────────────────────── -scenario_origin_price_row = price_df[price_df.index >= SCENARIO_ORIGIN] +# ── Scenario cards ──────────────────────────────────────────────────────────── +scenario_origin_price_row = price_df[price_df.index >= SCENARIO_AS_OF] scenario_origin_price = ( float(scenario_origin_price_row.iloc[0]["price"]) if not scenario_origin_price_row.empty else float("nan") ) display( Markdown( - f"#### Stream 3 — Scenario Analysis " - f"*(origin: {SCENARIO_ORIGIN.date()}, WTI ${scenario_origin_price:.2f}/bbl)*\n\n" + f"#### Agent response — Stream 3 " + f"*(origin: {SCENARIO_AS_OF.date()}, WTI ${scenario_origin_price:.2f}/bbl)*\n\n" f"Base case: **{scenario_payload.get('base_case', '?')}**" ) ) @@ -375,15 +681,22 @@ if overall: display(Markdown(f"---\n\n> **Overall reasoning:** {overall}")) ``` -## Cell 13 (markdown) +## Cell 21 (markdown) --- ## Summary -One agent identity (`build_wti_multitask_news_config` / `build_wti_news_config`) with -three task-specific prompt builders and output schemas demonstrates the bootcamp -pattern for multi-task agentic forecasting. Continue to -[`04_systematic_backtest_eval.ipynb`](04_systematic_backtest_eval.ipynb) for the -stateless backtest harness, then Notebooks 5–6 for the adaptive agent training and -protected evaluation. +**One identity, three roles.** The shared `analyst_config` (system instruction + +`search_web` toolbelt) never changes across streams. Each stream assigns a role +with an editable **task spec** in the user payload via `WtiMultitaskPromptBuilder`, +plus a stream-specific `output_schema`. + +That is the bootcamp pattern for multi-task agentic forecasting. Notebooks 02/04 +still use a trajectory-specialized system prompt (`build_wti_news_config`) for +scored backtests — a useful contrast: bake the contract into identity, or keep +identity stable and swap the user-message ask. + +Continue to [`04_systematic_backtest_eval.ipynb`](04_systematic_backtest_eval.ipynb) +for the stateless backtest harness, then Notebooks 5–6 for the adaptive agent +training and protected evaluation. diff --git a/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__05_adaptive_agent_training.ipynb.md b/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__05_adaptive_agent_training.ipynb.md index ee2ded37..5d3eb216 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__05_adaptive_agent_training.ipynb.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__05_adaptive_agent_training.ipynb.md @@ -8,12 +8,12 @@ kind: notebook > **Part 5 of 7.** Builds on the stateless backtest in [`04_systematic_backtest_eval.ipynb`](04_systematic_backtest_eval.ipynb). -Every method in Notebook 4 was **stateless** — configured once, run the same way each time. +Every method in Notebook 4 was **stateless** — configured once, run the same way each time. This notebook introduces an agent that is different: it can **learn from experience**. -The paradigm shift: instead of configuring a model, we onboard an analyst. -We give the analyst a task, historical data, and a set of tools. -The analyst explores the data, draws conclusions, and decides whether to update +The paradigm shift: instead of configuring a model, we onboard an analyst. +We give the analyst a task, historical data, and a set of tools. +The analyst explores the data, draws conclusions, and decides whether to update its own forecasting strategy — governed by evidence rules in its `meta-learning` skill. **What this notebook produces:** @@ -73,11 +73,11 @@ print(f" Trained: {TRAINED_STRATEGY_DIR}") --- ## 1. Before — The Agent's Starting State -The seed strategy (`wti-strategy/`) contains domain priors: a sensible initial -approach, but no evidence-backed calibration corrections. +The seed strategy (`wti-strategy/`) contains domain priors: a sensible initial +approach, but no evidence-backed calibration corrections. It is the same strategy the **untrained agent** uses in Notebook 6. -The trained variant starts from an identical copy of this seed. +The trained variant starts from an identical copy of this seed. Set `RESEED = True` in Setup if you want to reset it before a fresh study run. ## Cell 5 (code) @@ -108,7 +108,7 @@ print((SEED_STRATEGY_DIR / "SKILL.md").read_text()) --- ## 2. Self-Directed Study -We give the agent one open-ended analytical task: explore 2025 WTI price data +We give the agent one open-ended analytical task: explore 2025 WTI price data and assess whether its current forecasting approach is well-calibrated. The agent has access to: @@ -118,10 +118,10 @@ The agent has access to: - `meta-learning` — evidence governance rules for updating strategy - Strategy mutation tools — to record observations, open hypotheses, and apply corrections -The agent decides what to compute, what conclusions to draw, and whether any +The agent decides what to compute, what conclusions to draw, and whether any finding clears the evidence bar for updating its `wti-strategy-trained/` skill. -> **Run guard:** `RUN_STUDY = False` by default — the trained strategy state +> **Run guard:** `RUN_STUDY = False` by default — the trained strategy state > is committed so this notebook runs reproducibly without live API calls. ## Cell 7 (code) @@ -175,7 +175,7 @@ else: --- ## 3. After — What the Agent Learned -The cell below shows the trained strategy state. +The cell below shows the trained strategy state. Look at what changed relative to the clean seed: - **Observations**: patterns the agent noticed during analysis @@ -183,7 +183,7 @@ Look at what changed relative to the clean seed: - **Calibration corrections**: confirmed adjustments now applied at inference - **Approach narrative**: how the agent describes its own strategy in its own words -These are the changes that will be active when the agent makes predictions +These are the changes that will be active when the agent makes predictions in Notebook 6. ## Cell 9 (code) @@ -228,9 +228,9 @@ print((TRAINED_STRATEGY_DIR / "SKILL.md").read_text()) --- ## 4. Optional: Robustness Testing -In the self-directed study, the agent examined 2025 WTI data and recorded at -least one open hypothesis. The two cells below run follow-up tasks to test -whether those findings are robust — the standard scientific check before +In the self-directed study, the agent examined 2025 WTI data and recorded at +least one open hypothesis. The two cells below run follow-up tasks to test +whether those findings are robust — the standard scientific check before promoting any pattern to an active calibration correction. | Task | Structure | Goal | @@ -238,7 +238,7 @@ promoting any pattern to an active calibration correction. | A — Cross-period | Re-run the same analysis on 2023-2024 data | `record_hypothesis_outcome` for each open hypothesis | | B — Scope check | Identify untested boundary conditions and fill the gap | Second confirmation → attempt `graduate_hypothesis` | -> **Run guard:** `RUN_FOLLOWUP = False` by default. Both tasks use the same +> **Run guard:** `RUN_FOLLOWUP = False` by default. Both tasks use the same > agent session and must run together — outputs are committed after first run. ## Cell 12 (code) @@ -254,8 +254,8 @@ RUN_FOLLOWUP = False ### Task A — Cross-Period Robustness (2023–2024) -Ask the agent to review its open hypotheses and replicate the relevant -analysis on 2023-2024 WTI data, recording whether the earlier data confirms +Ask the agent to review its open hypotheses and replicate the relevant +analysis on 2023-2024 WTI data, recording whether the earlier data confirms or contradicts each finding. ## Cell 14 (code) @@ -304,9 +304,9 @@ else: ### Task B — Scope Check and Graduation Attempt -Ask the agent to identify the untested boundary conditions of its open -hypotheses — horizons, regimes, or market conditions not yet examined — -run a targeted analysis to fill the most important gap, and then attempt +Ask the agent to identify the untested boundary conditions of its open +hypotheses — horizons, regimes, or market conditions not yet examined — +run a targeted analysis to fill the most important gap, and then attempt graduation if the confirmation threshold is met. ## Cell 16 (code) @@ -369,8 +369,8 @@ print((TRAINED_STRATEGY_DIR / "SKILL.md").read_text()) --- ## 5. Continue Interactively -The notebook has walked the agent through a structured study session. But the -best way to understand what the agent has learned — and to push it further — +The notebook has walked the agent through a structured study session. But the +best way to understand what the agent has learned — and to push it further — is to have a direct conversation. Launch the ADK web interface from the repo root: @@ -386,8 +386,10 @@ WTI_STRATEGY_DIR=adaptive_agent/skills/wti-strategy-trained \\ uv run adk web adaptive_agent/ ``` -Open `http://localhost:8000` in your browser. The agent has its full skill -set available: code execution, web search, and mutation tools. +Open `http://localhost:8000` in your browser. The agent has its full skill +set available: code execution, web search, and mutation tools. On a **Coder +workspace** that URL is inside the VM — [guide 5](../../guides/05-access-adk-web-via-ssh-tunnel.md) +tunnels it to your laptop. **Suggested conversation starters:** @@ -400,10 +402,10 @@ set available: code execution, web search, and mutation tools. --- ## Next: Protected Evaluation -Notebook 6 evaluates both the **untrained agent** (uses `wti-strategy/`) -and the **trained agent** (uses `wti-strategy-trained/`) on the 2026 eval spec — +Notebook 6 evaluates both the **untrained agent** (uses `wti-strategy/`) +and the **trained agent** (uses `wti-strategy-trained/`) on the 2026 eval spec — a period of significant market volatility the agent has never seen. -The eval is deliberately **frozen**: the agent cannot update its strategy -during evaluation, so the comparison is a clean before/after of what +The eval is deliberately **frozen**: the agent cannot update its strategy +during evaluation, so the comparison is a clean before/after of what the self-directed study session contributed. diff --git a/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__06_protected_eval.ipynb.md b/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__06_protected_eval.ipynb.md index fa126a84..da45d960 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__06_protected_eval.ipynb.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__06_protected_eval.ipynb.md @@ -470,4 +470,4 @@ WTI_STRATEGY_DIR=adaptive_agent/skills/wti-strategy-trained \\ uv run adk web adaptive_agent/ ``` -Open `http://localhost:8000`. See Notebook 5 for suggested conversation starters. +Open `http://localhost:8000`. See Notebook 5 for suggested conversation starters. On a **Coder workspace** that URL is inside the VM — [guide 5](../../guides/05-access-adk-web-via-ssh-tunnel.md) tunnels it to your laptop. diff --git a/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__README.md.md b/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__README.md.md index 959dd3fb..25c4505d 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__README.md.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__README.md.md @@ -31,7 +31,7 @@ introduced in notebook 2. |----------|-------|---------| | **[`01_wti_case_study.ipynb`](01_wti_case_study.ipynb)** | **The Case Study Narrative** — rolling Prophet backtest animation, annotated context chart, 2025 vs 2026 coverage punchline, futures curve | No | | **[`02_intro_agentic_predictor.ipynb`](02_intro_agentic_predictor.ipynb)** | **The Agentic Staircase** — 4 capability levels on Mar 2, 2026; inspect configs and prompts | Yes | -| **[`03_one_agent_three_tasks.ipynb`](03_one_agent_three_tasks.ipynb)** | **One Agent, Three Tasks** — trajectory, binary shock, scenario analysis via shared agent identity | Yes | +| **[`03_one_agent_three_tasks.ipynb`](03_one_agent_three_tasks.ipynb)** | **One Agent, Three Tasks** — shared identity once; three editable inline task specs (trajectory, shock, scenario) | Yes | | **[`04_systematic_backtest_eval.ipynb`](04_systematic_backtest_eval.ipynb)** | **Systematic Competition** — 2025 backtest → leaderboard → 2026 protected eval | Yes | ### Adaptive-agent track @@ -59,26 +59,30 @@ An earlier set of information-session notebooks is archived in [`playground/ener ## The Forecasting Tasks -Each forecasting origin defines a strict information cutoff (`as_of`). Predictors receive price history up to `as_of` and answer up to three tasks: +Each forecasting origin defines a strict information cutoff (`as_of`). Predictors receive price history up to `as_of` and answer up to three tasks. News-grounded agents apply the same fence to `search_web` when `as_of` is in the past (an independent verifier, visible as `search_web.leakage_verifier` under the Langfuse agent trace) and skip it for a live origin. ### Task A: Trajectory Forecast (Track 1) - **Horizons:** 5, 10, 21 business days - **Output:** Point estimate + standard quantile grid (via `ContinuousAgentForecastOutput`) - **Evaluation:** CRPS and MAE (Notebook 4 backtest) +- **Notebook 03:** editable `TRAJECTORY_TASK_SPEC` in the user payload (same identity as Streams 2–3) ### Task B: Binary Up-shock Probability (Track 1) - **Question:** P(WTI closes > $5/bbl higher in 5 business days) - **Output:** `DiscreteAgentForecastOutput` → `BinaryForecast` - **Evaluation:** Brier score (Notebook 3) +- **Notebook 03:** editable `SHOCK_TASK_SPEC` in the user payload ### Task C: Scenario Analysis (Track 2) -- **Output:** Three scenario cards with probabilities and 60-day ranges +- **Question:** What three scenarios are oil-market analysts debating for WTI over the next 60 days? +- **Output:** Named scenario cards with probabilities, 60-day WTI ranges, point estimates, and key drivers - **Evaluation:** Display / qualitative (Track 2 — not head-to-head scored in backtest) +- **Notebook 03:** editable `SCENARIO_TASK_SPEC` in the user payload -The **one-agent-three-tasks** pattern lives in [`tasks.py`](tasks.py): one `AgentConfig` identity, three `(prompt_builder, output_schema)` pairs via `build_wti_news_predictor(task)`. +The **one-agent-three-tasks** pattern: notebook 03 defines the shared identity once (system instruction + `search_web` toolbelt), then each stream assigns a role with an inline **task spec** via `WtiMultitaskPromptBuilder`. Library defaults live in [`tasks.py`](tasks.py) (`TASK_SPECS` / `build_wti_news_predictor(task)`); notebooks 02/04 keep a trajectory-specialized system prompt (`build_wti_news_config`) for scored trajectory backtests. --- @@ -113,7 +117,7 @@ notebook 05). |-------|--------|------| | Package | `aieng.forecasting.methods.agentic` | `AgentPredictor`, `AgentConfig`, output schema base classes | | Stateless identity | `analyst_agent/agent.py` | Instructions, capability presets, skills — fixed at config time | -| Role per task | `tasks.py` | Prompt builders, `build_wti_news_predictor(task)` | +| Role per task | `tasks.py` + notebook 03 inline specs | `WtiMultitaskPromptBuilder(task_spec=...)`, `build_wti_news_predictor(task)` | | Learning agent | `adaptive_agent/` | Persistent, mutable strategy state updated via self-directed study (notebooks 05–06) | --- diff --git a/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__analyst_agent__agent.py.md b/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__analyst_agent__agent.py.md index e05eeae0..474d03d2 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__analyst_agent__agent.py.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__analyst_agent__agent.py.md @@ -73,6 +73,8 @@ You are an expert WTI crude oil market analyst. You will receive a JSON payload containing: - `task_spec`: the exact question and required JSON output schema - `as_of`: the forecast origin date (temporal cutoff) +- `horizons`: integer horizon steps (business days ahead) +- `standard_quantiles`: quantile levels for continuous forecasts (when applicable) - `origin_price_usd_bbl`: WTI close on the origin date - `target_history_csv`: compressed WTI daily close history @@ -447,10 +449,12 @@ def build_wti_news_config( """Build an :class:`AgentConfig` with bounded Google Search. Wires a :class:`~aieng.forecasting.methods.agentic.agent_factory.ContextRetrievalConfig` - sub-agent that enforces a temporal cutoff on every search call, preventing - future information from contaminating historical backtests. An - independent verifier call audits each search result against the cutoff - before it reaches the analyst (see :class:`ContextRetrievalConfig`). + sub-agent that enforces a temporal cutoff on retrospective search calls + (``as_of`` strictly before UTC today), preventing future information + from contaminating historical backtests. Live origins skip the fence. + An independent verifier call audits each historical search result + against the cutoff before it reaches the analyst (see + :class:`ContextRetrievalConfig`). Parameters ---------- @@ -539,9 +543,7 @@ def build_wti_code_exec_config( return AgentConfig( name="wti_analyst_code", model=model, - instruction=( - _WTI_ANALYST_INSTRUCTION + _CONTEXT_RETRIEVAL_SUPPLEMENT + _CODE_EXEC_SKILLS_SUPPLEMENT - ), + instruction=(_WTI_ANALYST_INSTRUCTION + _CONTEXT_RETRIEVAL_SUPPLEMENT + _CODE_EXEC_SKILLS_SUPPLEMENT), max_output_tokens=max_output_tokens, context_retrieval=ContextRetrievalConfig( enabled=True, @@ -616,9 +618,7 @@ def build_wti_tool_config( return AgentConfig( name="wti_analyst_tool", model=model, - instruction=( - _WTI_ANALYST_INSTRUCTION + _CONTEXT_RETRIEVAL_SUPPLEMENT + _FORECAST_TOOL_SUPPLEMENT - ), + instruction=(_WTI_ANALYST_INSTRUCTION + _CONTEXT_RETRIEVAL_SUPPLEMENT + _FORECAST_TOOL_SUPPLEMENT), context_retrieval=ContextRetrievalConfig( enabled=True, instruction=_WTI_CONTEXT_RETRIEVAL_INSTRUCTION, @@ -668,6 +668,11 @@ def build_wti_agent_predictor(config: AgentConfig) -> AgentPredictor: def __getattr__(name: str) -> Any: """Expose ``root_agent`` lazily for schema-free interactive use via ``adk web``.""" if name == "root_agent": - return build_adk_agent(build_wti_basic_config()) + # return build_adk_agent(build_wti_basic_config()) + return build_adk_agent( + build_wti_multitask_news_config( + model=ADVANCED_MODEL, search_model=ADVANCED_MODEL, verifier_model=ADVANCED_MODEL + ) + ) raise AttributeError(f"module {__name__!r} has no attribute {name!r}") ``` diff --git a/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__starter_agent__skills__research-playbook__SKILL.md.md b/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__starter_agent__skills__research-playbook__SKILL.md.md index f43e96ae..fa2d25d7 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__starter_agent__skills__research-playbook__SKILL.md.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__starter_agent__skills__research-playbook__SKILL.md.md @@ -21,11 +21,13 @@ Always pass `cutoff_date` equal to the `as_of` date in your payload. It is the temporal fence that keeps post-origin information out of a historical forecast. A forecast that "knew" what happened after `as_of` is not a forecast. -`search_web` runs an independent verifier on every result and returns +On historical origins (strictly before today's date), `search_web` runs an +independent verifier on every result and returns `[SEARCH_VERIFICATION_FAILED]` instead of content it couldn't confirm as pre-cutoff. Treat that as no verified news for the query — proceed on your other signals and say so, never filling the gap from your own background -knowledge. +knowledge. On a live origin (`as_of` is today), the verifier is skipped so +current news is not stripped. ## How to search diff --git a/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__tasks.py.md b/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__tasks.py.md index f51bbc93..21c3f349 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__tasks.py.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__tasks.py.md @@ -18,7 +18,7 @@ from typing import Any, ClassVar, Literal import pandas as pd from aieng.forecasting.data.context import ForecastContext -from aieng.forecasting.evaluation.prediction import BinaryForecast, Prediction +from aieng.forecasting.evaluation.prediction import STANDARD_QUANTILES, BinaryForecast, Prediction from aieng.forecasting.evaluation.task import ForecastingTask from aieng.forecasting.methods.agentic import ( AgentPredictor, @@ -29,31 +29,24 @@ from aieng.forecasting.methods.agentic.agent_factory import AgentConfig from aieng.forecasting.methods.agentic.outputs import AgentForecastOutput from aieng.forecasting.models import LITE_MODEL from energy_oil_forecasting.analyst_agent import ( - WtiPriceForecastPromptBuilder, build_wti_multitask_news_config, - build_wti_news_config, compress_history, ) from energy_oil_forecasting.paths import SHOCK_HORIZON, SHOCK_THRESHOLD from pydantic import BaseModel, Field -# ── Task specification strings (embedded in user prompts for NB3) ─────────── -# Each spec uses the corresponding output class's prompt_schema_json() so the -# required JSON format in the prompt is always in sync with the Pydantic schema. - -TASK_TRAJECTORY_SPEC = ( - "Forecast the WTI crude oil price at the horizons listed in the payload.\n\n" - "If a `set_model_response` tool is available, call it with your complete " - "JSON as `json_response`. Otherwise return the JSON directly as plain text.\n\n" - "Required JSON format:\n" + ContinuousAgentForecastOutput.prompt_schema_json() -) - TaskKind = Literal["trajectory", "shock", "scenario"] class WtiMultitaskPromptBuilder(BaseModel): - """Prompt builder for task-spec-driven agent calls (NB3).""" + """Prompt builder for task-spec-driven agent calls (NB3). + + The system instruction is task-agnostic; the ask lives in ``task_spec``. + The payload also includes ``horizons`` and ``standard_quantiles`` so + trajectory (and any horizon-aware) tasks can read them without baking the + forecasting contract into the system prompt. + """ task_spec: str @@ -66,6 +59,8 @@ class WtiMultitaskPromptBuilder(BaseModel): "task": task.task_id, "task_spec": self.task_spec, "as_of": str(context.as_of)[:10], + "horizons": list(task.horizons), + "standard_quantiles": list(STANDARD_QUANTILES), "origin_price_usd_bbl": float(last_row["value"]), "target_history_csv": compress_history(df), } @@ -160,19 +155,50 @@ class ScenarioAgentForecastOutput(AgentForecastOutput): # Task specification strings embedded in user prompts for NB3. # Defined after the output classes so each spec can reference the # corresponding prompt_schema_json() classmethod — single source of truth. +# Notebook 03 copies these into editable cells; the factory uses these defaults. + +TASK_TRAJECTORY_SPEC = ( + "Forecast the WTI crude oil price at each horizon listed in the payload " + "(`horizons`, business days ahead).\n\n" + "Rules:\n" + " - Produce one forecast for each horizon in `horizons`.\n" + " - Use exactly the quantile levels from `standard_quantiles` — " + "no additions, no omissions.\n" + " - `point_forecast` must exactly equal the 0.50 quantile value.\n" + " - Quantile values must be strictly non-decreasing as quantile levels increase.\n" + " - Document your reasoning in the `rationale` fields.\n\n" + "If a `set_model_response` tool is available, call it with your complete " + "JSON as `json_response`. Otherwise return the JSON directly as plain text.\n\n" + "Required JSON format:\n" + ContinuousAgentForecastOutput.prompt_schema_json() +) TASK_SHOCK_SPEC = ( f"Estimate P(up) — the probability that WTI will close MORE THAN\n" f"${int(SHOCK_THRESHOLD)}/bbl HIGHER than today's price at the end of\n" f"{SHOCK_HORIZON} trading days.\n\n" + "This is a directional upside question only.\n\n" + "Calibration guidance:\n" + " - No unusual upside catalyst -> base rate ~10-15%\n" + " - Escalating unconfirmed risk -> 20-40%\n" + " - Confirmed supply disruption -> 60-85%\n\n" "If a `set_model_response` tool is available, call it with your complete " "JSON as `json_response`. Otherwise return the JSON directly as plain text.\n\n" "Required JSON format:\n" + DiscreteAgentForecastOutput.prompt_schema_json() ) TASK_SCENARIOS_SPEC = ( - "Identify the three scenarios oil market analysts are debating for WTI " - "over the next 60 days.\n\n" + "Identify the three scenarios that oil market analysts and experts are most " + "actively debating for WTI crude over the next 60 days, given the current " + "market context and price history.\n\n" + "For each scenario:\n" + " - Give it a concise name (3-6 words)\n" + " - Describe it in 1-2 sentences\n" + " - Assign a probability (all three must sum to <= 1.0)\n" + " - Provide an expected WTI price range at the 60-day horizon as [low, high]\n" + " - Give your point estimate for WTI at 60 days under this scenario\n" + " - List 1-2 key drivers that would cause this scenario to materialise\n\n" + "Also identify which scenario is the base case and provide an overall " + "one-paragraph reasoning summary.\n\n" "If a `set_model_response` tool is available, call it with your complete " "JSON as `json_response`. Otherwise return the JSON directly as plain text.\n\n" "Required JSON format:\n" + ScenarioAgentForecastOutput.prompt_schema_json() @@ -198,6 +224,10 @@ def build_wti_news_predictor( ) -> AgentPredictor: """Build a news-grounded agent predictor for the given task kind. + All three task kinds share the same multitask news identity + (:func:`~energy_oil_forecasting.analyst_agent.build_wti_multitask_news_config`); + only the user-payload ``task_spec`` and output schema change. + Parameters ---------- task : TaskKind @@ -208,12 +238,6 @@ def build_wti_news_predictor( Defaults to the lite model (``"gemini-3.1-flash-lite-preview"``); pass the advanced model (``"gemini-3.5-flash"``) when more capability is needed. """ - if task == "trajectory": - return AgentPredictor( - agent_config=build_wti_news_config(model=model), - prompt_builder=WtiPriceForecastPromptBuilder(), - output_schema=ContinuousAgentForecastOutput, - ) return AgentPredictor( agent_config=build_wti_multitask_news_config(model=model), prompt_builder=WtiMultitaskPromptBuilder(task_spec=TASK_SPECS[task]), @@ -222,13 +246,11 @@ def build_wti_news_predictor( def build_wti_agent_predictor_for_task(config: AgentConfig, task: TaskKind) -> AgentPredictor: - """Wire any WTI agent config to a task-specific predictor.""" - if task == "trajectory": - return AgentPredictor( - agent_config=config, - prompt_builder=WtiPriceForecastPromptBuilder(), - output_schema=ContinuousAgentForecastOutput, - ) + """Wire any WTI agent config to a task-specific predictor. + + Uses the multitask prompt builder for every task kind so the ask rides in + ``task_spec`` rather than in the system instruction. + """ return AgentPredictor( agent_config=config, prompt_builder=WtiMultitaskPromptBuilder(task_spec=TASK_SPECS[task]), diff --git a/implementations/getting_started/concierge_agent/context/artifacts/implementations__food_price_forecasting__starter_agent__skills__research-playbook__SKILL.md.md b/implementations/getting_started/concierge_agent/context/artifacts/implementations__food_price_forecasting__starter_agent__skills__research-playbook__SKILL.md.md index 2c9e2463..3f1daadf 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/implementations__food_price_forecasting__starter_agent__skills__research-playbook__SKILL.md.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/implementations__food_price_forecasting__starter_agent__skills__research-playbook__SKILL.md.md @@ -21,11 +21,13 @@ Always pass `cutoff_date` equal to the `as_of` date in your payload. It is the temporal fence that keeps post-origin information out of a historical forecast. A forecast that "knew" what happened after `as_of` is not a forecast. -`search_web` runs an independent verifier on every result and returns +On historical origins (strictly before today's date), `search_web` runs an +independent verifier on every result and returns `[SEARCH_VERIFICATION_FAILED]` instead of content it couldn't confirm as pre-cutoff. Treat that as no verified news for the query — proceed on your other signals and say so, never filling the gap from your own background -knowledge. +knowledge. On a live origin (`as_of` is today), the verifier is skipped so +current news is not stripped. ## How to search diff --git a/implementations/getting_started/concierge_agent/context/artifacts/implementations__getting_started__99_repo_concierge.ipynb.md b/implementations/getting_started/concierge_agent/context/artifacts/implementations__getting_started__99_repo_concierge.ipynb.md index 8cd98d35..d6d3198f 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/implementations__getting_started__99_repo_concierge.ipynb.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/implementations__getting_started__99_repo_concierge.ipynb.md @@ -170,7 +170,9 @@ That loads the same `repo_concierge` agent (`gemini-3.1-flash-lite-preview`) wit **Alternative:** `uv run adk web implementations/getting_started/concierge_agent` opens a browser UI (same agent). From `implementations/getting_started/`, you can -also use the shorter `uv run adk run concierge_agent`. +also use the shorter `uv run adk run concierge_agent`. [Guide 5](../../guides/05-access-adk-web-via-ssh-tunnel.md) +is the full walkthrough: serve the concierge (or any other bootcamp agent) and, +on a Coder workspace, tunnel the UI to your laptop. --- diff --git a/implementations/getting_started/concierge_agent/context/artifacts/implementations__getting_started__README.md.md b/implementations/getting_started/concierge_agent/context/artifacts/implementations__getting_started__README.md.md index 17338152..45baaab6 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/implementations__getting_started__README.md.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/implementations__getting_started__README.md.md @@ -158,7 +158,9 @@ and modules, and can quote snippets from the committed public-`main` catalog. ``` (`uv run adk web implementations/getting_started/concierge_agent` opens the same - agent in a browser.) + agent in a browser. [Guide 5](../../guides/05-access-adk-web-via-ssh-tunnel.md) + walks through serving the concierge and other bootcamp agents in interactive + mode, including the Coder SSH tunnel.) From `implementations/getting_started/`, the shorter `uv run adk run concierge_agent` works too. diff --git a/implementations/getting_started/concierge_agent/context/artifacts/implementations__sp500_forecasting__starter_agent__skills__research-playbook__SKILL.md.md b/implementations/getting_started/concierge_agent/context/artifacts/implementations__sp500_forecasting__starter_agent__skills__research-playbook__SKILL.md.md index 359dd966..573af043 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/implementations__sp500_forecasting__starter_agent__skills__research-playbook__SKILL.md.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/implementations__sp500_forecasting__starter_agent__skills__research-playbook__SKILL.md.md @@ -21,11 +21,13 @@ Always pass `cutoff_date` equal to the `as_of` date in your payload. It is the temporal fence that keeps post-origin information out of a historical forecast. A forecast that "knew" what happened after `as_of` is not a forecast. -`search_web` runs an independent verifier on every result and returns +On historical origins (strictly before today's date), `search_web` runs an +independent verifier on every result and returns `[SEARCH_VERIFICATION_FAILED]` instead of content it couldn't confirm as pre-cutoff. Treat that as no verified news for the query — proceed on your other signals and say so, never filling the gap from your own background -knowledge. +knowledge. On a live origin (`as_of` is today), the verifier is skipped so +current news is not stripped. ## How to search diff --git a/implementations/getting_started/concierge_agent/context/catalog.yaml b/implementations/getting_started/concierge_agent/context/catalog.yaml index 21831953..27614239 100644 --- a/implementations/getting_started/concierge_agent/context/catalog.yaml +++ b/implementations/getting_started/concierge_agent/context/catalog.yaml @@ -1,8 +1,8 @@ source_url: https://github.com/VectorInstitute/agentic-forecasting -git_ref: 04507bf406cc5524c3d9523fc4835f29462fe3ea +git_ref: fc1e62211abf3c1d60d4425bcbd186f83f5cd7ea branch: main -built_at: '2026-08-24T20:54:40+00:00' -ingest_source: /home/akore/vscodeprojects/agentic-forecasting +built_at: '2026-08-26T18:48:11+00:00' +ingest_source: /Users/ethanjackson/agentic-forecasting entry_count: 197 entries: - path: AGENTS.md @@ -47,7 +47,7 @@ entries: - Extending the foundation - Code quality - Documentation - chars: 14063 + chars: 14678 artifact: artifacts/README.md.md - path: aieng-forecasting/aieng/forecasting/__init__.py kind: python @@ -425,11 +425,13 @@ entries: domain: core.root summary: Langfuse-oriented tracing bootstrap for LiteLLM and Google ADK. symbols: + - _NoOpObservation - _LangfuseTracingBootstrap + - langfuse_generation - init_langfuse_tracing - print_langfuse_trace_url sections: [] - chars: 6087 + chars: 8048 artifact: artifacts/aieng-forecasting__aieng__forecasting__langfuse_tracing.py.md - path: aieng-forecasting/aieng/forecasting/methods/README.md kind: markdown @@ -446,7 +448,7 @@ entries: - Numerical - LLM Processes - Agentic - chars: 7424 + chars: 7717 artifact: artifacts/aieng-forecasting__aieng__forecasting__methods__README.md.md - path: aieng-forecasting/aieng/forecasting/methods/__init__.py kind: python @@ -512,7 +514,7 @@ entries: - AgentConfig - build_adk_agent sections: [] - chars: 35772 + chars: 41485 artifact: artifacts/aieng-forecasting__aieng__forecasting__methods__agentic__agent_factory.py.md - path: aieng-forecasting/aieng/forecasting/methods/agentic/curriculum.py kind: python @@ -805,7 +807,7 @@ entries: - Directory layout - Relationship to `aieng-forecasting` - Adding a new use case - chars: 3921 + chars: 4270 artifact: artifacts/implementations__README.md.md - path: implementations/__init__.py kind: python @@ -850,7 +852,7 @@ entries: symbols: [] sections: - "Bank of Canada Rate Decisions \u2014 Your Starter Agent" - chars: 8086 + chars: 8085 artifact: artifacts/implementations__boc_rate_decisions__99_starter_agent.ipynb.md - path: implementations/boc_rate_decisions/README.md kind: markdown @@ -1242,7 +1244,7 @@ entries: - How to search - Domain focus (edit this for your use case) - Room to grow - chars: 2250 + chars: 2398 artifact: artifacts/implementations__boc_rate_decisions__starter_agent__skills__research-playbook__SKILL.md.md - path: implementations/energy_oil_forecasting/01_wti_case_study.ipynb kind: notebook @@ -1274,7 +1276,7 @@ entries: symbols: [] sections: - "WTI Oil Price Forecasting \u2014 One Agent, Three Tasks" - chars: 14481 + chars: 28199 artifact: artifacts/implementations__energy_oil_forecasting__03_one_agent_three_tasks.ipynb.md - path: implementations/energy_oil_forecasting/04_systematic_backtest_eval.ipynb kind: notebook @@ -1298,7 +1300,7 @@ entries: - "Task A \u2014 Cross-Period Robustness (2023\u20132024)" - "Task B \u2014 Scope Check and Graduation Attempt" - Strategy state after robustness testing - chars: 15882 + chars: 16026 artifact: artifacts/implementations__energy_oil_forecasting__05_adaptive_agent_training.ipynb.md - path: implementations/energy_oil_forecasting/05_forecast_tool_demo.ipynb kind: notebook @@ -1316,7 +1318,7 @@ entries: symbols: [] sections: - "WTI Crude Oil \u2014 Protected Evaluation (Notebook 6 of 7)" - chars: 15724 + chars: 15864 artifact: artifacts/implementations__energy_oil_forecasting__06_protected_eval.ipynb.md - path: implementations/energy_oil_forecasting/99_starter_agent.ipynb kind: notebook @@ -1346,7 +1348,7 @@ entries: - Module Layout - Agent layering - Data Source & Setup - chars: 7707 + chars: 8624 artifact: artifacts/implementations__energy_oil_forecasting__README.md.md - path: implementations/energy_oil_forecasting/__init__.py kind: python @@ -1621,7 +1623,7 @@ entries: - build_wti_tool_config - build_wti_agent_predictor sections: [] - chars: 27592 + chars: 27984 artifact: artifacts/implementations__energy_oil_forecasting__analyst_agent__agent.py.md - path: implementations/energy_oil_forecasting/analyst_agent/skills/statistical-analysis/SKILL.md kind: markdown @@ -1903,7 +1905,7 @@ entries: - How to search - Domain focus (edit this for your use case) - Room to grow - chars: 2133 + chars: 2281 artifact: artifacts/implementations__energy_oil_forecasting__starter_agent__skills__research-playbook__SKILL.md.md - path: implementations/energy_oil_forecasting/starter_agent/tools.py kind: python @@ -1939,7 +1941,7 @@ entries: - build_wti_agent_predictor_for_task - build_wti_news_predictor sections: [] - chars: 8968 + chars: 10398 artifact: artifacts/implementations__energy_oil_forecasting__tasks.py.md - path: implementations/energy_oil_forecasting/viz.py kind: python @@ -2283,7 +2285,7 @@ entries: - How to search - Domain focus (edit this for your use case) - Room to grow - chars: 2174 + chars: 2322 artifact: artifacts/implementations__food_price_forecasting__starter_agent__skills__research-playbook__SKILL.md.md - path: implementations/getting_started/00_environment_check.ipynb kind: notebook @@ -2346,7 +2348,7 @@ entries: symbols: [] sections: - "Repo Concierge \u2014 ask questions about this codebase" - chars: 5804 + chars: 5996 artifact: artifacts/implementations__getting_started__99_repo_concierge.ipynb.md - path: implementations/getting_started/README.md kind: markdown @@ -2369,7 +2371,7 @@ entries: - Where to go next - Directory layout - Key interfaces (from `aieng-forecasting`) - chars: 9367 + chars: 9546 artifact: artifacts/implementations__getting_started__README.md.md - path: implementations/getting_started/__init__.py kind: python @@ -2880,7 +2882,7 @@ entries: - How to search - Domain focus (edit this for your use case) - Room to grow - chars: 2163 + chars: 2311 artifact: artifacts/implementations__sp500_forecasting__starter_agent__skills__research-playbook__SKILL.md.md - path: planning-docs/roadmap.md kind: markdown diff --git a/implementations/getting_started/concierge_agent/skills/repo-navigation/references/catalog-summary.yaml b/implementations/getting_started/concierge_agent/skills/repo-navigation/references/catalog-summary.yaml index a715b400..2745598a 100644 --- a/implementations/getting_started/concierge_agent/skills/repo-navigation/references/catalog-summary.yaml +++ b/implementations/getting_started/concierge_agent/skills/repo-navigation/references/catalog-summary.yaml @@ -1,8 +1,8 @@ # Concierge catalog summary (regenerated by scripts/build_concierge_context.py) source_url: https://github.com/VectorInstitute/agentic-forecasting branch: main -built_at: '2026-08-24T20:54:40+00:00' -git_ref: 04507bf406cc5524c3d9523fc4835f29462fe3ea +built_at: '2026-08-26T18:48:11+00:00' +git_ref: fc1e62211abf3c1d60d4425bcbd186f83f5cd7ea entry_count: 197 domains: docs: 4 From 9a3f9af3fe162e91eaddd5f246b4bed6ae2638d6 Mon Sep 17 00:00:00 2001 From: Ethan Jackson Date: Wed, 26 Aug 2026 14:49:24 -0400 Subject: [PATCH 2/2] lint --- ...sting__05_adaptive_agent_training.ipynb.md | 60 +++++++++---------- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__05_adaptive_agent_training.ipynb.md b/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__05_adaptive_agent_training.ipynb.md index 5d3eb216..c80f66d6 100644 --- a/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__05_adaptive_agent_training.ipynb.md +++ b/implementations/getting_started/concierge_agent/context/artifacts/implementations__energy_oil_forecasting__05_adaptive_agent_training.ipynb.md @@ -8,12 +8,12 @@ kind: notebook > **Part 5 of 7.** Builds on the stateless backtest in [`04_systematic_backtest_eval.ipynb`](04_systematic_backtest_eval.ipynb). -Every method in Notebook 4 was **stateless** — configured once, run the same way each time. +Every method in Notebook 4 was **stateless** — configured once, run the same way each time. This notebook introduces an agent that is different: it can **learn from experience**. -The paradigm shift: instead of configuring a model, we onboard an analyst. -We give the analyst a task, historical data, and a set of tools. -The analyst explores the data, draws conclusions, and decides whether to update +The paradigm shift: instead of configuring a model, we onboard an analyst. +We give the analyst a task, historical data, and a set of tools. +The analyst explores the data, draws conclusions, and decides whether to update its own forecasting strategy — governed by evidence rules in its `meta-learning` skill. **What this notebook produces:** @@ -73,11 +73,11 @@ print(f" Trained: {TRAINED_STRATEGY_DIR}") --- ## 1. Before — The Agent's Starting State -The seed strategy (`wti-strategy/`) contains domain priors: a sensible initial -approach, but no evidence-backed calibration corrections. +The seed strategy (`wti-strategy/`) contains domain priors: a sensible initial +approach, but no evidence-backed calibration corrections. It is the same strategy the **untrained agent** uses in Notebook 6. -The trained variant starts from an identical copy of this seed. +The trained variant starts from an identical copy of this seed. Set `RESEED = True` in Setup if you want to reset it before a fresh study run. ## Cell 5 (code) @@ -108,7 +108,7 @@ print((SEED_STRATEGY_DIR / "SKILL.md").read_text()) --- ## 2. Self-Directed Study -We give the agent one open-ended analytical task: explore 2025 WTI price data +We give the agent one open-ended analytical task: explore 2025 WTI price data and assess whether its current forecasting approach is well-calibrated. The agent has access to: @@ -118,10 +118,10 @@ The agent has access to: - `meta-learning` — evidence governance rules for updating strategy - Strategy mutation tools — to record observations, open hypotheses, and apply corrections -The agent decides what to compute, what conclusions to draw, and whether any +The agent decides what to compute, what conclusions to draw, and whether any finding clears the evidence bar for updating its `wti-strategy-trained/` skill. -> **Run guard:** `RUN_STUDY = False` by default — the trained strategy state +> **Run guard:** `RUN_STUDY = False` by default — the trained strategy state > is committed so this notebook runs reproducibly without live API calls. ## Cell 7 (code) @@ -175,7 +175,7 @@ else: --- ## 3. After — What the Agent Learned -The cell below shows the trained strategy state. +The cell below shows the trained strategy state. Look at what changed relative to the clean seed: - **Observations**: patterns the agent noticed during analysis @@ -183,7 +183,7 @@ Look at what changed relative to the clean seed: - **Calibration corrections**: confirmed adjustments now applied at inference - **Approach narrative**: how the agent describes its own strategy in its own words -These are the changes that will be active when the agent makes predictions +These are the changes that will be active when the agent makes predictions in Notebook 6. ## Cell 9 (code) @@ -228,9 +228,9 @@ print((TRAINED_STRATEGY_DIR / "SKILL.md").read_text()) --- ## 4. Optional: Robustness Testing -In the self-directed study, the agent examined 2025 WTI data and recorded at -least one open hypothesis. The two cells below run follow-up tasks to test -whether those findings are robust — the standard scientific check before +In the self-directed study, the agent examined 2025 WTI data and recorded at +least one open hypothesis. The two cells below run follow-up tasks to test +whether those findings are robust — the standard scientific check before promoting any pattern to an active calibration correction. | Task | Structure | Goal | @@ -238,7 +238,7 @@ promoting any pattern to an active calibration correction. | A — Cross-period | Re-run the same analysis on 2023-2024 data | `record_hypothesis_outcome` for each open hypothesis | | B — Scope check | Identify untested boundary conditions and fill the gap | Second confirmation → attempt `graduate_hypothesis` | -> **Run guard:** `RUN_FOLLOWUP = False` by default. Both tasks use the same +> **Run guard:** `RUN_FOLLOWUP = False` by default. Both tasks use the same > agent session and must run together — outputs are committed after first run. ## Cell 12 (code) @@ -254,8 +254,8 @@ RUN_FOLLOWUP = False ### Task A — Cross-Period Robustness (2023–2024) -Ask the agent to review its open hypotheses and replicate the relevant -analysis on 2023-2024 WTI data, recording whether the earlier data confirms +Ask the agent to review its open hypotheses and replicate the relevant +analysis on 2023-2024 WTI data, recording whether the earlier data confirms or contradicts each finding. ## Cell 14 (code) @@ -304,9 +304,9 @@ else: ### Task B — Scope Check and Graduation Attempt -Ask the agent to identify the untested boundary conditions of its open -hypotheses — horizons, regimes, or market conditions not yet examined — -run a targeted analysis to fill the most important gap, and then attempt +Ask the agent to identify the untested boundary conditions of its open +hypotheses — horizons, regimes, or market conditions not yet examined — +run a targeted analysis to fill the most important gap, and then attempt graduation if the confirmation threshold is met. ## Cell 16 (code) @@ -369,8 +369,8 @@ print((TRAINED_STRATEGY_DIR / "SKILL.md").read_text()) --- ## 5. Continue Interactively -The notebook has walked the agent through a structured study session. But the -best way to understand what the agent has learned — and to push it further — +The notebook has walked the agent through a structured study session. But the +best way to understand what the agent has learned — and to push it further — is to have a direct conversation. Launch the ADK web interface from the repo root: @@ -386,9 +386,9 @@ WTI_STRATEGY_DIR=adaptive_agent/skills/wti-strategy-trained \\ uv run adk web adaptive_agent/ ``` -Open `http://localhost:8000` in your browser. The agent has its full skill -set available: code execution, web search, and mutation tools. On a **Coder -workspace** that URL is inside the VM — [guide 5](../../guides/05-access-adk-web-via-ssh-tunnel.md) +Open `http://localhost:8000` in your browser. The agent has its full skill +set available: code execution, web search, and mutation tools. On a **Coder +workspace** that URL is inside the VM — [guide 5](../../guides/05-access-adk-web-via-ssh-tunnel.md) tunnels it to your laptop. **Suggested conversation starters:** @@ -402,10 +402,10 @@ tunnels it to your laptop. --- ## Next: Protected Evaluation -Notebook 6 evaluates both the **untrained agent** (uses `wti-strategy/`) -and the **trained agent** (uses `wti-strategy-trained/`) on the 2026 eval spec — +Notebook 6 evaluates both the **untrained agent** (uses `wti-strategy/`) +and the **trained agent** (uses `wti-strategy-trained/`) on the 2026 eval spec — a period of significant market volatility the agent has never seen. -The eval is deliberately **frozen**: the agent cannot update its strategy -during evaluation, so the comparison is a clean before/after of what +The eval is deliberately **frozen**: the agent cannot update its strategy +during evaluation, so the comparison is a clean before/after of what the self-directed study session contributed.