-
Notifications
You must be signed in to change notification settings - Fork 10
feat(evaluation): Improve iteration loop efficiency #1101
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
AkhileshNegi
wants to merge
4
commits into
main
Choose a base branch
from
feature/evaluation-iteration-loop
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
127 changes: 127 additions & 0 deletions
127
backend/app/alembic/versions/082_add_evaluation_iteration_run.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,127 @@ | ||
| """Add evaluation_iteration_run table | ||
|
|
||
| Revision ID: 082 | ||
| Revises: 081 | ||
| Create Date: 2026-08-02 00:00:00.000000 | ||
|
|
||
| Thin tracking row for the eval-iterate-improve LangGraph loop (see | ||
| docs/srd-ai-prompt-improvement.md follow-on: the Evaluation Iteration Loop). The | ||
| round-by-round trajectory itself lives in the LangGraph checkpoint (owned by | ||
| `langgraph-checkpoint-postgres`, set up separately via `checkpointer.setup()`, | ||
| not this migration) — this table only tracks enough to create/look up a loop, | ||
| scope it by org/project, and let the cron tick find loops still in flight. | ||
| """ | ||
|
|
||
| import sqlalchemy as sa | ||
| import sqlmodel.sql.sqltypes | ||
| from alembic import op | ||
|
|
||
| revision = "082" | ||
| down_revision = "081" | ||
| branch_labels = None | ||
| depends_on = None | ||
|
|
||
|
|
||
| def upgrade(): | ||
| op.create_table( | ||
| "evaluation_iteration_run", | ||
| sa.Column("id", sa.Integer(), nullable=False), | ||
| sa.Column("dataset_id", sa.Integer(), nullable=False), | ||
| sa.Column( | ||
| "experiment_name", | ||
| sqlmodel.sql.sqltypes.AutoString(length=255), | ||
| nullable=False, | ||
| ), | ||
| sa.Column("config_id", sa.Uuid(), nullable=False), | ||
| sa.Column("initial_config_version", sa.Integer(), nullable=False), | ||
| sa.Column( | ||
| "status", | ||
| sa.Enum( | ||
| "processing", | ||
| "completed", | ||
| "failed", | ||
| name="evaluationiterationstatusenum", | ||
| ), | ||
| nullable=False, | ||
| comment="Loop bookkeeping status: processing, completed, or failed", | ||
| ), | ||
| sa.Column( | ||
| "stop_reason", | ||
| sqlmodel.sql.sqltypes.AutoString(), | ||
| nullable=True, | ||
| comment="Copied from the final graph state once terminal: ceiling_reached, max_rounds_reached, or round_failed", | ||
| ), | ||
| sa.Column( | ||
| "callback_url", | ||
| sqlmodel.sql.sqltypes.AutoString(), | ||
| nullable=False, | ||
| comment="HTTPS webhook validated via validate_callback_url before create", | ||
| ), | ||
| sa.Column("error_message", sa.Text(), nullable=True), | ||
| sa.Column("organization_id", sa.Integer(), nullable=False), | ||
| sa.Column("project_id", sa.Integer(), nullable=False), | ||
| sa.Column("inserted_at", sa.DateTime(), nullable=False), | ||
| sa.Column("updated_at", sa.DateTime(), nullable=False), | ||
| sa.ForeignKeyConstraint( | ||
| ["dataset_id"], | ||
| ["evaluation_dataset.id"], | ||
| name="fk_evaluation_iteration_run_dataset_id", | ||
| ondelete="CASCADE", | ||
| ), | ||
| sa.ForeignKeyConstraint( | ||
| ["config_id"], | ||
| ["config.id"], | ||
| name="fk_evaluation_iteration_run_config_id", | ||
| ondelete="RESTRICT", | ||
| ), | ||
| sa.ForeignKeyConstraint( | ||
| ["organization_id"], ["organization.id"], ondelete="CASCADE" | ||
| ), | ||
| sa.ForeignKeyConstraint(["project_id"], ["project.id"], ondelete="CASCADE"), | ||
| sa.PrimaryKeyConstraint("id"), | ||
| ) | ||
| op.create_index( | ||
| op.f("ix_evaluation_iteration_run_dataset_id"), | ||
| "evaluation_iteration_run", | ||
| ["dataset_id"], | ||
| unique=False, | ||
| ) | ||
| op.create_index( | ||
| op.f("ix_evaluation_iteration_run_config_id"), | ||
| "evaluation_iteration_run", | ||
| ["config_id"], | ||
| unique=False, | ||
| ) | ||
| op.create_index( | ||
| op.f("ix_evaluation_iteration_run_organization_id"), | ||
| "evaluation_iteration_run", | ||
| ["organization_id"], | ||
| unique=False, | ||
| ) | ||
| op.create_index( | ||
| op.f("ix_evaluation_iteration_run_project_id"), | ||
| "evaluation_iteration_run", | ||
| ["project_id"], | ||
| unique=False, | ||
| ) | ||
|
|
||
|
|
||
| def downgrade(): | ||
| op.drop_index( | ||
| op.f("ix_evaluation_iteration_run_project_id"), | ||
| table_name="evaluation_iteration_run", | ||
| ) | ||
| op.drop_index( | ||
| op.f("ix_evaluation_iteration_run_organization_id"), | ||
| table_name="evaluation_iteration_run", | ||
| ) | ||
| op.drop_index( | ||
| op.f("ix_evaluation_iteration_run_config_id"), | ||
| table_name="evaluation_iteration_run", | ||
| ) | ||
| op.drop_index( | ||
| op.f("ix_evaluation_iteration_run_dataset_id"), | ||
| table_name="evaluation_iteration_run", | ||
| ) | ||
| op.drop_table("evaluation_iteration_run") | ||
| sa.Enum(name="evaluationiterationstatusenum").drop(op.get_bind(), checkfirst=True) | ||
59 changes: 59 additions & 0 deletions
59
backend/app/api/docs/evaluation/create_evaluation_iteration_v2.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| Kick off a self-driving eval -> improve-prompt -> eval loop. | ||
|
|
||
| Each round runs a v2 judged evaluation against `dataset_id` with the current | ||
| config version, then — unless the round's stop score already clears the | ||
| ceiling or `max_rounds` is reached — hands the config to the prompt-improvement | ||
| step to produce the next version and repeats. The loop runs entirely in the | ||
| background; this endpoint only validates the request, creates the tracking | ||
| row, and dispatches round 1. | ||
|
|
||
| The stop score is the mean of **Adherence to Ground Truth** and **Adherence to | ||
| Prompt** from the round's judge summary (Adherence to Knowledge Base is | ||
| reported but never gates). The loop stops on whichever comes first: | ||
|
|
||
| - the stop score reaches the configured ceiling (`ceiling_reached`) | ||
| - `max_rounds` is exhausted (`max_rounds_reached`) | ||
| - a round fails to produce a usable score (`round_failed`) | ||
|
|
||
| `dataset_id` and `config_id`/`config_version` are validated the same way as | ||
| `POST /api/v2/evaluations` (dataset must exist and be accessible, config must | ||
| resolve to a text OpenAI config within the fast-eval row limit) — the loop | ||
| always runs judged. | ||
|
|
||
| ## Rounds cap (optional) | ||
|
|
||
| `max_rounds` bounds how many eval/improve rounds the loop may run. Omit it to | ||
| use `EVAL_ITERATION_MAX_ROUNDS_DEFAULT`; an oversized value is silently | ||
| clamped to `EVAL_ITERATION_MAX_ROUNDS_HARD_CAP` rather than rejected. | ||
|
|
||
| ## Completion webhook (required) | ||
|
|
||
| `callback_url` (HTTPS only) receives the round-by-round report once the loop | ||
| stops, for any reason. Same delivery semantics as the v2 evaluation callback — | ||
| best-effort, at-least-once, signed with `X-Webhook-Signature` / | ||
| `X-Webhook-Timestamp` when a `webhook_secret` credential is configured for the | ||
| project. The URL is rejected with `422 invalid_callback_url` if it is not a | ||
| public HTTPS endpoint (SSRF guard). | ||
|
|
||
| ## Example | ||
|
|
||
| ```json | ||
| { | ||
| "dataset_id": 123, | ||
| "experiment_name": "iterate-smoke-1", | ||
| "config_id": "f54f0d67-4817-4103-9fdf-b74b3d46733e", | ||
| "config_version": 1, | ||
| "max_rounds": 5, | ||
| "callback_url": "https://example.com/webhooks/eval-iteration-complete" | ||
| } | ||
| ``` | ||
|
|
||
| ## Error responses | ||
|
|
||
| | Status | When | | ||
| | --- | --- | | ||
| | 404 | `dataset_id` does not exist or is not accessible to this organization/project | | ||
| | 400 | The config fails to resolve for the given `config_id`/`config_version` | | ||
| | 422 | `invalid_callback_url` — not a public HTTPS endpoint (SSRF guard) | | ||
| | 422 | The config is not a text OpenAI config, or the dataset exceeds the fast-eval row limit | | ||
| | 500 | `evaluation_iteration_enqueue_failed` — the tracking row was created but round 1 could not be queued | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| """v2 evaluation iteration loop trigger — chains eval -> improve-prompt -> eval.""" | ||
|
|
||
| import logging | ||
|
|
||
| from asgi_correlation_id import correlation_id | ||
| from fastapi import APIRouter, Depends, HTTPException | ||
|
|
||
| from app.api.deps import AuthContextDep, SessionDep | ||
| from app.api.permissions import Permission, require_permission | ||
| from app.core.rate_monitor import monitor_rate | ||
| from app.models.evaluation_iteration import ( | ||
| EvaluationIterationCreateRequest, | ||
| EvaluationIterationRunImmediatePublic, | ||
| ) | ||
| from app.services.evaluations.iteration import validate_and_start_evaluation_iteration | ||
| from app.utils import APIResponse, load_description, validate_callback_url | ||
|
|
||
| logger = logging.getLogger(__name__) | ||
|
|
||
| router = APIRouter(prefix="/evaluations", tags=["Evaluation v2"]) | ||
|
|
||
|
|
||
| @router.post( | ||
| "/iterations", | ||
| description=load_description("evaluation/create_evaluation_iteration_v2.md"), | ||
| response_model=APIResponse[EvaluationIterationRunImmediatePublic], | ||
| status_code=202, | ||
| dependencies=[ | ||
| Depends(require_permission(Permission.REQUIRE_PROJECT)), | ||
| Depends(monitor_rate("evaluations")), | ||
| ], | ||
| ) | ||
| def create_evaluation_iteration_v2( | ||
| session: SessionDep, | ||
| auth_context: AuthContextDep, | ||
| request: EvaluationIterationCreateRequest, | ||
| ) -> APIResponse[EvaluationIterationRunImmediatePublic]: | ||
| """Kick off a self-driving eval -> improve-prompt -> eval loop.""" | ||
| try: | ||
| validate_callback_url(str(request.callback_url)) | ||
| except ValueError as exc: | ||
| raise HTTPException(status_code=422, detail=f"invalid_callback_url: {exc}") | ||
|
|
||
| iteration_run = validate_and_start_evaluation_iteration( | ||
| session=session, | ||
| dataset_id=request.dataset_id, | ||
| experiment_name=request.experiment_name, | ||
| config_id=request.config_id, | ||
| config_version=request.config_version, | ||
| max_rounds=request.max_rounds, | ||
| callback_url=str(request.callback_url), | ||
| organization_id=auth_context.organization_.id, | ||
| project_id=auth_context.project_.id, | ||
| trace_id=correlation_id.get() or "N/A", | ||
| ) | ||
|
|
||
| return APIResponse.success_response( | ||
| data=EvaluationIterationRunImmediatePublic( | ||
| iteration_run_id=iteration_run.id, | ||
| status=iteration_run.status, | ||
| message=( | ||
| "Evaluation iteration loop is running; the round-by-round report " | ||
| "will be delivered to your callback_url." | ||
| ), | ||
| inserted_at=iteration_run.inserted_at, | ||
| updated_at=iteration_run.updated_at, | ||
| ) | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
create index on status too, since cron is going to check status on every tick, so having index might improve the query performance.