feat(optimization): GEPA prompt optimization for templates and crews

[nehmetohme]

Summary

Adds a prompt-optimization stack built on MLflow GenAI + GEPA (reflective prompt evolution) that runs entirely inside Kasal — from mining real usage data, through crew-in-the-loop evaluation, to human grading that steers the next optimization round.

Template optimization (Configuration → Prompts)

• The six generation templates (detect_intent, generate_agent, generate_task, generate_crew, generate_crew_plan, generate_job_name) each get an Optimize (✨) action.
• Training data is mined from the LLM interaction logs (with junk filtering); scoring combines output-format validity and an LLM judge on correctness.
• The Prompts configuration surface is consolidated into a single list; optimization opens in a dialog scoped to the chosen template.

Crew optimization (crew catalog → Optimize Prompts)

• Full crew-in-the-loop GEPA: every metric call executes the crew for real and judges score the actual deliverable. Agent role/goal/backstory and task description/expected_output evolve together via a labeled crew-document serialization; a strict parser scores malformed candidates 0 without spending an execution.
• Hard execution cap — the budget the user selects is a promise. GEPA evaluates candidates in parallel and can overshoot its metric-call budget, so the predict_fn refuses executions beyond the cap. A Stop button cancels between iterations. Runs show N/M executions live.
• Apply writes the winning candidate back to the crew's agents and tasks through the repository layer.

Human judgment loop (all inside Kasal, persisted as MLflow assessments)

• Grade past evaluation answers 0–10 with per-judge attribution, comments, and ground-truth Expectations (mlflow.log_feedback + mlflow.log_expectation on the eval traces).
• Create custom LLM judges (mlflow.genai.make_judge) in-app; judges live in a shared library and are assigned per crew via a crew-scoped registered copy. A built-in graded 0–10 quality judge is always on.
• Harvested crew 👍/👎 feedback and human assessments feed both the judge rubric and the GEPA reflection objective — so a non-expert's plain-language feedback autonomously steers prompt evolution.

Infrastructure notes

• Respects the launch MLFLOW_TRACKING_URI (preserved as KASAL_LAUNCH_MLFLOW_TRACKING_URI before main.py forces databricks); a local OSS registry is only used when explicitly configured, otherwise databricks-uc.
• Cross-loop safety: optimization worker threads reach the app DB only via run_coroutine_threadsafe onto the main loop with UserContext re-established; crew execution polling uses fresh sessions per tick.
• MLflow autolog import-hook deadlocks avoided during optimization spans; a 1-token reflection-model preflight fails fast on a dead provider instead of burning budget.
• New dependency: gepa (required by MLflow's GepaPromptOptimizer). The uv.lock churn beyond the gepa addition is uv re-canonicalization only — no other package versions changed.
• Known Phase-1 tradeoff (documented in code): run state is kept in an in-memory registry and is lost on restart; a DB-backed run table is planned as a follow-up.

Test plan

• 25 backend unit tests (tests/unit/services/test_prompt_optimization_service_unit.py): crew-doc serialization round-trips, judge value grading (categorical + numeric), budget accounting, registry gating, apply/cancel flows
• python -c "from src.api import api_router" imports cleanly
• Frontend tsc clean; ESLint 0 errors (remaining warnings pre-exist on main)
• Manually exercised end-to-end locally: template runs with real improvements (e.g. 0.65 → 0.77), crew runs with hard-cap enforcement, stop/cancel, judge creation/assignment/deletion, grading with expectations feeding the next run's reflection objective

🤖 Generated with Claude Code

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}