Evaluation docs revamp#755
Merged
Merged
Conversation
Task-oriented guide for building a composite eval from the eval builder: switch to Composite, configure the child type and aggregation function, add child evals, set weights, save. Grounded in the live eval builder (EvalCreatePage.jsx, CompositeDetailPanel.jsx) and the create-composite endpoint.
… References section
…s/eval-ref-output-types Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…move its nav entry Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
docs(evaluation): Output types & scoring (reference)
…s/eval-ref-sdk-api
…st wheel, fix reader flow) Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
docs(evaluation): SDK & API (reference)
…s/eval-builtin-polish
…methods in the intro Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
… span/trace/session level) Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…coring Reference-shaped rewrite verified on dev (normalized output types, pass threshold, choice scores, aggregates). Repoints every reference, adds redirects for the moved URL and a direct eval-results entry to avoid a two-hop chain at rollup, restores the SDK & API footer card, and bridges the underlying 0-1 score model per reader review. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
docs(evaluation): built-in evals reference — intro + Next Steps
…s/eval-guide-ground-truth
…re flow Replaces the five stale screenshots with eight current annotated ones (3024x1964, descriptive names), rewrites the guide step-per-image to match the shipped Ground Truth tab (Inputs, Reference output, Retrieval), completes the evaluator model rename, and keeps footer cards to sidebar pages only. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
docs(evaluation): Add ground truth (how-to)
Replace the fictional example (response_quality_check / Score children) with the actual builder flow: customer_evals_composite, Pass/Fail children customer_agent_task_completion and customer_agent_human_escalation on a weighted average. Add the Select Evaluation drawer search-and-add loop, the Added-vs-Add state, variable mapping into Test Data, and a Test-and-save step. Swap the 7 bad annotation images for clean 3024x1964 annotated captures. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
docs(evaluation): Build a composite evals (how-to)
…eenshots Rewrite the how-to around the actual builder flow with a hateful_speech Agent Evaluator example: open builder, name and type, instructions with the Mustache/Jinja format choice, reasoning level (Auto/Agent/Quick) and evaluator model, output type, test, and save. Add the copyable evaluator prompt, list the reasoning levels, link Eval templates & versions for the variable formats, and match the SDK example to the same hateful_speech eval. Swap the stale numbered screenshots for annotated 3024x1964 captures and delete the orphaned old ones. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
docs(evaluation): rewrite Create a custom eval guide from real UI screenshots
Renaming to Advanced usage
…2 templates - Rewrite 77 built-in eval pages to a uniform template (hero lead, What it does with clean Input/Output tables, Run-from-code note + CodeGroup, When to use, What to do when it fails) - Add 13 new pages covering the previously undocumented evals, including 6 grouped reference pages for statistical, NLP, code-output, ASR, retrieval, and similarity/image metrics - Use registry template ids (backticked) consistently in all grouped metric tables - Add a setup Note above every code snippet (SDK install, API keys, evaluator model) - Restructure the built-in evals nav into 7 use-case folders + Statistical & NLP metrics, with 4-level sidebar nesting - Update the catalog Overview to all 132 templates
…up flow - Replace the Core SDK Functions + Full Implementation split with a single Steps walkthrough (requirements, eval script, workflow, secrets, open PR) - Define eval_data once instead of duplicating it - Move the evaluate_pipeline / get_pipeline_results param tables into a reference section after the walkthrough - Fix the tone eval input key (output, not input) and give groundedness a real supporting context passage, both verified against the eval registry - Convert About to a hero lead, rename the footer to Dive deeper, embed the screenshot with a caption, and note that Future AGI never runs your pipeline
…ate metric inputs - Hide 6 nav items whose pages aren't built yet (Explore playground folder, guardrails and feedback guides), commented with a re-add note so the sidebar has no 404s while shipping - Flag the CI/CD walkthrough as GitHub Actions specific and note post_github_comment is the part to swap for other CI systems - Correct the required-inputs columns on the statistical and code-output metric pages: add required params positive_label (f_beta_score, precision_score) and max_latency_ms (latency_check), and mark fleiss_kappa/image_properties input shapes
|
Automatic Review Skipped Too many files for automatic review. If you would still like a review, you can trigger one manually by commenting: |
atharva-bhange
previously approved these changes
Jul 16, 2026
- Delete 6 nav items for unbuilt pages (Explore playground folder, guardrails, feedback); the link audit reads hrefs out of // comments, so they had to be removed rather than commented - Repoint every features/* link the merge pulled in to its guides/* home (custom->custom-evals, custom-models, evaluate->running-evaluations, cicd) across eval concepts, index, faq, dataset, simulation, and cookbook pages - Drop the unbuilt Playground row from the run-surfaces table in running-evaluations
atharva-bhange
previously approved these changes
Jul 16, 2026
- Redirect the 4 deleted features/ pages to their guides/ homes (cicd, custom->custom-evals, evaluate->running-evaluations, futureagi-models-> concepts/evaluator-models) so live inbound URLs don't 404 - Repoint 5 stale /future-agi/get-started redirects that pointed at the now-deleted features/* pages
atharva-bhange
approved these changes
Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Description
Describe the changes in this pull request:
Checklist
Related Issues
Closes #<issue_number>