Add golden workflow prompt dataset - #115
Open
nilramoai wants to merge 4 commits into
Open
Conversation
Summary: - Add a golden dataset fixture for representative Linear issues across each active workflow state. - Document the dataset schema and add an ExUnit test that validates the fixture and renders it through the in-repo WORKFLOW.md prompt. Rationale: - The ticket requested golden dataset creation, and the repo previously had only dashboard snapshot fixtures. Keeping this as fixture data plus validation avoids changing runtime behavior while making prompt regressions detectable. Tests: - cd elixir && mix test test/symphony_elixir/golden_dataset_test.exs - cd elixir && mix test test/symphony_elixir/status_dashboard_snapshot_test.exs - cd elixir && mix format --check-formatted - cd elixir && mix test Co-authored-by: Codex <codex@openai.com>
…n-dataset-creation
Summary: - Update golden prompt expectations for the current follow-up wording in WORKFLOW.md after merging origin/main. - Refer to normalized tracker issues in the dataset metadata and docs. Rationale: - The main branch changed the prompt from continuation/retry wording to follow-up wording and moved the normalized issue type under Tracker. - Keeping the fixture synchronized with the live workflow contract keeps the golden dataset deterministic without changing runtime behavior. Tests: - cd elixir && mix test test/symphony_elixir/golden_dataset_test.exs - cd elixir && mix build - cd elixir && mix format --check-formatted - cd elixir && mix lint - cd elixir && mix test --cover - cd elixir && mix dialyzer --format short - cd elixir && make all (blocked: repo.hex.pm network denied during deps.get) Co-authored-by: Codex <codex@openai.com>
Summary: - Load the repo WORKFLOW.md directly inside the golden dataset test. - Parse active states from the same workflow config used for rendering. - Render the template without depending on shared WorkflowStore state. Rationale: - CI exposed that the golden test could render against a cached test workflow when run in the full suite. - Keeping the fixture validation self-contained makes it deterministic while still proving the checked-in workflow can render every case. Tests: - cd elixir && mix test test/symphony_elixir/golden_dataset_test.exs - cd elixir && mix build - cd elixir && mix format --check-formatted - cd elixir && mix lint - cd elixir && mix test --cover --seed 39961 --max-cases 8 - cd elixir && mix dialyzer --format short - cd elixir && make all (blocked: repo.hex.pm network denied during deps.get) Co-authored-by: Codex <codex@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
The issue requests a golden dataset. The repo had prompt rendering tests but no durable fixture
covering the active workflow states.
TL;DR
Adds a golden workflow prompt dataset and a renderability test.
Summary
Alternatives
Test Plan
make -C elixir allin GitHub Actionscd elixir && mix buildcd elixir && mix format --check-formattedcd elixir && mix lintcd elixir && mix test test/symphony_elixir/golden_dataset_test.exscd elixir && mix test --cover --seed 39961 --max-cases 8cd elixir && mix dialyzer --format short