fix(harness): improve auto-rename prompt quality

[dimakis]

Summary

• Rewrites the Haiku system prompt with few-shot examples and strict constraints to produce specific, action-oriented session titles instead of conversational responses
• Adds sanitizeSessionName() that strips quotes, truncates at newlines, and rejects known bad patterns (apologies, generic starters, conversational filler)
• Separates prompts with --- delimiters instead of bare newlines for clearer multi-prompt input
• Logs rejected/empty titles for observability

Problem

Haiku was producing garbage session titles in production:

• "I apologize, but I cannot access the specific Jira ticket li" — full conversational response
• "\"OpenShift OpenCode Agent Integration\"\n\nWould you like me to" — embedded quotes + trailing conversation
• "Start a Conversation" — generic filler when it couldn't extract intent

Test plan

• 31 tests pass including new sanitization + integration tests
• Full suite shows no regressions (pre-existing failures unrelated)
• Deploy and observe auto-rename log quality over a day of sessions

🤖 Generated with Claude Code

[dimakis]

Centaur Review

Found 2 issue(s).

packages/harness/src/auto-rename.ts

Clean quality-of-life improvement to auto-rename. Sanitization logic is well-tested and the prompt upgrade is solid. Two minor suggestions: tighten the unanchored would you like rejection pattern, and simplify the now-identity Vertex model map.

• 🔵 unsafe_assumptions (L33): The /would you like/i pattern is the only REJECTED_PATTERN not anchored to ^. This means a legitimate title like "Fix Would You Like Dialog" would be falsely rejected. Anchoring it to end-of-string (/would you like.*$/i) or keeping it unanchored but with more context (/would you like (me |to )/i) would reduce false positives while still catching Haiku run-on. [fixable]

packages/harness/src/providers/anthropic-vertex.ts

Clean quality-of-life improvement to auto-rename. Sanitization logic is well-tested and the prompt upgrade is solid. Two minor suggestions: tighten the unanchored would you like rejection pattern, and simplify the now-identity Vertex model map.

• 🔵 style (L17): VERTEX_MODEL_MAP is now an identity mapping — every key maps to itself. The fallback VERTEX_MODEL_MAP[this.model] ?? this.model would return the same value with or without the map. It documents "supported models" but a Set or inline comment would communicate that intent more clearly and avoid confusion about whether the map is supposed to transform anything. [fixable]

---

Reviewed at 55ff47c

[dimakis]

🔵 unsafe_assumptions: The /would you like/i pattern is the only REJECTED_PATTERN not anchored to ^. This means a legitimate title like "Fix Would You Like Dialog" would be falsely rejected. Anchoring it to end-of-string (/would you like.*$/i) or keeping it unanchored but with more context (/would you like (me |to )/i) would reduce false positives while still catching Haiku run-on. [fixable]

[dimakis]

🔵 style: VERTEX_MODEL_MAP is now an identity mapping — every key maps to itself. The fallback VERTEX_MODEL_MAP[this.model] ?? this.model would return the same value with or without the map. It documents "supported models" but a Set or inline comment would communicate that intent more clearly and avoid confusion about whether the map is supposed to transform anything. [fixable]

[dimakis]

Centaur Review

Found 5 issue(s) (2 warning).

packages/harness/src/providers/anthropic-vertex.ts

Solid improvement to auto-rename robustness with good sanitization logic and test coverage. Main concern is whether the Vertex model map removal could break callers that pass standard (non-Vertex) model names to Vertex clients.

• 🟡 regressions (L54): The VERTEX_MODEL_MAP removal means callers must now pass Vertex-format model names (e.g. 'claude-haiku-4-5@20251001') directly. If any caller passes standard API names like 'claude-opus-4-6' to a Vertex client, the API call will fail — previously the map translated these. Verify that all call sites that construct AnthropicVertexModelProvider with Vertex enabled pass Vertex-compatible model names.

server/auto-rename.ts

Solid improvement to auto-rename robustness with good sanitization logic and test coverage. Main concern is whether the Vertex model map removal could break callers that pass standard (non-Vertex) model names to Vertex clients.

• 🟡 missing_tests: sanitizeSessionName is not re-exported from server/auto-rename.ts (nor from packages/harness/src/index.ts). The server test file imports from ../auto-rename.js and cannot test sanitizeSessionName. Either add it to the re-export list in both files, or accept that sanitizeSessionName is only tested via the harness test suite. If it's intentionally internal, the export in auto-rename.ts should be test-only — but then the harness test imports it as a public export. [fixable]

packages/harness/__tests__/auto-rename.test.ts

Solid improvement to auto-rename robustness with good sanitization logic and test coverage. Main concern is whether the Vertex model map removal could break callers that pass standard (non-Vertex) model names to Vertex clients.

• 🔵 missing_tests: No test for sanitizeSessionName with smart/curly quotes (U+201C, U+201D, U+2018, U+2019) even though the regex explicitly handles them. Adding a test like sanitizeSessionName('\u201CHello\u201D') === 'Hello' would verify the Unicode quote stripping works. [fixable]

packages/harness/src/auto-rename.ts

Solid improvement to auto-rename robustness with good sanitization logic and test coverage. Main concern is whether the Vertex model map removal could break callers that pass standard (non-Vertex) model names to Vertex clients.

• 🔵 unsafe_assumptions (L243): The quote-stripping regex uses a character class that includes both whitespace (\s) and quote chars with + quantifier: ^[\s"'\u201C\u201D\u2018\u2019]+. This means a title like " Spaced Title " correctly produces Spaced Title, but a title that legitimately starts with an apostrophe (e.g. 'Twas) would lose it. Low-risk for session names but worth noting.

server/__tests__/auto-rename.test.ts

Solid improvement to auto-rename robustness with good sanitization logic and test coverage. Main concern is whether the Vertex model map removal could break callers that pass standard (non-Vertex) model names to Vertex clients.

• 🔵 style (L161): The server test relaxed its assertion from exact argument matching (toHaveBeenCalledWith) to individual field checks (call[0].model, call[0].messages[0].content). This is fine for resilience against prompt changes, but the system prompt (NAMING_SYSTEM_PROMPT) is now completely unverified in the server test suite. Consider adding a minimal assertion like expect(call[0].system).toContain('3-6 word title') so the server tests catch accidental system prompt deletions. [fixable]

---

Reviewed at 474210d

[dimakis]

🟡 regressions: The VERTEX_MODEL_MAP removal means callers must now pass Vertex-format model names (e.g. 'claude-haiku-4-5@20251001') directly. If any caller passes standard API names like 'claude-opus-4-6' to a Vertex client, the API call will fail — previously the map translated these. Verify that all call sites that construct AnthropicVertexModelProvider with Vertex enabled pass Vertex-compatible model names.

[dimakis]

🔵 unsafe_assumptions: The quote-stripping regex uses a character class that includes both whitespace (\s) and quote chars with + quantifier: ^[\s"'\u201C\u201D\u2018\u2019]+. This means a title like " Spaced Title " correctly produces Spaced Title, but a title that legitimately starts with an apostrophe (e.g. 'Twas) would lose it. Low-risk for session names but worth noting.

[dimakis]

🔵 style: The server test relaxed its assertion from exact argument matching (toHaveBeenCalledWith) to individual field checks (call[0].model, call[0].messages[0].content). This is fine for resilience against prompt changes, but the system prompt (NAMING_SYSTEM_PROMPT) is now completely unverified in the server test suite. Consider adding a minimal assertion like expect(call[0].system).toContain('3-6 word title') so the server tests catch accidental system prompt deletions. [fixable]