The problem
Long answers were truncated with 504 Request failed; short answers were fine.
The cause
One line of guardrails config. rails.output.streaming had context_size: 300 — larger than chunk_size: 200. NeMo's buffer drains with buffer[-context_size:], so a 200-item buffer never shrank below its own flush threshold. After the first 200 tokens, every single token triggered its own guardrail LLM call.
A 360-token answer made 162 serial LLM calls instead of 3, taking 74s instead of seconds. That blew past the gateway's 60s timeout on the trigger POST → 504 → the GUI closed a perfectly healthy stream mid-answer.
The fix
| Metric |
Before |
After |
| Guardrail calls (360 tokens) |
162 |
3 |
| Total request |
74.2s |
seconds |
| Cost |
$0.202 |
~$0.004 |
$0.199 of every request was wasted re-validation, hidden inside the streaming_generation cost bucket.
Four parts:
- context_size: 300 → 50 — the actual fix, plus a startup guard that clamps and logs loudly if it ever regresses
- Generator fix — extract_guardrails_prompts.py had 300 hardcoded, so re-running the optimizer would have silently reintroduced the bug
- Streaming resilience — client disconnect now aborts upstream generation, real heartbeats, idle watchdog, explicit timeouts; a failed POST no longer kills a healthy stream
- Typewriter pacing in the GUI — word-by-word rendering, since guardrails inherently release tokens in blocks
The problem
Long answers were truncated with 504 Request failed; short answers were fine.
The cause
One line of guardrails config. rails.output.streaming had context_size: 300 — larger than chunk_size: 200. NeMo's buffer drains with buffer[-context_size:], so a 200-item buffer never shrank below its own flush threshold. After the first 200 tokens, every single token triggered its own guardrail LLM call.
A 360-token answer made 162 serial LLM calls instead of 3, taking 74s instead of seconds. That blew past the gateway's 60s timeout on the trigger POST → 504 → the GUI closed a perfectly healthy stream mid-answer.
The fix
$0.199 of every request was wasted re-validation, hidden inside the streaming_generation cost bucket.
Four parts: