[harlan-zw]

🔗 Linked issue

Related to #440

❓ Type of change

• 📖 Documentation
• 🐞 Bug fix
• 👌 Enhancement
• ✨ New feature
• 🧹 Chore
• ⚠️ Breaking change

📚 Description

After #440 shipped, GET /api/skills/search returned "understood": null for most sentences, so the search box answered them without the model. Three causes combined:

• Understandings were cached per data center. They lived in the edge-cache mount, which is the Workers Cache API. A sentence understood in one colo was a miss in the next, so the model ran again. The same query sent twice within seconds hit the cached understanding on one request and re-ran the model on the other.
• The budget sat under the model's real latency. Workers AI analytics for @cf/meta/llama-3.1-8b-instruct-fast showed inference alone at p50 614ms, p90 815ms and max 1256ms over 24 calls. With an 800ms budget, many of those reruns timed out.
• The answer cache hid the late reply. The minute-long answer cache, plus its five-minute stale window, was keyed by the query alone, so it kept serving the fallback after the late model reply was cached.

The fix:

• Understandings now live in the KV-backed cache mount, which every colo reads. The volume is one write per new sentence, so the KV write cost stays negligible.
• The budget is 1200ms.
• The route reads the understanding first and keys the answer cache on it (skills-box:v2). A fallback answer and an understood answer for the same query no longer share an entry.
• Every model call and skip now emits a search-intent wide event with its result and model.durationMs. Previously only fallbacks were logged, which is why skips and slow calls were invisible.

The AI binding and the rate limiter work. The model also still serves with no error codes, although it is missing from the current Workers AI catalog listing.

🤖 AI disclosure: Harlan Agent Kit modified this description. My AI open-source policy.

[harlan-zw]

Agent check, run by hand because CI cannot cover it.

Production before this change, read only:

• write better commit messages with limit=3 returned the cached understanding in 323ms. The same query with limit=4, seconds later, re-ran the model and fell back in 1395ms. That is one cache entry seen by one colo and missed by another.
• Workers AI analytics for the model over 3 hours: 24 calls, inference p50 614ms, p90 815ms, max 1256ms, no error codes.

This branch, as a local build reading production D1, Vectorize and Workers AI:

• 10 of 12 cold sentences came back understood on the first request. The two misses were model calls over 1200ms through the local remote-binding proxy.
• Both misses were understood on the very next request, inside the minute the old answer cache would have kept serving the fallback.
• The new events log each call, for example outcome: understood, reason: model-call, model.durationMs: 571 and outcome: understood, reason: late-reply, model.durationMs: 1276.

To confirm after deploy: send one new sentence twice with different limit values. Both answers should carry the same understood terms.

[harlan-github-agent]

🤖 MERGED

Harlan Agent Kit posted this automated review. It is not Harlan's personal review or approval. AI open source policy. Last updated: 2026-10-05 17:53 UTC.

GitHub merged this pull request.

• Logged (22/100): Concurrent identical intent queries no longer share one model call per isolate. View code
  • Ask an agent to verify and fix this finding

Review completed. No finding above 80/100.

Selected findings run after merge and open separate pull requests.