Repository navigation
X discovery projects past the accepted $40 monthly spend line #179
Description
X discovery projects $42.60/mo, over the $40 accepted line, because the 2000-read DAILY_DISCOVERY_READ_BUDGET never binds at the observed 284 reads/day and needs lowering to about 250.
Target: shared/server/x-ingest.ts
Verify with: pnpm vitest run test/unit/x-ingest.test.ts
Estimated to change 1 file.
The daily-checkin routine opened this issue automatically on its 2026-09-07T21:40:00.000Z run. It is not Harlan's own report. Close it to reject the proposal, and the reason you give stops it being offered again.
Activity
harlan-github-agent commentedon Sep 7, 2026
🤖 ISSUE TRIAGE
harlan-zw commentedon Oct 5, 2026
Agent triage by Codex, 2026-10-05. This is an agent assessment.
Keep as the canonical X budget and discovery-scope decision. Difficulty 3/5, impact 4/5.
Production reads show 1992/2000 discovery reads spent today and 32 hot posts. The current collector formula projects $303.60/month. This is an estimate, not a billing statement.
A lower read ceiling alone cannot solve the issue. Current discovery saturates the ceiling. Reducing it grows the backlog beyond X's seven-day retention window.
Next step: choose narrower discovery coverage within the accepted $40/month line, or explicitly revise that line. Narrower coverage is the recommended direction under VISION.md's cost constraint. Do not implement an arbitrary 250-read ceiling.
Telemetry follow-up: eight remaining reads cannot satisfy X's ten-result minimum. The exhaustion signal should use the minimum request size.
harlan-zw commentedon Oct 5, 2026
Harlan Agent Kit wrote this investigation. It is an agent assessment.
The current projection does not establish actual X spend.
X pricing lists $0.005 per Post and $0.010 per User.
X usually deduplicates each resource within its UTC day. This guarantee has exceptions.
Our 2,000 ceiling counts returned Posts, including repeat IDs. It does not count invoice charges.
Both discovery and refresh request author expansions. Their billable User IDs are not recorded separately.
Read-only production measurements on 2026-10-05, filtered to platform X:
Discovery and refresh share observations. These counts cannot separate their billed usage.
This table corrects earlier measurements that included Bluesky in the shared tables.
The earlier 463 new Posts covered a rolling 24 hours and both platforms.
It cannot be compared directly with today's 1,992 returned X reads.
Text extracts do not identify which query arm returned a Post.
Literal GitHub links in those extracts cannot establish query selectivity.
#409 addresses repeated first-page reads. Measure complete UTC days after that fix.
A lower ceiling alone can hide unread results until X's seven-day search retention expires.
Recommended sequence:
#410 treats fewer than ten remaining returned reads as exhausted.
It records distinct observed X Posts separately from the Post-only spend estimate.
That exhaustion flag warns of possible backlog. It does not prove unread results remain.
The remaining decision is query coverage if actual billed spend exceeds $40 after continuation recovery.
Changing refresh timing alone cannot remove discovery charges. A provider spending limit pauses API access when reached.
harlan-zw commentedon Oct 5, 2026
Codex wrote this agent investigation update, October 5 at 08:15 UTC.
#409 and #410 are merged. #410's final pull request checks passed.
At 08:31 UTC, the telemetry change was verified live on descendant commit
48b0590d.Deploy 37283498548 and production smoke passed.
Cloudflare serves version
7d52a867at 100% traffic.A complete measurement period is still needed.
At 08:11 UTC, the returned-read budget remained 1,992 of 2,000.
Eight remaining reads cannot satisfy the minimum request size.
Distinct observed X Posts remain 498 for October 4 and 499 for October 5.
No completed post-pagination UTC day is available yet.
The next budget reset is October 6 at 00:00 UTC, 11:00 Melbourne.
The first complete day can be sampled October 7 at 00:15 UTC, 11:15 Melbourne.
Check cursor and window exhaustion to distinguish catch-up from steady discovery.
Compare three complete days with actual Developer Console resource counts and spend.
The budget and query remain unchanged pending those measurements.