Add Jev job-lead shadow observations - #418
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_85ed22b1-a972-44f5-9e70-5fbda0b46adf) |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 51f6842027
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_154d5dc7-6390-4910-b749-6674ed4b4a18) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d9e0450ed0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
d9e0450 to
88d919d
Compare
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_3cc78085-5f98-44e1-8881-7fba085264c8) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 88d919d103
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
88d919d to
e2cabae
Compare
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_db3849e3-e9cc-445a-8b1c-72544e6adec9) |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_deaf352e-a6b5-4b88-9ce2-6c75e3717fba) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1c05c6c4d2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| value_type="float", | ||
| env_names=("JOB_LEAD_JEV_SHADOW_SAMPLE_RATE",), | ||
| min_value=0.0, | ||
| max_value=1.0, |
There was a problem hiding this comment.
When an administrator enters NaN for this dashboard-managed float, coerce_runtime_config_value accepts it because both range comparisons against NaN are false. _bounded_float then converts that NaN to the configured maximum, so a NaN sample rate unexpectedly becomes 1.0; the new timeout and run-budget fields have the same issue and become their maximums. This can turn a partial or paused experiment into 100% sampling with maximum execution limits, so these float settings should reject non-finite values before persistence.
Useful? React with 👍 / 👎.
| agrees_with_primary=( | ||
| decision.is_contractor_friendly == classification.is_contractor_friendly | ||
| ), |
There was a problem hiding this comment.
When Jev and the primary classifier produce the same contractor-friendly boolean but different posting types, this records an agreement and the report creates no review item unless the confidence gate also falls back. Posting type affects downstream forum selection, so differences such as part_time versus part_time_or_full_time are production-relevant, yet they disappear from the aggregate experiment results even though both values are collected. Track posting-type or joint agreement separately and surface those disagreements for review.
Useful? React with 👍 / 👎.
| result = float(value) | ||
| except (TypeError, ValueError): | ||
| return None | ||
| return result if result >= 0 else None |
There was a problem hiding this comment.
If OpenRouter returns an overflowing JSON number such as 1e999, or the string "Infinity", for usage.cost, float(value) produces positive infinity and this accepts it as a valid cost. The infinite value is then placed in lead metadata and the scrape-job result; Python serializes it as the non-JSON token Infinity, which PostgreSQL jsonb rejects when the lead or job result is persisted. Thus a malformed non-authoritative shadow response can fail the authoritative scrape instead of being isolated as a shadow failure; require provider usage values to be finite.
Useful? React with 👍 / 👎.
1 participant
Summary
Safety
Viewing shadow data
Enable the Jev shadow under Dashboard > Configuration > AI. Then open Dashboard > Background tasks, select a scrape_job_leads_job run, choose Details, and inspect Result > classifier_shadow.
Validation
Dependency
This PR is stacked on #417 because that PR introduces the evaluated corpus and harness contract reused here. PR #417 is green and mergeable, but GitHub requires approval from someone other than its last pusher before merge-queue admission.
Note
Medium Risk
Extends the scrape-time classifier with bounded OpenRouter calls and new metadata, but production decisions are unchanged; misconfiguration or provider outages only affect shadow telemetry and scrape latency within configured budgets.
Overview
Adds an optional, disabled-by-default Jev shadow on HN job-lead scraping: after the authoritative LLM/heuristic classifier runs, a shared
job_lead_jevmodule can call OpenRouter Decisions on a deterministic sample and attach normalized observations without changing production labels or persistence.Configuration and ops: New
JOB_LEAD_JEV_SHADOW_*settings land in worker config, dashboard runtime config (AI),.env.example, and docs. Scrape results gainResult → classifier_shadowwith agreement rates, confidence-gate stats, latency, token/cost usage, bounded review items (IDs/URLs only), and run-level caps (default 25 calls / 20s). One provider failure disables further Jev calls for that scrape run.Refactor: Eval harness Jev path delegates to the same contract/transport as production shadow.
Reviewed by Cursor Bugbot for commit 1c05c6c. Bugbot is set up for automated code reviews on this repo. Configure here.