feat(backchecks): track progress against the backcheck target

[iabaako]

Pull Request Summary 🚀

Closes #318. Stacked on #319 (#300): this PR targets fix/300-backcheck-date-join and should be retargeted to main once #300 merges.

What does this PR do? 📝

The backchecks report now tracks progress against the backcheck target.

• Targets section in the Backchecks Summary, with two metrics side by side:
  • Backcheck Coverage: the share of eligible unique survey IDs that have at least one matching backcheck, with points above or below the target % as the delta (for example "23.0%", "+13.0 pts vs 10%").
  • Backchecks vs Expected: backchecks done against the expected total (survey target × target %, rounded up), with how many above or below as the delta (for example "29 / 20 (145%)", "+9 backchecks vs target"). When the survey target is not set, a hint replaces it.
• Counts row: the Survey Observations, Backcheck Observations, Total Enumerators and Total Back Checkers cards now share one row.
• Target % resolution: the settings panel value wins, then the page configuration, then 10% with a warning. The panel input is now "Backcheck target (%)", limited to 0–100. Clearing it falls back to the page configuration.
• Eligibility filter (optional): a survey column plus the values that mark a survey eligible (for example consent = 1), saved like the other panel settings.
• Enumerator Backchecker Error Statistics table:
  • Enumerator view: "Surveys" is now the enumerator's eligible unique submissions and "Backchecks" how many of those were backchecked. New "Coverage %" and "vs target" columns are added; coverage below target is highlighted, and enumerators with no backchecks show 0%.
  • Backchecker view: backchecks done only.
  • The table renders before comparison columns are configured, with a hint to configure them for error rates.

Why is this change needed? 🤔

The backcheck target % was set in two places but nothing read it (#299). The old "Backcheck Coverage %" divided matched keys by raw survey rows and read 0% until comparison columns were configured. The table's "Surveys" and "Backchecks" columns were always equal. Teams had no way to see whether backchecks were keeping pace with the target, overall or per enumerator.

How was this implemented? 🛠️

• New module checks/backchecks/coverage.py, with no Streamlit dependency:
  • Applies duplicate handling to both datasets (as the backcheck comparison does), applies the eligibility filter, and flags each unique survey ID as backchecked or not. Matching is by survey ID only, so no comparison columns are needed.
  • Computes the overall coverage (eligible, backchecked, on-track %, points vs target, expected backchecks and progress) and per-staff coverage.
  • Resolves the effective target % and builds the backcheck settings from the page configuration. A page-configuration target of 0 counts as not set, because that form stores 0 when the field is left blank.
• Settings:
  • The backcheck target % can now be "not set", and the settings gain the survey target and the eligibility fields.
  • A cleared panel value, or a saved value outside 0–100 (from the old count-based input), falls back to the page configuration.
  • Clearing the input restores the page-configuration target straight away.
• Report UI: the summary and the enumerator table read from the coverage module. The error-rate statistics no longer produce the always-equal "Surveys" and "Backchecks" columns; the coverage module supplies them and the error columns are joined on.
• Page wiring: the generated report view now passes the survey target through to the backchecks report.
• Docs: CHANGELOG (Added, plus two Breaking entries under Changed), the user guide's Back Checks section, and the onboarding text.

Breaking for internal callers: the target % setting no longer defaults to 10, and the error-rate statistics no longer return "Surveys" and "Backchecks". Both are listed under Changed in the CHANGELOG.

How to test or reproduce ? 🧪

1. Run just test. The new tests/checks/backchecks/test_coverage.py covers target resolution (panel, page config, default, cleared and out-of-range values), the eligibility filter, on-track %, points vs target, expected backchecks (rounding up, over 100%, unset), and per-enumerator and per-backchecker coverage.
2. In the app, open a page with backcheck data (for example the demo survey and demo backcheck datasets, with Survey ID "hhid", Enumerator "enum_name" and Back Checker "bcer_name") and go to the Backchecks tab:
   • Before adding any backcheck columns, the Targets section and the enumerator table should already show coverage, and the table should show a hint about error rates.
   • With a survey target of 200 and a 10% target, the demo data shows 23.0% coverage (+13.0 pts vs 10%) and 29 / 20 (145%) backchecks (+9 backchecks vs target).
   • Raise the panel target to 40%: both deltas turn red (−17.0 pts and −51 backchecks).
   • Clear the panel target: it falls back to the page-configuration target. With neither set, a warning says the 10% default is used.
   • Set the eligibility filter: the coverage base shrinks to eligible surveys only.
   • Remove the survey target from the page configuration: the Backchecks vs Expected card is replaced by a hint.

Screenshots (if applicable) 📷

Checklist ✅

• I have run and tested my changes locally
• I have limit this PR to less than 1000 lines of code change (if not, explain why)
  • 1,022 changed lines in total, but only about 620 are in src/; the rest are the new coverage tests (about 300 lines) and the CHANGELOG and user guide updates.
• I have updated/added tests to cover my changes (if applicable)
• I have updated/added requirements to cover my changes (if applicable)
  • N/A: no new dependencies.
• I have run linting and formatting on any code changes (if applicable)
• I have updated the documentation (README, etc.) accordingly
• I have reviewed and resolved any merge conflict
  • Branched from fix/300-backcheck-date-join; no conflicts.

Reviewer Emoji Legend

	:code:	Meaning
😃👍💯	:smiley: :+1: :100:	I like this... ...and I want the author to know it! This is a way to highlight positive parts of a code review.
⭐⭐⭐	:star: :star: :star:	Important to fix before PR can be approved... And I am providing reasons why it needs to be addressed as well as suggested improvements.
⭐⭐	:star: :star:	Important to fix but non-blocking for PR approval... And I am providing suggestions where it could be improved either in this PR or later.
⭐	:star:	Give this some thought but non-blocking for PR approval... ...and consider this a suggestion, not a requirement.
❓	:question:	I have a question. This should be a fully formed question with sufficient information and context that requires a response.
📝	:memo:	This is an explanatory note, fun fact, or relevant commentary that does not require any action.
⛏	:pick:	This is a nitpick. This does not require any changes and is often better left unsaid. This may include stylistic, formatting, or organization suggestions and should likely be prevented/enforced by linting if they really matter
♻️	:recycle:	Suggestion for refactoring. Should include enough context to be actionable and not be considered a nitpick.

🤖 Generated with Claude Code

[Copilot]

Copilot review overview

🟡 Changes recommended

A valid 0% target crashes the summary, and the required expected-total progress bar is missing.

Review effort: Balanced
Findings: 1

Open (1)

• Add clamped progress bar to expected-total card · New

What changed in this PR

Adds target-based backcheck coverage tracking across calculations, settings, reports, and documentation.

Changes:

• Adds overall and per-staff coverage calculations with eligibility filtering.
• Adds target configuration and expected-backcheck UI metrics.
• Updates tests, documentation, and page wiring.

File	Description
src/​datasure/​checks/​backchecks/​coverage.py	Implements coverage calculations.
src/​datasure/​checks/​backchecks/​models.py	Extends backcheck settings.
src/​datasure/​checks/​backchecks/​compute.py	Updates settings resolution and statistics.
src/​datasure/​checks/​backchecks/​settings_ui.py	Adds target and eligibility controls.
src/​datasure/​checks/​backchecks/​report_ui.py	Renders target metrics and staff coverage.
src/​datasure/​views/​output_view_template.py	Passes the survey target to backchecks.
src/​datasure/​utils/​onboarding_utils.py	Updates onboarding guidance.
tests/​checks/​backchecks/​test_coverage.py	Tests coverage behavior.
tests/​checks/​backchecks/​test_settings_ui.py	Updates settings tests.
tests/​checks/​backchecks/​test_report_ui.py	Updates summary tests.
tests/​checks/​backchecks/​test_models.py	Updates model defaults.
tests/​checks/​backchecks/​test_compute.py	Updates statistics expectations.
tests/​checks/​backchecks/​conftest.py	Extends Streamlit mocks.
docs/​USER_GUIDE.md	Documents target tracking.
CHANGELOG.md	Records features and breaking changes.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[sonarqubecloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
78.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud