diff --git a/CHANGELOG.md b/CHANGELOG.md index babd7d8f..2f60578f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,7 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `src/datasure/processing/correction_log.py`, shared with the replication package, which now exports legacy logs with these columns) — #296 - **Accept action**: `CorrectionProcessor.accept_value` records that a flagged - value (outliers, constraints, backchecks, duplicates, GPS) was reviewed and + value (outliers, constraints, duplicates, GPS) was reviewed and is correct, with a required reason. An acceptance is rejected if the data no longer holds the value being accepted. `get_active_acceptances` returns the acceptances whose recorded value still matches the data (for GPS, both @@ -102,9 +102,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 sidebar, saved in `cache/user_settings.json`, else the OS login (`getpass.getuser()`). Existing logs load with a null `user`. The Correction Log table and `correction_log.csv` include it — #321 +- **Backcheck mismatch attribution**: New Streamlit-free + `checks/backchecks/attribution.py`. Reviewers attribute mismatches in + Comparison Results Details to an `ErrorSource` (Enumerator, Backchecker, + Respondent, Unattributed) from a pinned Review button that opens a dialog; + selecting several mismatch rows and clicking Review on one of them + attributes them together. Only `match_status == "mismatch"` rows can be + attributed, and Backchecker and Respondent need a note. Entries are appended + to `bc_attribution_{page_name_id}` in the `logs` db with the survey and + backcheck KEYs, column, both values (as text), source, note, user + (`get_reviewer_name`) and date. `mark_error_sources` adds an `error_source` + column: the latest entry per KEY pair and column applies only while both + values still equal the stored ones, otherwise the mismatch is Unattributed. + An Attribution log expander lists the history. Attribution never changes + the data, the mismatch counts or the regular error rate — #301 +- **Adjusted error rate**: `compute_enumerator_backchecker_stats` adds + "Adjusted Error Rate % (Cat n)" and "(Total)": for enumerators + (mismatches − Backchecker − Respondent) ÷ values compared, for backcheckers + (mismatches − Enumerator − Respondent) ÷ values compared. Unattributed + mismatches always count. `compute_column_stats` adds "Enumerator / Backchecker + / Respondent / Unattributed Mismatches" counts. The Backchecks Summary shows + "Mismatches Attributed" (% of mismatches with a source). New optional + `BackcheckSettings.error_rate_target_percent` ("Error rate target (%)" in + Tracking Options): each regular and adjusted rate column above it is + highlighted in both the enumerator and backchecker views — #301 +- **Overall backcheck error rates**: `compute_overall_error_rates` returns an + `OverallErrorRate` (compared, mismatches, error rate, enumerator and + backchecker adjusted rates) for the total over categories 1–3 and for each + category. The Backchecks Summary gains an Error Rates section below Targets + with one card each, the enumerator adjusted rate as a grey, arrowless delta + and the backchecker adjusted rate in the help — #301 ### Changed +- **Breaking**: `backchecks` is no longer in `ACCEPT_CHECK_TYPES`, so a + backcheck mismatch can't be accepted; it can only be attributed — #301 + - **Breaking**: `BackcheckSettings.backcheck_target_percent` is now `float | None` (0–100), defaulting to None instead of 10; `effective_target_percent` applies the 10% default. The target resolves from diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index e66ce4e8..18559d01 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -96,8 +96,10 @@ Per project (UUID-keyed): - `cache/{project_id}/data/` — the DuckDB databases `raw.duckdb`, `prep.duckdb`, `corrected.duckdb` -- `cache/{project_id}/settings/` — `logs.duckdb` (import/prep logs), JSON - settings, and credential metadata +- `cache/{project_id}/settings/` — `logs.duckdb` (import/prep logs, and the + append-only backcheck attribution logs `bc_attribution_{page_name_id}` + written by `checks/backchecks/attribution.py`), JSON settings, and + credential metadata - `cache/projects.json` — the project registry - `cache/user_settings.json` — per-user preferences, currently the "Reviewer name" recorded as `user` in correction logs diff --git a/docs/USER_GUIDE.md b/docs/USER_GUIDE.md index 3c4395d1..88800af8 100644 --- a/docs/USER_GUIDE.md +++ b/docs/USER_GUIDE.md @@ -1001,6 +1001,10 @@ Configure validation: Pre-filled from the page configuration. A value you enter here is saved and overrides the page configuration; clear it to fall back. If neither is set, 10% is used and a warning is shown. +- **Error rate target (%)**: Optional. The highest acceptable error rate. In + the enumerator and back checker statistics, every error rate and adjusted + error rate above it is highlighted, each column on its own. Leave blank to + highlight nothing. - **Eligibility Filter**: Optional survey column and values that mark a survey eligible for back checks (e.g., `consent` in `1`). Only eligible surveys count towards coverage. @@ -1009,7 +1013,9 @@ Configure validation: ##### Backchecks Summary A row of counts (survey observations, back check observations, enumerators and -back checkers), then a **Targets** section with two metrics side by side: +back checkers) and **Mismatches Attributed**, the share of mismatches given an +error source (see [Attributing Mismatches](#attributing-mismatches)). Then a +**Targets** section with two metrics side by side: - **Backcheck Coverage**: Share of eligible unique survey IDs (after duplicate handling) with at least one matching back check, and how many points it is @@ -1020,6 +1026,14 @@ back checkers), then a **Targets** section with two metrics side by side: expected (survey target × target %, rounded up), and how many back checks above or below that it is. Values over 100% are shown as is. +Below the targets, an **Error Rates** section has one card for the total error +rate and one for each category. Each card shows mismatches ÷ values compared +over every back check. Its grey delta is the enumerator adjusted error rate, +and the card's help gives the back checker adjusted error rate (see +[Attributing Mismatches](#attributing-mismatches)). A category with no values +compared shows N/A, with "No values compared" in place of the delta. The cards +appear once back check columns are configured. + **Add Back Check Columns**: Click "Add a back check column" (+ button): @@ -1056,6 +1070,8 @@ Detailed column-level validation: - \# surveys, backchecks, compared - \# different values - Error rate (%) +- \# mismatches attributed to the enumerator, the back checker and the + respondent, and \# still unattributed (there is no adjusted rate per column) ##### Enumerator Statistics @@ -1068,7 +1084,7 @@ Performance by original enumerator: enumerators with no back checks show 0%) - \# values compared - \# different values -- Error rate (%) +- Error rate (%) and adjusted error rate (%), by category and in total ##### Back Checker Statistics @@ -1078,7 +1094,7 @@ Performance by validator: - Backchecks: unique surveys back checked - \# values compared - \# discrepancies -- Error rate (%) +- Error rate (%) and adjusted error rate (%), by category and in total ##### Comparison Details @@ -1090,8 +1106,58 @@ Record-level validation results: - Survey value - Back check value - Comparison result +- Error source, for mismatches - Column name +##### Attributing Mismatches + +Back check results measure how well data was collected, so the Back Checks +page never changes survey or back check data. There is no way to accept a +mismatch as valid or to replace a survey value with the back check value. +Expected differences, such as "Don't know" against "Refused", belong in the +exclude and no-differences lists in the settings. The comparison uses the +corrected survey data, so corrections made on the Correct Data page are +reflected. + +What you can record is who caused each mismatch: + +- **Enumerator**: the survey value is wrong. +- **Backchecker**: the back check value is wrong. A note is required. +- **Respondent**: the respondent gave different answers. A note is required. +- **Unattributed**: no source yet. Every mismatch starts here, and choosing it + clears an earlier attribution. + +Click **Review** on a mismatch in the Comparison Details table to open the +attribution dialog. It shows the survey and back check values, which you +can't edit. To attribute several mismatches at once, select their rows, then +click **Review** on one of the selected rows. Only mismatches can be +attributed. + +Each attribution records your reviewer name and the date. The latest +attribution for a mismatch applies only while the survey and back check values +are the ones you attributed. If either value changes, the mismatch is +Unattributed again; if the values now match, there is no mismatch to +attribute. The **Attribution log** expander below the table lists every +attribution, newest first, including the ones later replaced. + +Attribution never changes the mismatch counts or the regular error rate. It +only affects the **adjusted error rate**, which has the same denominator as +the error rate (values compared): + +- Enumerator adjusted error rate = (mismatches − Backchecker − Respondent) ÷ + values compared +- Back checker adjusted error rate = (mismatches − Enumerator − Respondent) ÷ + values compared + +Unattributed mismatches always count against both. For example, an enumerator +with 10 values compared and 4 mismatches, 1 attributed to the respondent and 1 +to the back checker, has an error rate of 40% and an adjusted error rate of +20%. + +Duplicate and unmatched back check IDs are not handled here. Until they can be +resolved from the Duplicates page, the **Handle Duplicates** setting decides +which duplicates are compared. + --- ### 9. GPS Checks Report diff --git a/src/datasure/checks/backchecks/attribution.py b/src/datasure/checks/backchecks/attribution.py new file mode 100644 index 00000000..dc321d30 --- /dev/null +++ b/src/datasure/checks/backchecks/attribution.py @@ -0,0 +1,308 @@ +"""Attribution of backcheck mismatches to an error source. + +Reviewers record whether a mismatch came from the enumerator, the +backchecker or the respondent. Attribution never changes the survey or +backcheck data, the mismatch counts or the regular error rate: it only feeds +the adjusted error rate. + +Attributions are kept in an append-only table in the `logs` db. The latest +entry for each survey KEY, backcheck KEY and column wins, and applies only +while the survey and backcheck values still equal the ones attributed. + +Kept free of Streamlit so the logic can be tested without a running app. +""" + +from datetime import datetime +from enum import StrEnum + +import polars as pl + +from datasure.checks.backchecks.models import merged_backcheck_name +from datasure.utils.duckdb_utils import duckdb_get_table, duckdb_save_table + + +class ErrorSource(StrEnum): + """Who caused a backcheck mismatch. Values are stored in the log.""" + + ENUMERATOR = "Enumerator" + BACKCHECKER = "Backchecker" + RESPONDENT = "Respondent" + UNATTRIBUTED = "Unattributed" + + +# Sources that excuse a mismatch from someone's adjusted rate need a reason. +_NOTE_REQUIRED_SOURCES = (ErrorSource.BACKCHECKER, ErrorSource.RESPONDENT) + +# Mismatches attributed to these sources are left out of each staff type's +# adjusted error rate. +_EXCUSED_SOURCES: dict[str, tuple[ErrorSource, ...]] = { + "enumerator": (ErrorSource.BACKCHECKER, ErrorSource.RESPONDENT), + "backchecker": (ErrorSource.ENUMERATOR, ErrorSource.RESPONDENT), +} + +# Added to the comparison results by `mark_error_sources`: the error source +# of each mismatch, null for other rows. +ERROR_SOURCE_COL = "error_source" + +MISMATCH = "mismatch" + +# Schema of a persisted attribution log, in column order. Keys and values +# are stored as text, so they are compared as text. +ATTRIBUTION_LOG_SCHEMA: dict[str, pl.DataType] = { + "survey_key": pl.String, + "backcheck_key": pl.String, + "column_name": pl.String, + "survey_value": pl.String, + "backcheck_value": pl.String, + "source": pl.String, + "note": pl.String, + # Who attributed: the reviewer name set in the app, else the OS login. + "user": pl.String, + "date": pl.Datetime("us"), +} + +_CELL = ["survey_key", "backcheck_key", "column_name"] + + +def attribution_table(page_name_id: str) -> str: + """Return the name of a page's attribution log table in the `logs` db.""" + return f"bc_attribution_{page_name_id}" + + +def note_required(source: ErrorSource) -> bool: + """Whether attributing a mismatch to `source` needs a note.""" + return source in _NOTE_REQUIRED_SOURCES + + +def backcheck_key_col(data: pl.DataFrame, survey_key: str) -> str: + """Return the backcheck KEY column of the comparison results. + + The merge only adds `{survey_key}__BCCL` when `survey_key` is not the + merge ID; otherwise the survey and backcheck share `survey_key`. + """ + backcheck_key = merged_backcheck_name(survey_key) + return backcheck_key if backcheck_key in data.columns else survey_key + + +def _cell_values(data: pl.DataFrame, survey_key: str) -> list[pl.Expr]: + """Select the keys, column and values of comparison results as log text.""" + return [ + pl.col(survey_key).cast(pl.String).alias("survey_key"), + pl.col(backcheck_key_col(data, survey_key)) + .cast(pl.String) + .alias("backcheck_key"), + pl.col("column_name").cast(pl.String), + pl.col("survey_value").cast(pl.String), + pl.col("backcheck_value").cast(pl.String), + ] + + +def is_mismatch() -> pl.Expr: + """Whether a row of comparison results is a mismatch.""" + return pl.col("match_status") == MISMATCH + + +def rows_to_review( + table: pl.DataFrame, clicked_row: int, selected_rows: list[int] +) -> pl.DataFrame: + """Return the mismatch rows of `table` that a Review click covers. + + Clicking Review on a selected row covers every selected row; on any + other row, just that row. Only mismatches can be attributed, so other + rows are dropped. Positions no longer in `table` are ignored. + """ + rows = selected_rows if clicked_row in selected_rows else [clicked_row] + rows = [row for row in rows if 0 <= row < table.height] + if not rows: + return table.clear() + return table[rows].filter(is_mismatch()) + + +def build_attribution_entries( + rows: pl.DataFrame, + survey_key: str, + source: ErrorSource, + note: str | None, + user: str, + date: datetime, +) -> pl.DataFrame: + """Return the attribution log entries for attributing `rows` to `source`. + + Parameters + ---------- + rows : pl.DataFrame + Comparison results to attribute, each a mismatch. + survey_key : str + The Survey KEY column in `rows`. + source : ErrorSource + The error source to record. + note : str | None + The reviewer's note; required for Backchecker and Respondent. + user : str + Who made the attribution. + date : datetime + When the attribution was made. + + Raises + ------ + ValueError + If there are no rows, a row is not a mismatch, or a required note is + blank. + """ + if rows.is_empty(): + raise ValueError("No mismatches to attribute") + if not rows.select(is_mismatch().all()).item(): + raise ValueError("Only a mismatch can be attributed to an error source") + note = (note or "").strip() or None + if note_required(source) and note is None: + raise ValueError(f"A note is required to attribute a mismatch to {source}") + + return rows.select( + *_cell_values(rows, survey_key), + pl.lit(str(source), dtype=pl.String).alias("source"), + pl.lit(note, dtype=pl.String).alias("note"), + pl.lit(user, dtype=pl.String).alias("user"), + pl.lit(date, dtype=ATTRIBUTION_LOG_SCHEMA["date"]).alias("date"), + ) + + +def mark_error_sources( + analysis: pl.DataFrame, log: pl.DataFrame, survey_key: str +) -> pl.DataFrame: + """Add `ERROR_SOURCE_COL` to comparison results. + + Each mismatch gets the source of its latest attribution, if the survey + and backcheck values still equal the attributed ones, else + Unattributed. Other rows get null, whatever the log holds. + + Parameters + ---------- + analysis : pl.DataFrame + Output of `compute_backcheck_analysis`. + log : pl.DataFrame + The attribution log, in the order entries were made. + survey_key : str + The Survey KEY column in `analysis`. + + Returns + ------- + pl.DataFrame + `analysis` in the same order, plus `ERROR_SOURCE_COL`. + """ + if analysis.is_empty(): + return analysis + + latest = log.select( + *(pl.col(col).cast(pl.String) for col in _CELL), + pl.col("survey_value").cast(pl.String).alias("_survey_value"), + pl.col("backcheck_value").cast(pl.String).alias("_backcheck_value"), + pl.col("source").cast(pl.String).alias("_source"), + ).unique(subset=_CELL, keep="last", maintain_order=True) + + # Match in a frame of our own columns, so they can't clash with a + # survey KEY of the same name. + cells = analysis.select( + *_cell_values(analysis, survey_key), + pl.col("match_status"), + ).join(latest, on=_CELL, how="left", maintain_order="left") + + still_applies = pl.col("survey_value").eq_missing(pl.col("_survey_value")) & pl.col( + "backcheck_value" + ).eq_missing(pl.col("_backcheck_value")) + source = cells.select( + pl.when(~is_mismatch()) + .then(pl.lit(None, dtype=pl.String)) + .when(pl.col("_source").is_not_null() & still_applies) + .then(pl.col("_source")) + .otherwise(pl.lit(str(ErrorSource.UNATTRIBUTED))) + .alias(ERROR_SOURCE_COL) + ) + return analysis.with_columns(source.get_columns()) + + +def _mismatch_sources(rows: pl.DataFrame) -> pl.Series: + """Return the error source of each mismatch in `rows`.""" + mismatches = rows.filter(is_mismatch()) if "match_status" in rows.columns else rows + if ERROR_SOURCE_COL not in mismatches.columns: + return pl.Series([str(ErrorSource.UNATTRIBUTED)] * mismatches.height) + return mismatches.get_column(ERROR_SOURCE_COL).fill_null( + str(ErrorSource.UNATTRIBUTED) + ) + + +def count_error_sources(rows: pl.DataFrame) -> dict[ErrorSource, int]: + """Count the mismatches in `rows` by error source. + + Without `ERROR_SOURCE_COL`, every mismatch is Unattributed. + """ + if rows.is_empty(): + return dict.fromkeys(ErrorSource, 0) + sources = _mismatch_sources(rows).to_list() + return {source: sources.count(str(source)) for source in ErrorSource} + + +def excused_mismatches(rows: pl.DataFrame, staff_type: str) -> int: + """Count the mismatches left out of `staff_type`'s adjusted error rate.""" + if staff_type not in _EXCUSED_SOURCES: + raise ValueError( + f"Unknown staff type '{staff_type}'; expected one of " + f"{', '.join(_EXCUSED_SOURCES)}" + ) + counts = count_error_sources(rows) + return sum(counts[source] for source in _EXCUSED_SOURCES[staff_type]) + + +def adjusted_error_rate( + mismatches: int, compared: int, rows: pl.DataFrame, staff_type: str +) -> float: + """Return `staff_type`'s adjusted error rate, in percent. + + The denominator is the values compared, as for the regular rate. + Enumerators: (mismatches - Backchecker - Respondent) / compared. + Backcheckers: (mismatches - Enumerator - Respondent) / compared. + Unattributed mismatches always count. + """ + excused = excused_mismatches(rows, staff_type) + if compared <= 0: + return 0.0 + return round((mismatches - excused) / compared * 100, 2) + + +def attributed_share(analysis: pl.DataFrame) -> float | None: + """Return the % of mismatches attributed to a source, None if none.""" + if analysis.is_empty() or "match_status" not in analysis.columns: + return None + counts = count_error_sources(analysis) + total = sum(counts.values()) + if total == 0: + return None + return (total - counts[ErrorSource.UNATTRIBUTED]) / total * 100 + + +def load_attribution_log(project_id: str, page_name_id: str) -> pl.DataFrame: + """Return a page's attribution log, in the order entries were made.""" + log = duckdb_get_table(project_id, attribution_table(page_name_id), "logs") + if log.width == 0: + return pl.DataFrame(schema=ATTRIBUTION_LOG_SCHEMA) + return log.select( + pl.col(col).cast(dtype) for col, dtype in ATTRIBUTION_LOG_SCHEMA.items() + ) + + +def save_attributions( + project_id: str, page_name_id: str, entries: pl.DataFrame +) -> None: + """Append `entries` to a page's attribution log. Earlier entries are kept.""" + log = pl.concat( + [load_attribution_log(project_id, page_name_id), entries], how="vertical" + ) + duckdb_save_table(project_id, log, attribution_table(page_name_id), "logs") + + +def attribution_history(log: pl.DataFrame) -> pl.DataFrame: + """Return the attribution log newest first, for display.""" + return ( + log.with_row_index("_order") + .sort(["date", "_order"], descending=True, nulls_last=True) + .drop("_order") + ) diff --git a/src/datasure/checks/backchecks/compute.py b/src/datasure/checks/backchecks/compute.py index 8c693eef..6c8b35bd 100644 --- a/src/datasure/checks/backchecks/compute.py +++ b/src/datasure/checks/backchecks/compute.py @@ -2,11 +2,17 @@ import re from contextlib import suppress +from dataclasses import dataclass from typing import Any import polars as pl from scipy import stats +from datasure.checks.backchecks.attribution import ( + ErrorSource, + adjusted_error_rate, + count_error_sources, +) from datasure.checks.backchecks.models import ( BACKCHECK_SUFFIX, TAB_NAME, @@ -30,7 +36,8 @@ def load_default_backchecks_settings( Loads previously saved backcheck report settings from the settings file and merges them with the provided default configuration. Saved settings take precedence over defaults, except a cleared or invalid backcheck - target, which falls back to the configured one. + target and an invalid error rate target, which fall back to the + configured one. A cleared error rate target stays cleared. Parameters ---------- @@ -45,11 +52,19 @@ def load_default_backchecks_settings( Merged settings combining saved and default configurations. """ saved_settings = load_check_settings(settings_file, TAB_NAME) - # A cleared target falls back to the configured one, as does a value saved - # by the old count-based input that is not a valid percentage. - saved_target = saved_settings.get("backcheck_target_percent") - if not isinstance(saved_target, int | float) or not 0 <= saved_target <= 100: - saved_settings.pop("backcheck_target_percent", None) + # A cleared backcheck target falls back to the configured one, as does a + # value saved by the old count-based input that is not a valid percentage. + # The error rate target is optional: clearing it means "highlight + # nothing", so a saved None is kept and only invalid values fall back. + for target, keep_cleared in ( + ("backcheck_target_percent", False), + ("error_rate_target_percent", True), + ): + saved_target = saved_settings.get(target) + if saved_target is None and keep_cleared: + continue + if not isinstance(saved_target, int | float) or not 0 <= saved_target <= 100: + saved_settings.pop(target, None) default_settings: dict = dict(config) default_settings.update(saved_settings) @@ -839,7 +854,7 @@ def _calculate_average_days( def _calculate_category_statistics( - cat_data: pl.DataFrame, category: int + cat_data: pl.DataFrame, category: int, staff_type: str = "enumerator" ) -> dict[str, int | float]: """Calculate statistics for a single category. @@ -849,6 +864,9 @@ def _calculate_category_statistics( Category-specific data. category : int Category number (1, 2, or 3). + staff_type : str + Either "enumerator" or "backchecker", whose adjusted error rate to + report. Returns ------- @@ -862,6 +880,7 @@ def _calculate_category_statistics( f"Values Compared (Cat {category})": 0, f"Mismatches (Cat {category})": 0, f"Error Rate % (Cat {category})": 0.0, + f"Adjusted Error Rate % (Cat {category})": 0.0, } # Count non-missing values @@ -887,6 +906,9 @@ def _calculate_category_statistics( f"Values Compared (Cat {category})": n_cat_compared, f"Mismatches (Cat {category})": n_mismatches, f"Error Rate % (Cat {category})": round(error_rate, 2), + f"Adjusted Error Rate % (Cat {category})": adjusted_error_rate( + n_mismatches, n_cat_compared, cat_data, staff_type + ), } @@ -897,6 +919,7 @@ def _calculate_staff_statistics( survey_key: str, survey_date: str | None, backcheck_date: str | None, + staff_type: str = "enumerator", ) -> dict[str, Any]: """Calculate all statistics for a single staff member. @@ -914,6 +937,9 @@ def _calculate_staff_statistics( Survey date column name. backcheck_date : str | None Backcheck date column name. + staff_type : str + Either "enumerator" or "backchecker", whose adjusted error rate to + report. Returns ------- @@ -935,7 +961,7 @@ def _calculate_staff_statistics( # Calculate statistics for each category for category in [1, 2, 3]: cat_data = staff_data.filter(pl.col("category") == category) - cat_stats = _calculate_category_statistics(cat_data, category) + cat_stats = _calculate_category_statistics(cat_data, category, staff_type) # Add category stats to staff_stats staff_stats.update(cat_stats) @@ -960,6 +986,12 @@ def _calculate_staff_statistics( "Values Compared (Total)": total_compared, "Mismatches (Total)": total_mismatches, "Error Rate % (Total)": round(total_error_rate, 2), + "Adjusted Error Rate % (Total)": adjusted_error_rate( + total_mismatches, + total_compared, + staff_data.filter(pl.col("category").is_in([1, 2, 3])), + staff_type, + ), } ) @@ -1049,6 +1081,7 @@ def compute_enumerator_backchecker_stats( survey_key, backcheck_settings.survey_date, backcheck_settings.backcheck_date, + staff_type, ) stats_list.append(staff_stats) @@ -1224,6 +1257,7 @@ def _build_column_stats_dict( n_mismatches: int, error_rate: float, test_results_str: str, + source_counts: dict[ErrorSource, int] | None = None, ) -> dict[str, Any]: """Build statistics dictionary for a column. @@ -1245,12 +1279,17 @@ def _build_column_stats_dict( Error rate percentage. test_results_str : str Formatted test results string. + source_counts : dict[ErrorSource, int] | None + Mismatches by error source; all Unattributed if not given. Returns ------- dict[str, Any] Statistics dictionary. """ + if source_counts is None: + source_counts = dict.fromkeys(ErrorSource, 0) + source_counts[ErrorSource.UNATTRIBUTED] = n_mismatches return { "Column Name": col_name, "Category": category, @@ -1259,6 +1298,7 @@ def _build_column_stats_dict( "Values Compared": n_compared, "Mismatches": n_mismatches, "Error Rate (%)": round(error_rate, 2), + **{f"{source} Mismatches": source_counts[source] for source in ErrorSource}, "Test Results": test_results_str, } @@ -1311,6 +1351,7 @@ def compute_column_stats( n_mismatches, error_rate, test_results_str, + count_error_sources(col_data), ) stats_list.append(stats_dict) @@ -1320,6 +1361,72 @@ def compute_column_stats( return pl.DataFrame(stats_list) +@dataclass(frozen=True) +class OverallErrorRate: + """Error rates over every compared value in a category, or in total. + + `error_rate` is None when no values were compared; the adjusted rates + are then 0. + """ + + label: str + compared: int + mismatches: int + error_rate: float | None + enumerator_adjusted: float + backchecker_adjusted: float + + +def _overall_error_rate(label: str, rows: pl.DataFrame) -> OverallErrorRate: + _, compared, mismatches, error_rate = _calculate_column_statistics(rows) + return OverallErrorRate( + label=label, + compared=compared, + mismatches=mismatches, + error_rate=round(error_rate, 2) if compared > 0 else None, + enumerator_adjusted=adjusted_error_rate( + mismatches, compared, rows, "enumerator" + ), + backchecker_adjusted=adjusted_error_rate( + mismatches, compared, rows, "backchecker" + ), + ) + + +def compute_overall_error_rates( + backcheck_analysis: pl.DataFrame, +) -> list[OverallErrorRate]: + """Compute the error rates over all comparisons, in total and by category. + + Parameters + ---------- + backcheck_analysis : pl.DataFrame + Results from compute_backcheck_analysis, optionally marked with + their error sources. + + Returns + ------- + list[OverallErrorRate] + The total over categories 1-3, then each category; empty if there + are no results. + """ + if backcheck_analysis.is_empty(): + return [] + + categories = [1, 2, 3] + in_categories = backcheck_analysis.filter(pl.col("category").is_in(categories)) + return [ + _overall_error_rate("Total", in_categories), + *( + _overall_error_rate( + f"Cat {category}", + backcheck_analysis.filter(pl.col("category") == category), + ) + for category in categories + ), + ] + + # ============================================================================== # COMPARISON AND ANALYSIS HELPERS # ============================================================================== diff --git a/src/datasure/checks/backchecks/models.py b/src/datasure/checks/backchecks/models.py index d75e5b3a..03e3aba4 100644 --- a/src/datasure/checks/backchecks/models.py +++ b/src/datasure/checks/backchecks/models.py @@ -85,6 +85,12 @@ class BackcheckSettings(BaseModel): survey_target: int | None = Field( None, ge=0, description="Target number of survey responses" ) + error_rate_target_percent: float | None = Field( + None, + ge=0, + le=100, + description="Highest acceptable error rate, in percent; None if not set", + ) eligibility_column: str | None = Field( None, description="Survey column that marks a survey eligible" ) diff --git a/src/datasure/checks/backchecks/report_ui.py b/src/datasure/checks/backchecks/report_ui.py index dfccfaeb..cdcc6809 100644 --- a/src/datasure/checks/backchecks/report_ui.py +++ b/src/datasure/checks/backchecks/report_ui.py @@ -1,15 +1,35 @@ """Report-rendering UI for the backchecks report.""" -from typing import Literal +import logging +from collections.abc import Callable +from dataclasses import dataclass +from datetime import datetime +from typing import Any, Literal +import duckdb import polars as pl import streamlit as st +from datasure.checks.backchecks.attribution import ( + ERROR_SOURCE_COL, + ErrorSource, + attributed_share, + attribution_history, + backcheck_key_col, + build_attribution_entries, + is_mismatch, + load_attribution_log, + mark_error_sources, + note_required, + rows_to_review, + save_attributions, +) from datasure.checks.backchecks.compute import ( compute_backcheck_analysis, compute_backchecker_productivity, compute_column_stats, compute_enumerator_backchecker_stats, + compute_overall_error_rates, expand_col_names, ) from datasure.checks.backchecks.coverage import ( @@ -35,12 +55,20 @@ from datasure.utils.dataframe_utils import ColumnByType from datasure.utils.duckdb_utils import duckdb_get_table, duckdb_save_table from datasure.utils.navigations_utils import demo_callout, show_demo_next_action +from datasure.utils.reviewer import get_reviewer_name from datasure.utils.settings_utils import ( load_check_settings, save_check_settings, trigger_save, ) -from datasure.utils.ui_utils import styled_dataframe +from datasure.utils.ui_utils import ( + metric_row, + queue_notice, + show_queued_notices, + styled_dataframe, +) + +logger = logging.getLogger(__name__) # ============================================================================== # COLUMN CONFIGURATION FUNCTIONS @@ -576,6 +604,7 @@ def _render_backcheck_summary( survey_data: pl.DataFrame, backcheck_data: pl.DataFrame, backcheck_settings: BackcheckSettings, + backcheck_analysis: pl.DataFrame | None = None, ) -> None: """Render summary metrics and progress against the backcheck target. @@ -587,6 +616,9 @@ def _render_backcheck_summary( Backcheck dataset. backcheck_settings : BackcheckSettings Backcheck settings including the staff columns and targets. + backcheck_analysis : pl.DataFrame | None + Comparison results marked with their error sources, for the share of + mismatches attributed. """ coverage = compute_backcheck_coverage( survey_data, backcheck_data, backcheck_settings @@ -606,25 +638,29 @@ def _render_backcheck_summary( else: n_backcheckers = 0 - # Display metrics in columns - c1, c2, c3, c4 = st.columns(4) - - with c1, st.container(border=True): - st.metric("Survey Observations", f"{len(survey_data):,}") - - with c2, st.container(border=True): - st.metric("Backcheck Observations", f"{len(backcheck_data):,}") - - with c3, st.container(border=True): - st.metric( - "Total Enumerators", f"{n_enumerators:,}" if n_enumerators > 0 else "N/A" - ) - - with c4, st.container(border=True): - st.metric( - "Total Back Checkers", - f"{n_backcheckers:,}" if n_backcheckers > 0 else "N/A", - ) + share = attributed_share( + backcheck_analysis if backcheck_analysis is not None else pl.DataFrame() + ) + metric_row( + [ + ("Survey Observations", f"{len(survey_data):,}"), + ("Backcheck Observations", f"{len(backcheck_data):,}"), + ( + "Total Enumerators", + f"{n_enumerators:,}" if n_enumerators > 0 else "N/A", + ), + ( + "Total Back Checkers", + f"{n_backcheckers:,}" if n_backcheckers > 0 else "N/A", + ), + ( + "Mismatches Attributed", + f"{share:.1f}%" if share is not None else "N/A", + "Share of mismatches attributed to the enumerator, backchecker " + "or respondent in Comparison Results Details.", + ), + ] + ) st.markdown("##### Targets") tc1, tc2 = st.columns(2) @@ -636,6 +672,48 @@ def _render_backcheck_summary( with tc2: _render_expected_backchecks(coverage) + _render_overall_error_rates( + backcheck_analysis if backcheck_analysis is not None else pl.DataFrame() + ) + + +def _render_overall_error_rates(backcheck_analysis: pl.DataFrame) -> None: + """Render one card per error rate, total then by category. + + Each card shows the regular error rate, with the enumerator adjusted + rate as a grey delta; the backchecker adjusted rate is in the help. + """ + st.markdown("##### Error Rates") + rates = compute_overall_error_rates(backcheck_analysis) + if not rates: + st.info( + "Error rates appear here once backcheck columns are configured in " + "the Backchecks Columns Configuration section above." + ) + return + + # Bordered columns stretch to the tallest card, and every card has a + # delta line, so the cards in the row are the same height. + for col, rate in zip(st.columns(len(rates), border=True), rates, strict=True): + compared = rate.error_rate is not None + with col: + st.metric( + f"Error Rate ({rate.label})", + f"{rate.error_rate:.2f}%" if compared else "N/A", + delta=f"{rate.enumerator_adjusted:.2f}% adjusted" + if compared + else "No values compared", + delta_color="off", + delta_arrow="off", + help=f"{rate.mismatches:,} mismatches out of {rate.compared:,} " + "values compared. The delta is the adjusted error rate for " + "enumerators, which leaves out mismatches attributed to the " + "backchecker or respondent. Adjusted error rate for " + f"backcheckers: {rate.backchecker_adjusted:.2f}%." + if compared + else "No values compared in this category.", + ) + def _render_coverage_metric(coverage: BackcheckCoverage | None) -> None: """Render the on-track coverage card with its delta against the target.""" @@ -909,17 +987,44 @@ def _render_enum_bcer_stats( ) -def _highlight_below_target(target_percent: float): - """Return a Styler cell function that flags coverage below the target.""" +_ADJUSTED_RATE_HELP = { + "enumerator": "Mismatches not attributed to the backchecker or respondent, " + "divided by values compared.", + "backchecker": "Mismatches not attributed to the enumerator or respondent, " + "divided by values compared.", +} + + +def _highlight_where(off_target: Callable[[float], bool]): + """Return a Styler cell function that flags numbers `off_target` accepts.""" def style(value: object) -> str: - if isinstance(value, int | float) and value < target_percent: + if isinstance(value, int | float) and off_target(value): return "background-color: #f8d7da; color: #842029" return "" return style +def _highlight_below_target(target_percent: float): + """Return a Styler cell function that flags coverage below the target.""" + return _highlight_where(lambda value: value < target_percent) + + +def _highlight_above_target(target_percent: float): + """Return a Styler cell function that flags error rates above the target.""" + return _highlight_where(lambda value: value > target_percent) + + +def _error_rate_columns(columns: list[str]) -> list[str]: + """Return the regular and adjusted error rate columns of a stats table.""" + return [ + col + for col in columns + if col.startswith(("Error Rate % (", "Adjusted Error Rate % (")) + ] + + @st.fragment def _render_enum_bcer_stats_table( survey_data: pl.DataFrame, @@ -1033,6 +1138,13 @@ def _render_enum_bcer_stats_table( column_config[f"Error Rate % (Cat {category})"] = st.column_config.NumberColumn( f"Error % (Cat {category})", format="%.2f" ) + column_config[f"Adjusted Error Rate % (Cat {category})"] = ( + st.column_config.NumberColumn( + f"Adjusted Error % (Cat {category})", + format="%.2f", + help=_ADJUSTED_RATE_HELP[staff_type], + ) + ) # Add total columns column_config["Non-Missing Survey (Total)"] = st.column_config.NumberColumn( @@ -1050,8 +1162,14 @@ def _render_enum_bcer_stats_table( column_config["Error Rate % (Total)"] = st.column_config.NumberColumn( "Error % (Total)", format="%.2f" ) + column_config["Adjusted Error Rate % (Total)"] = st.column_config.NumberColumn( + "Adjusted Error % (Total)", + format="%.2f", + help=_ADJUSTED_RATE_HELP[staff_type], + ) - if staff_type == "backchecker": + error_target = backcheck_settings.error_rate_target_percent + if staff_type == "backchecker" and error_target is None: st.dataframe( stats_df, hide_index=True, width="stretch", column_config=column_config ) @@ -1059,16 +1177,25 @@ def _render_enum_bcer_stats_table( # st.dataframe shows a Styler's formatted text, so format every cell here: # plain text by default, blank error rates for unbackchecked enumerators. - target_percent = effective_target_percent(backcheck_settings) formatters = {col: str for col in stats_df.columns} - formatters.update({"Coverage %": "{:.1f}%", "vs target": "{:+.1f}"}) - styler = ( - stats_df.to_pandas(use_pyarrow_extension_array=True) - .style.map(_highlight_below_target(target_percent), subset=["Coverage %"]) - .format(formatters, na_rep="") - ) + styler = stats_df.to_pandas(use_pyarrow_extension_array=True).style + if staff_type == "enumerator": + target_percent = effective_target_percent(backcheck_settings) + formatters.update({"Coverage %": "{:.1f}%", "vs target": "{:+.1f}"}) + styler = styler.map( + _highlight_below_target(target_percent), subset=["Coverage %"] + ) + # Each rate column is compared with the target on its own, so a rate can + # be over it while its adjusted rate is not. + rate_columns = _error_rate_columns(stats_df.columns) + formatters.update(dict.fromkeys(rate_columns, "{:.2f}")) + if error_target is not None and rate_columns: + styler = styler.map(_highlight_above_target(error_target), subset=rate_columns) styled_dataframe( - styler, hide_index=True, width="stretch", column_config=column_config + styler.format(formatters, na_rep=""), + hide_index=True, + width="stretch", + column_config=column_config, ) @@ -1114,6 +1241,12 @@ def _render_column_stats( "Error Rate (%)": st.column_config.NumberColumn( "Error Rate (%)", format="%.2f" ), + **{ + f"{source} Mismatches": st.column_config.NumberColumn( + f"{source} Mismatches", format="%d" + ) + for source in ErrorSource + }, "Test Results": st.column_config.TextColumn("Test Results", width="large"), } @@ -1155,6 +1288,7 @@ def _get_available_additional_columns( "backcheck_value", "match_status", "category", + ERROR_SOURCE_COL, } return sorted( @@ -1408,6 +1542,7 @@ def _build_display_columns( "survey_value", "backcheck_value", "match_status", + ERROR_SOURCE_COL, "category", ] @@ -1462,8 +1597,20 @@ def _prepare_display_data( return display_data +def _error_source_column(*, pinned: bool = False) -> Any: + """Return the column config of the Error Source column.""" + return st.column_config.TextColumn( + "Error Source", + help="Who caused the mismatch, as attributed by a reviewer.", + pinned=pinned, + ) + + def _build_column_config( - survey_key: str, survey_id: str, backcheck_key: str, filtered_data: pl.DataFrame + survey_key: str, + survey_id: str | None, + backcheck_key: str, + filtered_data: pl.DataFrame, ) -> dict: """Build column configuration for dataframe display. @@ -1488,6 +1635,7 @@ def _build_column_config( "survey_value": st.column_config.TextColumn("Survey Value"), "backcheck_value": st.column_config.TextColumn("Backcheck Value"), "match_status": st.column_config.TextColumn("Match Status"), + ERROR_SOURCE_COL: _error_source_column(), "category": st.column_config.NumberColumn("Category", format="%d"), } @@ -1506,16 +1654,37 @@ def _build_column_config( return column_config +@st.fragment +def _render_comparison_results_section( + survey_data: pl.DataFrame, + backcheck_data: pl.DataFrame, + backcheck_analysis: pl.DataFrame, + backcheck_settings: BackcheckSettings, + review: "AttributionContext | None", +) -> None: + """Render Comparison Results Details as a fragment. + + Filtering, selecting rows and opening the Review dialog rerun only this + section, not the whole report. Saving an attribution reruns the whole app + so the rates elsewhere on the page are refreshed. + """ + _render_backcheck_comparison_results( + survey_data, backcheck_data, backcheck_analysis, backcheck_settings, review + ) + + def _render_backcheck_comparison_results( survey_data: pl.DataFrame, backcheck_data: pl.DataFrame, backcheck_analysis: pl.DataFrame, backcheck_settings: BackcheckSettings, + review: "AttributionContext | None" = None, ) -> None: """Render detailed backcheck comparison results with filtering options. Displays a table showing each individual comparison with options to filter by match status, select specific columns, and add additional data columns. + With `review`, mismatches can be attributed to an error source. Parameters ---------- @@ -1527,6 +1696,8 @@ def _render_backcheck_comparison_results( Results from compute_backcheck_analysis. backcheck_settings : BackcheckSettings Backcheck configuration settings. + review : AttributionContext | None + Where attributions are saved; None for a read-only table. """ if backcheck_analysis.is_empty(): st.info( @@ -1593,20 +1764,259 @@ def _render_backcheck_comparison_results( st.info("No results match the selected filters.") return + # A stable order, so a row position reported by a Review click resolves + # to the same comparison on the rerun it triggers. + sort_cols = [ + col + for col in ("column_name", survey_key, backcheck_key) + if col in display_data.columns + ] + display_data = display_data.sort(sort_cols, nulls_last=True, maintain_order=True) + # Display results st.caption(f"Showing {len(display_data):,} comparison records") + if review is not None: + st.caption( + "Click **Review** on a mismatch to attribute it to the enumerator, " + "backchecker or respondent. To attribute several at once, select " + "their rows, then click **Review** on one of them. Attribution " + "never changes the data or the regular error rate." + ) column_config = _build_column_config( survey_key, survey_id, backcheck_key, display_data ) - st.dataframe( - display_data, + _render_comparison_table(display_data, column_config, survey_key, review) + + +# ============================================================================== +# MISMATCH ATTRIBUTION +# ============================================================================== + +# `queue_notice` scope of the confirmation shown after the post-save rerun. +_NOTICE_SCOPE = "backchecks_attribution" + +# First column of the comparison table: a button that opens the dialog. +REVIEW_BUTTON_COL = "_review" +REVIEW_BUTTON_LABEL = ":material/edit_note: Review" +# Pixels that fit the label; left unset, the button column is sized too wide. +REVIEW_BUTTON_WIDTH = 100 +_REVIEW_CLICK_KEY = "backchecks_attribution_review_click" +_COMPARISON_TABLE_KEY = "backchecks_comparison_table" + + +@dataclass(frozen=True) +class AttributionContext: + """Where the Backchecks page saves attributions, and the current log.""" + + project_id: str + page_name_id: str + log: pl.DataFrame + + +def _render_comparison_table( + display_data: pl.DataFrame, + column_config: dict[str, Any], + survey_key: str, + review: AttributionContext | None, +) -> None: + """Render the comparison table; with `review`, mismatches can be reviewed. + + Each mismatch row has a pinned Review button, and rows can be selected. + Clicking Review opens the attribution dialog for that row, or for every + selected mismatch if the row is selected. + """ + if review is None: + st.dataframe( + display_data, hide_index=True, width="stretch", column_config=column_config + ) + return + + # Survey and backcheck fields can be added to the table, so avoid their names. + button_col = REVIEW_BUTTON_COL + while button_col in display_data.columns: + button_col = f"_{button_col}" + # Review and Error Source come first and stay pinned while scrolling. + shown = display_data.select( + pl.when(is_mismatch()).then(pl.lit(REVIEW_BUTTON_LABEL)).alias(button_col), + pl.col(ERROR_SOURCE_COL), + pl.all().exclude(ERROR_SOURCE_COL), + ) + column_config = { + **column_config, + button_col: st.column_config.ButtonColumn( + "", + type="tertiary", + pinned=True, + width=REVIEW_BUTTON_WIDTH, + key=_REVIEW_CLICK_KEY, + help="Attribute this mismatch, or every selected one, to an error source.", + ), + ERROR_SOURCE_COL: _error_source_column(pinned=True), + } + event = st.dataframe( + shown, hide_index=True, width="stretch", column_config=column_config, + key=_COMPARISON_TABLE_KEY, + on_select="rerun", + selection_mode="multi-row", ) + # The click is only present during the rerun it triggers, so the dialog + # opens once per click; widgets inside the dialog rerun just the dialog. + click = st.session_state.get(_REVIEW_CLICK_KEY) + if not click: + return + selected = list(event.selection.rows) if event is not None else [] + rows = rows_to_review(display_data, click["row"], selected) + if not rows.is_empty(): + _attribution_dialog(rows, survey_key, review) + + +@st.dialog("Attribute mismatches to an error source", width="large") +def _attribution_dialog( + rows: pl.DataFrame, survey_key: str, review: AttributionContext +) -> None: + """Show the attribution form for `rows` in a dialog.""" + _render_attribution_form(rows, survey_key, review) + + +_SOURCE_HELP = { + ErrorSource.ENUMERATOR: "The survey value is wrong.", + ErrorSource.BACKCHECKER: "The backcheck value is wrong.", + ErrorSource.RESPONDENT: "The respondent gave different answers.", + ErrorSource.UNATTRIBUTED: "Clear an earlier attribution.", +} + + +def _plural_mismatches(n: int) -> str: + return f"{n:,} mismatch{'es' if n != 1 else ''}" + + +def _render_attribution_form( + rows: pl.DataFrame, survey_key: str, review: AttributionContext +) -> None: + """Render the attribution form for mismatch `rows` and save on submit. + + The values are shown read-only: attribution never changes the data. A + note is required for Backchecker and Respondent. A successful save + reruns the page so the tables and rates reflect it. + """ + st.markdown( + f"Attribute **{_plural_mismatches(rows.height)}** to an error source. " + "This changes only the adjusted error rate: the data, the mismatch " + "counts and the regular error rate stay as they are." + ) + backcheck_key = backcheck_key_col(rows, survey_key) + # When the survey KEY is the merge ID, it is also the backcheck KEY. + shown_cols = [ + col + for col in dict.fromkeys( + ( + survey_key, + backcheck_key, + "column_name", + "survey_value", + "backcheck_value", + ERROR_SOURCE_COL, + ) + ) + if col in rows.columns + ] + st.dataframe( + rows.select(shown_cols), + hide_index=True, + width="stretch", + column_config=_build_column_config(survey_key, None, backcheck_key, rows), + ) + + sources = list(ErrorSource) + current = ( + rows[ERROR_SOURCE_COL].unique().to_list() + if ERROR_SOURCE_COL in rows.columns + else [] + ) + default = sources.index(ErrorSource(current[0])) if len(current) == 1 else 0 + source = st.radio( + "Error source", + options=sources, + index=default, + horizontal=True, + captions=[_SOURCE_HELP[option] for option in sources], + key="backchecks_attribution_source", + ) + needs_note = note_required(source) + note = st.text_area( + "Note (required)" if needs_note else "Note (optional)", + key="backchecks_attribution_note", + placeholder="Why the mismatch has this source", + ) + missing_note = needs_note and not note.strip() + if missing_note: + st.caption(f"A note is required to attribute a mismatch to {source}.") + + if not st.button( + "Save attribution", + type="primary", + width="stretch", + disabled=missing_note, + key="backchecks_attribution_save", + ): + return + + try: + entries = build_attribution_entries( + rows, survey_key, source, note, get_reviewer_name(), datetime.now() + ) + save_attributions(review.project_id, review.page_name_id, entries) + except ValueError as e: + st.error(str(e)) + return + except (OSError, duckdb.Error, pl.exceptions.PolarsError): + logger.exception("Could not save backcheck attributions") + st.error("Could not save the attribution. Check the cache folder.") + return + + # A full rerun closes the dialog and refreshes the tables and rates. The + # dialog opens from a fragment, so ask for the whole app explicitly. + queue_notice( + _NOTICE_SCOPE, + "toast", + f"Attributed {_plural_mismatches(rows.height)} to {source}.", + ) + st.rerun(scope="app") + + +def _render_attribution_log(log: pl.DataFrame) -> None: + """Render the attribution history, newest first, in an expander.""" + with st.expander("Attribution log", icon=":material/history:"): + if log.is_empty(): + st.info("No mismatches have been attributed yet.") + return + st.caption( + "Every attribution, newest first. The latest one for a mismatch " + "applies while its survey and backcheck values are unchanged." + ) + st.dataframe( + attribution_history(log), + hide_index=True, + width="stretch", + column_config={ + "survey_key": st.column_config.TextColumn("Survey Key"), + "backcheck_key": st.column_config.TextColumn("Backcheck Key"), + "column_name": st.column_config.TextColumn("Column Name"), + "survey_value": st.column_config.TextColumn("Survey Value"), + "backcheck_value": st.column_config.TextColumn("Backcheck Value"), + "source": st.column_config.TextColumn("Error Source"), + "note": st.column_config.TextColumn("Note"), + "user": st.column_config.TextColumn("User"), + "date": st.column_config.DatetimeColumn("Date"), + }, + ) + # ============================================================================== # MAIN ENTRY POINT @@ -1642,6 +2052,7 @@ def backchecks_report( Configuration dictionary. """ st.title("Backchecks Report") + show_queued_notices(_NOTICE_SCOPE) demo_callout( """ @@ -1729,6 +2140,12 @@ def backchecks_report( _backcheck_analysis = compute_backcheck_analysis( survey_data, backcheck_data, backcheck_settings, backcheck_column_settings ) + review = AttributionContext( + project_id, page_name_id, load_attribution_log(project_id, page_name_id) + ) + _backcheck_analysis = mark_error_sources( + _backcheck_analysis, review.log, backcheck_settings.survey_key + ) st.subheader("Backchecks Summary") @@ -1742,6 +2159,9 @@ def backchecks_report( backcheck target. When the survey's target number of responses is set in the page configuration, it also shows backchecks done against the total number of backchecks expected, with how many above or below. + Under **Error Rates**, one card each shows the total error rate and the + error rate for each category, with the adjusted error rate for + enumerators as the delta. Below the metrics, a **Backchecker Productivity** table shows submission counts per backchecker over time. Use the **Daily / Weekly / Monthly** pills @@ -1749,7 +2169,9 @@ def backchecks_report( """ ) - _render_backcheck_summary(survey_data, backcheck_data, backcheck_settings) + _render_backcheck_summary( + survey_data, backcheck_data, backcheck_settings, _backcheck_analysis + ) _render_backchecker_productivity( backcheck_data, @@ -1812,9 +2234,10 @@ def backchecks_report( type="success", ) - _render_backcheck_comparison_results( - survey_data, backcheck_data, _backcheck_analysis, backcheck_settings + _render_comparison_results_section( + survey_data, backcheck_data, _backcheck_analysis, backcheck_settings, review ) + _render_attribution_log(review.log) st.write("---") demo_callout( diff --git a/src/datasure/checks/backchecks/settings_ui.py b/src/datasure/checks/backchecks/settings_ui.py index 67ea4201..92e5c054 100644 --- a/src/datasure/checks/backchecks/settings_ui.py +++ b/src/datasure/checks/backchecks/settings_ui.py @@ -271,7 +271,7 @@ def _render_tracking_options( settings_file: str, default_settings: BackcheckSettings, page_config_target: float | None = None, -) -> float | None: +) -> tuple[float | None, float | None]: """Render tracking options section. The target input is pre-filled with the saved panel value, else the page @@ -287,14 +287,18 @@ def _render_tracking_options( page_config_target : float | None Backcheck target % from the page configuration, if set. + The error rate target is optional: error rates above it are highlighted + in the enumerator and backchecker tables. + Returns ------- - float | None - Backcheck target percent, or None if not set anywhere. + tuple[float | None, float | None] + Backcheck target percent, or None if not set anywhere, and the error + rate target percent, or None if not set. """ with st.container(border=True): st.subheader("Tracking Options") - to1, _, _ = st.columns(3) + to1, to2, _ = st.columns(3) with to1: default_target = default_settings.backcheck_target_percent @@ -317,13 +321,37 @@ def _render_tracking_options( {"backcheck_target_percent": backcheck_target_percent}, ) + with to2: + default_error_target = default_settings.error_rate_target_percent + error_rate_target_percent = st.number_input( + "Error rate target (%)", + min_value=0.0, + max_value=100.0, + step=1.0, + format="%.1f", + help="Highest acceptable error rate. Error rates and adjusted " + "error rates above it are highlighted in the enumerator and " + "backchecker statistics. Leave blank to highlight nothing.", + key="backcheck_error_rate_target", + value=float(default_error_target) + if default_error_target is not None + else None, + on_change=trigger_save, + kwargs={"state_name": TAB_NAME + "_error_rate_target_percent"}, + ) + save_check_settings( + settings_file, + TAB_NAME, + {"error_rate_target_percent": error_rate_target_percent}, + ) + if backcheck_target_percent is None: st.warning( "No backcheck target is set here or in the page configuration, " f"so the default of {DEFAULT_TARGET_PERCENT:g}% is used." ) - return backcheck_target_percent + return backcheck_target_percent, error_rate_target_percent def _render_eligibility_filter( @@ -561,7 +589,7 @@ def backchecks_report_settings( - Survey identifiers (key and ID columns) - Survey date column selection - Enumerator and backchecker columns - - Tracking options (backcheck target % and eligibility filter) + - Tracking options (backcheck and error rate targets, eligibility filter) - Additional options (duplicate handling and value comparison) Settings are automatically saved to the settings file when changed @@ -618,7 +646,7 @@ def backchecks_report_settings( backcheck_categorical_columns, ) - backcheck_target_percent = _render_tracking_options( + backcheck_target_percent, error_rate_target_percent = _render_tracking_options( settings_file, default_settings, config.backcheck_target_percent ) @@ -642,6 +670,7 @@ def backchecks_report_settings( backchecker=backchecker, backcheck_target_percent=backcheck_target_percent, survey_target=default_settings.survey_target, + error_rate_target_percent=error_rate_target_percent, eligibility_column=eligibility_column, eligibility_values=eligibility_values, drop_duplicates_option=drop_duplicates_option, diff --git a/src/datasure/processing/correction_log.py b/src/datasure/processing/correction_log.py index 0700fc7c..6c144997 100644 --- a/src/datasure/processing/correction_log.py +++ b/src/datasure/processing/correction_log.py @@ -33,7 +33,10 @@ class Action(StrEnum): Action.REMOVE_ROW, ) -ACCEPT_CHECK_TYPES = ("outliers", "constraints", "backchecks", "duplicates", "gps") +# Backchecks is deliberately absent: backcheck results measure data quality, +# so a mismatch can't be accepted away. The Backchecks page attributes +# mismatches to an error source instead (see checks/backchecks/attribution.py). +ACCEPT_CHECK_TYPES = ("outliers", "constraints", "duplicates", "gps") # `severity` of an acceptance that overrides a hard constraint bound. HARD_SEVERITY = "hard" diff --git a/tests/checks/backchecks/test_attribution.py b/tests/checks/backchecks/test_attribution.py new file mode 100644 index 00000000..193071a3 --- /dev/null +++ b/tests/checks/backchecks/test_attribution.py @@ -0,0 +1,388 @@ +"""Tests for datasure.checks.backchecks.attribution.""" + +from datetime import datetime +from unittest.mock import patch + +import polars as pl +import pytest + +from datasure.checks.backchecks.attribution import ( + ATTRIBUTION_LOG_SCHEMA, + ERROR_SOURCE_COL, + ErrorSource, + adjusted_error_rate, + attributed_share, + attribution_history, + attribution_table, + build_attribution_entries, + count_error_sources, + excused_mismatches, + load_attribution_log, + mark_error_sources, + note_required, + rows_to_review, + save_attributions, +) + +SURVEY_KEY = "KEY" +BC_KEY = "KEY__BCCL" +NOW = datetime(2026, 10, 6, 12, 0) + + +def _analysis(**overrides) -> pl.DataFrame: + """Comparison results: two mismatches, a match and a missing value.""" + data = { + SURVEY_KEY: ["s1", "s2", "s3", "s4"], + BC_KEY: ["b1", "b2", "b3", "b4"], + "column_name": ["age", "age", "age", "age"], + "survey_value": [30, 41, 25, None], + "backcheck_value": [31, 40, 25, 50], + "category": [1, 1, 1, 1], + "match_status": ["mismatch", "mismatch", "match", "missing"], + } + data.update(overrides) + return pl.DataFrame(data) + + +def _log(rows: list[dict]) -> pl.DataFrame: + defaults = {"note": None, "user": "ana", "date": NOW} + return pl.DataFrame( + [{**defaults, **row} for row in rows], schema=ATTRIBUTION_LOG_SCHEMA + ) + + +def _entry(survey_key, backcheck_key, survey_value, backcheck_value, source, **kw): + return { + "survey_key": survey_key, + "backcheck_key": backcheck_key, + "column_name": kw.pop("column_name", "age"), + "survey_value": survey_value, + "backcheck_value": backcheck_value, + "source": source, + **kw, + } + + +# --------------------------------------------------------------------------- +# Vocabulary +# --------------------------------------------------------------------------- + + +def test_attribution_table_is_per_page(): + assert attribution_table("page_1") == "bc_attribution_page_1" + + +@pytest.mark.parametrize( + ("source", "required"), + [ + (ErrorSource.ENUMERATOR, False), + (ErrorSource.BACKCHECKER, True), + (ErrorSource.RESPONDENT, True), + (ErrorSource.UNATTRIBUTED, False), + ], +) +def test_note_required_for_backchecker_and_respondent(source, required): + assert note_required(source) is required + + +# --------------------------------------------------------------------------- +# Selecting rows to attribute +# --------------------------------------------------------------------------- + + +def test_rows_to_review_clicked_mismatch_alone(): + rows = rows_to_review(_analysis(), clicked_row=1, selected_rows=[]) + assert rows[SURVEY_KEY].to_list() == ["s2"] + + +def test_rows_to_review_clicked_non_mismatch_is_empty(): + rows = rows_to_review(_analysis(), clicked_row=2, selected_rows=[2]) + assert rows.is_empty() + + +def test_rows_to_review_clicked_inside_selection_takes_selected_mismatches(): + rows = rows_to_review(_analysis(), clicked_row=0, selected_rows=[0, 1, 2, 3]) + assert rows[SURVEY_KEY].to_list() == ["s1", "s2"] + + +def test_rows_to_review_clicked_outside_selection_takes_clicked_row_only(): + rows = rows_to_review(_analysis(), clicked_row=1, selected_rows=[0]) + assert rows[SURVEY_KEY].to_list() == ["s2"] + + +def test_rows_to_review_ignores_stale_positions(): + assert rows_to_review(_analysis(), clicked_row=9, selected_rows=[]).is_empty() + rows = rows_to_review(_analysis(), clicked_row=0, selected_rows=[0, 9]) + assert rows[SURVEY_KEY].to_list() == ["s1"] + + +# --------------------------------------------------------------------------- +# Building log entries +# --------------------------------------------------------------------------- + + +def test_build_attribution_entries_records_values_user_and_date(): + rows = _analysis().filter(pl.col("match_status") == "mismatch") + entries = build_attribution_entries( + rows, SURVEY_KEY, ErrorSource.ENUMERATOR, "", user="ana", date=NOW + ) + assert entries.schema == pl.Schema(ATTRIBUTION_LOG_SCHEMA) + assert entries.to_dicts() == [ + { + "survey_key": "s1", + "backcheck_key": "b1", + "column_name": "age", + "survey_value": "30", + "backcheck_value": "31", + "source": "Enumerator", + "note": None, + "user": "ana", + "date": NOW, + }, + { + "survey_key": "s2", + "backcheck_key": "b2", + "column_name": "age", + "survey_value": "41", + "backcheck_value": "40", + "source": "Enumerator", + "note": None, + "user": "ana", + "date": NOW, + }, + ] + + +@pytest.mark.parametrize("source", [ErrorSource.BACKCHECKER, ErrorSource.RESPONDENT]) +@pytest.mark.parametrize("note", ["", " ", None]) +def test_build_attribution_entries_requires_note(source, note): + rows = _analysis().head(1) + with pytest.raises(ValueError, match="note"): + build_attribution_entries(rows, SURVEY_KEY, source, note, user="a", date=NOW) + + +def test_build_attribution_entries_strips_note(): + rows = _analysis().head(1) + entries = build_attribution_entries( + rows, SURVEY_KEY, ErrorSource.RESPONDENT, " changed answer ", "a", NOW + ) + assert entries["note"].to_list() == ["changed answer"] + + +def test_build_attribution_entries_rejects_non_mismatch_rows(): + with pytest.raises(ValueError, match="mismatch"): + build_attribution_entries( + _analysis(), SURVEY_KEY, ErrorSource.ENUMERATOR, "", "a", NOW + ) + + +def test_build_attribution_entries_rejects_no_rows(): + with pytest.raises(ValueError, match="No"): + build_attribution_entries( + _analysis().clear(), SURVEY_KEY, ErrorSource.ENUMERATOR, "", "a", NOW + ) + + +def test_build_attribution_entries_shared_key_when_survey_key_is_merge_id(): + rows = _analysis().drop(BC_KEY).head(1) + entries = build_attribution_entries( + rows, SURVEY_KEY, ErrorSource.ENUMERATOR, None, "a", NOW + ) + assert entries["backcheck_key"].to_list() == ["s1"] + + +# --------------------------------------------------------------------------- +# Marking error sources, latest entry and lapse +# --------------------------------------------------------------------------- + + +def test_mark_error_sources_without_log_marks_mismatches_unattributed(): + marked = mark_error_sources(_analysis(), _log([]), SURVEY_KEY) + assert marked[ERROR_SOURCE_COL].to_list() == [ + "Unattributed", + "Unattributed", + None, + None, + ] + assert marked.drop(ERROR_SOURCE_COL).equals(_analysis()) + + +def test_mark_error_sources_latest_entry_wins(): + log = _log( + [ + _entry("s1", "b1", "30", "31", "Respondent", note="moved"), + _entry("s1", "b1", "30", "31", "Enumerator"), + ] + ) + marked = mark_error_sources(_analysis(), log, SURVEY_KEY) + assert marked[ERROR_SOURCE_COL].to_list()[:2] == ["Enumerator", "Unattributed"] + + +def test_mark_error_sources_can_be_reset_to_unattributed(): + log = _log( + [ + _entry("s1", "b1", "30", "31", "Backchecker", note="typo"), + _entry("s1", "b1", "30", "31", "Unattributed"), + ] + ) + marked = mark_error_sources(_analysis(), log, SURVEY_KEY) + assert marked[ERROR_SOURCE_COL][0] == "Unattributed" + + +@pytest.mark.parametrize( + ("survey_value", "backcheck_value"), [("29", "31"), ("30", "32")] +) +def test_mark_error_sources_attribution_lapses_when_a_value_changes( + survey_value, backcheck_value +): + log = _log([_entry("s1", "b1", survey_value, backcheck_value, "Backchecker")]) + marked = mark_error_sources(_analysis(), log, SURVEY_KEY) + assert marked[ERROR_SOURCE_COL][0] == "Unattributed" + + +def test_mark_error_sources_ignores_attribution_once_pair_matches(): + log = _log([_entry("s3", "b3", "25", "25", "Enumerator")]) + marked = mark_error_sources(_analysis(), log, SURVEY_KEY) + assert marked[ERROR_SOURCE_COL][2] is None + + +def test_mark_error_sources_matches_column_and_backcheck_key(): + log = _log( + [ + _entry("s1", "b1", "30", "31", "Enumerator", column_name="income"), + _entry("s2", "bX", "41", "40", "Enumerator"), + ] + ) + marked = mark_error_sources(_analysis(), log, SURVEY_KEY) + assert marked[ERROR_SOURCE_COL].to_list()[:2] == ["Unattributed", "Unattributed"] + + +def test_mark_error_sources_when_survey_key_is_merge_id(): + analysis = _analysis().drop(BC_KEY) + log = _log([_entry("s1", "s1", "30", "31", "Enumerator")]) + marked = mark_error_sources(analysis, log, SURVEY_KEY) + assert marked[ERROR_SOURCE_COL][0] == "Enumerator" + + +def test_mark_error_sources_empty_analysis(): + assert mark_error_sources(pl.DataFrame(), _log([]), SURVEY_KEY).is_empty() + + +# --------------------------------------------------------------------------- +# Counts and rates +# --------------------------------------------------------------------------- + + +def _marked(sources: list[str | None], statuses: list[str]) -> pl.DataFrame: + return pl.DataFrame({"match_status": statuses, ERROR_SOURCE_COL: sources}) + + +def test_count_error_sources(): + rows = _marked( + ["Enumerator", "Backchecker", "Backchecker", "Unattributed", None], + ["mismatch", "mismatch", "mismatch", "mismatch", "match"], + ) + assert count_error_sources(rows) == { + ErrorSource.ENUMERATOR: 1, + ErrorSource.BACKCHECKER: 2, + ErrorSource.RESPONDENT: 0, + ErrorSource.UNATTRIBUTED: 1, + } + + +def test_count_error_sources_without_source_column_counts_unattributed(): + rows = pl.DataFrame({"match_status": ["mismatch", "match", "mismatch"]}) + assert count_error_sources(rows)[ErrorSource.UNATTRIBUTED] == 2 + + +# Ten values compared: 6 mismatches, of which 1 Enumerator, 2 Backchecker, +# 1 Respondent and 2 Unattributed. +_RATE_ROWS = _marked( + ["Enumerator", "Backchecker", "Backchecker", "Respondent"] + + ["Unattributed"] * 2 + + [None] * 4, + ["mismatch"] * 6 + ["match"] * 4, +) + + +def test_excused_mismatches_per_staff_type(): + assert excused_mismatches(_RATE_ROWS, "enumerator") == 3 + assert excused_mismatches(_RATE_ROWS, "backchecker") == 2 + + +def test_adjusted_error_rate_for_enumerators(): + # (6 mismatches - 2 Backchecker - 1 Respondent) / 10 compared + assert adjusted_error_rate(6, 10, _RATE_ROWS, "enumerator") == 30.0 + + +def test_adjusted_error_rate_for_backcheckers(): + # (6 mismatches - 1 Enumerator - 1 Respondent) / 10 compared + assert adjusted_error_rate(6, 10, _RATE_ROWS, "backchecker") == 40.0 + + +def test_adjusted_error_rate_counts_unattributed_and_handles_no_values(): + rows = _marked(["Unattributed"], ["mismatch"]) + assert adjusted_error_rate(1, 4, rows, "enumerator") == 25.0 + assert adjusted_error_rate(0, 0, rows.clear(), "enumerator") == 0.0 + + +def test_adjusted_error_rate_rejects_unknown_staff_type(): + with pytest.raises(ValueError, match="staff type"): + adjusted_error_rate(1, 1, _RATE_ROWS, "respondent") + + +def test_attributed_share(): + rows = _marked( + ["Enumerator", "Respondent", "Unattributed", "Unattributed", None], + ["mismatch"] * 4 + ["match"], + ) + assert attributed_share(rows) == 50.0 + + +def test_attributed_share_none_without_mismatches(): + assert attributed_share(_marked([None], ["match"])) is None + assert attributed_share(pl.DataFrame()) is None + + +# --------------------------------------------------------------------------- +# Storage +# --------------------------------------------------------------------------- + + +def test_save_attributions_appends_to_existing_log(): + existing = _log([_entry("s1", "b1", "30", "31", "Enumerator")]) + new = _log([_entry("s1", "b1", "30", "31", "Respondent", note="moved")]) + with ( + patch( + "datasure.checks.backchecks.attribution.duckdb_get_table", + return_value=existing, + ) as get_table, + patch("datasure.checks.backchecks.attribution.duckdb_save_table") as save, + ): + save_attributions("proj", "page", new) + + get_table.assert_called_once_with("proj", "bc_attribution_page", "logs") + saved = save.call_args.args[1] + assert save.call_args.args[0] == "proj" + assert save.call_args.args[2:] == ("bc_attribution_page", "logs") + assert saved["source"].to_list() == ["Enumerator", "Respondent"] + + +def test_load_attribution_log_empty_table_has_schema(): + with patch( + "datasure.checks.backchecks.attribution.duckdb_get_table", + return_value=pl.DataFrame(), + ): + log = load_attribution_log("proj", "page") + assert log.is_empty() + assert log.schema == pl.Schema(ATTRIBUTION_LOG_SCHEMA) + + +def test_attribution_history_lists_newest_first(): + log = _log( + [ + _entry("s1", "b1", "30", "31", "Enumerator", date=datetime(2026, 1, 1)), + _entry("s2", "b2", "41", "40", "Respondent", date=datetime(2026, 2, 1)), + ] + ) + assert attribution_history(log)["survey_key"].to_list() == ["s2", "s1"] diff --git a/tests/checks/backchecks/test_compute.py b/tests/checks/backchecks/test_compute.py index 1e4ed135..039d4ef8 100644 --- a/tests/checks/backchecks/test_compute.py +++ b/tests/checks/backchecks/test_compute.py @@ -97,6 +97,39 @@ def test_load_default_backchecks_settings_partial_saved(tmp_path): assert result.backcheck_target_percent == 20 +@pytest.mark.parametrize("saved_target", [150, -1, "5"]) +def test_load_default_backchecks_settings_invalid_error_rate_target( + tmp_path, saved_target +): + """An invalid saved error rate target falls back to the configured one.""" + file_path = tmp_path / "settings.json" + file_path.write_text( + json.dumps({"backchecks": {"error_rate_target_percent": saved_target}}) + ) + config = BackcheckSettings(survey_key="KEY", error_rate_target_percent=5) + result = load_default_backchecks_settings(str(file_path), config) + assert result.error_rate_target_percent == 5 + + +def test_load_default_backchecks_settings_cleared_error_rate_target(tmp_path): + """A cleared error rate target stays cleared, so nothing is highlighted.""" + file_path = tmp_path / "settings.json" + file_path.write_text( + json.dumps({"backchecks": {"error_rate_target_percent": None}}) + ) + config = BackcheckSettings(survey_key="KEY", error_rate_target_percent=5) + result = load_default_backchecks_settings(str(file_path), config) + assert result.error_rate_target_percent is None + + +def test_load_default_backchecks_settings_saved_error_rate_target(tmp_path): + file_path = tmp_path / "settings.json" + file_path.write_text(json.dumps({"backchecks": {"error_rate_target_percent": 2.5}})) + config = BackcheckSettings(survey_key="KEY") + result = load_default_backchecks_settings(str(file_path), config) + assert result.error_rate_target_percent == 2.5 + + def test_expand_col_names_exact(): """Test expand_col_names with exact match.""" col_names = ["age", "income", "age_group", "income_total"] @@ -3027,3 +3060,209 @@ def test_compute_backchecker_productivity_month_period(): ) assert isinstance(result, pl.DataFrame) assert result.height > 0 + + +# ============================================================================== +# ERROR SOURCE ATTRIBUTION IN THE STATISTICS TABLES +# ============================================================================== + + +def _attributed_staff_data() -> pl.DataFrame: + """One staff member: 5 category-1 values compared, 4 mismatches, plus a + category-2 mismatch. Sources: Enumerator, Backchecker, Respondent, + Unattributed in category 1 and Backchecker in category 2. + """ + return pl.DataFrame( + { + "staff": ["S1"] * 6, + "category": [1, 1, 1, 1, 1, 2], + "match_status": ["mismatch"] * 4 + ["match", "mismatch"], + "survey_value": [1, 2, 3, 4, 5, 6], + "backcheck_value": [9, 9, 9, 9, 5, 9], + "error_source": [ + "Enumerator", + "Backchecker", + "Respondent", + "Unattributed", + None, + "Backchecker", + ], + } + ) + + +def test_staff_statistics_adjusted_error_rate_for_enumerators(): + stats = _calculate_staff_statistics( + _attributed_staff_data(), "staff", "S1", "KEY", None, None, "enumerator" + ) + # Regular rates ignore attribution. + assert stats["Mismatches (Cat 1)"] == 4 + assert stats["Error Rate % (Cat 1)"] == 80.0 + assert stats["Error Rate % (Total)"] == round(5 / 6 * 100, 2) + # Cat 1: (4 - 1 Backchecker - 1 Respondent) / 5 + assert stats["Adjusted Error Rate % (Cat 1)"] == 40.0 + # Cat 2: (1 - 1 Backchecker) / 1 + assert stats["Adjusted Error Rate % (Cat 2)"] == 0.0 + assert stats["Adjusted Error Rate % (Cat 3)"] == 0.0 + # Total: (5 - 2 Backchecker - 1 Respondent) / 6 + assert stats["Adjusted Error Rate % (Total)"] == round(2 / 6 * 100, 2) + + +def test_staff_statistics_adjusted_error_rate_for_backcheckers(): + stats = _calculate_staff_statistics( + _attributed_staff_data(), "staff", "S1", "KEY", None, None, "backchecker" + ) + # Cat 1: (4 - 1 Enumerator - 1 Respondent) / 5 + assert stats["Adjusted Error Rate % (Cat 1)"] == 40.0 + # Cat 2: the Backchecker mismatch counts against the backchecker. + assert stats["Adjusted Error Rate % (Cat 2)"] == 100.0 + # Total: (5 - 1 Enumerator - 1 Respondent) / 6 + assert stats["Adjusted Error Rate % (Total)"] == 50.0 + + +def test_staff_statistics_adjusted_rate_equals_regular_without_attribution(): + stats = _calculate_staff_statistics( + _attributed_staff_data().drop("error_source"), + "staff", + "S1", + "KEY", + None, + None, + ) + assert stats["Adjusted Error Rate % (Total)"] == stats["Error Rate % (Total)"] + + +def test_compute_enumerator_backchecker_stats_reports_adjusted_rate( + sample_survey_data_pl, + sample_backcheck_data_pl, + sample_backcheck_settings, +): + analysis = compute_backcheck_analysis( + sample_survey_data_pl, + sample_backcheck_data_pl, + sample_backcheck_settings, + _age_column_settings(), + ) + marked = analysis.with_columns( + pl.when(pl.col("match_status") == "mismatch") + .then(pl.lit("Respondent")) + .alias("error_source") + ) + + unmarked_stats = compute_enumerator_backchecker_stats( + sample_survey_data_pl, + sample_backcheck_data_pl, + analysis, + sample_backcheck_settings, + "enumerator", + ) + stats = compute_enumerator_backchecker_stats( + sample_survey_data_pl, + sample_backcheck_data_pl, + marked, + sample_backcheck_settings, + "enumerator", + ) + + # Attribution never changes the regular rate or the mismatch counts. + regular = ["Mismatches (Total)", "Error Rate % (Total)"] + assert ( + stats.sort("enumerator") + .select(regular) + .equals(unmarked_stats.sort("enumerator").select(regular)) + ) + # Every mismatch is the respondent's, so no enumerator is charged. + assert stats["Adjusted Error Rate % (Total)"].to_list() == [0.0] * stats.height + + +def test_compute_column_stats_counts_mismatches_by_error_source(): + analysis = pl.DataFrame( + { + "column_name": ["age"] * 5 + ["income"], + "category": [1] * 6, + "match_status": ["mismatch"] * 4 + ["match", "mismatch"], + "survey_value": [1, 2, 3, 4, 5, 6], + "backcheck_value": [9, 9, 9, 9, 5, 9], + "error_source": [ + "Enumerator", + "Backchecker", + "Backchecker", + "Unattributed", + None, + "Respondent", + ], + } + ) + + stats = compute_column_stats(pl.DataFrame(), analysis).sort("Column Name") + + assert stats.select( + "Column Name", + "Mismatches", + "Error Rate (%)", + "Enumerator Mismatches", + "Backchecker Mismatches", + "Respondent Mismatches", + "Unattributed Mismatches", + ).rows() == [ + ("age", 4, 80.0, 1, 2, 0, 1), + ("income", 1, 100.0, 0, 0, 1, 0), + ] + assert not any("Adjusted" in col for col in stats.columns) + + +def test_compute_column_stats_without_attribution_counts_unattributed( + sample_survey_data_pl, + sample_backcheck_data_pl, + sample_backcheck_settings, +): + analysis = compute_backcheck_analysis( + sample_survey_data_pl, + sample_backcheck_data_pl, + sample_backcheck_settings, + _age_column_settings(), + ) + stats = compute_column_stats(sample_survey_data_pl, analysis) + assert stats["Unattributed Mismatches"].to_list() == stats["Mismatches"].to_list() + + +def test_compute_overall_error_rates_total_and_categories(): + from datasure.checks.backchecks.compute import compute_overall_error_rates + + # Cat 1: 5 compared, 4 mismatches (Enumerator, Backchecker, Respondent, + # Unattributed). Cat 2: 1 compared, a Backchecker mismatch. Cat 3: none. + rates = compute_overall_error_rates(_attributed_staff_data()) + + assert [r.label for r in rates] == ["Total", "Cat 1", "Cat 2", "Cat 3"] + total, cat1, cat2, cat3 = rates + assert (total.compared, total.mismatches) == (6, 5) + assert total.error_rate == round(5 / 6 * 100, 2) + # Enumerators: (5 - 2 Backchecker - 1 Respondent) / 6 + assert total.enumerator_adjusted == round(2 / 6 * 100, 2) + # Backcheckers: (5 - 1 Enumerator - 1 Respondent) / 6 + assert total.backchecker_adjusted == 50.0 + assert (cat1.error_rate, cat1.enumerator_adjusted) == (80.0, 40.0) + assert (cat2.error_rate, cat2.enumerator_adjusted) == (100.0, 0.0) + assert cat2.backchecker_adjusted == 100.0 + assert (cat3.compared, cat3.error_rate) == (0, None) + + +def test_compute_overall_error_rates_ignores_missing_and_excluded(): + from datasure.checks.backchecks.compute import compute_overall_error_rates + + analysis = pl.DataFrame( + { + "category": [1, 1, 1, 1], + "match_status": ["mismatch", "match", "missing", "excluded"], + } + ) + total = compute_overall_error_rates(analysis)[0] + assert (total.compared, total.error_rate) == (2, 50.0) + # Without attribution, the adjusted rates equal the regular one. + assert total.enumerator_adjusted == total.backchecker_adjusted == 50.0 + + +def test_compute_overall_error_rates_empty_analysis(): + from datasure.checks.backchecks.compute import compute_overall_error_rates + + assert compute_overall_error_rates(pl.DataFrame()) == [] diff --git a/tests/checks/backchecks/test_report_ui_attribution.py b/tests/checks/backchecks/test_report_ui_attribution.py new file mode 100644 index 00000000..5d4a7c27 --- /dev/null +++ b/tests/checks/backchecks/test_report_ui_attribution.py @@ -0,0 +1,357 @@ +"""Tests for mismatch attribution on the Backchecks page (report_ui).""" + +from datetime import datetime +from unittest.mock import MagicMock, patch + +import polars as pl +import pytest + +from datasure.checks.backchecks.attribution import ( + ATTRIBUTION_LOG_SCHEMA, + ERROR_SOURCE_COL, + ErrorSource, +) +from datasure.checks.backchecks.models import BackcheckSettings +from datasure.checks.backchecks.report_ui import ( + AttributionContext, + _build_display_columns, + _error_rate_columns, + _highlight_above_target, + _render_attribution_form, + _render_attribution_log, + _render_backcheck_summary, + _render_comparison_table, +) +from tests.checks.backchecks.conftest import make_mock_st + +MODULE = "datasure.checks.backchecks.report_ui" + + +@pytest.fixture +def mock_st(): + st = make_mock_st() + st.session_state = {} + st.text_area.return_value = "" + with patch(f"{MODULE}.st", st): + yield st + + +@pytest.fixture +def review(): + return AttributionContext( + "proj", "page", pl.DataFrame(schema=ATTRIBUTION_LOG_SCHEMA) + ) + + +def _table() -> pl.DataFrame: + return pl.DataFrame( + { + "KEY": ["s1", "s2", "s3"], + "KEY__BCCL": ["b1", "b2", "b3"], + "column_name": ["age", "age", "age"], + "survey_value": [30, 41, 25], + "backcheck_value": [31, 40, 25], + "match_status": ["mismatch", "mismatch", "match"], + "category": [1, 1, 1], + ERROR_SOURCE_COL: ["Unattributed", "Unattributed", None], + } + ) + + +# --------------------------------------------------------------------------- +# Comparison table +# --------------------------------------------------------------------------- + + +def test_display_columns_show_error_source_after_match_status(): + columns = _build_display_columns(_table(), "KEY", None, "KEY__BCCL") + assert columns.index(ERROR_SOURCE_COL) == columns.index("match_status") + 1 + + +def test_comparison_table_review_buttons_only_on_mismatches(mock_st, review): + _render_comparison_table(_table(), {}, "KEY", review) + + shown = mock_st.dataframe.call_args.args[0] + labels = shown.to_series(0).to_list() + assert labels[2] is None + assert labels[0] is not None + assert labels[1] == labels[0] + assert mock_st.dataframe.call_args.kwargs["selection_mode"] == "multi-row" + + +def test_comparison_table_pins_review_then_error_source(mock_st, review): + _render_comparison_table(_table(), {}, "KEY", review) + + shown = mock_st.dataframe.call_args.args[0] + button_col = shown.columns[0] + assert shown.columns[1] == ERROR_SOURCE_COL + config = mock_st.dataframe.call_args.kwargs["column_config"] + assert mock_st.column_config.ButtonColumn.call_args.kwargs["pinned"] is True + assert isinstance(mock_st.column_config.ButtonColumn.call_args.kwargs["width"], int) + assert config[button_col] is mock_st.column_config.ButtonColumn.return_value + error_source_config = next( + c + for c in mock_st.column_config.TextColumn.call_args_list + if c.args and c.args[0] == "Error Source" + ) + assert error_source_config.kwargs["pinned"] is True + assert config[ERROR_SOURCE_COL] is mock_st.column_config.TextColumn.return_value + + +def test_comparison_table_without_click_opens_no_dialog(mock_st, review): + with patch(f"{MODULE}._attribution_dialog") as dialog: + _render_comparison_table(_table(), {}, "KEY", review) + dialog.assert_not_called() + + +def test_comparison_table_click_inside_selection_reviews_selected_mismatches( + mock_st, review +): + mock_st.session_state["backchecks_attribution_review_click"] = {"row": 1} + mock_st.dataframe.return_value = MagicMock(selection=MagicMock(rows=[0, 1, 2])) + with patch(f"{MODULE}._attribution_dialog") as dialog: + _render_comparison_table(_table(), {}, "KEY", review) + + rows = dialog.call_args.args[0] + assert rows["KEY"].to_list() == ["s1", "s2"] + + +def test_comparison_table_without_review_is_plain(mock_st): + _render_comparison_table(_table(), {}, "KEY", None) + assert "on_select" not in mock_st.dataframe.call_args.kwargs + + +# --------------------------------------------------------------------------- +# Attribution form +# --------------------------------------------------------------------------- + + +def _mismatches() -> pl.DataFrame: + return _table().head(2) + + +@pytest.mark.parametrize("source", [ErrorSource.BACKCHECKER, ErrorSource.RESPONDENT]) +def test_attribution_form_needs_note_to_save(mock_st, review, source): + mock_st.radio.return_value = source + mock_st.text_area.return_value = " " + with patch(f"{MODULE}.save_attributions") as save: + _render_attribution_form(_mismatches(), "KEY", review) + + assert mock_st.button.call_args.kwargs["disabled"] is True + save.assert_not_called() + + +def test_attribution_form_saves_enumerator_without_note(mock_st, review): + mock_st.radio.return_value = ErrorSource.ENUMERATOR + mock_st.button.return_value = True + with ( + patch(f"{MODULE}.save_attributions") as save, + patch(f"{MODULE}.get_reviewer_name", return_value="ana"), + patch(f"{MODULE}.queue_notice") as notice, + ): + _render_attribution_form(_mismatches(), "KEY", review) + + assert mock_st.button.call_args.kwargs["disabled"] is False + project_id, page_name_id, entries = save.call_args.args + assert (project_id, page_name_id) == ("proj", "page") + assert entries["source"].to_list() == ["Enumerator", "Enumerator"] + assert entries["user"].to_list() == ["ana", "ana"] + assert isinstance(entries["date"][0], datetime) + assert notice.call_args.args[1] == "toast" + mock_st.rerun.assert_called_once() + + +def test_attribution_form_saves_note(mock_st, review): + mock_st.radio.return_value = ErrorSource.RESPONDENT + mock_st.text_area.return_value = "respondent changed the answer" + mock_st.button.return_value = True + with ( + patch(f"{MODULE}.save_attributions") as save, + patch(f"{MODULE}.get_reviewer_name", return_value="ana"), + patch(f"{MODULE}.queue_notice"), + ): + _render_attribution_form(_mismatches(), "KEY", review) + + entries = save.call_args.args[2] + assert entries["note"].to_list() == ["respondent changed the answer"] * 2 + + +def test_attribution_form_shows_error_when_save_fails(mock_st, review): + mock_st.radio.return_value = ErrorSource.ENUMERATOR + mock_st.button.return_value = True + with ( + patch(f"{MODULE}.save_attributions", side_effect=OSError("disk full")), + patch(f"{MODULE}.get_reviewer_name", return_value="ana"), + ): + _render_attribution_form(_mismatches(), "KEY", review) + + mock_st.error.assert_called_once() + mock_st.rerun.assert_not_called() + + +def test_attribution_form_shows_values_read_only(mock_st, review): + mock_st.radio.return_value = ErrorSource.ENUMERATOR + _render_attribution_form(_mismatches(), "KEY", review) + + shown = mock_st.dataframe.call_args.args[0] + assert {"survey_value", "backcheck_value"} <= set(shown.columns) + mock_st.data_editor.assert_not_called() + + +# --------------------------------------------------------------------------- +# Attribution log, summary metric and rate highlighting +# --------------------------------------------------------------------------- + + +def test_attribution_log_empty(mock_st): + _render_attribution_log(pl.DataFrame(schema=ATTRIBUTION_LOG_SCHEMA)) + mock_st.info.assert_called_once() + + +def test_attribution_log_lists_history(mock_st): + log = pl.DataFrame( + [ + { + "survey_key": "s1", + "backcheck_key": "b1", + "column_name": "age", + "survey_value": "30", + "backcheck_value": "31", + "source": source, + "note": None, + "user": "ana", + "date": datetime(2026, 1, day), + } + for day, source in [(1, "Enumerator"), (2, "Unattributed")] + ], + schema=ATTRIBUTION_LOG_SCHEMA, + ) + _render_attribution_log(log) + shown = mock_st.dataframe.call_args.args[0] + assert shown["source"].to_list() == ["Unattributed", "Enumerator"] + + +def _summary_metric(analysis): + survey = pl.DataFrame({"key": [1, 2]}) + backcheck = pl.DataFrame({"key": [1]}) + settings = BackcheckSettings(survey_key="key", survey_id="key") + with patch(f"{MODULE}.metric_row") as metric_row: + _render_backcheck_summary(survey, backcheck, settings, analysis) + metrics = metric_row.call_args.args[0] + return next(m for m in metrics if m[0] == "Mismatches Attributed") + + +def test_summary_shows_share_of_mismatches_attributed(mock_st): + analysis = _table().with_columns( + pl.Series(ERROR_SOURCE_COL, ["Respondent", "Unattributed", None]) + ) + assert _summary_metric(analysis)[1] == "50.0%" + + +def test_summary_attributed_share_na_without_mismatches(mock_st): + assert _summary_metric(pl.DataFrame())[1] == "N/A" + + +def test_error_rate_columns_cover_regular_and_adjusted_rates(): + columns = [ + "enumerator", + "Coverage %", + "Error Rate % (Cat 1)", + "Adjusted Error Rate % (Cat 1)", + "Error Rate % (Total)", + "Adjusted Error Rate % (Total)", + ] + assert _error_rate_columns(columns) == columns[2:] + + +def test_highlight_above_target(): + style = _highlight_above_target(5.0) + assert style(5.01) + assert style(5.0) == "" + assert style(None) == "" + + +def test_attribution_form_when_survey_key_is_merge_id(mock_st, review): + mock_st.radio.return_value = ErrorSource.ENUMERATOR + _render_attribution_form(_mismatches().drop("KEY__BCCL"), "KEY", review) + + shown = mock_st.dataframe.call_args.args[0] + assert shown.columns.count("KEY") == 1 + + +def test_comparison_results_section_is_a_fragment(): + """Selecting rows or clicking Review reruns only the comparison section.""" + import importlib + import sys + + report_ui = sys.modules[MODULE] + fragments = [] + st = make_mock_st() + st.fragment = lambda func: fragments.append(func.__name__) or func + original_st = sys.modules["streamlit"] + sys.modules["streamlit"] = st + try: + importlib.reload(report_ui) + finally: + sys.modules["streamlit"] = original_st + importlib.reload(report_ui) + + assert "_render_comparison_results_section" in fragments + + +def test_attribution_save_reruns_the_whole_app(mock_st, review): + """A saved attribution refreshes the rates outside the fragment too.""" + mock_st.radio.return_value = ErrorSource.ENUMERATOR + mock_st.button.return_value = True + with ( + patch(f"{MODULE}.save_attributions"), + patch(f"{MODULE}.get_reviewer_name", return_value="ana"), + patch(f"{MODULE}.queue_notice"), + ): + _render_attribution_form(_mismatches(), "KEY", review) + + mock_st.rerun.assert_called_once_with(scope="app") + + +def _error_rate_cards(mock_st, analysis): + survey = pl.DataFrame({"key": [1, 2]}) + backcheck = pl.DataFrame({"key": [1]}) + settings = BackcheckSettings(survey_key="key", survey_id="key") + with patch(f"{MODULE}.metric_row"): + _render_backcheck_summary(survey, backcheck, settings, analysis) + return { + c.args[0]: c + for c in mock_st.metric.call_args_list + if c.args[0].startswith("Error Rate") + } + + +def test_summary_error_rate_cards_show_adjusted_rate_as_delta(mock_st): + # Two mismatches out of three compared, one attributed to the respondent. + analysis = _table().with_columns( + pl.lit(1).alias("category"), + pl.Series(ERROR_SOURCE_COL, ["Respondent", "Unattributed", None]), + ) + cards = _error_rate_cards(mock_st, analysis) + + assert list(cards) == [ + "Error Rate (Total)", + "Error Rate (Cat 1)", + "Error Rate (Cat 2)", + "Error Rate (Cat 3)", + ] + total = cards["Error Rate (Total)"] + assert total.args[1] == "66.67%" + assert total.kwargs["delta"] == "33.33% adjusted" + assert total.kwargs["delta_color"] == "off" + assert total.kwargs["delta_arrow"] == "off" + assert "33.33%" in total.kwargs["help"] # backchecker adjusted rate + cat2 = cards["Error Rate (Cat 2)"] + assert cat2.args[1] == "N/A" + # Every card has a delta line, so the cards are the same height. + assert cat2.kwargs["delta"] == "No values compared" + + +def test_summary_error_rate_cards_need_configured_columns(mock_st): + assert _error_rate_cards(mock_st, pl.DataFrame()) == {} + assert any("Error rates" in c.args[0] for c in mock_st.info.call_args_list) diff --git a/tests/checks/backchecks/test_settings_ui.py b/tests/checks/backchecks/test_settings_ui.py index fc9df107..a1d9c474 100644 --- a/tests/checks/backchecks/test_settings_ui.py +++ b/tests/checks/backchecks/test_settings_ui.py @@ -156,12 +156,24 @@ def test_render_staff_identifiers(patched_bc): def test_render_tracking_options(patched_bc): - """_render_tracking_options returns a numeric backcheck_goal.""" - patched_bc.number_input.return_value = 50 + """_render_tracking_options returns the backcheck and error rate targets.""" + patched_bc.number_input.side_effect = [50, 5] result = _render_tracking_options( "settings.json", BackcheckSettings(survey_key=None) ) - assert result == 50 + assert result == (50, 5) + + +def test_render_tracking_options_saves_error_rate_target(patched_bc): + """A cleared error rate target is saved as None, so nothing is highlighted.""" + patched_bc.number_input.side_effect = [10, None] + with patch("datasure.checks.backchecks.settings_ui.save_check_settings") as save: + _, error_target = _render_tracking_options( + "settings.json", BackcheckSettings(survey_key=None) + ) + assert error_target is None + saved = [call.args[2] for call in save.call_args_list] + assert {"error_rate_target_percent": None} in saved def test_render_duplicate_handling(patched_bc): @@ -249,7 +261,7 @@ def report_settings_with_choices(patched_bc): patch(f"{module}._render_survey_identifiers", return_value=("key", "sid")), patch(f"{module}._render_date_columns", return_value=(None, None)), patch(f"{module}._render_staff_identifiers", return_value=(None, None)), - patch(f"{module}._render_tracking_options", return_value=35), + patch(f"{module}._render_tracking_options", return_value=(35, 4.5)), patch( f"{module}._render_additional_options", return_value=("last", [], [], StrCompareOptions()), @@ -282,6 +294,13 @@ def test_report_settings_keeps_selected_target_percent( assert report_settings_with_choices.backcheck_target_percent == 35 +def test_report_settings_keeps_selected_error_rate_target( + report_settings_with_choices, +): + """The error rate target reaches the returned settings.""" + assert report_settings_with_choices.error_rate_target_percent == 4.5 + + def test_render_tracking_options_persists_changed_target(tmp_path): """A changed target passes the real save guard and reloads from disk.""" settings_file = str(tmp_path / "settings.json") @@ -289,8 +308,10 @@ def test_render_tracking_options_persists_changed_target(tmp_path): mock_st = make_mock_st() mock_st.session_state = session_state - def change_target(*_args, on_change, args, **_widget_kwargs): - on_change(*args) + def change_target(*_args, key, on_change, **widget_kwargs): + if key != "backcheck_goal_backchecks": + return None + on_change(*widget_kwargs["args"]) return 35 mock_st.number_input.side_effect = change_target @@ -300,6 +321,7 @@ def change_target(*_args, on_change, args, **_widget_kwargs): ): _render_tracking_options(settings_file, BackcheckSettings(survey_key=None)) - assert load_check_settings(settings_file, "backchecks") == { - "backcheck_target_percent": 35 - } + assert ( + load_check_settings(settings_file, "backchecks")["backcheck_target_percent"] + == 35 + ) diff --git a/tests/processing/test_corrections.py b/tests/processing/test_corrections.py index 368841e7..af7ea814 100644 --- a/tests/processing/test_corrections.py +++ b/tests/processing/test_corrections.py @@ -1311,7 +1311,10 @@ def test_accept_requires_a_reason(self, store, sample_data): assert processor.get_correction_log("survey").is_empty() - def test_accept_rejects_unknown_check_type(self, store, sample_data): + # Backcheck results measure data quality, so a mismatch can't be accepted + # away; it can only be attributed on the Backchecks page. + @pytest.mark.parametrize("check_type", ["missing", "backchecks"]) + def test_accept_rejects_unknown_check_type(self, store, sample_data, check_type): _seed_prep(store, sample_data) processor = CorrectionProcessor("p1") @@ -1320,7 +1323,7 @@ def test_accept_rejects_unknown_check_type(self, store, sample_data): alias="survey", key_col="survey_key", key_value="key1", - check_type="missing", + check_type=check_type, column="age", current_value=25, reason="ok",