You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(outliers): correct or accept outliers and constraint violations from the check page - #316
Lets users correct or accept outliers and constraint violations directly from the Outliers & Constraints tab, instead of switching to the Correct Data page.
Review button: each row of the constraint violations and outlier inspection tables starts with a pinned Review button. It opens the shared correction form (Corrections: add source and accept to the log, extract a shared correction form #296) in a dialog, prefilled with the row's KEY, column and current value. The actions are modify value, remove value and accept as valid. Entries are logged with source and check type outliers or constraints.
Accepted flags are hidden from the tables and left out of the metrics. They come back if the value changes or the acceptance is removed on the Correct Data page. Outlier and constraint acceptances are independent.
Hard constraint violations can only be accepted after a confirmation checkbox. They are logged with a new severity = "hard" column and highlighted red in the Correction Log.
Table toggles:
Show only flagged values is on by default. Turn it off to see every checked value.
Show reviewed shows accepted flags (Reviewed badge) and corrected values (Corrected badge) highlighted green, with the reason. A corrected value that is still flagged stays visible and counted, because it still needs attention.
Show only reviewed lists only accepted and corrected rows. The other two toggles are disabled while it's on.
After a save: a full rerun, then a toast and an Open the Correction Log page link.
Bug fixes included
Hard bounds read as soft:compute_constraint_violations tested soft bounds before hard ones. With a soft max set, a value above the hard max was reported as "above soft maximum", which also undercounted hard violations. This predates the PR; the confirmation step depends on it.
Styled tables could crash: the Summary, Missing and Progress tabs lowered pandas' process-wide styler.render.max_elements to fit their own tables, which could crash other styled tables. A single helper, ensure_styler_limit, now only raises the limit, under a lock, and all of them use it.
Styled tables changed values: they no longer reformat values. pandas' default Styler formatting showed 150 as 150.000000 and missing values as nan.
Why is this change needed? 🤔
Issue #298: reviewing a flag currently means leaving the check page, finding the KEY on the Correct Data page, and re-entering the column and value. Values that are confirmed correct have no way to be marked, so they keep showing up as flags and inflating the metrics on every run.
How was this implemented? 🛠️
src/datasure/checks/outliers/review.py (new, no Streamlit): flag review logic. It covers marking accepted and corrected rows, clearing accepted flags for metrics, filtering visible and flagged rows, turning a clicked row into a form prefill, the allowed actions, and the hard-confirmation rule.
checks/outliers/report_ui.py:
outliers_report takes the dataset alias.
The tables render through _render_flags_table, which uses an st.column_config.ButtonColumn for Review and an @st.dialog for the form.
The unused _render_outlier_table is removed, and the outlier table now hides its index.
processing/corrections.py: new get_active_corrections. A modify or remove counts as active while the data still holds its result. Also new: CorrectionEntry.severity, which is rejected on non-accept actions.
processing/correction_log.py: new severity log column. Legacy logs are backfilled with null, and the replication correction_log.csv gains the column.
utils/ui_utils.py:
queue_notice gains a toast level, and show_queued_notices now returns whether it showed anything.
New styled_dataframe raises the Styler cell limit only for one call.
New row_styler keeps nullable types and shows values as plain text.
Toast without a link: Streamlit opens Markdown links in a new tab, which starts a new session and loses the selected project. So the link to the Correction Log is an st.page_link shown next to the toast.
How to test or reproduce ? 🧪
just test: 3484 passed, 6 skipped. just pre-commit-run: clean.
In the app, open a check page whose Outliers & Constraints tab has columns configured with soft and hard bounds.
Click Review on a soft violation, choose accept, enter a reason and click Apply. The row disappears, the metrics drop by one, and the entry appears in the Correction Log with source constraints.
Click Review on a hard violation and choose accept. Apply stays disabled until the confirmation box is ticked, and the log row is highlighted red.
Click Review on a flagged value and modify it into range. Turn off Show only flagged values and turn on Show reviewed. The row is green with a Corrected badge, and its values display exactly as with the toggle off.
Turn on Show only reviewed. Only the accepted and corrected rows remain, and the other two toggles are disabled.
Remove the acceptance on the Correct Data page. The flag comes back on the check page.
Screenshots (if applicable) 📷
None yet. The UI was checked with Streamlit's AppTest harness and manually in the app during development. A screenshot of the Review button, the dialog and the green Show reviewed rows would help reviewers.
Checklist ✅
I have run and tested my changes locally
I have limit this PR to less than 1000 lines of code change (if not, explain why). About 2,850 lines added: roughly 1,860 are tests and the rest is source and docs. The feature spans the check page, the correction processor and log schema, and shared UI helpers, and splitting it would leave the check page half-working.
I have updated/added tests to cover my changes (if applicable)
I have updated/added requirements to cover my changes (if applicable). N/A: no new dependencies (pyarrow and pandas are already declared).
I have run linting and formatting on any code changes (if applicable)
I have updated the documentation (README, etc.) accordingly. Updated: CHANGELOG.md, docs/USER_GUIDE.md and CONTRIBUTING.md.
I have reviewed and resolved any merge conflict
Reviewer Emoji Legend
:code:
Meaning
😃👍💯
:smiley::+1::100:
I like this...
...and I want the author to know it! This is a way to highlight positive parts of a code review.
⭐⭐⭐
:star: :star: :star:
Important to fix before PR can be approved...
And I am providing reasons why it needs to be addressed as well as suggested improvements.
⭐⭐
:star: :star:
Important to fix but non-blocking for PR approval...
And I am providing suggestions where it could be improved either in this PR or later.
⭐
:star:
Give this some thought but non-blocking for PR approval...
...and consider this a suggestion, not a requirement.
❓
:question:
I have a question.
This should be a fully formed question with sufficient information and context that requires a response.
📝
:memo:
This is an explanatory note, fun fact, or relevant commentary that does not require any action.
⛏
:pick:
This is a nitpick.
This does not require any changes and is often better left unsaid. This may include stylistic, formatting, or organization suggestions and should likely be prevented/enforced by linting if they really matter
♻️
:recycle:
Suggestion for refactoring.
Should include enough context to be actionable and not be considered a nitpick.
…from the check page
- Selecting a row in the constraint violations or outlier inspection table
opens the shared correction form, prefilled with the row's KEY, column and
current value: modify value, remove value or accept as valid. Entries are
logged with source and check type `outliers` or `constraints`
- Accepted flags are hidden from the tables and left out of the metrics; a
"Show reviewed" toggle shows them with a Reviewed badge and the reason.
They come back when the value changes or the acceptance is removed.
Outlier and constraint acceptances are independent
- Accepting a hard constraint violation needs a confirmation and is logged
with a new `severity` column ("hard"), highlighted in the Correction Log
- After a save the page reruns, the table selection resets, and a toast plus
a page link point to the Correction Log
- Flag review logic lives in the Streamlit-free checks/outliers/review.py
- outliers_report takes the dataset alias; remove unused _render_outlier_table
Refs #298
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Test hard constraint bounds before soft ones: a value above the hard
maximum was labelled "above soft maximum" whenever a soft maximum was
set, so it skipped the hard-acceptance confirmation and was undercounted
- Key each results table on the rows it shows, so a selected row position
never carries over to a different flag after "Show reviewed", a column
change or a save
- Queue the post-save toast with queue_notice (new "toast" level) per
CONTRIBUTING; show_queued_notices returns whether it showed anything
- Test that outlier metrics leave out accepted outliers
- Rename the report's wrapper to _with_review_status; reuse _is_flagged
in select_flag
Refs #298
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace row selection on the constraint violations and outlier inspection
tables with a pinned first column of Review buttons
(st.column_config.ButtonColumn). Clicking one opens the shared correction
form for that row in an st.dialog instead of below the table; a save
reruns the page, which closes the dialog.
A button click is only present during the rerun it triggers, so the
selection-reset nonce and row fingerprint in the table key are no longer
needed and are removed.
Refs #298
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a "Show only flagged values" toggle, on by default, above the
constraint violations and outlier inspection tables, next to "Show
reviewed". Turning it off shows every checked value; unflagged
constraint rows have a null violation type and can still be corrected
from their Review button, but not accepted.
The constraint table already showed only violations, so its default is
unchanged. The outlier inspection table now shows only outliers by
default instead of every value.
Refs #298
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s on
When "Show reviewed" is on, the constraint violations and outlier
inspection tables are passed to st.dataframe as a pandas Styler that
colours every cell of a reviewed flag light green (the success green used
for log statuses), so accepted values stand out from open flags. The
Review button column works unchanged on the styled table.
Refs #298
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…imit
With "Show reviewed" on, the outlier table crashed with "The dataframe has
3124 cells, but the maximum number of cells allowed to be rendered by
Pandas Styler is configured to 2395". The progress, summary and missing
tabs set the process-wide `styler.render.max_elements` option to fit their
own tables, so whichever tab rendered last capped every other Styler.
Add ui_utils.styled_dataframe, which raises the limit to fit the table
(never lowering it) only for the st.dataframe call, via
pd.option_context. Use it for the outlier and constraint results tables
and for the Correction Log, which also renders a Styler.
Refs #298
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…alues as displayed
- With "Show reviewed" on, a cell whose current value comes from a modify
or remove correction is highlighted green with a "Corrected" badge and
the correction reason (new CorrectionProcessor.get_active_corrections:
a correction is active while the data still holds its result). Unlike
accepted flags, a corrected value that is still flagged stays visible
and counted, and can still be accepted
- Fix values changing when "Show reviewed" is on: Streamlit shows a
Styler's formatted text, and pandas' defaults showed 150 as 150.000000,
rounded to six decimals and missing values as "nan". New
ui_utils.row_styler keeps nullable types and shows plain values; the
results tables and the Correction Log use it
- Row highlight and status styling ignore pd.NA instead of raising
- Hide the outlier inspection table's index, like the constraint table
Refs #298
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ables
Add "Show only reviewed" next to "Show only flagged values" and "Show
reviewed". It lists only accepted (Reviewed) and corrected (Corrected)
rows, highlighted green, whatever the other toggles say: a value
corrected into range is unflagged, so flagged-only would hide it. The
other two toggles are disabled while it is on. Offered only when
corrections are enabled.
The toggle values are now a review.TableFilters, applied by the
Streamlit-free review.filter_table.
Refs #298
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds direct correction and acceptance of outlier and constraint flags in DataSure’s check pages, building on the shared correction form and prep-replay foundation.
Changes:
Adds Review dialogs, reviewed-value toggles and acceptance-aware metrics.
Records hard-constraint acceptance severity in the Correction Log.
Improves styled-table rendering and adds tests and documentation.
- Numeric KEYs: the Review dialog passed the KEY to the shared form as
text, so modify/remove on KEY 7 failed validation with "Key value '7'
not found in data". The saved entry now carries the KEY's native value;
the form still shows it as text
- Float32 columns: a modification such as 70.1 is stored as
70.0999984741211, so get_active_corrections never matched it and the
Corrected badge was missing. Compare against the logged value cast to
the column's type, as _apply_modify_value does
- Performance: get_active_corrections filtered the whole dataset once per
logged correction. It now checks only the rows of logged KEYs (keeping
every row of a duplicated KEY), and the report looks it up once per run
instead of once per table
- Widget keys: the dialog namespace joined check, KEY and column with
underscores, so KEY "survey_1"/column "age" collided with KEY "survey"/
column "1_age". Encode the parts as JSON
Refs #298
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Hard violations need a hard acceptance: a value accepted as a soft
violation stayed accepted after bounds were tightened into a hard
violation, hiding it from the table and metrics without the hard
confirmation. An acceptance now covers a hard violation only if it was
logged with severity "hard"; otherwise the flag stays pending and can be
accepted through the confirmation
- Review button column: a survey field named "_review" added via "Show
more columns" duplicated the button column and Polars raised. The button
column now takes a name not already in the table
- Styler limit: replace the temporary pd.option_context in
styled_dataframe with ensure_styler_limit, which raises the process-wide
styler.render.max_elements under a lock and never lowers it, so
concurrent sessions can't restore a limit below what another render
needs. The summary, missing and progress checks now use it instead of
pd.set_option, which lowered the limit to fit their own tables
Refs #298
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This filter includes corrections marked Failed by replay if their intended result matches the current data. For example, remove age=150, then re-import that cell as null: replay skips the removal because its recorded current value no longer matches, but this helper returns it as active. The check page then shows a Corrected badge and the failed step's reason even though that step was not applied. Require Successful status as well as a matching result, and cover this replay case.
- Sort results tables by KEY, column name and flag reason so a Review
click resolves to the same flag on the rerun it triggers, whatever
order the join returns.
- Keep the computed flag columns authoritative when joining survey
display columns: rename clashing survey fields (and "violation type")
with a " (survey)" suffix instead of dropping the flag's column, so a
survey field named "column name" can no longer become the correction
target.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Validate correction severity before accepting entries
src/datasure/processing/corrections.py:783
apply_corrections accepts severity="hard" with check_type="outliers", because the acceptance branch returns before this guard. The Correction Log then highlights that outlier acceptance as a hard-constraint override. Other unsupported severity strings are also persisted unchecked. Validate non-null acceptance severity before returning: allow only "hard" with check_type="constraints", and reject other combinations. Add regression tests that invalid entries leave the log unchanged.
- Review metadata can't clash with survey names: mark_reviewed matches
the log in a frame of its own helper columns, so a KEY named like a
helper (e.g. "_review_key") is no longer replaced or dropped.
join_survey_columns always renames survey fields named "review status"
or "review reason", even while those columns are hidden, so survey text
is never read as review state. A Survey KEY named like a review column
turns review off with a warning instead of being overwritten
- Duplicate KEYs: Review on a KEY whose rows hold different values of
the column shows a warning instead of the form, since a correction
changes every row with the KEY (review.key_has_conflicting_values)
- Failed corrections: get_active_corrections only returns corrections
with status "Successful", so one that failed to reapply is never shown
as Corrected
- Acceptance severity: apply_corrections rejects a severity on anything
but a constraint acceptance, and any severity other than "hard"
Refs #298
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re the Copilot "previously missed" finding, Validate correction severity before accepting entries (src/datasure/processing/corrections.py): fixed in 77b39bf.
_validate_acceptance now takes the entry's severity. An acceptance with a severity is rejected unless it is hard on a constraints acceptance, before anything is applied or logged. accept_value passes no severity, so it is unaffected.
Regression test: test_invalid_acceptance_severity_is_rejected, parametrized over a hard outlier acceptance and an unknown severity (soft). Both raise and leave the log empty.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Summary 🚀
Closes #298
What does this PR do? 📝
Lets users correct or accept outliers and constraint violations directly from the Outliers & Constraints tab, instead of switching to the Correct Data page.
outliersorconstraints.severity = "hard"column and highlighted red in the Correction Log.Bug fixes included
compute_constraint_violationstested soft bounds before hard ones. With a soft max set, a value above the hard max was reported as "above soft maximum", which also undercounted hard violations. This predates the PR; the confirmation step depends on it.styler.render.max_elementsto fit their own tables, which could crash other styled tables. A single helper,ensure_styler_limit, now only raises the limit, under a lock, and all of them use it.150as150.000000and missing values asnan.Why is this change needed? 🤔
Issue #298: reviewing a flag currently means leaving the check page, finding the KEY on the Correct Data page, and re-entering the column and value. Values that are confirmed correct have no way to be marked, so they keep showing up as flags and inflating the metrics on every run.
How was this implemented? 🛠️
src/datasure/checks/outliers/review.py(new, no Streamlit): flag review logic. It covers marking accepted and corrected rows, clearing accepted flags for metrics, filtering visible and flagged rows, turning a clicked row into a form prefill, the allowed actions, and the hard-confirmation rule.checks/outliers/report_ui.py:outliers_reporttakes the datasetalias._render_flags_table, which uses anst.column_config.ButtonColumnfor Review and an@st.dialogfor the form._render_outlier_tableis removed, and the outlier table now hides its index.processing/corrections.py: newget_active_corrections. A modify or remove counts as active while the data still holds its result. Also new:CorrectionEntry.severity, which is rejected on non-accept actions.processing/correction_log.py: newseveritylog column. Legacy logs are backfilled with null, and the replicationcorrection_log.csvgains the column.utils/ui_utils.py:queue_noticegains atoastlevel, andshow_queued_noticesnow returns whether it showed anything.styled_dataframeraises the Styler cell limit only for one call.row_stylerkeeps nullable types and shows values as plain text.st.page_linkshown next to the toast.How to test or reproduce ? 🧪
just test: 3484 passed, 6 skipped.just pre-commit-run: clean.constraints.Screenshots (if applicable) 📷
None yet. The UI was checked with Streamlit's
AppTestharness and manually in the app during development. A screenshot of the Review button, the dialog and the green Show reviewed rows would help reviewers.Checklist ✅
pyarrowandpandasare already declared).CHANGELOG.md,docs/USER_GUIDE.mdandCONTRIBUTING.md.Reviewer Emoji Legend
:code::smiley::+1::100:...and I want the author to know it! This is a way to highlight positive parts of a code review.
:star: :star: :star:And I am providing reasons why it needs to be addressed as well as suggested improvements.
:star: :star:And I am providing suggestions where it could be improved either in this PR or later.
:star:...and consider this a suggestion, not a requirement.
:question:This should be a fully formed question with sufficient information and context that requires a response.
:memo::pick:This does not require any changes and is often better left unsaid. This may include stylistic, formatting, or organization suggestions and should likely be prevented/enforced by linting if they really matter
:recycle:Should include enough context to be actionable and not be considered a nitpick.
🤖 Generated with Claude Code