Read the warehouse catalog and schema from settings - #4005
Open
blarghmatey wants to merge 2 commits into
Open
blarghmatey wants to merge 2 commits into
blarghmatey wants to merge 2 commits into
Conversation
Every BaseWarehouseETLTask pinned its view to ol_data_lake_production.ol_warehouse_production_integrations, so RC had no way to run a warehouse-pull task against QA data. Tasks now declare a bare table_name and view_name composes it with WAREHOUSE_CATALOG and WAREHOUSE_SCHEMA, which default to the production pair. Each part is checked on its own because a dotted setting would still pass iter_rows's check once joined, but address a different catalog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fa1mSBw5fudATLND9pAqTM
A subclass assigning view_name as a class attribute shadows the property and skips WAREHOUSE_CATALOG/WAREHOUSE_SCHEMA without any error, which is the pattern the open ownership-guard PR and the closed Cohort 1 task PR both use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fa1mSBw5fudATLND9pAqTM
OpenAPI ChangesNo changes detected Unexpected changes? Ensure your branch is up-to-date with |
shanbady
self-requested a review
September 29, 2026 13:29
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What are the relevant tickets?
Follow-up to mitodl/ol-infrastructure#5719, which left mit-learn's QA StarRocks settings unset because of this.
Description (What does it do?)
BaseWarehouseETLTasksubclasses declare their view as a literalol_data_lake_production.ol_warehouse_production_integrations.<view>, with no setting to override it. RC can't run a warehouse-pull task against QA data as a result, so none of the Cohort 1 catalog sources (mitxonline, xpro, mit_edx, ocw, oll) can be rehearsed before cutover.This adds
WAREHOUSE_CATALOGandWAREHOUSE_SCHEMAsettings, defaulting to the production pair. Tasks now declare a baretable_name, andview_namebecomes a property that joins the three.SyncProgramCertificatesTaskis the only subclass and resolves to the same view as before, so production is unchanged.Each part is checked against
[A-Za-z0-9_]+separately, because a dotted value (e.g.WAREHOUSE_CATALOG=ol_data_lake_qa.x) would still passiter_rows's check once joined but point somewhere else.iter_rowskeeps its own check on the composed name.A subclass that still assigns
view_namewould shadow the property and skip the settings silently, so__init_subclass__raisesTypeErrorfor that. #3565 adds one such test task inwarehouse_test.py, which will needtable_name = "integrations__learn__test"after rebasing onto this.The production default is deliberate (it keeps production's config untouched), and it means a non-production deploy that sets
STARROCKS_HOSTwithout the two new settings reads production. QA's ol-infrastructure change should set all four together. It also has to wait until QA actually has the views:ol_warehouse_qa_integrationshas no tables today (Glue, 2026-09-28), vs. 11integrations__learn__*tables in production.Screenshots (if appropriate):
N/A, backend only.
How can this be tested?
learning_resources/lib/andprofiles/tasks_test.pyagainst scratch Postgres/Redis containers: 50 passed, 3 skipped. The skips are the live-StarRocks tests inwarehouse_integration_test.py, which needSTARROCKS_HOSTand run in CI's StarRocks service container.test_base_warehouse_etl_task_runs_against_real_starrocksnow goes through the composed name (default_catalog.<scratch db>.<table>), which I haven't run locally.table_name) before any connection opens, and theview_namesubclass guard.🤖 Generated with Claude Code
https://claude.ai/code/session_01Fa1mSBw5fudATLND9pAqTM