Skip to content

refactor: broker-first instrument model with explicit feed mapping - #18

Merged
maxkuttner merged 1 commit into
mainfrom
refactor/broker-first-instrument-model
Jul 19, 2026
Merged

refactor: broker-first instrument model with explicit feed mapping#18
maxkuttner merged 1 commit into
mainfrom
refactor/broker-first-instrument-model

Conversation

@maxkuttner

Copy link
Copy Markdown
Owner

Instrument seeding, data feeds and broker routing were entangled: instruments
were created from Databento dataset "universes", and a single overloaded
oms.instrument_xref table mixed market-data mapping with broker routing via a
source_type discriminator. Feed-to-instrument wiring was implicit, relying on
an instrument-class filter plus the coincidence that a provider's symbol
sometimes equalled ours.

Restructured around one canonical instrument catalog keyed (symbol, venue),
with two independent mappings hanging off it.

Instruments are now seeded broker-first. Connecting a broker adapter discovers
its tradable catalog, and that is the instrument set — sync-broker creates the
master rows and the broker_instrument routing mapping in one pass. Databento
demotes to a pure data feed. Dropped instrument_universe,
instrument_universe_symbol and oms.instrument_xref.

Data feeds are independent of brokers and map 1:n: map-feed populates
feed_instrument, so one feed symbol can price the same pair on several venues
(the Bybit feed marks the Binance-seeded BTCUSDT).

Each feed owns its own symbology. FeedSymbology splits "which instruments can
this feed price" (a declarative filter pushed into SQL) from "what does it call
them" (a pure function), and both live on the feed struct beside the code that
speaks that vendor's protocol. This replaces a central match holding every
feed's rules as SQL literals, and makes the fiddly cases — Databento's
space-padded OSI in particular — unit-testable without a database.

Venue resolution no longer fails silently. alpaca_exchange_to_mic returns
Option instead of passing unknown labels through, the unwrap_or("") that
manufactured empty-string venues is gone, and the catalog summary reports the
distinct venue/currency codes that failed the FK check rather than only counting
them.

Also removes the now-unreachable UniverseSource tree (its only consumer was an
example), seeds crypto exchange venues that are absent from the ISO MIC registry
but required by the venue FK, and splits feeds from broker connections in the
cockpit with a new Architecture page documenting the model.

Verified against the dev database: Alpaca sync 15,876 instruments, Binance 1,249,
feed mapping 31,488 rows unchanged across the symbology port (identical
fingerprint, zero new rows). 92 tests pass.

Instrument seeding, data feeds and broker routing were entangled: instruments
were created from Databento dataset "universes", and a single overloaded
`oms.instrument_xref` table mixed market-data mapping with broker routing via a
`source_type` discriminator. Feed-to-instrument wiring was implicit, relying on
an instrument-class filter plus the coincidence that a provider's symbol
sometimes equalled ours.

Restructured around one canonical instrument catalog keyed `(symbol, venue)`,
with two independent mappings hanging off it.

Instruments are now seeded broker-first. Connecting a broker adapter discovers
its tradable catalog, and that is the instrument set — `sync-broker` creates the
master rows and the `broker_instrument` routing mapping in one pass. Databento
demotes to a pure data feed. Dropped `instrument_universe`,
`instrument_universe_symbol` and `oms.instrument_xref`.

Data feeds are independent of brokers and map 1:n: `map-feed` populates
`feed_instrument`, so one feed symbol can price the same pair on several venues
(the Bybit feed marks the Binance-seeded BTCUSDT).

Each feed owns its own symbology. `FeedSymbology` splits "which instruments can
this feed price" (a declarative filter pushed into SQL) from "what does it call
them" (a pure function), and both live on the feed struct beside the code that
speaks that vendor's protocol. This replaces a central match holding every
feed's rules as SQL literals, and makes the fiddly cases — Databento's
space-padded OSI in particular — unit-testable without a database.

Venue resolution no longer fails silently. `alpaca_exchange_to_mic` returns
`Option` instead of passing unknown labels through, the `unwrap_or("")` that
manufactured empty-string venues is gone, and the catalog summary reports the
distinct venue/currency codes that failed the FK check rather than only counting
them.

Also removes the now-unreachable `UniverseSource` tree (its only consumer was an
example), seeds crypto exchange venues that are absent from the ISO MIC registry
but required by the venue FK, and splits feeds from broker connections in the
cockpit with a new Architecture page documenting the model.

Verified against the dev database: Alpaca sync 15,876 instruments, Binance 1,249,
feed mapping 31,488 rows unchanged across the symbology port (identical
fingerprint, zero new rows). 92 tests pass.
@maxkuttner
maxkuttner merged commit b648f33 into main Jul 19, 2026
1 check passed
@maxkuttner
maxkuttner deleted the refactor/broker-first-instrument-model branch July 20, 2026 06:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant