refactor: broker-first instrument model with explicit feed mapping - #18
Merged
Merged
Conversation
Instrument seeding, data feeds and broker routing were entangled: instruments
were created from Databento dataset "universes", and a single overloaded
`oms.instrument_xref` table mixed market-data mapping with broker routing via a
`source_type` discriminator. Feed-to-instrument wiring was implicit, relying on
an instrument-class filter plus the coincidence that a provider's symbol
sometimes equalled ours.
Restructured around one canonical instrument catalog keyed `(symbol, venue)`,
with two independent mappings hanging off it.
Instruments are now seeded broker-first. Connecting a broker adapter discovers
its tradable catalog, and that is the instrument set — `sync-broker` creates the
master rows and the `broker_instrument` routing mapping in one pass. Databento
demotes to a pure data feed. Dropped `instrument_universe`,
`instrument_universe_symbol` and `oms.instrument_xref`.
Data feeds are independent of brokers and map 1:n: `map-feed` populates
`feed_instrument`, so one feed symbol can price the same pair on several venues
(the Bybit feed marks the Binance-seeded BTCUSDT).
Each feed owns its own symbology. `FeedSymbology` splits "which instruments can
this feed price" (a declarative filter pushed into SQL) from "what does it call
them" (a pure function), and both live on the feed struct beside the code that
speaks that vendor's protocol. This replaces a central match holding every
feed's rules as SQL literals, and makes the fiddly cases — Databento's
space-padded OSI in particular — unit-testable without a database.
Venue resolution no longer fails silently. `alpaca_exchange_to_mic` returns
`Option` instead of passing unknown labels through, the `unwrap_or("")` that
manufactured empty-string venues is gone, and the catalog summary reports the
distinct venue/currency codes that failed the FK check rather than only counting
them.
Also removes the now-unreachable `UniverseSource` tree (its only consumer was an
example), seeds crypto exchange venues that are absent from the ISO MIC registry
but required by the venue FK, and splits feeds from broker connections in the
cockpit with a new Architecture page documenting the model.
Verified against the dev database: Alpaca sync 15,876 instruments, Binance 1,249,
feed mapping 31,488 rows unchanged across the symbology port (identical
fingerprint, zero new rows). 92 tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Instrument seeding, data feeds and broker routing were entangled: instruments
were created from Databento dataset "universes", and a single overloaded
oms.instrument_xreftable mixed market-data mapping with broker routing via asource_typediscriminator. Feed-to-instrument wiring was implicit, relying onan instrument-class filter plus the coincidence that a provider's symbol
sometimes equalled ours.
Restructured around one canonical instrument catalog keyed
(symbol, venue),with two independent mappings hanging off it.
Instruments are now seeded broker-first. Connecting a broker adapter discovers
its tradable catalog, and that is the instrument set —
sync-brokercreates themaster rows and the
broker_instrumentrouting mapping in one pass. Databentodemotes to a pure data feed. Dropped
instrument_universe,instrument_universe_symbolandoms.instrument_xref.Data feeds are independent of brokers and map 1:n:
map-feedpopulatesfeed_instrument, so one feed symbol can price the same pair on several venues(the Bybit feed marks the Binance-seeded BTCUSDT).
Each feed owns its own symbology.
FeedSymbologysplits "which instruments canthis feed price" (a declarative filter pushed into SQL) from "what does it call
them" (a pure function), and both live on the feed struct beside the code that
speaks that vendor's protocol. This replaces a central match holding every
feed's rules as SQL literals, and makes the fiddly cases — Databento's
space-padded OSI in particular — unit-testable without a database.
Venue resolution no longer fails silently.
alpaca_exchange_to_micreturnsOptioninstead of passing unknown labels through, theunwrap_or("")thatmanufactured empty-string venues is gone, and the catalog summary reports the
distinct venue/currency codes that failed the FK check rather than only counting
them.
Also removes the now-unreachable
UniverseSourcetree (its only consumer was anexample), seeds crypto exchange venues that are absent from the ISO MIC registry
but required by the venue FK, and splits feeds from broker connections in the
cockpit with a new Architecture page documenting the model.
Verified against the dev database: Alpaca sync 15,876 instruments, Binance 1,249,
feed mapping 31,488 rows unchanged across the symbology port (identical
fingerprint, zero new rows). 92 tests pass.