refactor: Nautilus-style instrument model — broker-first seeding, exp… - #17
Closed
maxkuttner wants to merge 1 commit into
Closed
refactor: Nautilus-style instrument model — broker-first seeding, exp…#17maxkuttner wants to merge 1 commit into
maxkuttner wants to merge 1 commit into
Conversation
…licit feed mapping
Instrument seeding, data feeds and broker routing were entangled: instruments
were created from Databento dataset "universes", and a single overloaded
`oms.instrument_xref` table mixed market-data mapping with broker routing via a
`source_type` discriminator. Feed-to-instrument wiring was implicit, relying on
an instrument-class filter plus the coincidence that a provider's symbol
sometimes equalled ours.
Restructured along the lines NautilusTrader uses: one canonical instrument
catalog keyed `(symbol, venue)`, with two independent mappings hanging off it.
Instruments are now seeded broker-first. Connecting a broker adapter discovers
its tradable catalog, and that is the instrument set — `sync-broker` creates the
master rows and the `broker_instrument` routing mapping in one pass. Databento
demotes to a pure data feed. Dropped `instrument_universe`,
`instrument_universe_symbol` and `oms.instrument_xref`.
Data feeds are independent of brokers and map 1:n: `map-feed` populates
`feed_instrument`, so one feed symbol can price the same pair on several venues
(the Bybit feed marks the Binance-seeded BTCUSDT).
Each feed owns its own symbology. `FeedSymbology` splits "which instruments can
this feed price" (a declarative filter pushed into SQL) from "what does it call
them" (a pure function), and both live on the feed struct beside the code that
speaks that vendor's protocol. This replaces a central match holding every
feed's rules as SQL literals, and makes the fiddly cases — Databento's
space-padded OSI in particular — unit-testable without a database.
Venue resolution no longer fails silently. `alpaca_exchange_to_mic` returns
`Option` instead of passing unknown labels through, the `unwrap_or("")` that
manufactured empty-string venues is gone, and the catalog summary reports the
distinct venue/currency codes that failed the FK check rather than only counting
them.
Also removes the now-unreachable `UniverseSource` tree (its only consumer was an
example), seeds crypto exchange venues that are absent from the ISO MIC registry
but required by the venue FK, and splits feeds from broker connections in the
cockpit with a new Architecture page documenting the model.
Verified against the dev database: Alpaca sync 15,876 instruments, Binance 1,249,
feed mapping 31,488 rows unchanged across the symbology port (identical
fingerprint, zero new rows). 92 tests pass.
Owner
Author
|
Superseded by #18 — same commit on a renamed branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…licit feed mapping
Instrument seeding, data feeds and broker routing were entangled: instruments were created from Databento dataset "universes", and a single overloaded
oms.instrument_xreftable mixed market-data mapping with broker routing via asource_typediscriminator. Feed-to-instrument wiring was implicit, relying on an instrument-class filter plus the coincidence that a provider's symbol sometimes equalled ours.Restructured along the lines NautilusTrader uses: one canonical instrument catalog keyed
(symbol, venue), with two independent mappings hanging off it.Instruments are now seeded broker-first. Connecting a broker adapter discovers its tradable catalog, and that is the instrument set —
sync-brokercreates the master rows and thebroker_instrumentrouting mapping in one pass. Databento demotes to a pure data feed. Droppedinstrument_universe,instrument_universe_symbolandoms.instrument_xref.Data feeds are independent of brokers and map 1:n:
map-feedpopulatesfeed_instrument, so one feed symbol can price the same pair on several venues (the Bybit feed marks the Binance-seeded BTCUSDT).Each feed owns its own symbology.
FeedSymbologysplits "which instruments can this feed price" (a declarative filter pushed into SQL) from "what does it call them" (a pure function), and both live on the feed struct beside the code that speaks that vendor's protocol. This replaces a central match holding every feed's rules as SQL literals, and makes the fiddly cases — Databento's space-padded OSI in particular — unit-testable without a database.Venue resolution no longer fails silently.
alpaca_exchange_to_micreturnsOptioninstead of passing unknown labels through, theunwrap_or("")that manufactured empty-string venues is gone, and the catalog summary reports the distinct venue/currency codes that failed the FK check rather than only counting them.Also removes the now-unreachable
UniverseSourcetree (its only consumer was an example), seeds crypto exchange venues that are absent from the ISO MIC registry but required by the venue FK, and splits feeds from broker connections in the cockpit with a new Architecture page documenting the model.Verified against the dev database: Alpaca sync 15,876 instruments, Binance 1,249, feed mapping 31,488 rows unchanged across the symbology port (identical fingerprint, zero new rows). 92 tests pass.