Deutsche Bahn on a self-hosted MOTIS, official Timetables API, import-time probe - #617
Merged
Merged
Conversation
…e official Timetables API, probe connectors at import
Deutsche Bahn is the most-installed adapter (82 installs, 76 organisations)
and it had not returned a successful call from the cloud since 30 August:
db-rest scrapes bahn.de, Deutsche Bahn blocks datacenter IPs, and even
/locations answered 500. db-vendo-client's own README now recommends a
self-hosted MOTIS. This does that.
Deutsche Bahn
- deploy/motis: official MOTIS image plus an entrypoint that downloads the
gtfs.de long-distance and regional feeds (CC BY 4.0, ~12 MB), imports them
in about a second, polls the GTFS-RT feed for live delays and cancellations
and re-imports weekly. Trains only on purpose: the full feed with every bus
peaks at 7 GB on import.
- docker-compose.cloud.yml runs it as the internal `motis` service in
db-rest's place; docker-compose.yml offers it as the `motis` profile.
- The adapter now targets the MOTIS v1 API (/geocode, /stoptimes, /plan) with
JMESPath response mappings, which cut a departure board from 12 KB to 2 KB.
baseUrl is {{MOTIS_URL}}: operator-provided (MOTIS_INTERNAL_URL) and hidden
from the install form where the operator runs MOTIS, asked for otherwise.
Never Transitous by default: its policy forbids commercial use.
- New deutsche-bahn-timetables adapter on DB's official Timetables API
(CC BY 4.0, free key, 60 req/min) for station boards and the live change
feed. The REST engine now parses XML responses into objects so response
mapping applies to them.
- scripts/ops/migrate-deutsche-bahn-cloud.mjs generates the SQL that moves
installed connectors to the new schema, following the re-sync rules.
Keyless claim
- scripts/probe-keyless.mjs calls one real tool of every keyless adapter and
fails CI when one is bot-blocked or auth-gated; runs weekly from a runner
(a datacenter address, like the cloud) with a MOTIS built from deploy/motis.
- Adapters that only answer residential IPs are marked selfHostOnly: hidden
from the cloud catalog and excluded from the advertised count, which is now
20, not 26. Adapters declare a `probe` call for this and for import.
Activation
- Import runs the adapter's probe with the credentials just entered and
reports the result on the install form; 63 stuck workspaces had only ever
learned their token was wrong from the agent.
- product_events records what users do on the post-attach page (copied the
endpoint, opened a client, generated a key, left without copying), so the
74% who never send a request can be read step by step.
- The activation reminder sends users who already have an MCP server to its
page with connect-a-client copy, not to the tool tester.
- The cron flips trials past expiresAt to status=expired; 1,021 of 1,094
"active" trials had ended, and every status-based count was wrong.
- Catalog re-sync applies a catalog responseMapping to tools that have none.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
A new adapter, a new engine feature (XML responses), a new table and the first infrastructure service the cloud compose has needed since db-rest: a minor, not a patch.
…OTIS image, CodeQL config CodeQL flagged the product-event metadata copy as remote property injection (the client chose the property names). A fixed key set is the right shape anyway: the events carry a client name and an id, nothing else. The probe script is excluded from CodeQL by config: its job is to send requests to URLs read from our own adapter catalog, which the file-access-to-http query cannot tell from untrusted input.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Deutsche Bahn is the most-installed adapter (82 installs across 76 organisations) and had not returned a successful call from the cloud since 30 August. db-rest scrapes bahn.de; Deutsche Bahn blocks datacenter IPs; even
/locationsanswered 500. The library's own README now recommends a self-hosted MOTIS.What
Deutsche Bahn → MOTIS + open data
deploy/motis: official MOTIS image + entrypoint that downloads the gtfs.de long-distance and regional feeds (CC BY 4.0, ~12 MB), imports in ~1 s, polls GTFS-RT for live delays/cancellations, re-imports weekly. Verified locally: 24 of 30 board rows carry live data, delays visible, Freiburg→Berlin plans with ICE legs.motisservice in db-rest's slot; self-host gets amotisprofile./api/v1/geocode,/stoptimes,/planwith JMESPath mappings (board 12 KB → 2 KB).baseUrlis{{MOTIS_URL}}, operator-provided viaMOTIS_INTERNAL_URLand hidden from the form when set. Transitous is never the default (non-commercial policy).deutsche-bahn-timetablesadapter on DB's official Timetables API (XML). The REST engine now parses XML responses.scripts/ops/migrate-deutsche-bahn-cloud.mjsprints the SQL to move installed connectors (same rules as re-sync).Keyless claim
scripts/probe-keyless.mjs+ weekly workflow: one real call per keyless adapter from a runner IP, builds MOTIS fromdeploy/motisto cover DB. Fails on bot-block/auth.selfHostOnlyfor adapters that only work from residential IPs (vinted, untappd, dpd-germany, idealista, reddit, sorare, trenitalia, opentable): hidden on cloud, out of the count. Count is 189 adapters / 20 keyless; README, banner, server.json, glama.json etc. regenerated.Activation
probewith the entered credentials and reports on the install form.product_eventstable +POST /api/product-events; the post-attach page reports copy/tab/key/leave events.expiresAtasexpired(after the lifecycle email).responseMappingto tools that have none.Verification
deutsche-bahn.live.spec.tsrun against a MOTIS built fromdeploy/motis(DB_LIVE_MOTIS_URL): all five tools, live rows present.node scripts/adapter-count.mjs --checkandvalidate-adapters.mjspass.probe-keyless.mjs --allfrom this machine: 6 ok, vinted bot-blocked, opentable 401, trenitalia 412 (both now selfHostOnly).After merge
Release, then on the cloud:
deploy-cloud(builds the motis image), run the migration SQL, restart the app, verify a real MCP call, run the probe from the droplet.