-
Notifications
You must be signed in to change notification settings - Fork 35
Deutsche Bahn on a self-hosted MOTIS, official Timetables API, import-time probe #617
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| name: AnythingMCP CodeQL config | ||
|
|
||
| paths-ignore: | ||
| # Calls one real tool of every keyless adapter, with the URL taken from the | ||
| # adapter JSON in this repo. CodeQL reads that as "file data flows into an | ||
| # outbound request" (js/file-access-to-http), which is precisely the job of | ||
| # the script: the file is our own catalog, not user input, and the script | ||
| # runs only in CI and from an operator's shell. | ||
| - scripts/probe-keyless.mjs |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| name: Keyless adapter probe | ||
|
|
||
| # Calls one real tool of every adapter that claims to need no API key, from a | ||
| # GitHub runner — a datacenter address, like the cloud. The README and the | ||
| # banner quote that number; this is what keeps it true. An adapter that only | ||
| # works from residential IPs must be marked `selfHostOnly` (hidden from the | ||
| # cloud catalog, excluded from the count) rather than advertised. | ||
| # | ||
| # The Deutsche Bahn adapter needs a MOTIS instance, so the job builds the one | ||
| # in deploy/motis and probes through it — which also proves that image still | ||
| # boots, downloads the feeds and imports them. | ||
|
|
||
| on: | ||
| schedule: | ||
| - cron: "0 6 * * 1" | ||
| workflow_dispatch: | ||
| pull_request: | ||
| paths: | ||
| - "deploy/motis/**" | ||
| - "scripts/probe-keyless.mjs" | ||
| - "packages/backend/src/adapters/de/deutsche-bahn.json" | ||
|
|
||
| jobs: | ||
| probe: | ||
| name: Probe keyless adapters from a datacenter IP | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 20 | ||
| steps: | ||
| - uses: actions/checkout@v6 | ||
|
|
||
| - uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: 22 | ||
|
|
||
| - name: Build and start MOTIS | ||
| run: | | ||
| docker build -t anythingmcp-motis:ci deploy/motis | ||
| docker run -d --name motis -p 8080:8080 anythingmcp-motis:ci | ||
| for i in $(seq 1 60); do | ||
| if curl -fsS "http://localhost:8080/api/v1/geocode?text=Berlin&type=STOP" > /dev/null 2>&1; then | ||
| echo "MOTIS up after ${i} checks"; break | ||
| fi | ||
| if [ "$i" = "60" ]; then | ||
| echo "::error::MOTIS did not come up"; docker logs motis; exit 1 | ||
| fi | ||
| sleep 5 | ||
| done | ||
|
|
||
| - name: Probe | ||
| env: | ||
| PROBE_MOTIS_URL: http://localhost:8080 | ||
| run: node scripts/probe-keyless.mjs --check --all | ||
|
|
||
| - name: MOTIS logs | ||
| if: always() | ||
| run: docker logs motis 2>&1 | grep -v "\[debug\]" | tail -40 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| # MOTIS routing engine for the Deutsche Bahn connector. | ||
| # | ||
| # The connector used to go through db-rest / db-vendo-client, which scrapes | ||
| # bahn.de. Deutsche Bahn blocks datacenter IPs and the library's own README now | ||
| # calls those endpoints "very unreliable" and recommends a self-hosted MOTIS. | ||
| # This image is that: the official MOTIS binary plus an entrypoint that pulls | ||
| # the open GTFS timetable for Germany's trains from gtfs.de (CC BY 4.0), runs | ||
| # the import, keeps a GTFS-RT feed polled for live delays and cancellations, | ||
| # and re-imports the static feed once a week so it never runs off the end of | ||
| # the 30-day window the free feeds cover. | ||
| # | ||
| # Trains only (long-distance + regional/S-Bahn, ~12 MB of GTFS): the import | ||
| # takes about a second and peaks under 400 MB. The full Germany feed with every | ||
| # bus and tram is 280 MB and peaks at 7 GB on import, which would not fit next | ||
| # to the app on an 8 GB host. See README.md for the feed URLs and how to swap | ||
| # them. | ||
| FROM ghcr.io/motis-project/motis:2.11.3 | ||
|
|
||
| USER root | ||
| COPY config.yml /motis-config/config.yml | ||
| COPY entrypoint.sh /entrypoint.sh | ||
| RUN chmod 0755 /entrypoint.sh && mkdir -p /data && chown motis:motis /data | ||
|
|
||
| USER motis | ||
| WORKDIR /data | ||
| EXPOSE 8080 | ||
| # The first boot downloads and imports the feeds before the server listens; | ||
| # the compose healthchecks give that a longer start_period. | ||
| HEALTHCHECK --interval=30s --timeout=5s --start-period=180s --retries=3 \ | ||
| CMD wget --quiet --tries=1 --spider http://localhost:8080/ || exit 1 | ||
| ENTRYPOINT ["/entrypoint.sh"] | ||
| CMD [] | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| # MOTIS for the Deutsche Bahn connector | ||
|
|
||
| The `deutsche-bahn` adapter talks to a [MOTIS](https://github.com/motis-project/motis) | ||
| instance (MIT) instead of scraping bahn.de. This directory builds one that is | ||
| ready to run: the official MOTIS binary plus an entrypoint that downloads the | ||
| open German train timetable, imports it, polls a GTFS-RT feed for live delays | ||
| and cancellations, and re-imports the static feed once a week. | ||
|
|
||
| ## Why not db-rest / bahn.de | ||
|
|
||
| Until v0.8.1 the connector went through `db-rest` (db-vendo-client), which | ||
| calls Deutsche Bahn's undocumented web and app endpoints. Deutsche Bahn blocks | ||
| datacenter IP ranges, so from a hosted server every call failed (HTTP 500 / | ||
| `OPS_BLOCKED`), and the library's own README now describes those endpoints | ||
| as very unreliable and recommends a self-hosted MOTIS. Open data does not get | ||
| blocked. | ||
|
|
||
| ## What it serves | ||
|
|
||
| | piece | source | licence | | ||
| |---|---|---| | ||
| | Long-distance trains (ICE, IC, EC, ECE, EN, railjet, night trains) | [gtfs.de `de_fv`](https://gtfs.de/en/feeds/de_fv/) | CC BY 4.0 | | ||
| | Regional trains and S-Bahn | [gtfs.de `de_rv`](https://gtfs.de/en/feeds/de_rv/) | CC BY 4.0 | | ||
| | Live delays, platform changes, cancellations, service alerts | [gtfs.de GTFS-RT](https://gtfs.de/en/realtime/) `realtime-free.pb` | CC BY-SA 4.0 | | ||
|
|
||
| The free static feeds cover the next 30 days and are re-downloaded every | ||
| `MOTIS_REFRESH_DAYS` (default 7). Buses, trams and ferries are deliberately | ||
| left out: adding them (`de_full`, 280 MB) takes the import from one second and | ||
| under 400 MB of RAM to about 7 GB. | ||
|
|
||
| Attribution: gtfs.de asks that data users name **DELFI e.V.** as the source; | ||
| the connector's instructions and the adapter's `docsUrl` do. | ||
|
|
||
| ## Running it | ||
|
|
||
| Cloud (`docker-compose.cloud.yml`) runs it as the `motis` service and points | ||
| the app at it with `MOTIS_INTERNAL_URL=http://motis:8080`. Self-host: | ||
|
|
||
| ```bash | ||
| # in .env | ||
| COMPOSE_PROFILES=motis | ||
| MOTIS_INTERNAL_URL=http://motis:8080 | ||
| SSRF_ALLOWED_HOSTS=motis | ||
| docker compose up -d | ||
| ``` | ||
|
|
||
| When `MOTIS_INTERNAL_URL` is set the adapter's `MOTIS_URL` is filled in | ||
| automatically at import and hidden from the install form. Without it, the | ||
| install form asks for the URL of a MOTIS instance you run elsewhere. | ||
|
|
||
| `http://localhost:8080/` serves the MOTIS UI once the import is done; the | ||
| connector uses `/api/v1/geocode`, `/api/v1/stoptimes` and `/api/v1/plan`. | ||
|
|
||
| ## Environment | ||
|
|
||
| | variable | default | meaning | | ||
| |---|---|---| | ||
| | `MOTIS_REFRESH_DAYS` | `7` | re-download and re-import the static feeds after this many days | | ||
| | `MOTIS_REFRESH_CHECK_SECONDS` | `3600` | how often the running container checks whether a refresh is due | | ||
| | `MOTIS_GTFS_FV_URL` | gtfs.de `fv_free/latest.zip` | long-distance feed | | ||
| | `MOTIS_GTFS_RV_URL` | gtfs.de `rv_free/latest.zip` | regional feed | | ||
| | `MOTIS_GTFS_RT_URL` | gtfs.de `realtime-free.pb` | GTFS-RT feed polled every 120 s | | ||
| | `MOTIS_DATA_DIR` | `/data` | volume with feeds, imported data and the refresh stamp | | ||
|
|
||
| gtfs.de also sells complete feeds with extended route types (so an ICE reports | ||
| `HIGHSPEED_RAIL` rather than `REGIONAL_RAIL`) and no 30-day limit; point the | ||
| URL variables at those and nothing else changes. | ||
|
|
||
| ## Not Transitous | ||
|
|
||
| [Transitous](https://transitous.org) runs the same stack as a public service | ||
| and is a good way to try the API, but its policy forbids commercial use and | ||
| asks for a User-Agent naming the app and a contact. The adapter therefore | ||
| never defaults to it; if you qualify, entering `https://api.transitous.org` as | ||
| `MOTIS_URL` works. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| # MOTIS import/server configuration, templated by entrypoint.sh: | ||
| # __FEEDS__ directory holding the downloaded GTFS zips | ||
| # __RT_URL__ GTFS-RT feed (TripUpdates + ServiceAlerts) | ||
| # | ||
| # Two datasets rather than gtfs.de's single "full" feed: fv (long-distance: | ||
| # ICE, IC, EC, ECE, EN, railjet, night trains) and rv (regional rail and | ||
| # S-Bahn). Together they are every train in Germany and about 12 MB; the full | ||
| # feed adds every bus and tram and needs 7 GB of RAM to import. | ||
| # | ||
| # No OpenStreetMap file on purpose. MOTIS only needs OSM for street routing, | ||
| # map tiles and address geocoding; station-name geocoding works from the | ||
| # timetable alone (adr_extend), and that is all the connector asks for. | ||
| timetable: | ||
| first_day: TODAY | ||
| # The free feeds cover the next 30 days; anything beyond that is simply | ||
| # absent, so a large window costs nothing and never truncates a feed. | ||
| num_days: 365 | ||
| railviz: false | ||
| with_shapes: false | ||
| # The GTFS-RT feed is ~55 MB (it carries every bus in the country); the | ||
| # default 30 s download timeout truncated it and MOTIS logged parser errors. | ||
| http_timeout: 180 | ||
| # Seconds between GTFS-RT polls. The upstream file updates every 10 s, but | ||
| # each poll is a 55 MB download, so this is a bandwidth courtesy to gtfs.de | ||
| # more than a freshness choice. | ||
| update_interval: 120 | ||
| datasets: | ||
| fv: | ||
| path: __FEEDS__/fv.zip | ||
| rt: | ||
| - url: __RT_URL__ | ||
| protocol: gtfsrt | ||
| rv: | ||
| path: __FEEDS__/rv.zip | ||
| rt: | ||
| - url: __RT_URL__ | ||
| protocol: gtfsrt | ||
| geocoding: true | ||
| reverse_geocoding: false | ||
| street_routing: false | ||
| osr_footpath: false | ||
| elevators: false |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.