test(e2e): run the Rust golden crate end to end, and assert http.route (#118)

.github/workflows/ci.yml

@@ -171,8 +171,9 @@ jobs:
     # the Java agent download + a .NET `dotnet restore`/build); a healthy run is ~4-8 min. 25 caps
     # a genuine hang — run.sh's internal waits bound a healthy run well under this, and without a
     # job cap a stuck container would run to the 6-hour default (as a run on PR #76 was heading
-    # toward). Bumped 20→25 when the .NET app was added (a fifth inline restore+build).
-    timeout-minutes: 25
+    # toward). Bumped 20→25 when the .NET app was added (a fifth inline restore+build), and 25→35
+    # when the Rust app was added: a cold `cargo run` compiles a few hundred crates.
+    timeout-minutes: 35
     steps:
       - uses: actions/checkout@v4
 

README.md

@@ -28,14 +28,14 @@ Terraform for four backends (Grafana, Datadog, New Relic, Dash0).
 
 Generated output is validated for **syntax and schema** — every backend module is
 `terraform validate`d in CI, and the OTel SDK/semconv pins are checked against current
-releases weekly (see the drift check). The Node.js, Python, Java, .NET, Go, and Ruby
+releases weekly (see the drift check). The Node.js, Python, Java, .NET, Go, Ruby, and Rust
 instrumentation is additionally **proven end-to-end in CI**: the generated SDK bootstrap (for
 Java, the OpenTelemetry Java agent; for .NET, the generated `OpenTelemetry.cs` wired into a real
 ASP.NET Core app; for Go, the generated `tracing.go` wired into a real `net/http` server; for
 Ruby, the generated `tracing.rb` wired into a real Sinatra app) exports through the generated
 Collector config into a running trace store, asserted on every push (see
-[`tests/e2e/`](tests/e2e/)). Rust is newer: its golden crate is compiled in CI, and the output
-was run once against a local Collector by hand, but there is no CI end-to-end run for it yet.
+[`tests/e2e/`](tests/e2e/)). For Rust, the generated `telemetry.rs` wired into a real axum app is run the same way, and the
+server span is checked for `http.route`.
 The **Terraform is not yet proven against live vendor backends** — before trusting it, run
 `terraform plan` / `apply` against your own account. Treat backend modules as a reviewed
 starting point, not turnkey infrastructure. An opt-in CI job

ROADMAP.md

@@ -87,8 +87,8 @@ just a larger surface to be wrong on. So the near-term work is depth.
 ## Later: toward v1 — [milestone](https://github.com/radra23/otel-as-code-plugin/milestone/3)
 
 - **PHP, and the rest of Rust.** Rust phase 1 shipped in 0.8.15: axum 0.8, with the golden crate
-  compiled in CI, and the weekly drift check watches its crate pins. Still to come for Rust are an
-  end-to-end CI run, actix-web, and semconv-lint support for Rust attribute setters
+  run end-to-end in CI, and the weekly drift check watches its crate pins. Still to come for Rust
+  are actix-web and semconv-lint support for Rust attribute setters
   ([#118](https://github.com/radra23/otel-as-code-plugin/issues/118)). PHP is at the design
   stage ([#117](https://github.com/radra23/otel-as-code-plugin/issues/117)).
 - **Cardinality and cost budgets as generated policy**, not just as guardrail comments — a

tests/e2e/README.md

@@ -89,6 +89,19 @@ every push/PR. #4 stays local-only.
    collector and both apps, runs `docker compose down -v --remove-orphans`, and
    removes `tests/e2e/.work/` — regardless of whether the assertions passed.
 
+## The Rust leg
+
+`rust-app` (products-api, publishes `8084:8080`) runs the **golden crate** from
+`tests/snapshots/instrument/rust/` (Cargo.lock included) with `cargo run --locked` on
+`rust:1.88`, exporting OTLP/HTTP to the collector's `4318`. It is the slowest leg: a cold
+build compiles a few hundred crates, so its readiness wait is 600 s and the job timeout
+allows for it. Its fixture's liveness route is `/healthz`, not `/health`.
+
+Besides the usual resource attributes, it passes `--expect-span 'http.route=/products/{id}'`
+to `assert-traces.sh`. That is deliberate: the tower crate's `axum` feature is off by
+default, and without it the server span still exports with the right resource but carries
+no route. Only a span-level assertion catches that.
+
 ## Expected collector errors (not a failure)
 
 During a `run.sh` run the golden collector config routes **all three** pipelines

tests/e2e/assert-traces.sh

@@ -5,17 +5,18 @@
 # response for offline unit testing. JSON parsed with python3 (repo convention).
 set -euo pipefail
 
-JAEGER=""; SERVICE=""; EXPECT=""; TRACES_JSON=""
+JAEGER=""; SERVICE=""; EXPECT=""; TRACES_JSON=""; SPAN_EXPECT=""
 while [ $# -gt 0 ]; do
   case "$1" in
     --jaeger) JAEGER="$2"; shift 2;;
     --service) SERVICE="$2"; shift 2;;
     --expect) EXPECT="$2"; shift 2;;
     --traces-json) TRACES_JSON="$2"; shift 2;;
+    --expect-span) SPAN_EXPECT="$2"; shift 2;;
     *) echo "unknown arg: $1" >&2; exit 2;;
   esac
 done
-[ -n "$SERVICE" ] && [ -n "$EXPECT" ] || { echo "usage: --service <name> --expect k=v,... (--jaeger <url> | --traces-json <file>)" >&2; exit 2; }
+[ -n "$SERVICE" ] && [ -n "$EXPECT" ] || { echo "usage: --service <name> --expect k=v,... (--jaeger <url> | --traces-json <file>) [--expect-span k=v,...]" >&2; exit 2; }
 
 # NOTE: the python source is captured into a variable via `$(cat <<'PY' ... PY)`
 # rather than run directly as `python3 - <<'PY'` — the latter feeds the heredoc as
@@ -48,12 +49,28 @@ if not matched:
 for k, v in expect.items():
     if tags.get(k) != v:
         print(f"FAIL: {service} expected {k}={v}, got {tags.get(k)!r} (tags present: {sorted(tags)})"); sys.exit(1)
-print(f"OK: {service} has {', '.join(f'{k}={v}' for k,v in expect.items())}")
+# Optional span-level check (--expect-span): some span of this service, in any returned trace,
+# must carry every key=value. Resource attrs alone cannot tell a route-named server span from an
+# anonymous one, which is the failure a missing instrumentation feature produces.
+span_expect = dict(kv.split("=", 1) for kv in os.environ.get("SPAN_EXPECT", "").split(",") if kv)
+if span_expect:
+    hit = False
+    for tr in traces:
+        pids = {pid for pid, proc in (tr.get("processes") or {}).items() if proc.get("serviceName") == service}
+        for sp in tr.get("spans") or []:
+            if sp.get("processID") in pids:
+                st = {t["key"]: str(t.get("value")) for t in sp.get("tags") or []}
+                if all(st.get(k) == v for k, v in span_expect.items()):
+                    hit = True
+    if not hit:
+        print(f"FAIL: {service} has no span with {', '.join(f'{k}={v}' for k,v in span_expect.items())}"); sys.exit(1)
+print(f"OK: {service} has {', '.join(f'{k}={v}' for k,v in expect.items())}"
+      + (f" and a span with {', '.join(f'{k}={v}' for k,v in span_expect.items())}" if span_expect else ""))
 PY
 )
 
 check() {  # stdin: /api/traces JSON. Args: service, expect. Exit 0 ok / 1 mismatch.
-  SERVICE="$1" EXPECT="$2" python3 -c "$PY_CHECK_SRC"
+  SERVICE="$1" EXPECT="$2" SPAN_EXPECT="$SPAN_EXPECT" python3 -c "$PY_CHECK_SRC"
 }
 
 if [ -n "$TRACES_JSON" ]; then

tests/e2e/assert-traces.test.sh

@@ -15,5 +15,24 @@ check "missing service.version fails" \
 check "absent service fails" \
   '! bash assert-traces.sh --traces-json fixtures/jaeger-ok.json --service nope --expect service.name=nope >/dev/null'
 
+# --expect-span: a span-level attribute must be present on some span of the service
+tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
+TMP="$tmp" python3 - <<'PY'
+import json, os
+d = json.load(open("fixtures/jaeger-ok.json"))
+d["data"][0]["spans"][0]["tags"] = [{"key": "http.route", "type": "string", "value": "/checkout"}]
+json.dump(d, open(os.environ["TMP"] + "/ok.json", "w"))
+d["data"][0]["spans"][0]["tags"] = []
+json.dump(d, open(os.environ["TMP"] + "/missing.json", "w"))
+PY
+check "span attr present passes" \
+  'bash assert-traces.sh --traces-json $tmp/ok.json --service checkout-api --expect service.name=checkout-api --expect-span http.route=/checkout >/dev/null'
+check "span attr absent fails (route-less server span)" \
+  '! bash assert-traces.sh --traces-json $tmp/missing.json --service checkout-api --expect service.name=checkout-api --expect-span http.route=/checkout >/dev/null'
+check "span attr with the wrong value fails" \
+  '! bash assert-traces.sh --traces-json $tmp/ok.json --service checkout-api --expect service.name=checkout-api --expect-span http.route=/other >/dev/null'
+check "no --expect-span keeps the old behaviour" \
+  'bash assert-traces.sh --traces-json $tmp/missing.json --service checkout-api --expect service.name=checkout-api >/dev/null'
+
 echo "Results: $pass passed, $fail failed"
 [ "$fail" -eq 0 ]

tests/e2e/docker-compose.yml

@@ -193,3 +193,27 @@ services:
     # collector.
     command: ["sh", "-c", "bundle install && ruby app.rb"]
     depends_on: [collector]
+
+  rust-app:
+    # SDK image because the container compiles the source inline, like the Go and Ruby legs.
+    # 1.88 is the floor the golden crate's locked dependency tree needs (reqwest -> url -> icu);
+    # the rust-golden CI job checks the same crate on 1.88 and stable.
+    image: rust:1.88-bookworm
+    working_dir: /app
+    environment:
+      # The golden telemetry.rs reads service.name from OTEL_SERVICE_NAME. Use a name DISTINCT
+      # from every other service (the golden's own default, search-api, is the Go leg's name).
+      OTEL_SERVICE_NAME: "products-api"
+      DEPLOYMENT_ENV: "e2e"
+      # Rust exports OTLP/HTTP protobuf, so point at the collector's HTTP port (4318), like the
+      # Ruby and Next.js legs and unlike the gRPC 4317 the rest use.
+      OTEL_EXPORTER_OTLP_ENDPOINT: "http://collector:4318"
+    ports:
+      - "8084:8080"   # load is driven from the host (see run.sh)
+    volumes:
+      - ./.work/rust:/app
+    # The golden crate: the pristine axum fixture plus the generated src/telemetry.rs, the marked
+    # Cargo.toml block and the three printed main.rs lines, with its Cargo.lock. --locked makes
+    # the run reproduce the dependency tree the rust-golden job compiles.
+    command: ["sh", "-c", "cargo run --locked"]
+    depends_on: [collector]

tests/e2e/run.sh

@@ -16,7 +16,7 @@ cleanup() {
   # metrics/logs pipelines (Jaeger's OTLP receiver only accepts traces) — see the
   # "Expected collector errors" section in README.md before treating those as the failure.
   echo "--- collector + app logs (tail, timestamped) ---"
-  docker compose logs --timestamps --tail=60 jaeger collector node-app python-app java-app nextjs-app dotnet-app go-app ruby-app 2>/dev/null || true
+  docker compose logs --timestamps --tail=60 jaeger collector node-app python-app java-app nextjs-app dotnet-app go-app ruby-app rust-app 2>/dev/null || true
   echo "--- jaeger /api/services (what actually landed) ---"
   curl -fsS "http://localhost:16686/api/services" 2>/dev/null && echo || echo "(jaeger /api/services unreachable)"
   docker compose down -v --remove-orphans 2>/dev/null || true
@@ -28,7 +28,7 @@ cleanup() {
 trap cleanup EXIT
 
 # 1. seed throwaway app copies (fixtures stay pristine)
-rm -rf "$WORK"; mkdir -p "$WORK/nodejs" "$WORK/python" "$WORK/java" "$WORK/nextjs" "$WORK/dotnet" "$WORK/go" "$WORK/ruby"
+rm -rf "$WORK"; mkdir -p "$WORK/nodejs" "$WORK/python" "$WORK/java" "$WORK/nextjs" "$WORK/dotnet" "$WORK/go" "$WORK/ruby" "$WORK/rust"
 # The collector's file exporter (logs-overlay) writes logs.json here. World-writable so the
 # collector container's non-root user can write regardless of the host UID that created it.
 mkdir -p "$WORK/collector-out"; chmod 777 "$WORK/collector-out"
@@ -90,6 +90,13 @@ cp -R ../../fixtures/ruby-greenfield/. "$WORK/ruby/"
 cp ../snapshots/instrument/ruby/tracing.rb "$WORK/ruby/tracing.rb"
 cp ../snapshots/instrument/ruby/Gemfile "$WORK/ruby/Gemfile"   # instrumented manifest
 
+# Rust: the golden crate IS the instrumented fixture (pristine axum app + generated telemetry.rs +
+# the marked Cargo.toml block + the three printed main.rs lines), so seed the whole golden
+# directory, Cargo.lock included, minus any local build output.
+mkdir -p "$WORK/rust/src"
+cp ../snapshots/instrument/rust/Cargo.toml ../snapshots/instrument/rust/Cargo.lock "$WORK/rust/"
+cp ../snapshots/instrument/rust/src/*.rs "$WORK/rust/src/"
+
 # 2. up + wait for jaeger health
 docker compose up -d
 echo "waiting for stack..."
@@ -101,15 +108,16 @@ done
 
 # 3. drive load from the host against both apps (both ports are published — see
 #    docker-compose.yml). Each wait is bounded so CI can't hang.
-wait_ready() { # base_url [timeout_s]
+wait_ready() { # base_url [timeout_s] [health_path]
   local base="$1"
+  local health_path="${3:-/health}"
   # Default 150s: the python-app runs `pip install` (grpcio + otlp proto exporter) inline
   # before serving, which on a cold CI runner can approach 90s alone — 90s was too
   # tight and risked spurious red on the first live run. The Next.js app does npm install +
   # next build inline, which is heavier, so it passes a larger timeout (2nd arg).
   local APP_READY_TIMEOUT_S="${2:-150}"
   local deadline=$(( $(date +%s) + APP_READY_TIMEOUT_S ))
-  until curl -fsS "$base/health" >/dev/null 2>&1; do
+  until curl -fsS "$base$health_path" >/dev/null 2>&1; do
     [ "$(date +%s)" -lt "$deadline" ] || { echo "FAIL: $base did not become ready within timeout" >&2; exit 1; }
     sleep 2
   done
@@ -173,6 +181,16 @@ for i in 1 2 3 4 5; do
   curl -fsS "http://localhost:8083/notify" >/dev/null || true
 done
 
+# Rust app (cargo run compiles the crate inline: a cold build fetches and compiles ~300 crates, so
+# allow the most time of any leg). Hitting /products/{id} makes opentelemetry-instrumentation-tower
+# emit a server span named after the route, exported over OTLP/HTTP (:4318). The fixture's liveness
+# route is /healthz, not /health.
+wait_ready "http://localhost:8084" 600 /healthz
+for i in 1 2 3 4 5; do
+  curl -fsS "http://localhost:8084/healthz" >/dev/null || true
+  curl -fsS "http://localhost:8084/products/42" >/dev/null || true
+done
+
 # 4. assert (Jaeger reachable on host 16686)
 JA="http://localhost:16686"
 # Check BOTH services in one run (don't fail-fast on the first) so a single CI run reports
@@ -204,6 +222,14 @@ POLL_TIMEOUT=60 bash assert-traces.sh --jaeger "$JA" --service search-api \
 POLL_TIMEOUT=60 bash assert-traces.sh --jaeger "$JA" --service notifications-api \
   --expect service.name=notifications-api,service.version=1.1.0,service.namespace=storefront,deployment.environment.name=e2e || rc=1
 
+# Rust SDK wiring (golden telemetry.rs). service.version and service.namespace are the golden's
+# defaults; service.name comes from OTEL_SERVICE_NAME and deployment.environment.name from
+# DEPLOYMENT_ENV. --expect-span proves the tower layer's `axum` feature is on: without it the
+# server span exists but carries no http.route.
+POLL_TIMEOUT=90 bash assert-traces.sh --jaeger "$JA" --service products-api \
+  --expect service.name=products-api,service.version=0.4.0,service.namespace=storefront,deployment.environment.name=e2e \
+  --expect-span 'http.route=/products/{id}' || rc=1
+
 # Logs (#12, --experimental): the Python app logs `reserved sku=...` on /inventory/reserve, the
 # --experimental bootstrap bridges it to OTLP, and the collector's file exporter writes it to
 # collector-out/logs.json. Assert that record actually arrived with the right service.name.