feat(otel): OpenTelemetry OTEL export support — independent of anonymous telemetry opt-out by MaxMoldmann · Pull Request #1510 · 1jehuang/jcode

MaxMoldmann · 2026-09-25T22:21:51Z

Summary

Adds a new `otel` module to `jcode-telemetry-core` that exports OpenTelemetry spans to a configurable OTLP collector endpoint. This is a completely separate, independent enterprise feature — it does not affect or depend on the existing anonymous usage telemetry. Features: - Session spans (`jcode.session`) emitted on session end with token counts, turn counts, tool call counts, duration, provider, model, and end reason - Turn spans (`jcode.turn`) emitted per user turn, linked to the session span as parent — forming a parent-child trace tree in the collector - Supports `http/json` (default) and `http/protobuf` wire formats; `grpc` warns and falls back to `http/json` - Transport: hand-rolled HTTP/1.1 POST over stdlib `TcpStream` — safe from both sync and async (tokio) calling contexts, zero new dependencies - Standard OTLP env var configuration: - `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` - `OTEL_EXPORTER_OTLP_PROTOCOL` (http/json or http/protobuf) - `OTEL_EXPORTER_OTLP_HEADERS` (e.g. Authorization=Basic ...) - `OTEL_EXPORTER_OTLP_TIMEOUT`, `OTEL_SERVICE_NAME`, `OTEL_SERVICE_VERSION` - `OTEL_RESOURCE_ATTRIBUTES`, `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` - Privacy: spans never contain prompt text, file contents, tool inputs/outputs, or any free-form user data — only numeric metrics, categorical labels, and UUIDs - Complete no-op when no endpoint env var is configured

Adds 64 tests across unit, integration-boundary, and field-verification levels: Unit tests: - Proto encoding: varint, length-delimited, fixed64, nested messages - JSON encoding: export request structure, resource attributes, header parsing - Span field correctness: trace/span ID derivation, timestamps, attribute values - Protocol selection: http/json vs http/protobuf routing, grpc fallback - URL resolution: OTLP_TRACES_ENDPOINT takes priority over base URL + suffix - Privacy: session_span_attributes_never_contain_arbitrary_string_content (fuzzes arbitrary strings through the span builder; panics if content leaks) - Edge cases: zero-token spans, parent session IDs, missing env vars HTTP integration-boundary tests (real TCP listener): - export_session_span_public_api_proto_sends_valid_otlp - export_turn_span_public_api_json_sends_valid_otlp - http_json_post_sends_correct_content_type_and_json_body - http_proto_post_sends_correct_content_type_and_binary_body - grpc_protocol_falls_back_to_http_json_at_tcp_layer - http_400_response_returns_false - post_otlp_works_from_within_tokio_runtime (proves no panic/deadlock) - uuid_span_ids_survive_proto_tcp_round_trip All tests are #[cfg(test)]-only with zero production build impact.

greptile-apps · 2026-09-26T17:00:06Z

Confidence Score: 3/5

[Medium risk] Adds OpenTelemetry span export alongside existing telemetry.

Do not merge until opted-out analytics delivery and loss of spans on immediate exit are fixed.

Findings

Opt-out sends anonymous telemetry ▶
Exit can discard spans ▶
Copilot runtime cannot compile ▶
Stalled exports accumulate threads ▶
Final turn outlives session ▶
Hostname exports exceed timeout ▶

Fix with agent prompt

### Issue 1
crates/jcode-telemetry-core/src/lifecycle.rs:23
When anonymous telemetry is opted out but an OTLP collector is configured, ending a session with a pending turn calls `finalize_current_turn`. It queues an anonymous `turn_end` payload for delivery to Jcode’s analytics backend before exporting the OTLP span. The user’s opt-out does not prevent that payload from entering the delivery path; separate OTLP finalization from anonymous event emission before merging.

### Issue 2
crates/jcode-telemetry-core/src/otel.rs:908-912
When the process exits immediately after ending a session, this detached export thread is not joined or flushed. The collector can receive neither the session span nor a pending final turn span. Provide a bounded shutdown flush so ending a session does not silently lose its trace.

### Issue 3
crates/jcode-provider-copilot-runtime/Cargo.toml:undefined-22
This removes `jcode-provider-openai`, but non-test Copilot runtime code still calls that crate to build requests and process responses. Those references cannot resolve, so the Copilot runtime cannot compile. Restore the dependency or replace the calls before merging.

### Issue 4
crates/jcode-telemetry-core/src/otel.rs:931-935
Each turn starts a separate export thread without a concurrency limit. While a collector withheld responses, 24 turn exports left 24 requests in flight and the process at 25 threads. Continued turns during a collector stall can accumulate threads and consume resources; bounding export concurrency would avoid this non-blocking operational cost.

### Issue 5
crates/jcode-telemetry-core/src/lifecycle.rs:23
For an opted-out session with a pending turn, session export captures the parent span’s end time before this call finalizes the turn. The exported child then ends after its parent, producing inconsistent trace durations. Finalize the turn before timestamping the session; this measurement correction is non-blocking.

### Issue 6
crates/jcode-telemetry-core/src/otel.rs:779-789
If hostname resolution is slow, it has no deadline tied to `OTEL_EXPORTER_OTLP_TIMEOUT`; each resolved address also receives a fresh full connection timeout. With a delayed lookup, an export completed after 250 ms despite a 50 ms setting. Slow or multi-address hostnames can therefore keep export threads occupied beyond the requested timeout, a non-blocking operational concern.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

This PR adds OTLP session and turn spans, HTTPS delivery, lifecycle handling, tests, and documentation. For a user who opts out of anonymous analytics but configures an OTLP collector, ending a session with a pending turn still queues an analytics event. Immediate process exit can also discard session spans. Both issues must be fixed before merging. Stalled-export threads, inconsistent span times, and hostname timeout overruns are non-blocking concerns.

_{Reviews (2) · Last reviewed commit: "fix(otel): resolve export bugs, add docs..."}

greptile-apps · 2026-09-26T17:00:09Z

 jcode-base = { path = "../jcode-base", default-features = false }
 jcode-message-types = { path = "../jcode-message-types" }
 jcode-provider-copilot = { path = "../jcode-provider-copilot" }
 jcode-provider-core = { path = "../jcode-provider-core" }


Copilot runtime cannot compile

This removes jcode-provider-openai, but non-test Copilot runtime code still calls that crate to build requests and process responses. Those references cannot resolve, so the Copilot runtime cannot compile. Restore the dependency or replace the calls before merging.

Artifacts

Authored Copilot dependency compilation check

The executed script attempts a locked package check and compiles the actual source expression with and without the direct crate binding, preserving the tracked lockfile.

Baseline and candidate dependency with non-test source references

A command captured the baseline manifest, candidate manifest and referenced source lines, showing the dependency was removed while its uses remain.

Locked Copilot package check blocked before compilation

The targeted offline Cargo check exited 101 because Cargo.lock needs updating under --locked, so it could not reach package compilation.

Source expression compiles with OpenAI crate supplied

Rustc compiled the source expression with an OpenAI crate binding and exited 0, establishing the comparison baseline.

Source expression fails without OpenAI crate supplied

Rustc compiled the same expression without the crate binding and exited 1 with E0433, confirming the missing-crate defect.

_{Ran code and verified through T-Rex}

Prompt To Fix With AI

This is a comment left during a code review. Path: crates/jcode-provider-copilot-runtime/Cargo.toml Line: 22 Comment: **Copilot runtime cannot compile** This removes `jcode-provider-openai`, but non-test Copilot runtime code still calls that crate to build requests and process responses. Those references cannot resolve, so the Copilot runtime cannot compile. Restore the dependency or replace the calls before merging. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-09-26T17:00:19Z

Comments Outside Diff

These findings sit on lines the diff does not cover, so they could not be posted inline. Each one leaves this list once its file changes.

Session-end span is lost on immediate process exit ▶
- Bug
  - After end_session returns, an immediately exiting child delivered no session spans in 30 trials; the same child delivered all 8 spans when kept alive for 250 ms.
- Cause
  - crates/jcode-telemetry-core/src/otel.rs:908-912 spawns the session export on a detached thread and does not wait for delivery before shutdown.
- Fix
  - Keep the session export nonblocking during normal operation, but provide a bounded flush or join at process shutdown so pending session-end exports can complete.
Turn exports create an unbounded number of detached threads ▶
- Bug
  - Each configured turn export spawns a new OS thread. A loopback collector that withheld responses left 24 exports in flight and the caller process at 25 threads, with no responses completed. Continued turns during a collector stall can therefore accumulate threads and associated resources.
- Cause
  - export_turn_span calls std::thread::spawn for every payload without a queue, worker limit, or backpressure. The HTTPS path in post_otlp also spawns a detached thread; that additional path was not runtime-tested.
- Fix
  - Use a bounded export queue and fixed-size worker pool, with an explicit policy for full queues and shutdown.
Opted-out pending turn outlives its parent session span ▶
- Bug
  - With an OTEL endpoint configured, ending an opted-out session exports a pending turn whose end timestamp is later than the parent session's end timestamp. The focused collector run measured a 474,087 ns overrun.
- Cause
  - end_session_with_reason captures and exports the session end before emit_lifecycle_event reaches lifecycle.rs:23 to finalize and timestamp the pending turn.
- Fix
  - Finalize the pending turn before capturing the session span's end timestamp, then clear session state after both exports are prepared.
Hostname HTTP exports have no overall timeout ▶
- Bug
  - A slow hostname lookup can keep an HTTP OTLP export in progress beyond cfg.timeout; after resolution, the export can still succeed. If several addresses are returned, each connection attempt can consume another full timeout. This matters when DNS is slow or a hostname resolves to multiple unreachable addresses; it is not a demonstrated delay for every environment or endpoint.
- Cause
  - to_socket_addrs() at line 779 has no deadline tied to cfg.timeout, and line 789 passes the unchanged timeout to every connect_timeout attempt.
- Fix
  - Apply one end-to-end deadline to DNS resolution and connection attempts, passing only the remaining time to each attempt.

…review comments

greptile-apps · 2026-09-26T20:23:51Z

+                    let now = Instant::now();
+                    if let Some(ref mut state) = *guard {
+                        if let Some(ref id) = id_opt {
+                            finalize_current_turn(id, state, now, reason.as_str(), DeliveryMode::Background);


Opt-out sends anonymous telemetry

When anonymous telemetry is opted out but an OTLP collector is configured, ending a session with a pending turn calls finalize_current_turn. It queues an anonymous turn_end payload for delivery to Jcode’s analytics backend before exporting the OTLP span. The user’s opt-out does not prevent that payload from entering the delivery path; separate OTLP finalization from anonymous event emission before merging.

Artifacts

Opted-out pending-turn test source

The authored test ends an opted-out session with a pending turn and captures both OTLP spans and anonymous lifecycle payloads.

Opted-out turn comparison command

The authored command runs the same focused test with the prior early return and with the current lifecycle branch.

Prior early-return output

The comparison received a session span and captured no lifecycle analytics event.

Current opt-out output

The current branch received session and turn spans and captured a turn_end analytics payload with Background delivery.

Loopback telemetry export test source

The authored Rust integration test starts an opted-out pending-turn session, captures real OTLP requests, and asserts that a child does not end after its parent.

Telemetry export test command

The authored command runner executes the focused Cargo test and records its command, working directory, exit code, and output.

Collector output with turn finalization omitted

The control run omitted only the line-23 finalization call and received one parent session span with no child turn span.

Collector output with current line-23 finalization

The current-code run received linked parent and child spans and failed because the child ended 474,087 ns after the parent.

_{Ran code and verified through T-Rex}

Prompt To Fix With AI

This is a comment left during a code review. Path: crates/jcode-telemetry-core/src/lifecycle.rs Line: 23 Comment: **Opt-out sends anonymous telemetry** When anonymous telemetry is opted out but an OTLP collector is configured, ending a session with a pending turn calls `finalize_current_turn`. It queues an anonymous `turn_end` payload for delivery to Jcode’s analytics backend before exporting the OTLP span. The user’s opt-out does not prevent that payload from entering the delivery path; separate OTLP finalization from anonymous event emission before merging. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-09-26T20:23:51Z

+        #[cfg(not(test))]
+        {
+            let cfg = cfg.clone();
+            std::thread::spawn(move || { let _ = post_otlp(&cfg, payload); });
+        }


Exit can discard spans

When the process exits immediately after ending a session, this detached export thread is not joined or flushed. The collector can receive neither the session span nor a pending final turn span. Provide a bounded shutdown flush so ending a session does not silently lose its trace.

Artifacts

Session-ending Rust child

This source calls the production session APIs and optionally remains alive for 250 ms, allowing the two exit conditions to be compared.

Loopback collector and child-process command

This script compiles the Rust child, runs each trial against a local OTLP collector, and records whether a session span arrived.

Collector output with a 250 ms exit grace period

The executed control command exited successfully and received a `jcode.session` POST in all 8 trials, showing the export can reach the collector.

Collector output after immediate process exit

The executed immediate-exit command reported successful child exits but received no request in 30 trials, confirming the session-end export was lost.

_{Ran code and verified through T-Rex}

Prompt To Fix With AI

This is a comment left during a code review. Path: crates/jcode-telemetry-core/src/otel.rs Line: 908-912 Comment: **Exit can discard spans** When the process exits immediately after ending a session, this detached export thread is not joined or flushed. The collector can receive neither the session span nor a pending final turn span. Provide a bounded shutdown flush so ending a session does not silently lose its trace. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-09-26T20:23:51Z

+        #[cfg(not(test))]
+        {
+            let cfg = cfg.clone();
+            std::thread::spawn(move || { let _ = post_otlp(&cfg, payload); });
+        }


Stalled exports accumulate threads

Each turn starts a separate export thread without a concurrency limit. While a collector withheld responses, 24 turn exports left 24 requests in flight and the process at 25 threads. Continued turns during a collector stall can accumulate threads and consume resources; bounding export concurrency would avoid this non-blocking operational cost.

Artifacts

Rust caller for 24 turn exports

The authored caller invoked the real non-test turn-export API 24 times and stayed alive for inspection, enabling the thread comparison.

Stalled loopback collector and Cargo artifact harness

The executed harness selected each revision’s Cargo-produced library, withheld HTTP responses, and sampled the caller’s threads, ensuring the comparison used the intended implementations.

Pre-change stalled-collector run

The pre-change run observed two process threads and one parsed request while responses were withheld, showing the synchronous baseline.

Current-code stalled-collector run

The HEAD run observed 25 process threads and 24 parsed requests before any response, confirming concurrent exporter-thread accumulation in this trial.

_{Ran code and verified through T-Rex}

Prompt To Fix With AI

This is a comment left during a code review. Path: crates/jcode-telemetry-core/src/otel.rs Line: 931-935 Comment: **Stalled exports accumulate threads** Each turn starts a separate export thread without a concurrency limit. While a collector withheld responses, 24 turn exports left 24 requests in flight and the process at 25 threads. Continued turns during a collector stall can accumulate threads and consume resources; bounding export concurrency would avoid this non-blocking operational cost. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-09-26T20:23:51Z

+                    let now = Instant::now();
+                    if let Some(ref mut state) = *guard {
+                        if let Some(ref id) = id_opt {
+                            finalize_current_turn(id, state, now, reason.as_str(), DeliveryMode::Background);


Final turn outlives session

For an opted-out session with a pending turn, session export captures the parent span’s end time before this call finalizes the turn. The exported child then ends after its parent, producing inconsistent trace durations. Finalize the turn before timestamping the session; this measurement correction is non-blocking.

Artifacts

Opted-out pending-turn test source

The authored test ends an opted-out session with a pending turn and captures both OTLP spans and anonymous lifecycle payloads.

Opted-out turn comparison command

The authored command runs the same focused test with the prior early return and with the current lifecycle branch.

Prior early-return output

The comparison received a session span and captured no lifecycle analytics event.

Current opt-out output

The current branch received session and turn spans and captured a turn_end analytics payload with Background delivery.

Loopback telemetry export test source

The authored Rust integration test starts an opted-out pending-turn session, captures real OTLP requests, and asserts that a child does not end after its parent.

Telemetry export test command

The authored command runner executes the focused Cargo test and records its command, working directory, exit code, and output.

Collector output with turn finalization omitted

The control run omitted only the line-23 finalization call and received one parent session span with no child turn span.

Collector output with current line-23 finalization

The current-code run received linked parent and child spans and failed because the child ended 474,087 ns after the parent.

_{Ran code and verified through T-Rex}

Prompt To Fix With AI

This is a comment left during a code review. Path: crates/jcode-telemetry-core/src/lifecycle.rs Line: 23 Comment: **Final turn outlives session** For an opted-out session with a pending turn, session export captures the parent span’s end time before this call finalizes the turn. The exported child then ends after its parent, producing inconsistent trace durations. Finalize the turn before timestamping the session; this measurement correction is non-blocking. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-09-26T20:23:52Z

+        let addrs: Vec<_> = match addr_str.to_socket_addrs() {
+            Ok(iter) => iter.collect(),
+            Err(e) => {
+                logging::warn(&format!("otel: DNS resolve of {addr_str} failed: {e}"));
+                return false;
+            }
+        };
+        let timeout = cfg.timeout;
+        let mut stream_opt = None;
+        for addr in &addrs {
+            match TcpStream::connect_timeout(addr, timeout) {


Hostname exports exceed timeout

If hostname resolution is slow, it has no deadline tied to OTEL_EXPORTER_OTLP_TIMEOUT; each resolved address also receives a fresh full connection timeout. With a delayed lookup, an export completed after 250 ms despite a 50 ms setting. Slow or multi-address hostnames can therefore keep export threads occupied beyond the requested timeout, a non-blocking operational concern.

Artifacts

Delayed DNS reproduction script

The script extracts and runs each revision’s actual `post_otlp` implementation against a loopback collector with a controlled DNS delay, making the timeout comparison reproducible.

Before-change export with delayed DNS

The parent revision returned false after 275 ms with a 50 ms timeout, and the collector received no request or HTTP response.

After-change export with delayed DNS

The changed revision sent the request and received HTTP/1.1 200 OK after 250 ms despite the 50 ms configured timeout.

_{Ran code and verified through T-Rex}

Prompt To Fix With AI

This is a comment left during a code review. Path: crates/jcode-telemetry-core/src/otel.rs Line: 779-789 Comment: **Hostname exports exceed timeout** If hostname resolution is slow, it has no deadline tied to `OTEL_EXPORTER_OTLP_TIMEOUT`; each resolved address also receives a fresh full connection timeout. With a delayed lookup, an export completed after 250 ms despite a 50 ms setting. Slow or multi-address hostnames can therefore keep export threads occupied beyond the requested timeout, a non-blocking operational concern. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

MaxMoldmann added 2 commits September 26, 2026 03:28

MaxMoldmann force-pushed the feat/otel-support branch from 07edc88 to 8340b48 Compare September 26, 2026 08:29

MaxMoldmann marked this pull request as ready for review September 26, 2026 16:49

greptile-apps Bot reviewed Sep 26, 2026

View reviewed changes

fix(otel): resolve export bugs, add docs/e2e tests, address all 8 PR …

0dcb1c2

…review comments

MaxMoldmann force-pushed the feat/otel-support branch from 8340b48 to 0dcb1c2 Compare September 26, 2026 20:11

greptile-apps Bot reviewed Sep 26, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(otel): OpenTelemetry OTEL export support — independent of anonymous telemetry opt-out - #1510

feat(otel): OpenTelemetry OTEL export support — independent of anonymous telemetry opt-out#1510
MaxMoldmann wants to merge 3 commits into
1jehuang:masterfrom
MaxMoldmann:feat/otel-support

MaxMoldmann commented Sep 25, 2026

Uh oh!

greptile-apps Bot commented Sep 26, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot Sep 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 26, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot Sep 26, 2026

Uh oh!

greptile-apps Bot Sep 26, 2026

Uh oh!

greptile-apps Bot Sep 26, 2026

Uh oh!

greptile-apps Bot Sep 26, 2026

Uh oh!

greptile-apps Bot Sep 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

MaxMoldmann commented Sep 25, 2026

Summary

Uh oh!

greptile-apps Bot commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 3/5

Findings

Summary

Uh oh!

greptile-apps Bot Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comments Outside Diff

Uh oh!

greptile-apps Bot Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

greptile-apps Bot commented Sep 26, 2026 •

edited

Loading

greptile-apps Bot commented Sep 26, 2026 •

edited

Loading