Repository navigation
fix: drop FastAPI OTLP export, audit SCIM writes - #789
Merged
Merged
Conversation
FastAPI 0.142 added its own OpenTelemetry setup. At startup it reads OTEL_EXPORTER_OTLP_ENDPOINT, assumes http/protobuf whatever the port, and adds a span, metric and log exporter beside the gRPC ones scalo already built. The charts point every app at the collector's gRPC port 4317, so each of those HTTP/1.1 posts got an HTTP/2 SETTINGS frame back and failed: "Failed to export metrics batch due to timeout" every minute on the engine and the KEDA shim. scalo's own exporters were never the problem.
The engine API and the KEDA shim now pass telemetry={"auto_configure": False}. FastAPI's request telemetry still records, into scalo's providers, and leaves on scalo's gRPC exporters only. The fastapi floor moves to the locked 0.142.2, because 0.141 rejects the argument.
The SCIM routes create, update and delete accounts and groups for an IdP and wrote no audit event for any of it. Every mutating SCIM route now emits the same auth.account.* or auth.group.* event the admin routes do, with details.via set to "scim" so an IdP push reads apart from an admin's. A PATCH that changes nothing emits nothing.
|
Released in v1.22.18 -- https://github.com/hyperi-io/dfe-engine/releases/tag/v1.22.18 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two fixes, both found on the 2.2.1-rc.2 deploy.
Engine OTLP export failing every minute
FastAPI 0.142 added its own OpenTelemetry setup. At startup it reads OTEL_EXPORTER_OTLP_ENDPOINT, assumes http/protobuf whatever the port, and adds a span, metric and log exporter beside the gRPC ones scalo already built. The charts point every app at the collector's gRPC port 4317, so those HTTP/1.1 posts get an HTTP/2 SETTINGS frame back (
BadStatusLine('\x00\x00\x06\x04...'), thenConnectionResetError(104)) and fail with "Failed to export metrics batch due to timeout". The engine and the KEDA shim both do it. The hunt runner has no FastAPI app and logs none of it. scalo's own exporters were fine the whole time: the live engine logsreaders=[otlp(grpc)->...:4317].telemetry={"auto_configure": False}(one constant,dfe_engine/fastapi_telemetry.py). FastAPI's request telemetry still records into scalo's providers and goes out on scalo's gRPC exporters.test_otlp_export_env.pyputs a real listener on the endpoint the chart env names and asserts the engine API and the shim send it nothing. With the fix reverted it catches/v1/tracesand/v1/metricsposts from both apps.SCIM writes left no audit trail
The SCIM routes create, update and delete accounts and groups for an IdP and wrote no audit event. Every mutating SCIM route now emits the same
auth.account.*/auth.group.*event the admin routes do (#787), withdetails.via = "scim"so an IdP push reads apart from an admin's.activechanges), DELETE deleted.test_scim_change_audit.pyfollowstest_admin_change_audit.py: one event per write, right name, actor and target, no password or hash in it, and nothing for a no-op PATCH or a refused create.