Skip to content

docs: request-context metric labels, HTTPS redirect guard, ORC metadata limit, scalar function memory, OTLP log record size - #2440

Merged
lukekim merged 2 commits into
trunkfrom
docs-sync/2026-10-27-limits-and-labels
Oct 27, 2026
Merged

lukekim merged 2 commits into
trunkfrom
docs-sync/2026-10-27-limits-and-labels

Conversation

@spiceemma

Copy link
Copy Markdown
Contributor

Summary

This PR documents five changes merged to spiceai/spiceai trunk. All five are vNext only (website/docs/): none of the merge commits is an ancestor of v2.4.0, so the version-2.4.x snapshot correctly documents the earlier behavior.

  1. Request-context metric labels (features/observability, monitoring/datadog): the (request context) labels are now protocol, cache_namespace_kind, client, runtime, runtime_version, and runtime_system. client is a known Spice client name or other. The client_version, client_system, and user_agent labels are gone, so the Datadog cardinality note and its blocklist example no longer name them.
  2. HTTPS redirect guard (components/data-connectors/https): a JSON API dataset refuses a redirect to a host name that resolves to a private, loopback, or link-local address, and the error is documented. The sentence "Cross-origin redirects are rejected." is removed: the redirect policy follows a cross-origin redirect to a public host (Unverified, code inspection only; see Verification).
  3. ORC metadata decompression limit (reference/file_format).
  4. Argument-sized scalar functions draw on the query memory pool (reference/memory).
  5. OTLP logs record size limit (features/data-ingestion).

Source PRs

Verification

Every value below was read on spiceai/spiceai origin/trunk at 814e639e8eb55eacbe3f3fadaad2ec0aaf712c05. These are name, default, and error-string claims. The behavior statements come from reading the code, Unverified, code inspection only, except where a source PR records its own run.

crates/runtime-request-context/src/user_agent.rs:107  const KNOWN_CLIENTS: [&str; 8] = ["spice","repl","spicepy","spice-rs","gospice","spice.js","spice-java","spice-dotnet"]
crates/runtime-request-context/src/user_agent.rs:118  const OTHER_CLIENT: &str = "other";
crates/runtime-request-context/src/user_agent.rs:94   Self::Absent => None,   (no User-Agent -> no client/runtime labels)
crates/runtime-request-context/src/context.rs:701-707 build(): pushes "client", "runtime", "runtime_version", "runtime_system"
crates/runtime-request-context/src/context.rs:253,259 to_dimensions(): "protocol", "cache_namespace_kind"
crates/runtime-request-context/src/cache_namespace.rs:64-66  "public" | "principal" | "system"
(no caller of with_baggage outside context.rs: `git grep -n 'with_baggage\|baggage_mut' origin/trunk -- crates bin` -> only context.rs)

crates/runtime-object-store/src/ssrf.rs:161  "the HTTPS data connector does not follow a redirect to a private, loopback, or link-local address",
crates/object_store_spawn/src/guard.rs:140   #[snafu(display("Refused to connect to {target}: {reason}"))]
crates/runtime/src/dataconnector/https.rs:986   async fn build_http_client(...)  (called from create_http_table_provider, "JSON API endpoints")
crates/runtime/src/dataconnector/https.rs:1017-1033  redirect Policy::custom: refuses internal targets, records new hosts, otherwise attempt.follow() (no same-origin check)

Cargo.toml:366  orc-rust = { git = "https://github.com/spiceai/orc-rust.git", rev = "5b35c6d10a8cbd9750d501216ff04e72e23829b6", ... } # orc-rust 0.9.0 + bounded metadata decompression
spiceai/orc-rust@5b35c6d src/compression.rs:39-43  MIN_METADATA_DECODE_SIZE = 64 * 1024 * 1024; MAX_METADATA_DECODE_RATIO = 100; limit = max(min, encoded * ratio)
spiceai/orc-rust@5b35c6d src/error.rs:174  "ORC metadata stream exceeds the decompression limit of {} bytes"

crates/runtime/src/datafusion/output_reservation.rs:58-63  "repeat" | "lpad" | "rpad" | "array_repeat" | "array_resize" | "range" | "generate_series"

crates/runtime/src/opentelemetry_logs.rs:84   const MAX_LOGS_BATCH_STRING_BYTES: usize = 256 * 1024 * 1024;   (= 268435456)
crates/runtime/src/opentelemetry_logs.rs:512-516  one record above the limit -> return Err(... "The OpenTelemetry logs export contains one record that expands to {} string bytes, above the per-batch limit of {max_string_bytes} bytes; reduce the record's body or attributes")
crates/runtime/src/opentelemetry_logs.rs:254-268  string_bytes includes resource.attributes and scope.attributes
crates/runtime/src/opentelemetry_logs.rs:567-568  ArrowError::InvalidArgumentError -> Status::invalid_argument

The source PRs ran their own reproductions. spiceai/spiceai#15594 ran a spiced debug build against a local 302 to localhost4 and recorded Refused to connect to localhost4: the HTTPS data connector does not follow a redirect … in its body. The runtime was not run for this docs PR.

Later revert, removal, or retune: gh pr list --repo spiceai/spiceai --state merged --search "<symbol>" for KNOWN_CLIENTS, redirect_destination_guard, MAX_LOGS_BATCH_STRING_BYTES, and output_reservation returned only the source PRs above.

Known limitations: spiceai/spiceai#15594 says the dataset's own host and a proxy are not checked at redirect time, and the page states this. The versioned https/index.md copies (2.0.x–2.4.x) still say "Cross-origin redirects are rejected." v2.4.0's https.rs:979 redirect policy has no same-origin check either, but that is code inspection only. This PR leaves the versioned copies for an audit pass that can run the connector.

Propagation (vNext only, since none of the merge commits is in v2.4.0):

$ grep -rn 'client_version\|`user_agent`' website/docs
(no output)
$ git diff --name-only origin/trunk..HEAD
website/docs/components/data-connectors/https/index.md
website/docs/features/data-ingestion/index.md
website/docs/features/observability/index.md
website/docs/monitoring/datadog/index.md
website/docs/reference/file_format.md
website/docs/reference/memory.md

Build: NODE_OPTIONS=--max-old-space-size=12288 npm run build (in website/) passed:

[SUCCESS] Generated static files in "build".
exit=0

Style grep (docs-pr-hygiene.md §"Local Docusaurus build gate") over the added lines: no hits.

Test plan

  • cd website && npm run build passes (output above)
  • Versioned-docs propagation checked: vNext only (grep above)
  • Files updated: 6 (matches the diff above)

Review gate

  • Adversarial review: skipped — codex not installed (receipt: attempt=codex result=not-installed, engine=none)
  • /security-review: no findings (docs-only Markdown diff)
  • /simplify: not run — docs-only diff, no code

…ta limit, scalar output reservations, OTLP log record size
@spiceemma
spiceemma requested a review from a team as a code owner October 27, 2026 14:49
Copilot AI balanced review requested due to automatic review settings October 27, 2026 14:49
@spiceemma spiceemma self-assigned this Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

✅ Pull with Spice Passed

Passing checks:

  • ✅ Title meets minimum length requirement (10 characters)
  • ✅ Has at least one of the required labels: area/blog, area/docs, area/cookbook, dependencies
  • ✅ No banned labels detected
  • ✅ Has at least one assignee: spiceemma

Copilot AI previously approved these changes Oct 27, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approved

The documentation updates have no blocking issues; the remaining lower-bound reservation wording clarification is minor.

1 open issue
What changed in this PR

This PR updates vNext documentation for five runtime changes covering metric labels, HTTPS redirect protection, and resource limits. Versioned documentation remains unchanged because these changes postdate v2.4.0.

Changes:

  • Refresh request-context metric labels and Datadog tag examples.
  • Document HTTPS redirect restrictions and their scope.
  • Describe ORC metadata, scalar-function memory, and OTLP log record limits.
File Description
website/​docs/​reference/​memory.md Describes scalar-function output reservations.
website/​docs/​reference/​file_format.md Documents ORC metadata decompression limits.
website/​docs/​monitoring/​datadog/​index.md Updates metric tag examples.
website/​docs/​features/​observability/​index.md Defines current request-context labels.
website/​docs/​features/​data-ingestion/​index.md Documents OTLP string-data limits and rejection behavior.
website/​docs/​components/​data-connectors/​https/​index.md Clarifies redirect protection and exclusions.

Review effort: Balanced

Comment thread website/docs/reference/memory.md Resolved
@github-actions

Copy link
Copy Markdown
Contributor

🚀 deployed to https://94f530b5.spiceai-org-website.pages.dev

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approved

The documentation matches the cited implementation changes, incorporates the prior memory-reservation correction, and has no unresolved findings.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@github-actions

Copy link
Copy Markdown
Contributor

🚀 deployed to https://f6bf00e3.spiceai-org-website.pages.dev

@lukekim
lukekim merged commit f54dba1 into trunk Oct 27, 2026
8 checks passed
@lukekim
lukekim deleted the docs-sync/2026-10-27-limits-and-labels branch October 27, 2026 16:54

This branch was successfully deployed

1 active deployment
preview — 90386a3f Deployed Oct 27, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants