From 7721d593483139651985c207baca0b1d46c860e3 Mon Sep 17 00:00:00 2001 From: Eduardo Silva Date: Wed, 4 Nov 2026 22:04:27 -0600 Subject: [PATCH 1/9] processor_fts: document the public Lua API for LuaLS Signed-off-by: Eduardo Silva --- docs/fts/README.md | 84 ++++++ docs/fts/fluentbit.d.lua | 565 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 649 insertions(+) create mode 100644 docs/fts/README.md create mode 100644 docs/fts/fluentbit.d.lua diff --git a/docs/fts/README.md b/docs/fts/README.md new file mode 100644 index 00000000000..d6fed1c2efc --- /dev/null +++ b/docs/fts/README.md @@ -0,0 +1,84 @@ +# FTS editor definitions and API manifest + +Add this directory to LuaLS `workspace.library` (or copy `fluentbit.d.lua` into +your project's definitions directory). For example, in `.luarc.json`: + +```json +{ + "workspace.library": ["/path/to/fluent-bit/docs/fts"], + "runtime.version": "Lua 5.1" +} +``` + +Use `Lua 5.4` for the portable browser WASM interpreter. The declarations are +IDE stubs; do not execute them or add them to the engine's runtime module path. +The engine supplies `require("fluentbit")`. Callback names come from the `call` +configuration (default `process`), and arguments are local to the callback: + +```lua +local fb = require("fluentbit") + +---@type FtsFilterCallback +function process(log) + if log.body:get_string("path") == "/health" then + return fb.DROP + end + if log.attributes then + log.attributes:set("reviewed", true) + end + return fb.KEEP +end +``` + +For metrics or traces, annotate the callback with `FtsMetricCallback` or +`FtsSpanCallback`. `record` is simply another possible callback argument name, +not a separate global API. Helpers beginning with `_` are implementation details +and are deliberately excluded. There are no public logging utilities. + +The definitions follow the bindings in `plugins/processor_fts/`: `KEEP` and +`DROP` are strings, edits commit on `nil`/`KEEP`, and Lua errors roll back edits. +There are no `MODIFY`/`ERROR`, `has`/`delete`, scalar comparison methods, or +hex-decoding bytes constructor. Use `get(key) ~= nil`, `remove(key)`, +`fb.numeric_compare`, and raw byte strings respectively. `parse_json` returns +typed userdata (including typed null), and raises on invalid JSON. Log bodies +must be maps; severity fields use `fb.log_field` and `fb.set_severity` rather +than direct properties. `FtsClassicFilterCallback` documents the separate +numeric-status `filter_lua` ABI and is not an FTS callback contract. + +Generate the machine-readable manifest from any working directory with Python +3.9 or later; no third-party packages, Lua executable or Fluent Bit binary are +needed: + +```sh +python3 tools/fts/export_api_schema.py +python3 tools/fts/export_api_schema.py --output /tmp/fts_api.json +python3 tools/fts/export_api_schema.py --check +python3 -m unittest discover -s tools/fts/tests -v +``` + +The default output is `build/fts_api.json` relative to the checkout. The manifest +format version is `1.0.0`; it is independent of Fluent Bit, pack, and recipe +versions. `constants` contains values and descriptions, `modules.fb.functions` +contains signatures, and `types` contains classes, enums, and callback aliases. +Functions also declare `signals` and `modes`: log helpers load for logs only, +telemetry helpers load for metrics and traces, and recipe helpers such as +`numeric_compare` and the state constructors load only for author packs. +Availability describes which VM exports a function; helpers still enforce their +documented signal and binding restrictions at invocation. +Types are LuaLS type expressions, not JSON Schema validation constraints. +Class methods/fields include `extends` for inheritance and `generics` where +applicable; consumers should resolve inherited members. Optional parameters +and fields have an explicit `optional` flag; return types encode nil using `?` +or a union. `nodiscard` marks pure reads or required results; emitting a metric +and removing a key have useful side effects even when their results are ignored. +The `fb.null` constant is a userdata singleton, so it has no JSON `value`. + +The exporter accepts the declaration subset used by this file and rejects +unknown annotations and unannotated parameters. Update both when introducing +new annotation syntax. Output is deterministic, and `--check` fails on missing +or stale artifacts. The runtime integration test compares all public module +functions and value methods with the real Lua VM. Keep generated JSON in build +output; downstream tools can distribute that artifact without parsing Lua. + +This API manifest complements the recipe/configuration model exported by +`tools/fts/fts_schema.py`; it does not replace native recipe preflight validation. diff --git a/docs/fts/fluentbit.d.lua b/docs/fts/fluentbit.d.lua new file mode 100644 index 00000000000..83345ca2ed9 --- /dev/null +++ b/docs/fts/fluentbit.d.lua @@ -0,0 +1,565 @@ +---@meta fluentbit + +-- Canonical public API for plugins/processor_fts. This is an IDE stub, not runtime code. +-- Use local fb = require("fluentbit"); fb, log and record are not injected globals. +-- A configured callback receives one log, metric family or span and commits edits +-- on nil/KEEP. Raise a Lua error to roll back and apply error_mode. +-- Log views expire after the callback. Keep copy() results for later callbacks. +-- Metric/span snapshots remain readable, but later edits cannot affect committed output. + +---@alias FtsFilterCallback fun(log: FtsLogRecord): FtsResultCode? +---@alias FtsMetricCallback fun(metric: FtsMetric): FtsResultCode? +---@alias FtsSpanCallback fun(span: FtsSpan): FtsResultCode? +---@alias FtsTelemetry FtsLogRecord|FtsMetric|FtsSpan +---@alias FtsBlockCallback fun(record: FtsTelemetry): FtsResultCode? +---@alias FtsWindowCallback fun(log: FtsLogRecord): "hold"? +-- Compatibility documentation for filter_lua only: FTS does not call this ABI. +---@alias FtsClassicResultCode -1|0|1|2 +---@alias FtsClassicFilterCallback fun(tag: string, timestamp: FtsTimestamp, record: table): FtsClassicResultCode, FtsTimestamp?, table? +---@alias FtsValueKind "map"|"array"|"string"|"bytes"|"bool"|"int64"|"uint64"|"double"|"null" +---@alias FtsWritableValue FtsValue|string|boolean|number +---@alias FtsNumeric FtsInt64|FtsUInt64|FtsDouble +---@alias FtsMetricKind "gauge"|"sum"|"histogram"|"exponential_histogram"|"summary" +---@alias FtsTemporality "unspecified"|"delta"|"cumulative" +---@alias FtsAggregation "sum"|"min"|"max"|"mean"|"count" + +---@class FtsTimestamp +---@field sec integer # Seconds since Unix epoch, 0..4294967293. +---@field nsec integer # Nanoseconds, 0..999999999. Assign a new timestamp table to change time. + +--- Typed userdata. Methods are registered on every value; incompatible kinds raise errors. +--- Lua tables and nil cannot be assigned as values; use typed containers and fb.null. +---@class FtsValue +local FtsValue = {} + +--- Return the stored kind without coercion. +---@return FtsValueKind +---@nodiscard +function FtsValue:type() end + +--- Scalars only. Integers return exact decimal strings, bytes return raw strings, +--- null returns nil. Containers raise errors. +---@return string|boolean|number|nil +---@nodiscard +function FtsValue:value() end + +--- Numeric values only. Integers outside +/-9007199254740991 raise errors. +---@return number +---@nodiscard +function FtsValue:number() end + +--- Independent writable copy, safe to retain beyond the callback. +---@return self +---@nodiscard +function FtsValue:copy() end + +--- Map lookup returns nil for missing keys. Arrays use existing 1-based indexes; +--- out-of-bounds indexes raise errors. Children are typed views, including scalars. +---@param key string|integer +---@return FtsValue? +---@nodiscard +function FtsValue:get(key) end + +--- Strict string accessor; missing map keys return nil, wrong types raise errors. +---@param key string|integer +---@return string? +---@nodiscard +function FtsValue:get_string(key) end + +--- Strict boolean accessor; missing map keys return nil, wrong types raise errors. +---@param key string|integer +---@return boolean? +---@nodiscard +function FtsValue:get_bool(key) end + +--- Numeric accessor; missing map keys return nil, unsafe integers raise errors. +---@param key string|integer +---@return number? +---@nodiscard +function FtsValue:get_number(key) end + +--- Maps replace all entries matching the literal key. Arrays replace an existing index. +--- Plain Lua numbers are stored as doubles. Does not return a value. +---@param key string|integer +---@param value FtsWritableValue +function FtsValue:set(key, value) end + +--- Maps only. Remove every entry matching the literal key; report whether any existed. +---@param key string +---@return boolean +function FtsValue:remove(key) end + +--- Maps only. Return an existing child map or create one; wrong types raise errors. +---@param key string +---@return FtsMap +---@nodiscard +function FtsValue:ensure_map(key) end + +--- Maps only. Return keys in stored order (including duplicates in unchanged input). +---@return string[] +---@nodiscard +function FtsValue:keys() end + +--- Maps and arrays only. Number of entries. +---@return integer +---@nodiscard +function FtsValue:len() end + +--- Arrays only. Append a typed value or Lua scalar. +---@param value FtsWritableValue +function FtsValue:append(value) end + +---@class FtsMap: FtsValue +local FtsMap = {} + +---@param key string +---@return FtsValue? +---@nodiscard +function FtsMap:get(key) end + +---@param key string +---@param value FtsWritableValue +function FtsMap:set(key, value) end + +---@class FtsArray: FtsValue +local FtsArray = {} + +---@param key integer +---@return T +---@nodiscard +function FtsArray:get(key) end + +---@param key integer +---@param value FtsWritableValue +function FtsArray:set(key, value) end + +---@class FtsInt64: FtsValue +local FtsInt64 = {} + +--- Lossless signed decimal representation. In packs, compare numbers with fb.numeric_compare. +---@return string +---@nodiscard +function FtsInt64:value() end + +---@class FtsUInt64: FtsValue +local FtsUInt64 = {} + +--- Lossless unsigned decimal representation. +---@return string +---@nodiscard +function FtsUInt64:value() end + +---@class FtsBytes: FtsValue +local FtsBytes = {} + +--- Raw bytes, including NUL; no hexadecimal or base64 decoding/encoding. +--- Compare extracted strings for byte equality; userdata has no comparison methods. +---@return string +---@nodiscard +function FtsBytes:value() end + +---@class FtsDouble: FtsValue +local FtsDouble = {} + +---@return number +---@nodiscard +function FtsDouble:value() end + +---@class FtsNull: FtsValue +local FtsNull = {} + +---@return nil +---@nodiscard +function FtsNull:value() end + +---@class FtsResource +---@field attributes FtsMap # Mutate through set/remove; do not replace this view. +---@field schema_url string? +---@field dropped_attributes_count integer? # Metric/span contexts; logs expose this through context maps. + +---@class FtsScope: FtsResource +---@field name string? +---@field version string? + +---@class FtsLogContext +---@field attributes FtsMap +---@field metadata FtsMap + +--- Log bodies must remain maps. OTel aliases are nil on ordinary JSON-line logs. +--- Read severity_number/severity_text with fb.log_field, not direct log properties. +--- resource, scope, attributes, metadata and context roots cannot be replaced. +---@class FtsLogRecord +---@field body FtsMap # Replaceable with another map; arbitrary scalar bodies are not accepted. +---@field metadata FtsMap +---@field timestamp FtsTimestamp +---@field tag string # Read-only routing tag. +---@field attributes? FtsMap # Only present with OTel context, e.g. opentelemetry_envelope. +---@field resource? FtsResource # Only present with OTel context. +---@field scope? FtsScope # Only present with OTel context. +---@field context? FtsLogContext # Present on grouped records; roots cannot be replaced. + +--- One metric family; datapoints are typed maps with OTLP snake_case keys. +--- time_unix_nano, start_time_unix_nano, count and bucket counts are exact integers. +---@class FtsMetric +---@field signal "metrics" # Read-only. +---@field tag string # Read-only. +---@field name string +---@field description string +---@field unit string +---@field type FtsMetricKind # Only gauge/sum conversion is supported. +---@field datapoints FtsArray +---@field aggregation_temporality? FtsTemporality # Absent on gauges and summaries. +---@field is_monotonic? boolean # Sums only. +---@field resource FtsResource +---@field scope FtsScope +local FtsMetric = {} + +--- Emit a family snapshot; returned view edits the emitted copy. Does not rerun the callback. +--- Emission from an already emitted view is unavailable. Errors discard all emissions. +---@return FtsMetric +function FtsMetric:emit() end + +---@class FtsSpan +---@field signal "traces" # Read-only. +---@field tag string # Read-only. +---@field name string +---@field trace_state string? +---@field kind integer # Numeric OpenTelemetry span kind. +---@field flags integer +---@field trace_id string # 32 hexadecimal characters, nonzero. +---@field span_id string # 16 hexadecimal characters, nonzero. +---@field parent_span_id string # Empty for root spans, otherwise 16 hex characters. +---@field start_time_unix_nano FtsUInt64 +---@field end_time_unix_nano FtsUInt64 +---@field attributes FtsMap +---@field status FtsMap # code (integer) and message (string). +---@field events FtsArray +---@field links FtsArray +---@field dropped_attributes_count integer +---@field dropped_events_count integer +---@field dropped_links_count integer +---@field resource FtsResource +---@field scope FtsScope + +---@class FtsSelector +---@field root string # Signal-appropriate root, e.g. body, attributes, resource.attributes. +---@field path string[] # Literal map keys, never a dotted-path expression. + +---@class FtsCondition +---@field all? FtsCondition[] +---@field any? FtsCondition[] +---@field selector? FtsSelector # Leaf condition; mutually exclusive with all/any. +---@field op? "eq"|"neq"|"in"|"not_in"|"gt"|"gte"|"lt"|"lte" +---@field value? any # Validated scalar literal for non-membership operators. +---@field values? any[] # Validated scalar literals for in/not_in. + +---@class FtsDedupeOptions +---@field keys FtsSelector[] # 1..16 selectors. +---@field tenant? FtsSelector +---@field ttl_ms integer # 1..86400000. +---@field max_keys? integer # Default 1024, maximum 65536. +---@field max_key_bytes? integer # Default 1024, maximum 65536. +---@field overflow? "keep" +---@field on_missing? "error"|"keep" + +---@class FtsThrottleOptions: FtsDedupeOptions +---@field rate integer # Tokens/second, 1..1000000. +---@field burst integer # 1..1000000; TTL must cover a full refill. + +---@class FtsWindowOptions +---@field keys FtsSelector[] +---@field tenant? FtsSelector +---@field hold_ms integer # 1..86400000. +---@field late_ms? integer # Default 60000; 0..86400000. +---@field ready_timeout_ms? integer # Default 60000; 1..86400000. +---@field max_keys? integer # Default 1024, maximum 65536. +---@field max_key_bytes? integer # Default 1024, maximum 65536. +---@field retain_when FtsCondition +---@field keep_when? FtsCondition +---@field overflow? "keep" +---@field on_missing? "error"|"keep" + +---@class FtsMetricWindowOptions +---@field interval_ms integer # 1..86400000. +---@field ready_timeout_ms? integer # Default 60000; 1..86400000. +---@field max_keys? integer # Default 1024, maximum 65536. +---@field max_key_bytes? integer # Default 4096, maximum 65536. +---@field max_samples? integer # Default 1000000, maximum 1000000. + +---@class FtsMetricWindow +local FtsMetricWindow = {} + +--- Read a previous retained family and its reduction state; missing key returns nil,nil,0. +---@param key string +---@return FtsMetric? previous +---@return FtsMap? state +---@return integer samples +---@nodiscard +function FtsMetricWindow:get(key) end + +--- Freeze a validated family snapshot for release at the following processor stage. +---@param key string +---@param metric FtsMetric +---@param state? FtsMap +---@return "hold" +---@nodiscard +function FtsMetricWindow:hold(key, metric, state) end + +--- require("fluentbit") module, conventionally bound locally as fb. +---@class FtsModule +local fb = {} + +--- Callback results are strings. MODIFY and ERROR are not exported: mutate then +--- return nil/KEEP to commit, or raise an error for rollback and error_mode. +---@enum FtsResultCode +local FtsResultCode = { + KEEP = "keep", -- Commit edits and pass the item to the next stage. + DROP = "drop", -- Drop this item; explicitly emitted metric families survive. +} + +--- Commit changes and forward the item. +fb.KEEP = FtsResultCode.KEEP +--- Drop the item from the pipeline. +fb.DROP = FtsResultCode.DROP + +--- Typed null singleton; distinct from missing map keys. +---@type FtsNull +fb.null = nil + +--- Stable fixed-seed fingerprint algorithm; provides no secrecy. +---@type string +fb.hash64_algorithm = "xxh3-64-v1-seed0" + +--- Empty typed map. +---@return FtsMap +---@nodiscard +function fb.map() end + +--- Empty typed array. +---@return FtsArray +---@nodiscard +function fb.array() end + +--- Exact signed 64-bit integer. Numeric arguments must be safe integral Lua numbers. +---@param value number|string # Decimal string for values beyond +/-9007199254740991. +---@return FtsInt64 +---@nodiscard +function fb.int64(value) end + +--- Exact unsigned 64-bit integer; use decimal strings for large values. +---@param value number|string +---@return FtsUInt64 +---@nodiscard +function fb.uint64(value) end + +--- Binary value from raw bytes; does not decode hexadecimal strings. +---@param value string +---@return FtsBytes +---@nodiscard +function fb.bytes(value) end + +--- Parse JSON into typed values, including scalar roots. Raises on invalid JSON, +--- duplicate keys, input over 1 MiB or nesting beyond 64 levels; never returns nil. +---@param text string +---@return FtsValue +---@nodiscard +function fb.parse_json(text) end + +--- Parse strict RFC3339 time; raises on invalid input or unsupported event time range. +---@param text string +---@return FtsTimestamp +---@nodiscard +function fb.parse_time(text) end + +--- Recursively redact matching map keys (case-insensitive); default replacement [REDACTED]. +---@param value FtsValue +---@param keys string[] +---@param replacement? FtsWritableValue +function fb.redact(value, keys, replacement) end + +--- Remove all keys except the listed literal names. +---@param map FtsMap +---@param keys string[] +function fb.keep_keys(map, keys) end + +--- Availability: logs; modes: script,pack. +--- Check before accessing optional OTel aliases. +---@param log FtsLogRecord +---@return boolean +---@nodiscard +function fb.is_otel(log) end + +--- Availability: logs; modes: script,pack. +--- OTel-only; raises for native logs. Same view as log.attributes. +---@param log FtsLogRecord +---@return FtsMap +---@nodiscard +function fb.attributes(log) end + +--- Availability: logs; modes: script,pack. +--- OTel-only; e.g. severity_number, severity_text, trace_id, observed_timestamp. +--- Returns a typed value or nil for a missing field. Raises for native logs. +---@param log FtsLogRecord +---@param name string +---@return FtsValue? +---@nodiscard +function fb.log_field(log, name) end + +--- Availability: logs; modes: script,pack. +--- OTel-only. Set severity text and number; unknown names raise errors. +---@param log FtsLogRecord +---@param level string # trace, debug, info, warn, error, fatal and common aliases. +function fb.set_severity(log, level) end + +--- Availability: logs; modes: script,pack. +--- OTel-only. Set nonzero correlation IDs from hexadecimal strings. +---@param log FtsLogRecord +---@param trace string # 32 hex characters. +---@param span string # 16 hex characters. +function fb.set_trace_context(log, trace, span) end + +--- Availability: metrics,traces; modes: script,pack. +--- Gauge/sum value, or nil if neither as_int nor as_double exists. +---@param point FtsMap +---@return FtsNumeric? +---@nodiscard +function fb.metric_value(point) end + +--- Availability: metrics,traces; modes: script,pack. +---@param point FtsMap +---@param value FtsNumeric|number +function fb.set_metric_value(point, value) end + +--- Availability: metrics,traces; modes: script,pack. +--- Scale values, bounds and exemplars; exponential histogram scaling is unsupported. +---@param metric FtsMetric +---@param factor number # Finite and positive. +---@param unit? string +function fb.scale_metric(metric, factor, unit) end + +--- Availability: metrics,traces; modes: script,pack. +--- Combine equal series after retaining attributes. Only gauges allow non-sum operations. +---@param metric FtsMetric +---@param keys string[] +---@param operation FtsAggregation +---@param max_exemplars? integer # 0..128; omitted preserves legacy exemplar behavior. +function fb.aggregate_metric(metric, keys, operation, max_exemplars) end + +--- Availability: metrics,traces; modes: script,pack. +--- Explicit histogram only; bounds must be a subset of current bounds. +---@param metric FtsMetric +---@param bounds number[] +function fb.merge_histogram_buckets(metric, bounds) end + +--- Availability: metrics,traces; modes: script,pack. +--- Emit a copy; returning DROP still preserves this emitted family. +---@param metric FtsMetric +---@param name? string +---@param description? string +---@param unit? string +---@return FtsMetric +function fb.copy_metric(metric, name, description, unit) end + +--- Availability: metrics,traces; modes: script,pack. +--- Emit a sum derived from a distribution. No usable points returns nil. +---@param metric FtsMetric +---@param field "count"|"sum"|"avg" +---@param name? string +---@param monotonic? boolean +---@return FtsMetric? +function fb.extract_metric(metric, field, name, monotonic) end + +--- Availability: logs,metrics,traces; modes: pack. +--- Exact ordering of typed numeric values: -1, 0, 1; nonfinite values return nil. +--- Same-kind nonnumeric scalars return 0 if equal, 1 otherwise; mismatched kinds return nil. +--- Containers raise errors. Integers are compared without Lua-number precision loss. +---@param left FtsValue +---@param right FtsValue +---@return integer? +---@nodiscard +function fb.numeric_compare(left, right) end + +--- Stable lowercase 16-character XXH3 fingerprint of up to 1 MiB of bytes. +---@param bytes string +---@return string +---@nodiscard +function fb.hash64(bytes) end + +--- Full-width unsigned hash remainder, computed before conversion to a Lua number. +---@param bytes string +---@param modulus integer # 1..1000000000. +---@return integer +---@nodiscard +function fb.hash64_mod(bytes, modulus) end + +--- Lowercase 64-character HMAC hex; key_ref names a configured secret_refs entry. +--- Secret bytes are never exposed to Lua. Message limit: 1 MiB. +---@param key_ref string +---@param bytes string +---@return string +---@nodiscard +function fb.hmac_sha256(key_ref, bytes) end + +--- Availability: logs,metrics,traces; modes: pack. +--- Materialize a validated literal parameter; for author packs (manifest grammar 2). +---@param value any # Literal descriptor or supported scalar; not an arbitrary Lua object. +---@return FtsWritableValue +---@nodiscard +function fb.literal(value) end + +--- Availability: logs,metrics,traces; modes: pack. +--- Bind a validated stateless core block during pack factory binding only. +--- Options follow the block registry model; requires compatible signal/privacy context. +---@param id string +---@param options? table +---@return FtsBlockCallback +---@nodiscard +function fb.block(id, options) end + +--- Availability: logs,metrics,traces; modes: pack. +--- Pack binding only; terminal input-log step with explicit state_commit: local. +---@param options FtsDedupeOptions +---@return FtsFilterCallback +---@nodiscard +function fb.dedupe(options) end + +--- Availability: logs,metrics,traces; modes: pack. +--- Pack binding only; terminal input-log step with explicit state_commit: local. +---@param options FtsThrottleOptions +---@return FtsFilterCallback +---@nodiscard +function fb.throttle(options) end + +--- Availability: logs,metrics,traces; modes: pack. +--- Pack binding only; terminal input-log step. Returned hold action is recipe-only. +---@param options FtsWindowOptions +---@return FtsWindowCallback +---@nodiscard +function fb.window(options) end + +--- Availability: logs,metrics,traces; modes: pack. +--- Pack binding only; terminal input-metric step with explicit state_commit: local. +---@param options FtsMetricWindowOptions +---@return FtsMetricWindow +---@nodiscard +function fb.metric_window(options) end + +--- Availability: metrics,traces; modes: script,pack. +--- Stable identity of a nonempty gauge family for temporal retention; may raise. +---@param metric FtsMetric +---@return string +---@nodiscard +function fb.metric_window_key(metric) end + +--- Availability: metrics,traces; modes: script,pack. +--- Reduce a gauge family in Lua using optional previous family/state; return new state. +---@param metric FtsMetric +---@param previous FtsMetric? +---@param state FtsMap? +---@param samples integer +---@param operation FtsAggregation +---@return FtsMap +---@nodiscard +function fb.gauge_rollup(metric, previous, state, samples, operation) end + +return fb From 442efe5cb2cf775e34d5bae4235439d0823b79ea Mon Sep 17 00:00:00 2001 From: Eduardo Silva Date: Wed, 4 Nov 2026 22:04:34 -0600 Subject: [PATCH 2/9] processor_fts: export Lua API declarations as JSON Signed-off-by: Eduardo Silva --- tools/fts/export_api_schema.py | 222 ++++++++++++++++++++++ tools/fts/tests/test_export_api_schema.py | 81 ++++++++ 2 files changed, 303 insertions(+) create mode 100644 tools/fts/export_api_schema.py create mode 100644 tools/fts/tests/test_export_api_schema.py diff --git a/tools/fts/export_api_schema.py b/tools/fts/export_api_schema.py new file mode 100644 index 00000000000..75f9ff6b839 --- /dev/null +++ b/tools/fts/export_api_schema.py @@ -0,0 +1,222 @@ +#!/usr/bin/env python3 +"""Export the public FTS LuaLS declarations without Lua, a compiler or dependencies. + +This deliberately reads the declaration subset used by fluentbit.d.lua, not +arbitrary Lua. Unsupported annotations/declarations fail instead of silently +omitting API. Type expressions retain LuaLS syntax for downstream consumers. +""" + +import argparse +import json +from pathlib import Path +import re +import sys + + +ROOT = Path(__file__).resolve().parents[2] +DEFAULT_SOURCE = ROOT / "docs/fts/fluentbit.d.lua" +DEFAULT_OUTPUT = ROOT / "build/fts_api.json" + + +def typed(text): + """Split a LuaLS type from its optional name/comment, respecting nesting.""" + quote = None + depth = 0 + for index, char in enumerate(text): + if quote: + if char == quote and (index == 0 or text[index - 1] != "\\"): + quote = None + elif char in "\"'": + quote = char + elif char in "(<[{": + depth += 1 + elif char in ")>]}": + depth -= 1 + elif char.isspace() and depth == 0: + return text[:index], text[index:].strip().removeprefix("#").strip() + return text, "" + + +def export(source): + manifest = {"version": "1.0.0", "engine": "fts", "constants": [], + "modules": {"fb": {"require": "fluentbit", "functions": []}}, "types": {}} + types = manifest["types"] + owners = {} + enums = {} + current = None + enum = None + pending = {} + description = [] + + def documentation(): + return " ".join(description) + + def reset(): + pending.clear() + description.clear() + + for number, raw in enumerate(source.splitlines(), 1): + line = raw.strip() + if not line or line.startswith("-- ") or line.startswith("-- Compatibility"): + continue + try: + if line.startswith("---@"): + annotation, _, text = line[4:].partition(" ") + text = text.strip() + if annotation == "meta": + continue + if annotation == "alias": + name, expression = text.split(None, 1) + if name in types: + raise ValueError(f"duplicate type {name}") + types[name] = {"kind": "alias", "type": expression, + "description": documentation()} + reset() + elif annotation == "class": + match = re.fullmatch(r"(\w+)(<[^>]+>)?(?:\s*:\s*(.+))?", text) + if not match: + raise ValueError("invalid class declaration") + name, generics, base = match.groups() + if name in types: + raise ValueError(f"duplicate type {name}") + current = {"kind": "class", "description": documentation(), + "fields": [], "methods": []} + if generics: + current["generics"] = generics[1:-1].split(",") + if base: + current["extends"] = base + types[name] = current + reset() + pending["class"] = name + elif annotation == "field": + if current is None: + raise ValueError("field outside class") + name, rest = text.split(None, 1) + expression, comment = typed(rest) + current["fields"].append({"name": name.rstrip("?"), "type": expression, + "optional": name.endswith("?"), "description": comment}) + elif annotation == "enum": + if text in types: + raise ValueError(f"duplicate type {text}") + enum = {"kind": "enum", "description": documentation(), "values": []} + types[text] = enum + pending["enum"] = text + elif annotation == "param": + name, rest = text.split(None, 1) + expression, comment = typed(rest) + pending.setdefault("parameters", []).append({"name": name.rstrip("?"), + "type": expression, "optional": name.endswith("?"), "description": comment}) + elif annotation == "return": + expression, comment = typed(text) + pending.setdefault("returns", []).append({"type": expression, "description": comment}) + elif annotation == "nodiscard": + pending["nodiscard"] = True + elif annotation == "type": + pending["type"] = text + else: + raise ValueError(f"unsupported annotation @{annotation}") + elif line.startswith("---"): + comment = line[3:].strip() + availability = re.fullmatch(r"Availability: ([a-z,]+); modes: ([a-z,]+)\.", comment) + if availability: + pending["signals"] = availability[1].split(",") + pending["modes"] = availability[2].split(",") + else: + description.append(comment) + elif enum is not None and line == "}": + enum = None + reset() + elif enum is not None and not line.startswith("local "): + match = re.fullmatch(r'(\w+)\s*=\s*("[^"]*"|-?\d+),\s*(?:--\s*(.*))?', line) + if not match: + raise ValueError("unsupported enum member") + name, value, comment = match.groups() + enum["values"].append({"name": name, "value": json.loads(value), + "description": comment or ""}) + elif line.startswith("local "): + match = re.fullmatch(r"local (\w+) = \{\}?", line) + if not match: + raise ValueError("unsupported local declaration") + name = match[1] + if "enum" in pending: + enums[name] = pending["enum"] + elif "class" in pending: + owners[name] = pending["class"] + reset() + else: + raise ValueError("local declaration without type") + elif line.startswith("function "): + match = re.fullmatch(r"function (\w+)([.:])(\w+)\(([^)]*)\) end", line) + if not match: + raise ValueError("unsupported function declaration") + owner, separator, name, arguments = match.groups() + params = pending.get("parameters", []) + if [p["name"] for p in params] != [a.strip() for a in arguments.split(",") if a.strip()]: + raise ValueError(f"parameter annotations do not match {owner}.{name}") + function = {"name": name, "description": documentation(), "parameters": params, + "returns": pending.get("returns", []), + "nodiscard": pending.get("nodiscard", False)} + if owner == "fb" and separator == ".": + function["signals"] = pending.get("signals", ["logs", "metrics", "traces"]) + function["modes"] = pending.get("modes", ["script", "pack"]) + manifest["modules"]["fb"]["functions"].append(function) + elif separator == ":" and owner in owners: + types[owners[owner]]["methods"].append(function) + else: + raise ValueError(f"unknown function owner {owner}") + reset() + elif line.startswith("fb."): + match = re.fullmatch(r"fb\.(\w+) = (.+)", line) + if not match: + raise ValueError("unsupported module constant") + name, expression = match.groups() + constant = {"name": name, "module": "fb", "description": documentation()} + if "." in expression: + owner, member = expression.split(".", 1) + enum_name = enums[owner] + value = next(v for v in types[enum_name]["values"] if v["name"] == member) + constant.update(value=value["value"], type=enum_name, category="action") + else: + constant.update(value=None if expression == "nil" else json.loads(expression), + type=pending["type"], category="value") + # null is a userdata singleton, not Lua nil; the stub uses nil as a placeholder. + if constant["type"] == "FtsNull": + constant.pop("value") + manifest["constants"].append(constant) + reset() + elif line == "return fb": + reset() + else: + raise ValueError("unsupported declaration") + except (ValueError, KeyError, StopIteration) as error: + raise ValueError(f"line {number}: {error}: {line}") from error + + if enum is not None: + raise ValueError("unterminated enum") + if not manifest["modules"]["fb"]["functions"]: + raise ValueError("no fb functions found") + return manifest + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--source", type=Path, default=DEFAULT_SOURCE) + parser.add_argument("--output", type=Path, default=DEFAULT_OUTPUT) + parser.add_argument("--check", action="store_true", help="fail if the output is missing or stale") + args = parser.parse_args() + try: + content = json.dumps(export(args.source.read_text(encoding="utf-8")), indent=2, + ensure_ascii=False, sort_keys=True) + "\n" + if args.check: + if not args.output.exists() or args.output.read_text(encoding="utf-8") != content: + raise ValueError(f"stale API manifest: {args.output}") + else: + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(content, encoding="utf-8") + except (OSError, ValueError) as error: + parser.exit(1, f"{error}\n") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/fts/tests/test_export_api_schema.py b/tools/fts/tests/test_export_api_schema.py new file mode 100644 index 00000000000..a56a2f74808 --- /dev/null +++ b/tools/fts/tests/test_export_api_schema.py @@ -0,0 +1,81 @@ +"""Public API export stays lossless, deterministic and strict about drift.""" + +import importlib.util +import json +from pathlib import Path +import subprocess +import sys +import unittest + + +TOOL = Path(__file__).resolve().parents[1] / "export_api_schema.py" +SPEC = importlib.util.spec_from_file_location("export_api_schema", TOOL) +API = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(API) + + +class ExportApiTests(unittest.TestCase): + def setUp(self): + self.source = API.DEFAULT_SOURCE.read_text(encoding="utf-8") + self.manifest = API.export(self.source) + + def test_runtime_contract_and_optional_context(self): + constants = {c["name"]: c for c in self.manifest["constants"]} + self.assertEqual(constants["KEEP"]["value"], "keep") + self.assertEqual(constants["DROP"]["value"], "drop") + self.assertNotIn("MODIFY", constants) + self.assertNotIn("ERROR", constants) + self.assertNotIn("value", constants["null"]) + fields = {f["name"]: f for f in self.manifest["types"]["FtsLogRecord"]["fields"]} + self.assertEqual(fields["body"]["type"], "FtsMap") + self.assertTrue(fields["attributes"]["optional"]) + self.assertTrue(fields["resource"]["optional"]) + self.assertNotIn("severity_number", fields) + + def test_function_contracts_are_preserved(self): + functions = {f["name"]: f for f in self.manifest["modules"]["fb"]["functions"]} + self.assertEqual(functions["parse_json"]["returns"][0]["type"], "FtsValue") + self.assertTrue(functions["parse_json"]["nodiscard"]) + self.assertFalse(functions["redact"]["nodiscard"]) + self.assertTrue(functions["redact"]["parameters"][-1]["optional"]) + self.assertEqual(functions["log_field"]["signals"], ["logs"]) + self.assertEqual(functions["literal"]["modes"], ["pack"]) + types = self.manifest["types"] + self.assertEqual(types["FtsArray"]["generics"], ["T"]) + self.assertEqual(types["FtsUInt64"]["extends"], "FtsValue") + self.assertIn("FtsClassicResultCode", types["FtsClassicFilterCallback"]["type"]) + window = {m["name"]: m for m in types["FtsMetricWindow"]["methods"]} + self.assertEqual(len(window["get"]["returns"]), 3) + + def test_cli_deterministic_export_and_stale_detection(self): + import tempfile + + with tempfile.TemporaryDirectory() as directory: + output = Path(directory) / "nested/api.json" + command = [sys.executable, str(TOOL), "--output", str(output)] + subprocess.run(command, check=True) + self.assertEqual(json.loads(output.read_text()), self.manifest) + original = output.read_bytes() + subprocess.run(command, check=True) + self.assertEqual(output.read_bytes(), original) + subprocess.run(command + ["--check"], check=True) + output.write_text("{}\n") + result = subprocess.run(command + ["--check"], capture_output=True, text=True) + self.assertEqual(result.returncode, 1) + self.assertIn("stale API manifest", result.stderr) + + def test_unsupported_or_incomplete_declarations_fail(self): + for old, new in [ + ("---@param text string", "---@unknown text string"), + ("function fb.map() end", "function fb.map(untyped) end"), + ("function fb.map() end", "function missing.map() end"), + ("---@class FtsMap: FtsValue", "---@class FtsValue"), + ]: + with self.subTest(new=new), self.assertRaises(ValueError): + API.export(self.source.replace(old, new, 1)) + with self.assertRaises(ValueError): + API.export("") + + +if __name__ == "__main__": + unittest.main() From 88540c3aa5db5e37531403cedfb3d65c4368eeed Mon Sep 17 00:00:00 2001 From: Eduardo Silva Date: Wed, 4 Nov 2026 22:04:34 -0600 Subject: [PATCH 3/9] tests: integration: check FTS API definitions against runtime Signed-off-by: Eduardo Silva --- .../tests/test_fts_api_definitions.py | 74 +++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 tests/integration/scenarios/processor_fts/tests/test_fts_api_definitions.py diff --git a/tests/integration/scenarios/processor_fts/tests/test_fts_api_definitions.py b/tests/integration/scenarios/processor_fts/tests/test_fts_api_definitions.py new file mode 100644 index 00000000000..5924f5a7684 --- /dev/null +++ b/tests/integration/scenarios/processor_fts/tests/test_fts_api_definitions.py @@ -0,0 +1,74 @@ +"""Compare the IDE's public symbol inventory against a real FTS Lua VM.""" + +import json +from pathlib import Path +import subprocess +import sys + +import pytest +import yaml + +from test_fts import log_text, make_service, records + + +ROOT = Path(__file__).resolve().parents[5] + + +@pytest.mark.parametrize("signal", ["logs", "metrics", "traces"]) +def test_public_api_definitions_match_runtime(tmp_path, signal): + output = tmp_path / "fts_api.json" + subprocess.run([sys.executable, str(ROOT / "tools/fts/export_api_schema.py"), + "--output", str(output)], check=True) + manifest = json.loads(output.read_text()) + functions = [function["name"] for function in manifest["modules"]["fb"]["functions"] + if signal in function["signals"] and "script" in function["modes"]] + methods = [method["name"] for method in manifest["types"]["FtsValue"]["methods"]] + code = ''' +local fb = require("fluentbit") +local expected = {} +for _, name in ipairs(FUNCTIONS) do + assert(type(fb[name]) == "function", "missing function: " .. name) + expected[name] = true +end +for name, value in pairs(fb) do + if name:sub(1, 1) ~= "_" and type(value) == "function" then + assert(expected[name], "undocumented public function: " .. name) + end +end +assert(fb.KEEP == "keep" and fb.DROP == "drop") +assert(fb.MODIFY == nil and fb.ERROR == nil) +assert(fb.hash64_algorithm == "xxh3-64-v1-seed0") +local map = fb.map() +for _, name in ipairs(METHODS) do + assert(type(map[name]) == "function", "missing value method: " .. name) +end +assert(map.has == nil and map.delete == nil) +assert(fb.parse_json("null"):type() == "null") +assert(not pcall(fb.parse_json, "invalid json")) +assert(fb.int64("-9223372036854775808"):value() == "-9223372036854775808") +assert(fb.bytes("ff"):value() == "ff") +function process_log(log) + assert(not fb.is_otel(log)) + assert(log.attributes == nil and log.resource == nil) + assert(log.severity_number == nil and log.severity_text == nil) + log.body:set("message", "api-verified") + return fb.KEEP +end +'''.replace("FUNCTIONS", "{" + ",".join(json.dumps(name) for name in functions) + "}") + code = code.replace("METHODS", "{" + ",".join(json.dumps(name) for name in methods) + "}") + service = make_service(tmp_path, code) + if signal != "logs": + config = tmp_path / "typed.yaml" + document = yaml.safe_load(config.read_text()) + processor = document["pipeline"]["inputs"][0]["processors"]["logs"] + document["pipeline"]["inputs"] = [{"name": "opentelemetry", + "port": "${FLUENT_BIT_TEST_LISTENER_PORT}", "processors": {signal: processor}}] + config.write_text(yaml.safe_dump(document)) + try: + service.start() + if signal == "logs": + result = service.wait_for_condition(lambda: records(service), timeout=30) + assert all(record["message"] == "api-verified" for record in result) + assert "edits rolled back" not in log_text(service) + finally: + service.stop() From fa5600605d2b7b21bc9f607493ec52882e2d37c9 Mon Sep 17 00:00:00 2001 From: Eduardo Silva Date: Wed, 4 Nov 2026 23:11:43 -0600 Subject: [PATCH 4/9] processor_fts: correct callback and optional field declarations Signed-off-by: Eduardo Silva --- docs/fts/README.md | 5 +++-- docs/fts/fluentbit.d.lua | 10 +++++----- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/fts/README.md b/docs/fts/README.md index d6fed1c2efc..f3ab6ae1066 100644 --- a/docs/fts/README.md +++ b/docs/fts/README.md @@ -39,13 +39,14 @@ The definitions follow the bindings in `plugins/processor_fts/`: `KEEP` and `DROP` are strings, edits commit on `nil`/`KEEP`, and Lua errors roll back edits. There are no `MODIFY`/`ERROR`, `has`/`delete`, scalar comparison methods, or hex-decoding bytes constructor. Use `get(key) ~= nil`, `remove(key)`, -`fb.numeric_compare`, and raw byte strings respectively. `parse_json` returns +`fb.numeric_compare` (author packs only, unavailable in script mode), and raw +byte strings respectively. `parse_json` returns typed userdata (including typed null), and raises on invalid JSON. Log bodies must be maps; severity fields use `fb.log_field` and `fb.set_severity` rather than direct properties. `FtsClassicFilterCallback` documents the separate numeric-status `filter_lua` ABI and is not an FTS callback contract. -Generate the machine-readable manifest from any working directory with Python +Generate the machine-readable manifest from the checkout root with Python 3.9 or later; no third-party packages, Lua executable or Fluent Bit binary are needed: diff --git a/docs/fts/fluentbit.d.lua b/docs/fts/fluentbit.d.lua index 83345ca2ed9..0ce8f280b32 100644 --- a/docs/fts/fluentbit.d.lua +++ b/docs/fts/fluentbit.d.lua @@ -15,7 +15,7 @@ ---@alias FtsWindowCallback fun(log: FtsLogRecord): "hold"? -- Compatibility documentation for filter_lua only: FTS does not call this ABI. ---@alias FtsClassicResultCode -1|0|1|2 ----@alias FtsClassicFilterCallback fun(tag: string, timestamp: FtsTimestamp, record: table): FtsClassicResultCode, FtsTimestamp?, table? +---@alias FtsClassicFilterCallback fun(tag: string, timestamp: FtsTimestamp|number, record: table): FtsClassicResultCode, FtsTimestamp|number|nil, table? ---@alias FtsValueKind "map"|"array"|"string"|"bytes"|"bool"|"int64"|"uint64"|"double"|"null" ---@alias FtsWritableValue FtsValue|string|boolean|number ---@alias FtsNumeric FtsInt64|FtsUInt64|FtsDouble @@ -174,12 +174,12 @@ function FtsNull:value() end ---@class FtsResource ---@field attributes FtsMap # Mutate through set/remove; do not replace this view. ----@field schema_url string? ----@field dropped_attributes_count integer? # Metric/span contexts; logs expose this through context maps. +---@field schema_url? string? +---@field dropped_attributes_count? integer? # Metric/span contexts; logs expose this through context maps. ---@class FtsScope: FtsResource ----@field name string? ----@field version string? +---@field name? string? +---@field version? string? ---@class FtsLogContext ---@field attributes FtsMap From a8f0c309683c6b48197dac01795b170f33bf4edd Mon Sep 17 00:00:00 2001 From: Eduardo Silva Date: Wed, 4 Nov 2026 23:11:43 -0600 Subject: [PATCH 5/9] processor_fts: reject incomplete API schema declarations Signed-off-by: Eduardo Silva --- tools/fts/export_api_schema.py | 12 ++++++-- tools/fts/tests/test_export_api_schema.py | 34 +++++++++++++++++++++++ 2 files changed, 44 insertions(+), 2 deletions(-) diff --git a/tools/fts/export_api_schema.py b/tools/fts/export_api_schema.py index 75f9ff6b839..4b23debecd2 100644 --- a/tools/fts/export_api_schema.py +++ b/tools/fts/export_api_schema.py @@ -33,11 +33,14 @@ def typed(text): elif char in ")>]}": depth -= 1 elif char.isspace() and depth == 0: + if text[:index].endswith("|") or text[index:].lstrip().startswith("|"): + raise ValueError("whitespace around union pipes is unsupported") return text[:index], text[index:].strip().removeprefix("#").strip() return text, "" def export(source): + """Build a manifest from the supported LuaLS declaration subset.""" manifest = {"version": "1.0.0", "engine": "fts", "constants": [], "modules": {"fb": {"require": "fluentbit", "functions": []}}, "types": {}} types = manifest["types"] @@ -49,9 +52,11 @@ def export(source): description = [] def documentation(): + """Join the documentation attached to the next declaration.""" return " ".join(description) def reset(): + """Clear annotations and documentation consumed by a declaration.""" pending.clear() description.clear() @@ -82,7 +87,7 @@ def reset(): current = {"kind": "class", "description": documentation(), "fields": [], "methods": []} if generics: - current["generics"] = generics[1:-1].split(",") + current["generics"] = [name.strip() for name in generics[1:-1].split(",")] if base: current["extends"] = base types[name] = current @@ -171,7 +176,7 @@ def reset(): raise ValueError("unsupported module constant") name, expression = match.groups() constant = {"name": name, "module": "fb", "description": documentation()} - if "." in expression: + if re.fullmatch(r"[A-Za-z_]\w*\.[A-Za-z_]\w*", expression): owner, member = expression.split(".", 1) enum_name = enums[owner] value = next(v for v in types[enum_name]["values"] if v["name"] == member) @@ -185,6 +190,8 @@ def reset(): manifest["constants"].append(constant) reset() elif line == "return fb": + if pending or description: + raise ValueError("unconsumed annotations or description before return fb") reset() else: raise ValueError("unsupported declaration") @@ -199,6 +206,7 @@ def reset(): def main(): + """Write the manifest or check that the existing artifact is current.""" parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--source", type=Path, default=DEFAULT_SOURCE) parser.add_argument("--output", type=Path, default=DEFAULT_OUTPUT) diff --git a/tools/fts/tests/test_export_api_schema.py b/tools/fts/tests/test_export_api_schema.py index a56a2f74808..ac0366bfef7 100644 --- a/tools/fts/tests/test_export_api_schema.py +++ b/tools/fts/tests/test_export_api_schema.py @@ -15,11 +15,15 @@ class ExportApiTests(unittest.TestCase): + """Exercise API contracts and strict declaration parsing.""" + def setUp(self): + """Load the canonical declarations and their exported manifest.""" self.source = API.DEFAULT_SOURCE.read_text(encoding="utf-8") self.manifest = API.export(self.source) def test_runtime_contract_and_optional_context(self): + """Preserve action values and optional telemetry context fields.""" constants = {c["name"]: c for c in self.manifest["constants"]} self.assertEqual(constants["KEEP"]["value"], "keep") self.assertEqual(constants["DROP"]["value"], "drop") @@ -31,8 +35,16 @@ def test_runtime_contract_and_optional_context(self): self.assertTrue(fields["attributes"]["optional"]) self.assertTrue(fields["resource"]["optional"]) self.assertNotIn("severity_number", fields) + for name, expected in [("FtsResource", {"schema_url": "string?", + "dropped_attributes_count": "integer?"}), + ("FtsScope", {"name": "string?", "version": "string?"})]: + fields = {f["name"]: f for f in self.manifest["types"][name]["fields"]} + for field, expression in expected.items(): + self.assertTrue(fields[field]["optional"]) + self.assertEqual(fields[field]["type"], expression) def test_function_contracts_are_preserved(self): + """Retain signatures, availability, inheritance and callback aliases.""" functions = {f["name"]: f for f in self.manifest["modules"]["fb"]["functions"]} self.assertEqual(functions["parse_json"]["returns"][0]["type"], "FtsValue") self.assertTrue(functions["parse_json"]["nodiscard"]) @@ -44,10 +56,13 @@ def test_function_contracts_are_preserved(self): self.assertEqual(types["FtsArray"]["generics"], ["T"]) self.assertEqual(types["FtsUInt64"]["extends"], "FtsValue") self.assertIn("FtsClassicResultCode", types["FtsClassicFilterCallback"]["type"]) + self.assertIn("timestamp: FtsTimestamp|number", types["FtsClassicFilterCallback"]["type"]) + self.assertIn("FtsTimestamp|number|nil", types["FtsClassicFilterCallback"]["type"]) window = {m["name"]: m for m in types["FtsMetricWindow"]["methods"]} self.assertEqual(len(window["get"]["returns"]), 3) def test_cli_deterministic_export_and_stale_detection(self): + """Check stable CLI output and failure on stale generated artifacts.""" import tempfile with tempfile.TemporaryDirectory() as directory: @@ -65,17 +80,36 @@ def test_cli_deterministic_export_and_stale_detection(self): self.assertIn("stale API manifest", result.stderr) def test_unsupported_or_incomplete_declarations_fail(self): + """Reject unsupported syntax and annotations without a declaration.""" for old, new in [ ("---@param text string", "---@unknown text string"), + ("---@param text string", "---@param text string | nil"), + ("---@param text string", "---@param text string| nil"), ("function fb.map() end", "function fb.map(untyped) end"), ("function fb.map() end", "function missing.map() end"), ("---@class FtsMap: FtsValue", "---@class FtsValue"), + ("return fb", "---@type string\nreturn fb"), + ("return fb", "--- Unattached description.\nreturn fb"), ]: with self.subTest(new=new), self.assertRaises(ValueError): API.export(self.source.replace(old, new, 1)) with self.assertRaises(ValueError): API.export("") + def test_generic_names_and_dotted_literals(self): + """Trim generic names and parse dotted values as JSON literals.""" + source = self.source.replace("FtsArray", "FtsArray", 1) + source = source.replace("return fb", '''---@type string +fb.test_string = "api.example.com" +---@type number +fb.test_number = 1.5 +return fb''') + manifest = API.export(source) + self.assertEqual(manifest["types"]["FtsArray"]["generics"], ["T", "U"]) + constants = {constant["name"]: constant for constant in manifest["constants"]} + self.assertEqual(constants["test_string"]["value"], "api.example.com") + self.assertEqual(constants["test_number"]["value"], 1.5) + if __name__ == "__main__": unittest.main() From 93ab057d762c924d528cd48dda993ab066403139 Mon Sep 17 00:00:00 2001 From: Eduardo Silva Date: Wed, 4 Nov 2026 23:11:43 -0600 Subject: [PATCH 6/9] tests: integration: verify FTS metric window pack methods Signed-off-by: Eduardo Silva --- .../tests/test_fts_api_definitions.py | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/tests/integration/scenarios/processor_fts/tests/test_fts_api_definitions.py b/tests/integration/scenarios/processor_fts/tests/test_fts_api_definitions.py index 5924f5a7684..29677fd9884 100644 --- a/tests/integration/scenarios/processor_fts/tests/test_fts_api_definitions.py +++ b/tests/integration/scenarios/processor_fts/tests/test_fts_api_definitions.py @@ -16,6 +16,7 @@ @pytest.mark.parametrize("signal", ["logs", "metrics", "traces"]) def test_public_api_definitions_match_runtime(tmp_path, signal): + """Verify each signal VM exposes the declared script functions and methods.""" output = tmp_path / "fts_api.json" subprocess.run([sys.executable, str(ROOT / "tools/fts/export_api_schema.py"), "--output", str(output)], check=True) @@ -72,3 +73,41 @@ def test_public_api_definitions_match_runtime(tmp_path, signal): assert "edits rolled back" not in log_text(service) finally: service.stop() + + +def test_metric_window_methods_in_pack_binding(tmp_path): + """Construct a window in an author factory and verify its declared methods.""" + if not (ROOT / "plugins/processor_fts/fts_recipe.lua").exists(): + pytest.skip("metric-window pack binding requires the separate FTS recipe implementation") + output = tmp_path / "fts_api.json" + subprocess.run([sys.executable, str(ROOT / "tools/fts/export_api_schema.py"), + "--output", str(output)], check=True) + manifest = json.loads(output.read_text()) + methods = [method["name"] for method in manifest["types"]["FtsMetricWindow"]["methods"]] + assert set(methods) == {"get", "hold"} + (tmp_path / "check.lua").write_text('''return function(parameters) + assert(type(fb.metric_window) == "function", "metric-window pack binding unavailable") + local window = fb.metric_window{interval_ms=1000} + for _, name in ipairs(METHODS) do + assert(type(window[name]) == "function", "missing window method: " .. name) + end + return function(metric) end +end +'''.replace("METHODS", "{" + ",".join(json.dumps(name) for name in methods) + "}")) + (tmp_path / "pack.json").write_text(json.dumps({"pack_version": 1, + "fts_api_version": 1, "id": "org.fluentbit.api-check", "version": "1.0.0", + "blocks": [{"name": "window", "signals": ["metrics"], + "source": "check.lua", "parameters": {}}]})) + service = make_service(tmp_path, "unused") + config = tmp_path / "typed.yaml" + document = yaml.safe_load(config.read_text()) + processor = {"name": "fts", "packages_dir": str(tmp_path), "packs": ["pack.json"], + "state_commit": "local", "recipe": [{"id": "check", "use": "org.fluentbit.api-check/window"}]} + document["pipeline"]["inputs"] = [{"name": "opentelemetry", + "port": "${FLUENT_BIT_TEST_LISTENER_PORT}", "processors": {"metrics": [processor]}}] + config.write_text(yaml.safe_dump(document)) + try: + # Initialization runs the author factory and its native window binding. + service.start() + finally: + service.stop() From d48f1238caecf4d236d4809e74cd97ec9a51278b Mon Sep 17 00:00:00 2001 From: Eduardo Silva Date: Thu, 5 Nov 2026 06:52:19 -0600 Subject: [PATCH 7/9] processor_fts: describe safe JSON decoding helpers in the Lua API Signed-off-by: Eduardo Silva --- docs/fts/README.md | 10 ++++++++++ docs/fts/fluentbit.d.lua | 31 +++++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+) diff --git a/docs/fts/README.md b/docs/fts/README.md index f3ab6ae1066..f506eda745f 100644 --- a/docs/fts/README.md +++ b/docs/fts/README.md @@ -46,6 +46,16 @@ must be maps; severity fields use `fb.log_field` and `fb.set_severity` rather than direct properties. `FtsClassicFilterCallback` documents the separate numeric-status `filter_lua` ABI and is not an FTS callback contract. +`fb.try_parse_json(text)` returns a typed value or `nil, error` for ordinary +parse failures. `fb.decode_json(container, key)` decodes one string field in +place; log-only `fb.unpack_json(log, field)` replaces the body with a decoded +object, reading `message` by default. Both mutation helpers return `true` on +success or `false, error` without mutation for ordinary decoding failures. +OTel log destinations reject unsigned integers above `INT64_MAX` before mutation. +Invalid mutation arguments and budget exhaustion still raise errors. Helper +boolean results are separate from callback actions: do not return them from +`process`. Ignoring a decoding failure preserves the original data. + Generate the machine-readable manifest from the checkout root with Python 3.9 or later; no third-party packages, Lua executable or Fluent Bit binary are needed: diff --git a/docs/fts/fluentbit.d.lua b/docs/fts/fluentbit.d.lua index 0ce8f280b32..897bfa42ac6 100644 --- a/docs/fts/fluentbit.d.lua +++ b/docs/fts/fluentbit.d.lua @@ -365,6 +365,37 @@ function fb.bytes(value) end ---@nodiscard function fb.parse_json(text) end +--- Parse JSON without throwing on ordinary parse errors or non-string input. +--- JSON null/false return typed values, not Lua nil/false. Same parser limits as parse_json. +--- Instruction-budget exhaustion still fails the callback. +---@param text any # JSON text; missing or non-string input returns nil and an error. +---@return FtsValue? value +---@return string? error +---@nodiscard +function fb.try_parse_json(text) end + +--- Decode a string field in place, preserving siblings. Return false,error without +--- mutation for missing/non-string fields, ordinary parse errors or OTel log integers +--- above INT64_MAX. Accepts any JSON value. Map keys are literal; arrays are 1-based. +--- Invalid containers/keys, expired views, mutation errors and budget exhaustion raise. +--- These boolean results are not callback actions; do not return them from process. +---@param container FtsMap|FtsArray +---@param key string|integer +---@return boolean success +---@return string? error +function fb.decode_json(container, key) end + +--- Availability: logs; modes: script,pack. +--- Decode a JSON object and replace log.body. Preserve the body and return false,error +--- for missing/non-string fields, ordinary parse errors, non-object JSON or OTel log +--- integers above INT64_MAX. Invalid arguments and budget exhaustion still raise. +--- Ignoring failure keeps the original body. Boolean results are not callback actions. +---@param log FtsLogRecord +---@param field? string # Literal source field; defaults to message. +---@return boolean success +---@return string? error +function fb.unpack_json(log, field) end + --- Parse strict RFC3339 time; raises on invalid input or unsupported event time range. ---@param text string ---@return FtsTimestamp From 574c53491e33740cdbf27eddd67a96cd3d564d29 Mon Sep 17 00:00:00 2001 From: Eduardo Silva Date: Thu, 5 Nov 2026 06:52:19 -0600 Subject: [PATCH 8/9] processor_fts: verify JSON helper schema contracts Signed-off-by: Eduardo Silva --- tools/fts/tests/test_export_api_schema.py | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/tools/fts/tests/test_export_api_schema.py b/tools/fts/tests/test_export_api_schema.py index ac0366bfef7..7551d40bfe2 100644 --- a/tools/fts/tests/test_export_api_schema.py +++ b/tools/fts/tests/test_export_api_schema.py @@ -79,6 +79,19 @@ def test_cli_deterministic_export_and_stale_detection(self): self.assertEqual(result.returncode, 1) self.assertIn("stale API manifest", result.stderr) + def test_json_helper_signatures_and_availability(self): + """Expose safe JSON decoding with its tuple returns and log-only helper.""" + functions = {f["name"]: f for f in self.manifest["modules"]["fb"]["functions"]} + for name, result in [("try_parse_json", "FtsValue?"), ("decode_json", "boolean"), + ("unpack_json", "boolean")]: + self.assertEqual([r["type"] for r in functions[name]["returns"]], [result, "string?"]) + self.assertTrue(functions["try_parse_json"]["nodiscard"]) + self.assertFalse(functions["decode_json"]["nodiscard"]) + self.assertFalse(functions["unpack_json"]["nodiscard"]) + self.assertEqual(functions["decode_json"]["signals"], ["logs", "metrics", "traces"]) + self.assertEqual(functions["unpack_json"]["signals"], ["logs"]) + self.assertTrue(functions["unpack_json"]["parameters"][1]["optional"]) + def test_unsupported_or_incomplete_declarations_fail(self): """Reject unsupported syntax and annotations without a declaration.""" for old, new in [ From 458f84b54cf72c4571541227fccd5e64ae8af55c Mon Sep 17 00:00:00 2001 From: Eduardo Silva Date: Thu, 5 Nov 2026 06:52:19 -0600 Subject: [PATCH 9/9] agents: require FTS API definition checks for FTS changes Signed-off-by: Eduardo Silva --- AGENTS.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index b9cf4b543b4..7346b5a902e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,6 +36,24 @@ Fluent Bit is a C/C++ monorepo built with CMake. Keep changes scoped: plugin logic in its plugin directory, shared behavior in `src/` or `lib/`. +## FTS API Definitions +- Whenever touching FTS, including `plugins/processor_fts/`, `packs/fts/`, + `tools/fts/`, or FTS tests, agents must check `docs/fts/fluentbit.d.lua` against the affected + implementation. Update declarations and `docs/fts/README.md` when exported + functions, constants, methods, record fields, callback contracts, argument or + return types, nil-safety, error behavior, or signal/pack availability change. +- Verify exports in the C bindings and embedded Lua helpers; do not infer API + availability solely from the declarations. Keep private `_` helpers excluded. +- Regenerate the API manifest and verify the exporter from the checkout root: + `python3 tools/fts/export_api_schema.py` + `python3 tools/fts/export_api_schema.py --check` + `python3 -m unittest discover -s tools/fts/tests -v` + Keep generated `build/fts_api.json` out of commits. +- Run `tests/integration/scenarios/processor_fts/tests/test_fts_api_definitions.py` + against a binary built from the matching source, normally and with the platform + memory checker as required below. Add focused contract coverage when necessary. + Report any verification blocker and whether definitions needed an update. + ## Bundled Library Changes - Treat `lib/` as bundled third-party or separately maintained code unless the specific path is clearly Fluent Bit-owned.