Additional notes regarding alignment with the config SDK

Author: stevejgordon

design/configuration-analysis-deep-dives.md

@@ -1883,8 +1883,209 @@ Key design constraints for the OpAMP package:
   telemetry policies) are free to implement their own `IConfigurationProvider`
   against the OpAMP client — the SDK's abstractions do not prevent this.
 
-→ [Deep Dive
+--> [Deep Dive
 C](configuration-analysis-deep-dives.md#c-otlp-exporter-snapshot-architecture-and-reload-path)
 covers OTLP-specific reload. [Deep Dive
 D](configuration-analysis-deep-dives.md#d-sampler-reloadability-design) covers
 sampler reload via `ReloadableSampler`.
+
+---
+
+## I. Configuration SDK Spec Alignment
+
+This section covers implementation details for aligning with the [Configuration
+SDK
+specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/configuration/sdk.md).
+The open design questions (in-memory model, Parse/Create operations,
+ConfigProvider) are discussed in [SS5.1 of the executive
+summary](configuration-analysis.md#51-spec-alignment--configuration-sdk-operations).
+This deep dive focuses on concrete implementation requirements.
+
+### I.1 YAML Environment Variable Substitution
+
+The spec defines a `${VAR:-default}` substitution syntax for YAML configuration
+files with detailed rules that differ from .NET's `IConfiguration` environment
+variable provider. The substitution must happen during YAML parsing, before
+values enter the `IConfiguration` tree.
+
+#### I.1.1 Spec Requirements Summary
+
+The substitution rules (from the [data model
+spec](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/configuration/data-model.md#environment-variable-substitution)):
+
+- `${VAR}` -- substitute the value of environment variable `VAR`
+- `${VAR:-fallback}` -- substitute `VAR`, or `fallback` if `VAR` is undefined,
+  empty, or null
+- `$$` -- escape sequence producing a literal `$`; the resolved `$` is not
+  considered for further substitution matching
+- Substitution applies only to scalar values, not mapping keys
+- Substitution is not recursive -- the result of one substitution is not
+  scanned for further `${...}` references
+- YAML structure injection via env vars is prohibited -- substituted values
+  are always scalars
+- Node types are interpreted after substitution (e.g. `${BOOL_VALUE}` where
+  `BOOL_VALUE=true` resolves to YAML boolean `true`, not string `"true"`)
+- Invalid substitution references (e.g. `${VAR:?error}`) must produce a parse
+  error with no partial results
+- The optional `env:` prefix is supported: `${env:VAR}` is equivalent to
+  `${VAR}`
+
+#### I.1.2 Implementation Approach
+
+The substitution should be implemented within the custom YAML
+`IConfigurationProvider`. The processing order is:
+
+```text
+1. Read YAML file from disk
+2. Perform ${VAR:-default} substitution on all scalar values
+3. Parse substituted YAML into IConfiguration key/value pairs
+4. Expose via standard IConfigurationProvider interface
+```
+
+Step 2 processes the raw YAML text (or the parsed YAML DOM scalar nodes)
+before projecting into `IConfiguration`. This ensures type coercion happens
+after substitution, matching the spec requirement.
+
+#### I.1.3 Interplay with IConfiguration Environment Variable Loading
+
+The SDK currently uses a vendored `EnvironmentVariablesConfigurationProvider` to
+make `OTEL_*` env vars available as `IConfiguration` keys. When `OTEL_CONFIG_FILE`
+is active, these two env-var mechanisms overlap:
+
+| Mechanism | When it runs | What it does |
+| --- | --- | --- |
+| YAML `${VAR}` substitution | YAML parse time | Replaces placeholders in YAML scalar values with env var values |
+| `EnvironmentVariablesConfigurationProvider` | `IConfiguration` build time | Makes all `OTEL_*` env vars available as `IConfiguration` keys that can override options |
+
+The spec states that when `OTEL_CONFIG_FILE` is set, the config file is the
+authoritative source and environment variables should not override values
+defined in the file (see [Risk
+3.1](configuration-analysis-risks.md#31-otel_config_file-vs-iconfiguration-hierarchy--resolution-via-sdk-option)).
+The YAML `${VAR}` substitution is the spec's mechanism for env-var-driven
+values within declarative config.
+
+**Recommended behaviour when `OTEL_CONFIG_FILE` is active:**
+
+- The `EnvironmentVariablesConfigurationProvider` for OTel-specific keys should
+  **not** be layered above the YAML provider in the `IConfiguration` source
+  chain. This prevents env vars from silently overriding YAML-defined values.
+- The YAML provider's `${VAR}` substitution is the only path for env vars to
+  influence values defined in the YAML file.
+- Env vars not represented in the YAML file (e.g. `OTEL_SERVICE_NAME` when
+  `resource.attributes.service.name` is absent from the YAML) should still be
+  available via the standard env var provider for properties that the YAML file
+  does not define. The priority model should be: YAML-defined values (with
+  `${VAR}` substitution already applied) take precedence over env var fallbacks
+  for the same logical property.
+
+**When `OTEL_CONFIG_FILE` is not set:** The current behaviour is unchanged.
+The `EnvironmentVariablesConfigurationProvider` remains the primary source for
+`OTEL_*` keys.
+
+#### I.1.4 Testing Considerations
+
+The substitution implementation needs test coverage for:
+
+- Basic substitution (`${VAR}` with defined and undefined vars)
+- Default values (`${VAR:-fallback}`)
+- Escape sequences (`$$`, `$$$`, `$$$$`)
+- Type preservation after substitution (bool, int, float, string)
+- Structure injection prevention (env var containing newlines/colons)
+- Non-recursive substitution (`${VAR}` where VAR's value contains `${...}`)
+- Invalid substitution references producing parse errors
+- The `env:` prefix variant
+
+The spec provides a comprehensive examples table that can be translated
+directly into test cases.
+
+### I.2 Schema Validation Requirements
+
+The spec requires two distinct validation layers, only one of which the current
+design addresses.
+
+#### I.2.1 Two-Layer Validation Model
+
+| Layer | What it validates | When it runs | Current status |
+| --- | --- | --- | --- |
+| **Schema validation** | YAML structure conforms to the config data model: required fields present, types correct, no unknown keys, valid nesting | During Parse, before values enter `IConfiguration` | **Not addressed** |
+| **Options validation** | Individual option values are semantically valid: port in range, endpoint is valid URI, sampler arg in `[0.0, 1.0]` | During `DelegatingOptionsFactory.Create`, via `IValidateOptions<T>` | Addressed in [Risk 1.1](configuration-analysis-risks.md#11-options-validation-is-completely-absent) (Step 0a) |
+
+Both layers are needed. Schema validation catches structural problems early
+(before any SDK component construction). Options validation catches semantic
+problems that schema alone cannot express.
+
+#### I.2.2 Schema Validation Implementation
+
+The spec's configuration data model is defined as a [JSON
+Schema](https://github.com/open-telemetry/opentelemetry-configuration). Schema
+validation can be implemented by:
+
+1. Deserializing the YAML file into a JSON-compatible DOM
+2. Validating against the published JSON Schema (or a compiled representation)
+3. Reporting all violations with file location context (line/column) before
+   any `IConfiguration` projection
+
+For Option C (typed model for Parse, `IConfiguration` for Create -- see
+[SS5.1.1](configuration-analysis.md#511-in-memory-configuration-model)), schema
+validation falls out naturally from typed deserialization: if the YAML cannot
+be deserialized into the typed `OpenTelemetryConfiguration` model, Parse returns
+an error.
+
+For Option A (`IConfiguration` only), schema validation must be a separate step
+before or during the YAML-to-`IConfiguration` projection.
+
+#### I.2.3 Unknown Key Handling
+
+The spec's JSON Schema defines the valid set of keys. The YAML provider should
+detect and report unknown keys rather than silently ignoring them, since
+unknown keys almost always indicate typos:
+
+```yaml
+tracer_provider:
+  processors:
+    - batch:
+        exporter:
+          otlp:
+            endpont: https://collector.example.com:4317  # typo: "endpont"
+```
+
+Without schema validation, `endpont` silently becomes an unused
+`IConfiguration` key and the endpoint defaults to `localhost:4317`.
+
+### I.3 `file_format` Versioning
+
+The spec requires a `file_format` field at the root of every configuration
+file:
+
+```yaml
+file_format: "0.4"
+sdk:
+  traces:
+    sampler:
+      type: parentbased_traceidratio
+      arg: 0.5
+```
+
+#### I.3.1 Requirements
+
+- The YAML `IConfigurationProvider` must read `file_format` before processing
+  the rest of the file
+- If `file_format` is missing, Parse must return an error
+- If `file_format` specifies a version the SDK does not support, Parse must
+  return an error with a clear message identifying supported versions
+- The SDK should document which `file_format` versions it supports per release
+
+#### I.3.2 Version Compatibility Policy
+
+The `file_format` version follows the `opentelemetry-configuration` repository's
+[versioning
+policy](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/VERSIONING.md).
+The SDK should support:
+
+- The current stable `file_format` version
+- Optionally, a limited set of prior versions with documented deprecation
+  timeline
+
+The version check should happen as the first step in Parse, before env var
+substitution or schema validation, so that version-incompatible files fail
+immediately with an actionable error.