diff --git a/.bumpy/land-pending-fixes.md b/.bumpy/land-pending-fixes.md new file mode 100644 index 000000000..4393d4111 --- /dev/null +++ b/.bumpy/land-pending-fixes.md @@ -0,0 +1,6 @@ +--- +varlock: patch +env-spec-language: patch +--- + +Fix enum/url/ip/md5/port coercion, imported @currentEnv (#428), and ServerResponse.end hang (#897) diff --git a/packages/varlock-website/src/content/docs/guides/import.mdx b/packages/varlock-website/src/content/docs/guides/import.mdx index bbe87d33c..3faaf8826 100644 --- a/packages/varlock-website/src/content/docs/guides/import.mdx +++ b/packages/varlock-website/src/content/docs/guides/import.mdx @@ -166,7 +166,7 @@ Meaning if there was a value for `ITEM` in all 4 files, the final value used wou ### `environment flag "..." must be defined within this schema` -If you use [`@currentEnv`](/reference/root-decorators/#currentenv) to point at a variable (e.g. `# @currentEnv=$DEPLOY_ENV`) and that variable is only brought in via a **partial** `@import()`, varlock validates the env flag during schema initialization, before imported values are merged. The flag must be **defined in the same `.env.schema` file** that declares `@currentEnv`, not only in an imported file. +If you use [`@currentEnv`](/reference/root-decorators/#currentenv) to point at a variable (e.g. `# @currentEnv=$DEPLOY_ENV`), that item must either be defined in the same file or brought in by `@import`. A partial import must include the flag in `pick=[...]` (or not omit it). This commonly appears in monorepos when a sub-package imports shared keys from a parent schema: @@ -177,24 +177,18 @@ This commonly appears in monorepos when a sub-package imports shared keys from a MY_SERVICE_URL=... ``` -Running `varlock load` fails with: +That works: `DEPLOY_ENV` arrives via the import, and varlock uses it to load `.env.` files after imports finish. + +If the pick list omits the flag, `varlock load` fails with: ```txt -environment flag "DEPLOY_ENV" must be defined within this schema +environment flag "DEPLOY_ENV" must be defined within this schema or imported via @import ``` **Fixes:** -- Define the env flag locally in the file that uses `@currentEnv`, even if the value comes from elsewhere: - -```env-spec title=".env.schema (sub-package)" -# @currentEnv=$DEPLOY_ENV -# @import(../../../, pick=[DEPLOY_ENV, AWS_REGION]) -# --- -DEPLOY_ENV= -MY_SERVICE_URL=... -``` - +- Add the flag to the import filter: `pick=[DEPLOY_ENV, ...]` +- Define the env flag locally in the file that uses `@currentEnv` - Move `@currentEnv` to the shared schema where the flag is already defined - Import the full directory (omit the key list) if the sub-package should inherit the parent's `@currentEnv` handling diff --git a/packages/varlock-website/src/content/docs/guides/secrets.mdx b/packages/varlock-website/src/content/docs/guides/secrets.mdx index f4ff8717f..337cb4435 100644 --- a/packages/varlock-website/src/content/docs/guides/secrets.mdx +++ b/packages/varlock-website/src/content/docs/guides/secrets.mdx @@ -214,7 +214,7 @@ To disable runtime log redaction, set the [`@redactLogs`](/reference/root-decora _Only available in JavaScript/Node.js projects using varlock's runtime integrations._ -Varlock scans outgoing HTTP responses at runtime to detect if any sensitive values are being accidentally sent to clients. If a leak is detected, varlock throws an error with a detailed diagnostic message including the config item key and where the leak was detected. +Varlock scans outgoing HTTP responses at runtime to detect if any sensitive values are being accidentally sent to clients. If a leak is detected, varlock throws an error with a detailed diagnostic message including the config item key and where the leak was detected. On `ServerResponse.end` (for example Next.js Pages Router `res.json()`), the response is finished with a 500 (or the socket is destroyed if headers were already sent) so the client is not left hanging. This works by patching: - **Node.js `ServerResponse`**: intercepts `write()` and `end()` calls, scanning text and JSON response bodies (including gzip-compressed responses) diff --git a/packages/varlock-website/src/content/docs/reference/data-types.mdx b/packages/varlock-website/src/content/docs/reference/data-types.mdx index 737768745..93ab33c05 100644 --- a/packages/varlock-website/src/content/docs/reference/data-types.mdx +++ b/packages/varlock-website/src/content/docs/reference/data-types.mdx @@ -166,7 +166,7 @@ MY_BOOL=true **Options:** - `prependHttps` (boolean): Automatically prepend "https://" if no protocol is specified - `allowedProtocols` (string[]): List of allowed protocols. Protocol names are case-insensitive and can include the trailing colon. If omitted, any valid URL protocol is allowed -- `allowedDomains` (string[]): List of allowed domains +- `allowedDomains` (string[] or comma-separated string): List of allowed hosts. A quoted comma-string (`allowedDomains="a.com,b.com"`) is treated as a host list, not a substring match - `noTrailingSlash` (boolean): Disallow a trailing slash on the URL path (except root `/`) - `matches` (string|RegExp): Regular expression pattern the full URL must match. Use `/pattern/flags` syntax or a quoted string pattern (see [regex-like strings](/reference/functions#regex-like-strings)) @@ -207,13 +207,18 @@ DB_HOST=10.0.3.12
### `enum` -Checks a value is contained in a list of possible values - it must match one exactly. Members can also be sourced from other items (see [Dynamic type options](#dynamic-type-options)). +Checks a value is contained in a list of possible values. It must match one exactly. Members can also be sourced from other items (see [Dynamic type options](#dynamic-type-options)). + +`process.env` and `overrideValues` are always strings. Numeric and boolean members still match those overrides (`LEVEL=1`, `FLAG=true`) after coercion. **NOTE** - this is the only type that cannot be used without any additional arguments ```env-spec # @type=enum(development, staging, production) ENV=development + +# @type=enum(1, 2, 3) +LEVEL=2 ```
@@ -230,7 +235,7 @@ MY_EMAIL=User@Example.com
### `port` -Checks for valid port number. Coerces to a number. +Checks for a valid integer port number (0-65535). Coerces to a number. Fractional values like `80.5` are rejected. **Options:** - `min` (number): Minimum port number (default: 0) @@ -244,7 +249,7 @@ MY_PORT=3000
### `ip` -Checks for a valid [IP address](https://en.wikipedia.org/wiki/IP_address). +Checks for a valid [IP address](https://en.wikipedia.org/wiki/IP_address). IPv6 accepts IPv4-mapped addresses such as `::ffff:192.168.1.1`. **Options:** - `version` (`4|6`): IPv4 or IPv6 @@ -253,6 +258,9 @@ Checks for a valid [IP address](https://en.wikipedia.org/wiki/IP_address). ```env-spec # @type=ip(version=4, normalize=true) MY_IP=192.168.1.1 + +# @type=ip(version=6) +MAPPED=::ffff:192.168.1.1 ```
@@ -286,7 +294,8 @@ MY_UUID=123e4567-e89b-12d3-a456-426614174000
### `md5` -Checks for valid [MD5 hash](https://en.wikipedia.org/wiki/MD5). +Checks for a valid [MD5 hash](https://en.wikipedia.org/wiki/MD5) (32 hex digits). Uppercase hex is accepted and normalized to lowercase. + ```env-spec # @type=md5 MY_HASH=d41d8cd98f00b204e9800998ecf8427e diff --git a/packages/varlock-website/src/content/docs/reference/root-decorators.mdx b/packages/varlock-website/src/content/docs/reference/root-decorators.mdx index e4888d5c1..4ff688320 100644 --- a/packages/varlock-website/src/content/docs/reference/root-decorators.mdx +++ b/packages/varlock-website/src/content/docs/reference/root-decorators.mdx @@ -31,7 +31,7 @@ and also may affect other dynamic behaviour in your schema, such as the [`forEnv - It _must_ be set to a simple reference to a single config item (e.g. `$APP_ENV`). - This decorator should only be set in your `.env.schema` file. -- The referenced item _must_ be defined within the same file. +- The referenced item must be defined in the same file, or brought in by `@import` (and included by any `pick`/`omit` filter on that import). - This will override the `--env` CLI flag if it is set. - We do not recommend using `NODE_ENV` as your environment flag, as it has other implications, and is often set out of your control. @@ -43,6 +43,14 @@ See [environments guide](/guides/environments) for more info. # @type=enum(dev, preview, prod, test) APP_ENV=dev ``` + +In a monorepo, the flag can live in a shared schema and be imported: + +```env-spec +# @currentEnv=$DEPLOY_ENV +# @import(../../../.env.schema, pick=[DEPLOY_ENV]) +# --- +```
diff --git a/packages/varlock/src/env-graph/lib/data-source.ts b/packages/varlock/src/env-graph/lib/data-source.ts index 1817a979f..ff919dfbf 100644 --- a/packages/varlock/src/env-graph/lib/data-source.ts +++ b/packages/varlock/src/env-graph/lib/data-source.ts @@ -4,8 +4,10 @@ import path from 'node:path'; import _ from '@env-spec/utils/my-dash'; import { tryCatch } from '@env-spec/utils/try-catch'; import { + ParsedEnvSpecArrayLiteral, ParsedEnvSpecDecorator, ParsedEnvSpecDecoratorComment, ParsedEnvSpecFile, - ParsedEnvSpecFunctionCall, ParsedEnvSpecStaticValue, parseEnvSpecDotEnvFile, + ParsedEnvSpecFunctionCall, ParsedEnvSpecKeyValuePair, ParsedEnvSpecStaticValue, + parseEnvSpecDotEnvFile, } from '@env-spec/parser'; import { ConfigItem, type ConfigItemDef } from './config-item'; @@ -35,6 +37,66 @@ export function keyPassesImportFilter( return keyMatchesFilter(key, importFilter); } +/** + * Peek unprocessed `@import(...)` decorators on `source` to see if any would statically + * bring in `key`. Used during finishInit so `@currentEnv=$FLAG` can reference a flag that + * arrives via import (imports are processed later). + */ +export function importStaticallyProvidesKey(source: EnvGraphDataSource, key: string): boolean { + for (const importDec of source.getRootDecFns('import')) { + const args = importDec.parsedDecorator.bareFnArgs; + if (!args) continue; + + let enabled: boolean | 'dynamic' = true; + let pickPatterns: Array | undefined; + let omitPatterns: Array | undefined; + const positionalKeys: Array = []; + let sawPath = false; + + for (const arg of args.values) { + if (arg instanceof ParsedEnvSpecKeyValuePair) { + if (arg.key === 'enabled') { + if (arg.value instanceof ParsedEnvSpecStaticValue) { + enabled = arg.value.value === true; + } else { + enabled = 'dynamic'; + } + } else if (arg.key === 'pick' && arg.value instanceof ParsedEnvSpecArrayLiteral) { + pickPatterns = arg.value.simplifiedValue.filter((v): v is string => typeof v === 'string' && !!v.trim()); + } else if (arg.key === 'omit' && arg.value instanceof ParsedEnvSpecArrayLiteral) { + omitPatterns = arg.value.simplifiedValue.filter((v): v is string => typeof v === 'string' && !!v.trim()); + } + } else if (arg instanceof ParsedEnvSpecStaticValue) { + // first positional arg is the import path; later ones are deprecated key allowlist + if (!sawPath) { + sawPath = true; + } else if (typeof arg.value === 'string' && arg.value.trim()) { + positionalKeys.push(arg.value.trim()); + } + } + } + + // static enabled=false: this import will not run + if (enabled === false) continue; + + if (pickPatterns?.length) { + if (keyMatchesFilter(key, { mode: 'pick', patterns: pickPatterns })) return true; + continue; + } + if (omitPatterns?.length) { + if (keyMatchesFilter(key, { mode: 'omit', patterns: omitPatterns })) return true; + continue; + } + if (positionalKeys.length) { + if (positionalKeys.includes(key)) return true; + continue; + } + // full import (no pick/omit/positional filter) brings every key + return true; + } + return false; +} + const DATA_SOURCE_TYPES = Object.freeze({ schema: { fileSuffixes: ['schema'], @@ -317,6 +379,14 @@ export abstract class EnvGraphDataSource { // For files, @currentEnv won't take effect and forEnv will fall back to parent's env setting if (this.isPartialImport && !this.isKeyImported(envFlagItemKey)) { skipCurrentEnvProcessing = true; + } else if ( + // Flag arrives via @import later. Do not process ref() yet (it would SchemaError + // "invalid dependency" and mark this source invalid, which skips _processImports). + !this.configItemDefs[envFlagItemKey] + && !isBuiltinVar(envFlagItemKey) + && importStaticallyProvidesKey(this, envFlagItemKey) + ) { + skipCurrentEnvProcessing = true; } } } @@ -342,8 +412,14 @@ export abstract class EnvGraphDataSource { } if (envFlagItemKey) { - if (!this.configItemDefs[envFlagItemKey] && !isBuiltinVar(envFlagItemKey)) { - this._errors.push(new LoadingError(`environment flag "${envFlagItemKey}" must be defined within this schema`)); + const definedLocally = !!this.configItemDefs[envFlagItemKey] || isBuiltinVar(envFlagItemKey); + // Flag may arrive later via @import. Allow that without erroring or early-returning + // (early return used to skip @defaultSensitive processing and cascade into a crash). + const providedByImport = !definedLocally && importStaticallyProvidesKey(this, envFlagItemKey); + if (!definedLocally && !providedByImport) { + this._errors.push(new LoadingError( + `environment flag "${envFlagItemKey}" must be defined within this schema or imported via @import`, + )); return; } @@ -353,7 +429,7 @@ export abstract class EnvGraphDataSource { } // Always set the envFlagKey so parent directories can check it - // (even if we're skipping processing for a file partial import) + // (even if we're skipping processing for a file partial import, or waiting on @import) this.setEnvFlag(envFlagItemKey); } @@ -965,6 +1041,9 @@ export class DirectoryDataSource extends EnvGraphDataSource { if (!envFlagItem.resolvedValue) await envFlagItem.earlyResolve(); return { env: envFlagItem.resolvedValue?.toString(), fromFallback: false }; } + // Schema declared @currentEnv=$FLAG but FLAG is not in the graph yet (still waiting on + // @import). Do not fall back to parent/CLI env, or we would load the wrong .env.* files. + return { env: undefined, fromFallback: false }; } // Fall back to parent chain or fallback value const fromEnvFlagItem = !!this.envFlagConfigItem; @@ -1029,6 +1108,14 @@ export class DirectoryDataSource extends EnvGraphDataSource { for (const source of envSources) { await source._processImports(); } + } else if (this.schemaDataSource?._envFlagKey) { + const envFlagKey = this.schemaDataSource._envFlagKey; + if (!this.graph.configSchema[envFlagKey]) { + this._errors.push(new LoadingError( + `environment flag "${envFlagKey}" was expected from @import but was not provided. ` + + 'Include it in pick=[...] (or omit filters), or define it in this schema.', + )); + } } } } diff --git a/packages/varlock/src/env-graph/lib/data-types.ts b/packages/varlock/src/env-graph/lib/data-types.ts index 2ddab245c..7495a0182 100644 --- a/packages/varlock/src/env-graph/lib/data-types.ts +++ b/packages/varlock/src/env-graph/lib/data-types.ts @@ -421,12 +421,21 @@ const UrlDataType = createEnvGraphDataType( } } } - if ( - settings?.allowedDomains && !settings.allowedDomains.includes(url.host.toLowerCase()) - ) { - errors.push(new ValidationError(`Domain (${url.host}) is not in allowed list: ${settings.allowedDomains.join(',')}`)); + // allowedDomains may arrive as a comma-string (`"a.com,b.com"`) from schema + // syntax, or as a real array. Normalize before membership checks: string + // `.includes` is substring match and would allow e.g. "ample.com" for "example.com", + // and `.join` on a string throws when building the rejection message. + const allowedDomains = (() => { + const raw = settings?.allowedDomains as Array | string | undefined; + if (!raw) return [] as Array; + const list = Array.isArray(raw) ? raw : String(raw).split(','); + return list.map((d) => d.trim().toLowerCase()).filter(Boolean); + })(); + if (allowedDomains.length && !allowedDomains.includes(url.host.toLowerCase())) { + errors.push(new ValidationError(`Domain (${url.host}) is not in allowed list: ${allowedDomains.join(',')}`)); } - if (settings?.noTrailingSlash && val.endsWith('/')) { + // Docs and vscode exempt root pathname `/` (`https://example.com/` is OK). + if (settings?.noTrailingSlash && url.pathname.endsWith('/') && url.pathname !== '/') { errors.push(new ValidationError('URL must not have a trailing slash')); } if (settings?.matches) { @@ -584,8 +593,21 @@ const EnumDataType = createEnvGraphDataType( icon: 'material-symbols-light:category', // a few shapes... not sure about this one coercedType: { enum: enumOptions }, coerce(val) { - if (_.isString(val) || _.isNumber(val) || _.isBoolean(val)) return val; - return new CoercionError('Value must be a string, number, or boolean'); + if (_.isNumber(val) || _.isBoolean(val)) return val; + if (!_.isString(val)) { + return new CoercionError('Value must be a string, number, or boolean'); + } + // Exact string member (e.g. enum(dev, prod) + "dev") + if (enumOptions.includes(val)) return val; + // process.env / overrideValues are always strings. Schema file values like + // LEVEL=2 are auto-coerced to numbers by the parser, but CI overrides stay + // as "2" / "true" and must still match numeric/boolean members. + for (const opt of enumOptions) { + if (_.isNumber(opt) && String(opt) === val) return opt; + if (opt === true && val === 'true') return true; + if (opt === false && val === 'false') return false; + } + return val; }, validate(val) { const possibleValues: Array = enumOptions || []; @@ -622,7 +644,7 @@ const EmailDataType = createEnvGraphDataType( }), ); -const IP_V6_ADDRESS_REGEX = /^(?:(?:[a-fA-F\d]{1,4}:){7}(?:[a-fA-F\d]{1,4}|:)|(?:[a-fA-F\d]{1,4}:){6}(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)(?:\\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)){3}|:[a-fA-F\d]{1,4}|:)|(?:[a-fA-F\d]{1,4}:){5}(?::(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)(?:\\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)){3}|(?::[a-fA-F\d]{1,4}){1,2}|:)|(?:[a-fA-F\d]{1,4}:){4}(?:(?::[a-fA-F\d]{1,4}){0,1}:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)(?:\\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)){3}|(?::[a-fA-F\d]{1,4}){1,3}|:)|(?:[a-fA-F\d]{1,4}:){3}(?:(?::[a-fA-F\d]{1,4}){0,2}:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)(?:\\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)){3}|(?::[a-fA-F\d]{1,4}){1,4}|:)|(?:[a-fA-F\d]{1,4}:){2}(?:(?::[a-fA-F\d]{1,4}){0,3}:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)(?:\\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)){3}|(?::[a-fA-F\d]{1,4}){1,5}|:)|(?:[a-fA-F\d]{1,4}:){1}(?:(?::[a-fA-F\d]{1,4}){0,4}:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)(?:\\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)){3}|(?::[a-fA-F\d]{1,4}){1,6}|:)|(?::(?:(?::[a-fA-F\d]{1,4}){0,5}:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)(?:\\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)){3}|(?::[a-fA-F\d]{1,4}){1,7}|:)))(?:%[0-9a-zA-Z]{1,})?$/; +const IP_V6_ADDRESS_REGEX = /^(?:(?:[a-fA-F\d]{1,4}:){7}(?:[a-fA-F\d]{1,4}|:)|(?:[a-fA-F\d]{1,4}:){6}(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)){3}|:[a-fA-F\d]{1,4}|:)|(?:[a-fA-F\d]{1,4}:){5}(?::(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)){3}|(?::[a-fA-F\d]{1,4}){1,2}|:)|(?:[a-fA-F\d]{1,4}:){4}(?:(?::[a-fA-F\d]{1,4}){0,1}:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)){3}|(?::[a-fA-F\d]{1,4}){1,3}|:)|(?:[a-fA-F\d]{1,4}:){3}(?:(?::[a-fA-F\d]{1,4}){0,2}:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)){3}|(?::[a-fA-F\d]{1,4}){1,4}|:)|(?:[a-fA-F\d]{1,4}:){2}(?:(?::[a-fA-F\d]{1,4}){0,3}:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)){3}|(?::[a-fA-F\d]{1,4}){1,5}|:)|(?:[a-fA-F\d]{1,4}:){1}(?:(?::[a-fA-F\d]{1,4}){0,4}:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)){3}|(?::[a-fA-F\d]{1,4}){1,6}|:)|(?::(?:(?::[a-fA-F\d]{1,4}){0,5}:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]\d|\d)){3}|(?::[a-fA-F\d]{1,4}){1,7}|:)))(?:%[0-9a-zA-Z]{1,})?$/; const IpAddressDataType = createEnvGraphDataType( (settings?: { version?: 4 | 6, @@ -660,9 +682,18 @@ const PortDataType = createEnvGraphDataType( if (rawVal.includes('.')) throw new CoercionError('Port number must be an integer'); if (rawVal.includes('e')) throw new CoercionError('Port number should be an integer, not in exponential notation'); } - return coerceToNumber(rawVal); + const numVal = coerceToNumber(rawVal); + // Unquoted schema values like 80.5 are already numbers after parse auto-coerce; + // the string '.' check above does not cover that path. + if (!Number.isInteger(numVal)) { + throw new CoercionError('Port number must be an integer'); + } + return numVal; }, validate(val) { + if (!Number.isInteger(val)) { + return new ValidationError('Port number must be an integer'); + } if (settings?.min !== undefined && val < settings?.min) { return new ValidationError(`Min value is ${settings?.min}`); } @@ -723,12 +754,16 @@ const UuidDataType = createEnvGraphDataType({ }, }); -const MD5_REGEX = /^[a-f0-9]{32}$/; +const MD5_REGEX = /^[a-f0-9]{32}$/i; const Md5DataType = createEnvGraphDataType({ name: 'md5', typeDescription: 'MD5 hash string', // A deterministic, unique, valid 32-hex string derived from the seed. generatePlaceholder: (seed) => hexFromSeed(seed).slice(0, 32), + coerce(rawVal) { + // Accept uppercase hex (common from tools) and normalize like typical hash handling + return coerceToString(rawVal).toLowerCase(); + }, validate(val) { const result = MD5_REGEX.test(val); if (result) return true; diff --git a/packages/varlock/src/env-graph/test/data-types.test.ts b/packages/varlock/src/env-graph/test/data-types.test.ts index 3a8fea594..8746b2e81 100644 --- a/packages/varlock/src/env-graph/test/data-types.test.ts +++ b/packages/varlock/src/env-graph/test/data-types.test.ts @@ -10,8 +10,12 @@ import { describe, it, expect } from 'vitest'; import { outdent } from 'outdent'; import { DotEnvFileDataSource, EnvGraph, CoercionError } from '../index'; -async function loadAndResolve(envFileContent: string) { +async function loadAndResolve( + envFileContent: string, + opts?: { overrideValues?: Record }, +) { const g = new EnvGraph(); + if (opts?.overrideValues) g.overrideValues = opts.overrideValues; const testDataSource = new DotEnvFileDataSource('.env.schema', { overrideContents: outdent` # @defaultRequired=false @@ -47,6 +51,62 @@ describe('number data type - Infinity coercion', () => { }); }); +describe('enum data type - process.env string overrides', () => { + it('accepts numeric enum members from schema file values', async () => { + const g = await loadAndResolve(outdent` + # @type=enum(1, 2, 3) + LEVEL=2 + `); + expect(g.configSchema.LEVEL.isValid).toBe(true); + expect(g.configSchema.LEVEL.resolvedValue).toBe(2); + }); + + it('accepts numeric enum members from string overrides', async () => { + const g = await loadAndResolve(outdent` + # @type=enum(1, 2, 3) + LEVEL=2 + `, { overrideValues: { LEVEL: '1' } }); + expect(g.configSchema.LEVEL.isValid).toBe(true); + expect(g.configSchema.LEVEL.resolvedValue).toBe(1); + }); + + it('accepts boolean enum members from string overrides', async () => { + const g = await loadAndResolve(outdent` + # @type=enum(true, false) + FLAG=false + `, { overrideValues: { FLAG: 'true' } }); + expect(g.configSchema.FLAG.isValid).toBe(true); + expect(g.configSchema.FLAG.resolvedValue).toBe(true); + }); +}); + +describe('port data type', () => { + it('accepts integer ports', async () => { + const g = await loadAndResolve(outdent` + # @type=port + P=8080 + `); + expect(g.configSchema.P.isValid).toBe(true); + expect(g.configSchema.P.resolvedValue).toBe(8080); + }); + + it('rejects non-integer numeric ports from schema auto-coerce', async () => { + const g = await loadAndResolve(outdent` + # @type=port + P=80.5 + `); + expect(g.configSchema.P.isValid).toBe(false); + }); + + it('rejects non-integer string ports', async () => { + const g = await loadAndResolve(outdent` + # @type=port + P="80.5" + `); + expect(g.configSchema.P.isValid).toBe(false); + }); +}); + describe('url data type', () => { describe('prependHttps', () => { it('prepends https:// when missing', async () => { @@ -121,6 +181,34 @@ describe('url data type', () => { }); }); + describe('allowedDomains', () => { + it('accepts a host listed in a comma-string allowlist', async () => { + const g = await loadAndResolve(outdent` + # @type=url(allowedDomains="example.com,api.example.com") + MY_URL=https://api.example.com/v1 + `); + expect(g.configSchema.MY_URL.isValid).toBe(true); + }); + + it('rejects a host that is only a substring of an allowlist entry', async () => { + const g = await loadAndResolve(outdent` + # @type=url(allowedDomains="example.com") + MY_URL=https://ample.com/ + `); + expect(g.configSchema.MY_URL.isValid).toBe(false); + expect(g.configSchema.MY_URL.errors[0]?.message).toMatch(/not in allowed list/); + }); + + it('rejects a disallowed host without throwing on the error message', async () => { + const g = await loadAndResolve(outdent` + # @type=url(allowedDomains="example.com") + MY_URL=https://evil.com/ + `); + expect(g.configSchema.MY_URL.isValid).toBe(false); + expect(g.configSchema.MY_URL.errors[0]?.message).toContain('example.com'); + }); + }); + describe('noTrailingSlash', () => { it('accepts url without trailing slash', async () => { const g = await loadAndResolve(outdent` @@ -138,12 +226,12 @@ describe('url data type', () => { expect(g.configSchema.MY_URL.isValid).toBe(false); }); - it('rejects bare domain with trailing slash', async () => { + it('accepts root URL with trailing slash', async () => { const g = await loadAndResolve(outdent` # @type=url(noTrailingSlash=true) MY_URL=https://example.com/ `); - expect(g.configSchema.MY_URL.isValid).toBe(false); + expect(g.configSchema.MY_URL.isValid).toBe(true); }); it('accepts bare domain without trailing slash', async () => { @@ -245,6 +333,52 @@ describe('url data type - path values', () => { }); }); +describe('ip data type', () => { + it('accepts IPv4', async () => { + const g = await loadAndResolve(outdent` + # @type=ip(version=4) + IP=192.168.1.1 + `); + expect(g.configSchema.IP.isValid).toBe(true); + }); + + it('accepts plain IPv6', async () => { + const g = await loadAndResolve(outdent` + # @type=ip(version=6) + IP=2001:db8::1 + `); + expect(g.configSchema.IP.isValid).toBe(true); + }); + + it('accepts IPv4-mapped IPv6 addresses', async () => { + const g = await loadAndResolve(outdent` + # @type=ip(version=6) + IP=::ffff:192.168.1.1 + `); + expect(g.configSchema.IP.isValid).toBe(true); + }); +}); + +describe('md5 data type', () => { + it('accepts lowercase md5', async () => { + const g = await loadAndResolve(outdent` + # @type=md5 + H=d41d8cd98f00b204e9800998ecf8427e + `); + expect(g.configSchema.H.isValid).toBe(true); + expect(g.configSchema.H.resolvedValue).toBe('d41d8cd98f00b204e9800998ecf8427e'); + }); + + it('accepts uppercase md5 and normalizes to lowercase', async () => { + const g = await loadAndResolve(outdent` + # @type=md5 + H=D41D8CD98F00B204E9800998ECF8427E + `); + expect(g.configSchema.H.isValid).toBe(true); + expect(g.configSchema.H.resolvedValue).toBe('d41d8cd98f00b204e9800998ecf8427e'); + }); +}); + describe('domain data type', () => { it('accepts a basic domain', async () => { const g = await loadAndResolve(outdent` diff --git a/packages/varlock/src/env-graph/test/environments.test.ts b/packages/varlock/src/env-graph/test/environments.test.ts index 124cc1537..b5ebff7f2 100644 --- a/packages/varlock/src/env-graph/test/environments.test.ts +++ b/packages/varlock/src/env-graph/test/environments.test.ts @@ -16,6 +16,88 @@ describe('@currentEnv and .env.* file loading logic', () => { expectError: true, })); + // #428: @currentEnv may reference a flag key brought in via @import + test('@currentEnv can reference a key imported via pick=[]', envFilesTest({ + files: { + '.env.schema': outdent` + # @currentEnv=$DEPLOY_ENV + # @import(./.env.shared, pick=[DEPLOY_ENV]) + # --- + `, + '.env.shared': outdent` + # --- + DEPLOY_ENV=dev + `, + '.env.dev': outdent` + ITEM1=from-dev + `, + }, + expectValues: { + DEPLOY_ENV: 'dev', + ITEM1: 'from-dev', + }, + })); + + test('@currentEnv can reference a key imported via pick glob', envFilesTest({ + files: { + '.env.schema': outdent` + # @currentEnv=$DEPLOY_ENV + # @import(./.env.shared, pick=[DEPLOY_*]) + # --- + `, + '.env.shared': outdent` + # --- + DEPLOY_ENV=staging + DEPLOY_REGION=us + `, + '.env.staging': outdent` + ITEM1=from-staging + `, + }, + expectValues: { + DEPLOY_ENV: 'staging', + DEPLOY_REGION: 'us', + ITEM1: 'from-staging', + }, + })); + + test('@currentEnv can reference a key imported via deprecated positional keys', envFilesTest({ + files: { + '.env.schema': outdent` + # @currentEnv=$DEPLOY_ENV + # @import(./.env.shared, DEPLOY_ENV) + # --- + `, + '.env.shared': outdent` + # --- + DEPLOY_ENV=dev + `, + '.env.dev': outdent` + ITEM1=from-dev-positional + `, + }, + expectValues: { + DEPLOY_ENV: 'dev', + ITEM1: 'from-dev-positional', + }, + })); + + test('@currentEnv errors when imported pick list omits the flag key', envFilesTest({ + files: { + '.env.schema': outdent` + # @currentEnv=$DEPLOY_ENV + # @import(./.env.shared, pick=[OTHER]) + # --- + `, + '.env.shared': outdent` + # --- + DEPLOY_ENV=dev + OTHER=x + `, + }, + expectError: true, + })); + test('all .env.* files are loaded in correct precedence order', envFilesTest({ files: { '.env.schema': outdent` diff --git a/packages/varlock/src/runtime/patch-server-response.ts b/packages/varlock/src/runtime/patch-server-response.ts index 492f07167..79b893a56 100644 --- a/packages/varlock/src/runtime/patch-server-response.ts +++ b/packages/varlock/src/runtime/patch-server-response.ts @@ -183,6 +183,31 @@ function scanChunk(state: ScanState, chunkStr: string, o: { return emit; } +/** + * Leak detection on `end` used to throw before the original `end` ran, which left + * the HTTP client hanging (Next.js Pages Router `res.json()`). Finish the response + * first, then rethrow so callers still see the leak error. + */ +function finishResponseOnLeak( + res: ServerResponse, + originalEnd: typeof ServerResponse.prototype.end, + err: unknown, +): never { + if (!res.headersSent) { + res.statusCode = 500; + res.setHeader('Content-Type', 'text/plain; charset=utf-8'); + try { + // @ts-ignore Node's end overloads confuse Function.call + originalEnd.call(res, 'Internal Server Error'); + } catch { + res.destroy(); + } + } else { + res.destroy(); + } + throw err; +} + export function patchGlobalServerResponse(opts?: { ignoreUrlPatterns?: Array, redactInsteadOfThrow?: boolean, @@ -386,10 +411,14 @@ export function patchGlobalServerResponse(opts?: { } if (decompressed !== undefined) { // compressed output can't be scrubbed, so a detected leak always throws (see write above) - scanForLeaks(state.carry + decodeDecompressedDelta(state, decompressed, true), { - method: 'patched ServerResponse.end', - file: (this as any).req?.url, - }); + try { + scanForLeaks(state.carry + decodeDecompressedDelta(state, decompressed, true), { + method: 'patched ServerResponse.end', + file: (this as any).req?.url, + }); + } catch (err) { + finishResponseOnLeak(this, serverResponseEnd, err); + } } // @ts-ignore return serverResponseEnd.apply(this, args); @@ -409,11 +438,16 @@ export function patchGlobalServerResponse(opts?: { if (chunkStr || state.pending) { // last chunk, so nothing can be withheld for later - it all goes out now - const emit = scanChunk(state, chunkStr, { - canHoldBack: false, - redactInsteadOfThrow: opts?.redactInsteadOfThrow, - meta: { method: 'patched ServerResponse.end', file: (this as any).req?.url }, - }); + let emit: string; + try { + emit = scanChunk(state, chunkStr, { + canHoldBack: false, + redactInsteadOfThrow: opts?.redactInsteadOfThrow, + meta: { method: 'patched ServerResponse.end', file: (this as any).req?.url }, + }); + } catch (err) { + finishResponseOnLeak(this, serverResponseEnd, err); + } // for a string (or absent) final chunk, `chunkStr` may carry a flushed decoder tail // that the outgoing chunk needs to pick up, so compare against what was actually passed let originalStr = ''; diff --git a/packages/varlock/src/runtime/test/patch-server-response.test.ts b/packages/varlock/src/runtime/test/patch-server-response.test.ts index 662b859b4..5a53681c0 100644 --- a/packages/varlock/src/runtime/test/patch-server-response.test.ts +++ b/packages/varlock/src/runtime/test/patch-server-response.test.ts @@ -152,6 +152,13 @@ describe('patched ServerResponse.end', () => { const res = makeRes({ 'content-type': 'application/json' }); expect(() => res.end(Buffer.from(JSON.stringify({ leaked: SECRET })))).toThrow(/DETECTED LEAKED SENSITIVE CONFIG/); }); + + it('finishes the response so the client does not hang when leak detection throws', () => { + const res = makeRes({ 'content-type': 'application/json' }); + expect(() => res.end(JSON.stringify({ leaked: SECRET }))).toThrow(/DETECTED LEAKED SENSITIVE CONFIG/); + expect(res.writableEnded || res.destroyed).toBe(true); + expect(res.statusCode).toBe(500); + }); }); // a scan that only ever sees one chunk at a time misses any value that straddles a boundary, diff --git a/packages/vscode-plugin/src/intellisense-catalog.ts b/packages/vscode-plugin/src/intellisense-catalog.ts index ebaaf578a..f3b5fefb0 100644 --- a/packages/vscode-plugin/src/intellisense-catalog.ts +++ b/packages/vscode-plugin/src/intellisense-catalog.ts @@ -325,8 +325,8 @@ export const DATA_TYPES: Array = [ optionSnippets: [ { name: 'prependHttps', insertText: `prependHttps=${booleanChoiceSnippet()}`, documentation: 'Automatically add `https://` when missing.' }, { name: 'allowedProtocols', insertText: 'allowedProtocols=[${1:http}, ${2:https}]', documentation: 'Restrict the URL to an allowed protocol list.' }, - { name: 'allowedDomains', insertText: 'allowedDomains=${1:"example.com"}', documentation: 'Restrict the URL host to an allowed domain list.' }, - { name: 'noTrailingSlash', insertText: `noTrailingSlash=${booleanChoiceSnippet()}`, documentation: 'Disallow a trailing slash on the URL path.' }, + { name: 'allowedDomains', insertText: 'allowedDomains=${1:"example.com"}', documentation: 'Restrict the URL host to an allowed domain list. A quoted comma-string is treated as a host list.' }, + { name: 'noTrailingSlash', insertText: `noTrailingSlash=${booleanChoiceSnippet()}`, documentation: 'Disallow a trailing slash on the URL path (root `/` is allowed).' }, { name: 'matches', insertText: 'matches=${1:"pattern"}', documentation: 'A regular expression that the full URL must match.' }, ], },