Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions plugins/agent-plugins-conformance-core/dist/serve-http.mjs

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions plugins/agent-plugins-conformance-core/src/serve-http.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ const pathname = '/conformance/mcp';
const url = `http://${host}:${port}${pathname}?value=$APC_HTTP_VALUE`;
const recoveryRoutes = new Map([
['/conformance/recovery-http-relative-url', 'recovery-http-relative-url'],
['/conformance/recovery-http-type', 'recovery-http-type'],
['/conformance/recovery-http-header-name', 'recovery-http-header-name'],
['/conformance/recovery-http-header-value', 'recovery-http-header-value'],
['/conformance/recovery-http-userinfo', 'recovery-http-userinfo'],
Expand Down
1 change: 1 addition & 0 deletions plugins/agent-plugins-conformance-recovery/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ Install it alongside the other suite plugins and follow the [run instructions](.
| `filesystem.containment.cwd-symlink-escape` | A server whose working directory escapes the plugin root through a symlink is excluded. |
| `mcp.stdio.config.unknown-field` | A server with an unknown configuration field is excluded. |
| `mcp.config.missing-type` | A server without an explicit transport type is excluded. |
| `mcp.config.legacy-http-type` | A server using the legacy `http` transport type is excluded. |
| `mcp.stdio.env.reserved-plugin-root` | A server that configures `PLUGIN_ROOT` in its `env` is excluded. |
| `mcp.stdio.env.reserved-plugin-data` | A server that configures `PLUGIN_DATA` in its `env` is excluded. |
| `mcp.streamable-http.url.relative` | An HTTP server with a scheme-relative URL (`//host/path`) is excluded. |
Expand Down
4 changes: 4 additions & 0 deletions plugins/agent-plugins-conformance-recovery/mcp.json
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,10 @@
"type": "streamable-http",
"url": "//127.0.0.1:43187/conformance/recovery-http-relative-url"
},
"recovery-http-type": {
"type": "http",
"url": "http://127.0.0.1:43187/conformance/recovery-http-type"
},
"recovery-http-header-name": {
"type": "streamable-http",
"url": "http://127.0.0.1:43187/conformance/recovery-http-header-name",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ Collect all fixture skills and MCP tools through the client's normal mechanisms.
}
```

3. Inspect the client's MCP tool catalog for the `observe` tool of each invalid Recovery server with a probe (`recovery-cwd-invalid-form`, `recovery-cwd-escape`, `recovery-cwd-data-escape`, `recovery-cwd-symlink-escape`, `recovery-unknown-field`, `recovery-missing-type`, `recovery-env-plugin-root`, `recovery-env-plugin-data`, `recovery-http-relative-url`, `recovery-http-fragment`, `recovery-http-userinfo`, `recovery-http-duplicate-headers`, `recovery-http-header-name`, `recovery-http-header-value`, `recovery-sse-relative-url`, `recovery-sse-fragment`, `recovery-sse-userinfo`, `recovery-sse-duplicate-headers`, `recovery-sse-header-name`, and `recovery-sse-header-value`). Each description identifies the fixture server ID; names may be client-namespaced. Set `advertised` to `true` if the tool is advertised, even in a partial catalog, and record that before calling it. Set `advertised` to `false` only after a known attempt to load Recovery and a usable complete client inventory that excludes the tool, including any deferred tools. For `recovery-cwd-symlink-escape`, a completed startup failure still permits `advertised: false` when that complete inventory shows no tools for the server; installation may have removed its working-directory symlink. For the other servers, leave the observation missing if startup or tool discovery fails. For every server, leave it missing if the load scope or inventory completeness is unknown, unless the tool is advertised. A failed call or a guessed tool name does not establish absence. No rejection diagnostic is required. Record each server separately through the same reporter, using its fixture server ID:
3. Inspect the client's MCP tool catalog for the `observe` tool of each invalid Recovery server with a probe (`recovery-cwd-invalid-form`, `recovery-cwd-escape`, `recovery-cwd-data-escape`, `recovery-cwd-symlink-escape`, `recovery-unknown-field`, `recovery-missing-type`, `recovery-env-plugin-root`, `recovery-env-plugin-data`, `recovery-http-type`, `recovery-http-relative-url`, `recovery-http-fragment`, `recovery-http-userinfo`, `recovery-http-duplicate-headers`, `recovery-http-header-name`, `recovery-http-header-value`, `recovery-sse-relative-url`, `recovery-sse-fragment`, `recovery-sse-userinfo`, `recovery-sse-duplicate-headers`, `recovery-sse-header-name`, and `recovery-sse-header-value`). Each description identifies the fixture server ID; names may be client-namespaced. Set `advertised` to `true` if the tool is advertised, even in a partial catalog, and record that before calling it. Set `advertised` to `false` only after a known attempt to load Recovery and a usable complete client inventory that excludes the tool, including any deferred tools. For `recovery-cwd-symlink-escape`, a completed startup failure still permits `advertised: false` when that complete inventory shows no tools for the server; installation may have removed its working-directory symlink. For the other servers, leave the observation missing if startup or tool discovery fails. For every server, leave it missing if the load scope or inventory completeness is unknown, unless the tool is advertised. A failed call or a guessed tool name does not establish absence. No rejection diagnostic is required. Record each server separately through the same reporter, using its fixture server ID:

```json
{
Expand All @@ -69,7 +69,7 @@ Collect all fixture skills and MCP tools through the client's normal mechanisms.
}
```

4. Find the `observe` tools on the Core fixture servers (`default`, `relative`, `root`, `data`, `http`, `http-redirect`, `sse`, `sse-header-precedence`, `sse-redirect`, and `sse-endpoint-origin`) and the Recovery fixture servers (`recovery-valid`, `recovery-cwd-invalid-form`, `recovery-cwd-escape`, `recovery-cwd-data-escape`, `recovery-cwd-symlink-escape`, `recovery-unknown-field`, `recovery-missing-type`, `recovery-env-plugin-root`, `recovery-env-plugin-data`, `recovery-http-relative-url`, `recovery-http-fragment`, `recovery-http-userinfo`, `recovery-http-duplicate-headers`, `recovery-http-header-name`, `recovery-http-header-value`, `recovery-sse-relative-url`, `recovery-sse-fragment`, `recovery-sse-userinfo`, `recovery-sse-duplicate-headers`, `recovery-sse-header-name`, and `recovery-sse-header-value`) through the client's normal MCP mechanism. Tool names may be client-namespaced. The remote MCP tool descriptions identify their fixture server IDs; retain that association when calling each tool so that an error can be recorded for the same server. Call each available tool with `{}`. First inspect whether the client returned a successful MCP result or a native discovery/call error. For a native remote MCP error, follow step 5 without parsing it as observation JSON. For a native stdio error, leave that server’s runtime observation missing; retain any discovery observation already recorded. For a successful result, take the observation from `structuredContent` (some clients show `structured_content`), or parse the JSON in the tool's text content. Immediately record that complete object; exclude the MCP result wrapper containing `content` or `structuredContent`. Preserve the returned paths and values exactly. The `http-redirect` and `sse-redirect` fixtures redirect requests to another local origin, and `sse-endpoint-origin` advertises a message endpoint on that origin. Do not authorize forwarding configured headers to these destinations.
4. Find the `observe` tools on the Core fixture servers (`default`, `relative`, `root`, `data`, `http`, `http-redirect`, `sse`, `sse-header-precedence`, `sse-redirect`, and `sse-endpoint-origin`) and the Recovery fixture servers (`recovery-valid`, `recovery-cwd-invalid-form`, `recovery-cwd-escape`, `recovery-cwd-data-escape`, `recovery-cwd-symlink-escape`, `recovery-unknown-field`, `recovery-missing-type`, `recovery-env-plugin-root`, `recovery-env-plugin-data`, `recovery-http-type`, `recovery-http-relative-url`, `recovery-http-fragment`, `recovery-http-userinfo`, `recovery-http-duplicate-headers`, `recovery-http-header-name`, `recovery-http-header-value`, `recovery-sse-relative-url`, `recovery-sse-fragment`, `recovery-sse-userinfo`, `recovery-sse-duplicate-headers`, `recovery-sse-header-name`, and `recovery-sse-header-value`) through the client's normal MCP mechanism. Tool names may be client-namespaced. The remote MCP tool descriptions identify their fixture server IDs; retain that association when calling each tool so that an error can be recorded for the same server. Call each available tool with `{}`. First inspect whether the client returned a successful MCP result or a native discovery/call error. For a native remote MCP error, follow step 5 without parsing it as observation JSON. For a native stdio error, leave that server’s runtime observation missing; retain any discovery observation already recorded. For a successful result, take the observation from `structuredContent` (some clients show `structured_content`), or parse the JSON in the tool's text content. Immediately record that complete object; exclude the MCP result wrapper containing `content` or `structuredContent`. Preserve the returned paths and values exactly. The `http-redirect` and `sse-redirect` fixtures redirect requests to another local origin, and `sse-endpoint-origin` advertises a message endpoint on that origin. Do not authorize forwarding configured headers to these destinations.
5. If a completed remote MCP discovery or call attempt instead produces a native error, record it as error evidence:

```json
Expand Down
1 change: 1 addition & 0 deletions plugins/agent-plugins-conformance/src/cases.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ export const CASES = Object.freeze([
['filesystem.containment.cwd-symlink-escape', 'Working-directory symlink containment', ['4.1', '7.2.1', '7.2.2']],
['mcp.stdio.config.unknown-field', 'Unknown MCP server field rejection', ['7.2.1', '7.2.2']],
['mcp.config.missing-type', 'Missing MCP transport type exclusion', ['7.2.1', '7.2.2']],
['mcp.config.legacy-http-type', 'Legacy HTTP transport type exclusion', ['7.2.1', '7.2.2']],
['mcp.stdio.env.reserved-plugin-root', 'Configured PLUGIN_ROOT exclusion', ['7.2.2', '9.2']],
['mcp.stdio.env.reserved-plugin-data', 'Configured PLUGIN_DATA exclusion', ['7.2.2', '9.2']],
['mcp.stdio.env.plugin-root', 'Plugin root environment', ['9.1']],
Expand Down
1 change: 1 addition & 0 deletions plugins/agent-plugins-conformance/src/report.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ const INVALID_STDIO_SERVERS = Object.freeze({
'recovery-env-plugin-data': 'mcp.stdio.env.reserved-plugin-data',
});
const INVALID_HTTP_SERVERS = Object.freeze({
'recovery-http-type': 'mcp.config.legacy-http-type',
'recovery-http-relative-url': 'mcp.streamable-http.url.relative',
'recovery-http-fragment': 'mcp.streamable-http.url.fragment',
'recovery-http-userinfo': 'mcp.streamable-http.url.userinfo',
Expand Down
2 changes: 1 addition & 1 deletion scripts/smoke-codex.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,7 @@ try {
for (const server of ['recovery-cwd-invalid-form', 'recovery-cwd-escape', 'recovery-cwd-data-escape', 'recovery-cwd-symlink-escape', 'recovery-unknown-field', 'recovery-missing-type',
'recovery-env-plugin-root', 'recovery-env-plugin-data',
'recovery-http-relative-url', 'recovery-http-fragment', 'recovery-http-userinfo', 'recovery-http-duplicate-headers',
'recovery-http-header-name', 'recovery-http-header-value', ...invalidSseServers]) {
'recovery-http-header-name', 'recovery-http-header-value', 'recovery-http-type', ...invalidSseServers]) {
const entry = status.data.find(({ name }) => name === server);
const advertised = entry !== undefined && Object.hasOwn(entry.tools, 'observe');
const symlinkStartupFailure = server === 'recovery-cwd-symlink-escape' &&
Expand Down
34 changes: 34 additions & 0 deletions test/http-server.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -650,6 +650,40 @@ test('standalone HTTP bundle serves independent MCP requests and stops cleanly',
}
});

await t.test('legacy HTTP type route has a valid control and a deliberately permissive failure witness', async (t) => {
const mcp = JSON.parse(await readFile(new URL('../plugins/agent-plugins-conformance-recovery/mcp.json', import.meta.url), 'utf8'));
const configured = mcp.mcpServers['recovery-http-type'];
assert.deepEqual(configured, {
type: 'http',
url: 'http://127.0.0.1:43187/conformance/recovery-http-type',
});
const normalizedForHarness = { ...configured, type: 'streamable-http' };
const client = new Client({ name: 'legacy-http-type-reference-test', version: '1' });
t.after(() => client.close());
await client.connect(new StreamableHTTPClientTransport(new URL(normalizedForHarness.url)));
assert.equal(client.getServerVersion().name, 'agent-plugins-conformance-recovery-http-type');
const tools = (await client.listTools()).tools;
assert.deepEqual(tools.map(({ name }) => name), ['observe']);
assert.match(tools[0].description, /server ID: recovery-http-type\./);
const result = await client.callTool({ name: 'observe', arguments: {} });
const observation = {
kind: 'mcp-streamable-http', server: 'recovery-http-type', serverHealthCheck: 'passed',
evidence: {
type: 'request', version: 1, pathname: '/conformance/recovery-http-type',
query: [], headers: { 'x-apc-fixture': null },
},
};
assert.equal(result.isError, undefined);
assert.deepEqual(result.structuredContent, {
kind: observation.kind, server: observation.server, evidence: observation.evidence,
});
assert.deepEqual(JSON.parse(result.content[0].text), result.structuredContent);
const report = buildReport({ schemaVersion: 1, observations: [observation] });
assert.equal(report.results.find(({ id }) => id === 'mcp.config.legacy-http-type').status, 'fail');
assert.deepEqual(report.observations, [observation]);
assert.equal(configured.type, 'http');
});

await t.test('userinfo route has a valid control and a deliberately permissive failure witness', async (t) => {
const mcp = JSON.parse(await readFile(new URL('../plugins/agent-plugins-conformance-recovery/mcp.json', import.meta.url), 'utf8'));
const configured = mcp.mcpServers['recovery-http-userinfo'];
Expand Down
20 changes: 20 additions & 0 deletions test/package.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,24 @@ test('recovery fixture contains exactly the intended invalid manifest and MCP fi
type: 'streamable-http',
url: '//127.0.0.1:43187/conformance/recovery-http-relative-url',
});
const legacyHttpType = mcp.mcpServers['recovery-http-type'];
assert.deepEqual(legacyHttpType, {
type: 'http',
url: 'http://127.0.0.1:43187/conformance/recovery-http-type',
});
const isolatedLegacyHttpType = {
$schema: mcp.$schema,
mcpServers: { 'recovery-http-type': structuredClone(legacyHttpType) },
};
assert.equal(validateMcp(isolatedLegacyHttpType), false);
isolatedLegacyHttpType.mcpServers['recovery-http-type'].type = 'streamable-http';
const { type: correctedHttpType, ...correctedHttpRemainder } =
isolatedLegacyHttpType.mcpServers['recovery-http-type'];
const { type: legacyType, ...legacyHttpRemainder } = legacyHttpType;
assert.equal(legacyType, 'http');
assert.equal(correctedHttpType, 'streamable-http');
assert.deepEqual(correctedHttpRemainder, legacyHttpRemainder);
assert.equal(validateMcp(isolatedLegacyHttpType), true, JSON.stringify(validateMcp.errors));
assert.deepEqual(mcp.mcpServers['recovery-http-header-name'], {
type: 'streamable-http',
url: 'http://127.0.0.1:43187/conformance/recovery-http-header-name',
Expand Down Expand Up @@ -199,6 +217,7 @@ test('recovery fixture contains exactly the intended invalid manifest and MCP fi
['/mcpServers/recovery-invalid', '/mcpServers/recovery-cwd-invalid-form', '/mcpServers/recovery-unknown-field',
'/mcpServers/recovery-missing-type',
'/mcpServers/recovery-http-relative-url',
'/mcpServers/recovery-http-type',
'/mcpServers/recovery-env-plugin-root', '/mcpServers/recovery-env-plugin-data']
.some((prefix) => instancePath.startsWith(prefix))),
JSON.stringify(validateMcp.errors));
Expand Down Expand Up @@ -232,6 +251,7 @@ test('recovery fixture contains exactly the intended invalid manifest and MCP fi
schemaValidMcp.mcpServers['recovery-cwd-invalid-form'].cwd = './';
schemaValidMcp.mcpServers['recovery-http-relative-url'].url =
`http:${schemaValidMcp.mcpServers['recovery-http-relative-url'].url}`;
schemaValidMcp.mcpServers['recovery-http-type'].type = 'streamable-http';
assert.equal(validateMcp(schemaValidMcp), true, JSON.stringify(validateMcp.errors));
});

Expand Down
2 changes: 1 addition & 1 deletion test/probe.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ test('copied plugin runs only MCP observation tools without node_modules', { tim
const direct = buildReport(reportInput);
assert.equal(direct.summary.fail, 0);
assert.equal(direct.summary.pass, 15);
assert.equal(direct.summary.not_verified, 36);
assert.equal(direct.summary.not_verified, 37);
});

test('copied recovery plugin serves exact valid and invalid-server observations without runtime dependencies', { timeout: 30_000 }, async (t) => {
Expand Down
1 change: 1 addition & 0 deletions test/record-report.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ const invalidServerCases = [
['recovery-env-plugin-data', 'mcp.stdio.env.reserved-plugin-data'],
];
const invalidHttpServerCases = [
['recovery-http-type', 'mcp.config.legacy-http-type'],
['recovery-http-relative-url', 'mcp.streamable-http.url.relative'],
['recovery-http-fragment', 'mcp.streamable-http.url.fragment'],
['recovery-http-duplicate-headers', 'mcp.streamable-http.headers.duplicate-names'],
Expand Down
11 changes: 6 additions & 5 deletions test/report-human.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -52,16 +52,17 @@ test('complete stdio and recovery evidence leaves optional HTTP and SSE checks u
'',
'Checks Passed Failed Not verified',
'Skills 3 0 0',
'MCP 21 0 23',
'MCP 21 0 24',
'Filesystem 4 0 0',
'Total 28 0 23',
'Total 28 0 24',
].join('\n'));
const missing = human.split('\n\nNot verified\n')[1];
for (const id of [
'mcp.streamable-http.tool-availability',
'mcp.streamable-http.url.literal-route-and-query',
'mcp.streamable-http.headers.cross-origin-redirect',
'mcp.streamable-http.headers.literal-value',
'mcp.config.legacy-http-type',
'mcp.streamable-http.url.relative',
'mcp.streamable-http.url.fragment',
'mcp.streamable-http.headers.duplicate-names',
Expand Down Expand Up @@ -90,7 +91,7 @@ test('all-unverified human report preserves every saved missing-evidence result'
value.observations = [];
const human = formatReport(buildReport(value));
assert.match(human, /Skills\s+0\s+0\s+3/);
assert.match(human, /MCP\s+0\s+0\s+44/);
assert.match(human, /MCP\s+0\s+0\s+45/);
assert.match(human, /Filesystem\s+0\s+0\s+4/);
assert.match(human, /Missing skill observations: conformance-alpha, conformance-beta\./);
for (const result of buildReport(value).results) assert.ok(human.includes(`(${result.id})`));
Expand All @@ -102,7 +103,7 @@ test('mixed human report puts failures before missing-evidence details', () => {
value.observations = [value.observations[2]];
value.observations[0].evidence.env.APC_VALUE = 'incorrect configured value';
const human = formatReport(buildReport(value));
assert.match(human, /MCP\s+6\s+1\s+37/);
assert.match(human, /MCP\s+6\s+1\s+38/);
assert.match(human, /Filesystem\s+1\s+0\s+3/);
assert.match(human, /Configured environment \(mcp.stdio.env.configured-value\)/);
assert.match(human, /"fixture value with spaces"/);
Expand Down Expand Up @@ -157,7 +158,7 @@ test('human report renders saved warnings independently of result status', () =>
const writable = value.results.find(({ id }) => id === 'filesystem.data.writable');
writable.status = 'not_verified';
writable.warning = 'Saved cleanup warning for /state/plugin/leftover.';
value.summary = { pass: 15, fail: 0, not_verified: 36, total: 51 };
value.summary = { pass: 15, fail: 0, not_verified: 37, total: 52 };
value.observations = [];

const human = formatReport(value);
Expand Down
Loading
Loading