Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion plugins/agent-plugins-conformance-core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ Install this fixture alongside [Agent Plugins Conformance](../agent-plugins-conf

Node.js 22 or newer must be available as `node` on the client's executable search path. The committed MCP bundle includes the official MCP SDK and requires no dependency installation or build.

The command-token check includes POSIX and Windows servers. Only the server for the current platform is expected to run; the other may produce a startup error.
Comment thread
jonathanhefner marked this conversation as resolved.
Resolved

## Optional HTTP setup

To include checks for MCP servers using Streamable HTTP or legacy HTTP+SSE, start the bundled server in a separate terminal **before the client loads the plugin**:
Expand All @@ -28,9 +30,10 @@ Replace the path with a downloaded or installed copy of this plugin; the command
| `mcp.stdio.env.plugin-root` | Supplied `PLUGIN_ROOT` matches the independently determined package root. |
| `mcp.stdio.env.plugin-data-absolute` | Supplied `PLUGIN_DATA` is an absolute path. This check does not exercise writability, storage dedication, or persistence across updates. |
| `filesystem.data.writable` | The default server can create, write, and close a small temporary file directly in the supplied `PLUGIN_DATA` directory. |
| `filesystem.data.consistent-within-plugin` | All four servers resolve `PLUGIN_DATA` to the same path. |
| `filesystem.data.consistent-within-plugin` | Servers using omitted and data-rooted working directories resolve `PLUGIN_DATA` to the same filesystem path. |
| `mcp.stdio.env.configured-value` | The configured value reaches the subprocess. The fixture does not arrange a conflicting inherited value, so it does not establish override behavior. |
| `mcp.stdio.args.preservation-and-expansion` | Argument boundaries, including spaces and an empty argument, are preserved; recognized placeholders expand and unknown placeholder-like text stays literal. |
| `mcp.stdio.command.single-token` | A command path containing a space is preserved as a single executable token. |
| `mcp.stdio.env.expansion` | Repeated recognized placeholders expand in environment values and unknown placeholder-like text stays literal. |
| `mcp.streamable-http.tool-availability` | The client exposes the HTTP tool and it returns a valid observation despite a conflicting, mixed-case configured `Accept` header. A completed discovery or call attempt with no observation fails this check only when the reporter confirms that the local fixture is healthy. |
| `mcp.streamable-http.url.literal-route-and-query` | The configured URL path and query are preserved literally. |
Expand Down
2 changes: 2 additions & 0 deletions plugins/agent-plugins-conformance-core/bin/posix/probe
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
#!/bin/sh
exec node "$(dirname "$0")/../../dist/probe.mjs" command-token-posix posix-decoy split "$@"
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
#!/bin/sh
exec node "$(dirname "$0")/../../dist/probe.mjs" command-token-posix posix-exact intact "$@"
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
@echo off
node "%~dp0..\..\dist\probe.mjs" command-token-windows windows-exact intact %*
2 changes: 2 additions & 0 deletions plugins/agent-plugins-conformance-core/bin/windows/probe.cmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
@echo off
node "%~dp0..\..\dist\probe.mjs" command-token-windows windows-decoy split %*
4 changes: 2 additions & 2 deletions plugins/agent-plugins-conformance-core/dist/probe.mjs

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

18 changes: 18 additions & 0 deletions plugins/agent-plugins-conformance-core/mcp.json
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,24 @@
],
"cwd": "${PLUGIN_DATA}"
},
"command-token-posix": {
"type": "stdio",
"command": "./bin/posix/probe token.sh",
"args": [
"arg with spaces",
"",
"literal-value"
]
},
"command-token-windows": {
"type": "stdio",
"command": "./bin/windows/probe token.cmd",
"args": [
"arg with spaces",
"",
"literal-value"
]
},
"sse": {
"type": "sse",
"url": "http://127.0.0.1:43187/conformance/sse?value=$APC_SSE_VALUE",
Expand Down
4 changes: 2 additions & 2 deletions plugins/agent-plugins-conformance-core/src/probe.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ import { CallToolRequestSchema, ListToolsRequestSchema } from '@modelcontextprot
import { observeDataWrite } from './data-write.mjs';

const serverName = process.argv[2];
if (!['default', 'relative', 'root', 'data'].includes(serverName)) {
throw new Error('Expected probe mode: default, relative, root, or data');
if (!['default', 'relative', 'root', 'data', 'command-token-posix', 'command-token-windows'].includes(serverName)) {
throw new Error('Expected probe mode: default, relative, root, data, command-token-posix, or command-token-windows');
}
// Both src/ and dist/ are immediate children of the installed plugin root.
// This value is independent of the client-provided PLUGIN_ROOT environment.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ Collect all fixture skills and MCP tools through the client's normal mechanisms.
}
```

4. Find the `observe` tools on the Core fixture servers (`default`, `relative`, `root`, `data`, `http`, `http-redirect`, `sse`, `sse-header-precedence`, `sse-redirect`, and `sse-endpoint-origin`) and the Recovery fixture servers (`recovery-valid`, `recovery-cwd-invalid-form`, `recovery-cwd-escape`, `recovery-cwd-data-escape`, `recovery-cwd-symlink-escape`, `recovery-unknown-field`, `recovery-missing-type`, `recovery-env-plugin-root`, `recovery-env-plugin-data`, `recovery-http-type`, `recovery-http-relative-url`, `recovery-http-fragment`, `recovery-http-userinfo`, `recovery-http-duplicate-headers`, `recovery-http-header-name`, `recovery-http-header-value`, `recovery-sse-relative-url`, `recovery-sse-fragment`, `recovery-sse-userinfo`, `recovery-sse-duplicate-headers`, `recovery-sse-header-name`, and `recovery-sse-header-value`) through the client's normal MCP mechanism. Tool names may be client-namespaced. The remote MCP tool descriptions identify their fixture server IDs; retain that association when calling each tool so that an error can be recorded for the same server. Call each available tool with `{}`. First inspect whether the client returned a successful MCP result or a native discovery/call error. For a native remote MCP error, follow step 5 without parsing it as observation JSON. For a native stdio error, leave that server’s runtime observation missing; retain any discovery observation already recorded. For a successful result, take the observation from `structuredContent` (some clients show `structured_content`), or parse the JSON in the tool's text content. Immediately record that complete object; exclude the MCP result wrapper containing `content` or `structuredContent`. Preserve the returned paths and values exactly. The `http-redirect` and `sse-redirect` fixtures redirect requests to another local origin, and `sse-endpoint-origin` advertises a message endpoint on that origin. Do not authorize forwarding configured headers to these destinations.
4. Find the `observe` tools on the Core fixture servers (`default`, `relative`, `root`, `data`, `command-token-posix`, `command-token-windows`, `http`, `http-redirect`, `sse`, `sse-header-precedence`, `sse-redirect`, and `sse-endpoint-origin`) and the Recovery fixture servers (`recovery-valid`, `recovery-cwd-invalid-form`, `recovery-cwd-escape`, `recovery-cwd-data-escape`, `recovery-cwd-symlink-escape`, `recovery-unknown-field`, `recovery-missing-type`, `recovery-env-plugin-root`, `recovery-env-plugin-data`, `recovery-http-type`, `recovery-http-relative-url`, `recovery-http-fragment`, `recovery-http-userinfo`, `recovery-http-duplicate-headers`, `recovery-http-header-name`, `recovery-http-header-value`, `recovery-sse-relative-url`, `recovery-sse-fragment`, `recovery-sse-userinfo`, `recovery-sse-duplicate-headers`, `recovery-sse-header-name`, and `recovery-sse-header-value`) through the client's normal MCP mechanism. Tool names may be client-namespaced. The remote MCP tool descriptions identify their fixture server IDs; retain that association when calling each tool so that an error can be recorded for the same server. Call each available tool with `{}`. First inspect whether the client returned a successful MCP result or a native discovery/call error. For a native remote MCP error, follow step 5 without parsing it as observation JSON. For a native stdio error, leave that server’s runtime observation missing; retain any discovery observation already recorded. Only the platform-appropriate command-token server is expected to run; a startup error for the other is not a collection limitation. For a successful result, take the observation from `structuredContent` (some clients show `structured_content`), or parse the JSON in the tool's text content. Immediately record that complete object; exclude the MCP result wrapper containing `content` or `structuredContent`. Preserve the returned paths and values exactly. The `http-redirect` and `sse-redirect` fixtures redirect requests to another local origin, and `sse-endpoint-origin` advertises a message endpoint on that origin. Do not authorize forwarding configured headers to these destinations.
5. If a completed remote MCP discovery or call attempt instead produces a native error, record it as error evidence:

```json
Expand Down
1 change: 1 addition & 0 deletions plugins/agent-plugins-conformance/src/cases.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ export const CASES = Object.freeze([
['filesystem.data.distinct-across-plugins', 'Distinct data directories across plugins', ['9.1']],
['filesystem.data.consistent-within-plugin', 'Consistent data directory within a plugin', ['9.1']],
['mcp.stdio.env.configured-value', 'Configured environment', ['9.1']],
['mcp.stdio.command.single-token', 'Command token preservation', ['7.2.1']],
['mcp.stdio.args.preservation-and-expansion', 'Argument preservation and expansion', ['7.2.1', '9.2']],
['mcp.stdio.env.expansion', 'Environment expansion', ['9.2']],
['mcp.streamable-http.tool-availability', 'Streamable HTTP MCP tool evidence', ['6.1', '7.2.1']],
Expand Down
55 changes: 45 additions & 10 deletions plugins/agent-plugins-conformance/src/report.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ import { CASES, MCP_CWD_VARIANTS } from './cases.mjs';

export { CASES, CASE_IDS } from './cases.mjs';
const CORE_SERVERS = Object.keys(MCP_CWD_VARIANTS);
const COMMAND_TOKEN_SERVERS = Object.freeze({
'command-token-posix': Object.freeze({ exactOrigin: 'posix-exact', decoyOrigin: 'posix-decoy', splitTail: 'token.sh' }),
'command-token-windows': Object.freeze({ exactOrigin: 'windows-exact', decoyOrigin: 'windows-decoy', splitTail: 'token.cmd' }),
});
const INVALID_STDIO_SERVERS = Object.freeze({
'recovery-cwd-invalid-form': 'mcp.stdio.cwd.invalid-form',
'recovery-cwd-escape': 'mcp.stdio.cwd.plugin-relative-escape',
Expand Down Expand Up @@ -34,7 +38,7 @@ const INVALID_STDIO_SERVER_NAMES = Object.keys(INVALID_STDIO_SERVERS);
const RECOVERY_INVALID_SERVER_NAMES = [
...INVALID_STDIO_SERVER_NAMES, ...Object.keys(INVALID_HTTP_SERVERS), ...Object.keys(INVALID_SSE_SERVERS),
];
const SERVERS = [...CORE_SERVERS, 'recovery-valid', ...INVALID_STDIO_SERVER_NAMES];
const SERVERS = [...CORE_SERVERS, ...Object.keys(COMMAND_TOKEN_SERVERS), 'recovery-valid', ...INVALID_STDIO_SERVER_NAMES];
const HTTP_SERVERS = ['http', 'http-redirect', ...Object.keys(INVALID_HTTP_SERVERS)];
const SSE_SERVERS = [
'sse', 'sse-header-precedence', 'sse-redirect', 'sse-endpoint-origin', ...Object.keys(INVALID_SSE_SERVERS),
Expand Down Expand Up @@ -537,6 +541,34 @@ export function buildReport(input) {
'Working directory matches the resolved expected path.', mismatch('Working directory', target, evidence.cwd));
}
}
const defaultEvidence = runtime.get('default');
const commandTokenId = 'mcp.stdio.command.single-token';
if (defaultEvidence) {
const defaultFlavor = pathFlavor(defaultEvidence.root);
const platform = defaultFlavor === path.win32 ? 'windows' : 'posix';
const server = `command-token-${platform}`;
const variant = COMMAND_TOKEN_SERVERS[server];
const evidence = runtime.get(server);
if (!evidence) {
set(commandTokenId, 'not_verified', `No attributable ${platform} command-token observation was supplied.`);
} else if (!samePath(evidence.root, defaultEvidence.root, defaultFlavor)) {
set(commandTokenId, 'not_verified',
`The ${platform} command-token observation did not identify the same installed plugin root as the Core default observation.`);
} else {
const split = evidence.argv[0] === server && evidence.argv[1] === variant.decoyOrigin &&
evidence.argv[2] === 'split' && evidence.argv[3] === variant.splitTail;
const intact = evidence.argv[0] === server && evidence.argv[1] === variant.exactOrigin && evidence.argv[2] === 'intact';
if (split) {
set(commandTokenId, 'fail',
`The ${platform} decoy wrapper received ${JSON.stringify(variant.splitTail)} as an argument, showing that the configured command was split.`);
} else if (intact) {
set(commandTokenId, 'pass', `The ${platform} exact-name wrapper was selected.`);
} else {
set(commandTokenId, 'not_verified',
`The ${platform} command-token observation did not identify either the exact-name wrapper or an attributable split-name decoy.`);
}
}
}
if (skills.has('conformance-recovery-valid')) {
check('skills.recovery.valid-skill-available', skills.get('conformance-recovery-valid') === SKILLS['conformance-recovery-valid'],
'Agent reported the expected client-loaded marker for the valid recovery skill.',
Expand Down Expand Up @@ -630,15 +662,18 @@ export function buildReport(input) {
? `Resolved PLUGIN_DATA unavailable for: ${missingData.join(', ')}.`
: `Core servers ${coreData.map(([server]) => server).join(', ')} resolved PLUGIN_DATA to paths distinct from recovery-valid (${JSON.stringify(recoveryData)}).`);

const [firstCoreData, ...otherCoreData] = coreData;
const inconsistentData = firstCoreData
? otherCoreData.filter(([, data]) => !samePath(data, firstCoreData[1], pathFlavor(firstCoreData[1]))) : [];
set('filesystem.data.consistent-within-plugin', inconsistentData.length ? 'fail' : missingCoreData.length ? 'not_verified' : 'pass',
inconsistentData.length
? [firstCoreData, ...inconsistentData].map(([server, data]) => `${server} resolved PLUGIN_DATA to ${JSON.stringify(data)}.`).join(' ')
: missingCoreData.length
? `Resolved PLUGIN_DATA unavailable for: ${missingCoreData.join(', ')}.`
: 'All four core servers resolved PLUGIN_DATA to the same path.');
const consistencyServers = ['default', 'data'];
const missingConsistencyData = consistencyServers.filter((server) => runtime.get(server)?.resolvedData == null);
if (missingConsistencyData.length) {
set('filesystem.data.consistent-within-plugin', 'not_verified',
`Resolved PLUGIN_DATA unavailable for: ${missingConsistencyData.join(', ')}.`);
} else {
const defaultData = runtime.get('default').resolvedData;
const dataCwdData = runtime.get('data').resolvedData;
check('filesystem.data.consistent-within-plugin', samePath(dataCwdData, defaultData, pathFlavor(defaultData)),
'The default and data working-directory servers resolved PLUGIN_DATA to the same path.',
`default resolved PLUGIN_DATA to ${JSON.stringify(defaultData)}. data resolved PLUGIN_DATA to ${JSON.stringify(dataCwdData)}.`);
}
const evidence = runtime.get('default');
if (evidence) {
const { root, env, argv } = evidence;
Expand Down
19 changes: 16 additions & 3 deletions scripts/smoke-codex.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import assert from 'node:assert/strict';
import { spawn, spawnSync } from 'node:child_process';
import { createHash } from 'node:crypto';
import { closeSync, openSync, writeSync } from 'node:fs';
import { cp, mkdir, mkdtemp, readFile, readdir, readlink, realpath, rm, writeFile } from 'node:fs/promises';
import { cp, mkdir, mkdtemp, readFile, readdir, readlink, realpath, rm, stat, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { delimiter, dirname, isAbsolute, join, relative } from 'node:path';
import { createInterface } from 'node:readline';
Expand All @@ -16,15 +16,17 @@ const [codex, ...extra] = process.argv.slice(2);
assert.ok(codex && isAbsolute(codex) && extra.length === 0,
'Usage: node scripts/smoke-codex.mjs <absolute-codex-binary>');
const names = ['agent-plugins-conformance-core', 'agent-plugins-conformance', 'agent-plugins-conformance-recovery'];
const servers = ['default', 'relative', 'root', 'data', 'http', 'recovery-valid'];
const commandPlatform = process.platform === 'win32' ? 'windows' : 'posix';
const commandServer = `command-token-${commandPlatform}`;
const servers = ['default', 'relative', 'root', 'data', commandServer, 'http', 'recovery-valid'];
const invalidSseServers = [
'recovery-sse-relative-url', 'recovery-sse-fragment', 'recovery-sse-userinfo',
'recovery-sse-duplicate-headers', 'recovery-sse-header-name', 'recovery-sse-header-value',
];
// Redirect refusal requires the guiding agent to interpret and preserve the native error.
const coveredCases = CASE_IDS.filter((id) => (id.startsWith('mcp.') || id.startsWith('filesystem.')) &&
id !== 'mcp.streamable-http.headers.cross-origin-redirect' && !id.startsWith('mcp.sse.'));
const temporary = await realpath(await mkdtemp(join(tmpdir(), 'apc-codex-smoke-')));
const temporary = await realpath(await mkdtemp(join(tmpdir(), 'apc codex smoke ')));
const home = join(temporary, 'codex-home');
const workspace = join(temporary, 'workspace');
const marketplace = join(temporary, 'marketplace');
Expand Down Expand Up @@ -161,6 +163,12 @@ try {
if (!actual.some(([path]) => path === 'escape-link')) expected = original.filter(([path]) => path !== 'escape-link');
}
assert.deepEqual(actual, expected, `${name}: installed package differs from source`);
if (name === 'agent-plugins-conformance-core' && process.platform !== 'win32') {
for (const file of ['bin/posix/probe token.sh', 'bin/posix/probe']) {
assert.ok((await stat(join(installedPath, file))).mode & 0o111,
`${file}: installed wrapper is not executable`);
}
}
installed.push(installedPath);
}
const reporter = join(installed[1], 'skills/run-conformance/scripts/report.mjs');
Expand Down Expand Up @@ -226,6 +234,11 @@ try {
: invalidSseServers.includes(server) ? 'mcp-sse' : 'mcp-stdio',
`${server}: missing observation`);
assert.equal(observation.server, server, `${server}: unexpected observation server`);
if (server === commandServer) {
assert.deepEqual(observation.evidence.argv,
[commandServer, `${commandPlatform}-exact`, 'intact', 'arg with spaces', '', 'literal-value'],
'Native launch did not preserve the command token and configured arguments');
}
record({ action: 'record', observation });
}
// SSE is optional. Preserve native diagnostics even when the loader omits entries.
Expand Down
Loading
Loading