diff --git a/CHANGELOG.md b/CHANGELOG.md index f1974879..23807a57 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,8 +2,49 @@ ## 2026-08-30 +### New CLI Options + +- **`--endpoint`** (api) — The base API endpoint an `explorbot api` run tests, so the API boat no longer + needs a config file to know where the API is. `api test`, which takes a plan file rather than an + endpoint, reads it from here too, and a path prefix is kept: given + `https://api.example.com/v1`, a step on `/users` is sent to `https://api.example.com/v1/users`. + It sets the same value as `EXPLORBOT_URL`, and wins when both are given. + ```bash + explorbot api plan /users --endpoint https://api.example.com/v1 + explorbot api test output/plans/users.md --endpoint https://api.example.com/v1 + ``` +- **`--spec`** (api) — The OpenAPI spec for the run, as a local file or a URL. Chief plans from it and + Curler looks up schemas in it; given here it replaces `api.spec` from the config file. It sets the + same value as `EXPLORBOT_API_SPEC`. + ```bash + explorbot api plan /users --spec ./openapi.yaml + explorbot api plan /users --spec https://api.example.com/openapi.json + ``` +- **`--spec`** (prima) — The collected documentation a prima run reads as page knowledge: a Docbot + application spec directory, or its `index.md`. It is the flag form of the `--spec` that + `explorbot start` already takes, and the new `EXPLORBOT_SPEC` variable sets the same thing for + every browser command — `PRIMA_CLI_SPEC` for prima, like the other variables it mirrors. + ```bash + prima check "a project can be archived" --spec output/docs + PRIMA_CLI_SPEC=output/docs prima do "open the account menu" + ``` +- **`--url`** (docs collect) — The base URL to document when the path argument is relative, so the + site can come from the command line rather than only from an absolute path or the environment. An + absolute path argument still carries its own. Same value as `EXPLORBOT_URL`. + ```bash + explorbot docs collect /dashboard --url https://app.example.com + ``` + ### Changes +- Knowledge from `EXPLORBOT_KNOWLEDGE` and `EXPLORBOT_KNOWLEDGE_FILE` now reaches runs that use the + global configuration in `~/.explorbot` — exploration, prima, doc collection and API testing alike. + Each run writes what they carry into the site's knowledge directory, where the agents read it like + any other knowledge file; the next run rewrites it, and a run that sets neither variable removes + it, so `learn` and `know` remain the way to keep a fact. Until now those two variables only had an + effect when no configuration file existed at all. +- `config` no longer prints a directory as the project root with an absolute path glued onto the + end. An absolute `dirs` entry, such as an application spec outside the project, is shown as it is. - Doc Collector: a `docs collect` run streamed with `--ws` now sends the spec index it generates as a `docs` frame — the file path and the full markdown of `docs/index.md` — so a listening UI can show the finished documentation the same way an exploration run streams its session report. @@ -185,6 +226,35 @@ ## 2026-08-23 +### New CLI Options + +- **`--knowledge`** — Facts for one run, passed on the command line instead of stored in `knowledge/`. + Nothing is written to disk, so credentials and one-off test data stay out of the repository. Plain + text applies everywhere; frontmatter scopes it to a page (`url:`) or an API endpoint (`endpoint:`), + with the same patterns knowledge files use. `${env.VAR}` interpolation and page automation fields + such as `wait` work as they do in files. Repeat the flag for several facts. Like `--ws`, it is a + program-level option: it works on every command of `explorbot`, `explorbot api`, `explorbot docs` + and `prima`, and can go anywhere on the line. + ```bash + explorbot explore /pay --knowledge 'My credit card is 4111 1111 1111 1111' + explorbot explore / --knowledge '--- + url: /login + --- + Log in as admin@example.com / secret123' + explorbot api explore /orders --knowledge 'Send X-Api-Key on every request' + prima check "checkout completes" --knowledge 'Use the sandbox card 4111 1111 1111 1111' + ``` +- **`--save-knowledge`** (renamed) — `explorbot drill --knowledge ` is now + `explorbot drill --save-knowledge `, and `/drill --knowledge` is now `/drill --save-knowledge`. + It still saves the interactions drilling learned to a knowledge file at that URL path; the rename + frees `--knowledge` for the session facts above. + ```bash + explorbot drill /login --save-knowledge /login + ``` + ``` + /drill --save-knowledge /login --max-components 10 + ``` + ### Changes - Prima reads `PRIMA_CLI_*` environment variables. Each one mirrors the `EXPLORBOT_*` variable of the @@ -212,6 +282,10 @@ the test. - [Pilot] The Pilot no longer pushes a full page of HTML into the Tester mid-test. It can still attach the accessibility tree, a page summary, or the UI map when recent actions failed. +- [Chief] Now reads endpoint knowledge when planning API tests, so auth rules and business + constraints written with `explorbot api know` reach the plan. +- [Curler] Now reads endpoint knowledge when running an API test, so auth headers and payload rules + reach the requests themselves rather than only the plan. - Prima now ships as its own npm package, so `npx prima-cli` runs it without installing explorbot first. It is the same tool as the `prima` command that comes with explorbot, built from the same source and released alongside it — only the package name and the binary differ. diff --git a/CLAUDE.md b/CLAUDE.md index 93517364..723156b4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -99,7 +99,7 @@ Two tiers. **Data parts** are deterministic memory — they answer "what is true | Module | Does | Must never do | |---|---|---| | **StateManager** | Where am I / where have I been: states, transitions, hashes, loop detection, events | Call AI; interpret semantics ("this is a login page"); execute actions | -| **KnowledgeTracker** | Load/filter human facts by URL pattern; expose hints verbatim | Judge relevance semantically; act on hints; be written mid-run outside learn/drill flow | +| **KnowledgeTracker** | Load/filter human facts by URL or endpoint pattern, from `knowledge/` and from `--knowledge` session entries; expose hints verbatim | Judge relevance semantically; act on hints; be written mid-run outside learn/drill flow | | **ExperienceTracker** | Store/retrieve lessons per state hash; dedup blocks | Rank by meaning; compact itself (ExperienceCompactor's job); change behavior directly | | **Config** | Resolve immutable settings once at startup | Change mid-run; hold site-specific values; be read by agents directly (injected instead) | @@ -136,7 +136,7 @@ All persisted formats share one rule: **envelope keys (YAML frontmatter, HTML co | Format | Location & owner | Envelope | Body grammar | |---|---|---|---| -| Knowledge | `knowledge/*.md`, KnowledgeTracker | `url`/`path`, `wait`, `waitForElement`, `noExperienceReading/Writing` | Free prose facts | +| Knowledge | `knowledge/*.md`, KnowledgeTracker | `url`/`path`, `endpoint`, `wait`, `waitForElement`, `noExperienceReading/Writing` | Free prose facts | | Experience | `experience/.md`, ExperienceTracker | sparse frontmatter | `## FLOW:` / `## ACTION:` h2 blocks; bullets + ```js``` + `Solution:` line; h3 forbidden under blocks | | Test plan | `output/plans/*.md`, test-plan-markdown.ts | `` comment: `priority`, `style`; scenario heading, `url:` line, bullets as steps | Notes/results appended by runner | @@ -539,7 +539,7 @@ There are application commands available in TUI * /research [uri] - performs research on a current page or navigate to [uri] if uri is provided * /plan - plan testing feature starting from current page * /navigate - move to other page. Use AI to complete navigation -* /drill [--knowledge ] [--max-components ] - drill all components on page to learn interactions +* /drill [--save-knowledge ] [--max-components ] - drill all components on page to learn interactions There are also CodeceptJS commands available: @@ -563,7 +563,15 @@ Consequently `isInteractive()` (`src/ai/task-agent.ts`) is **`INK_RUNNING || exe **There are no frame types.** A frame is `{type, ts, ...whatever}`; `send(type, data)` puts data on the wire and the UI renders what it recognises. Neither side validates the other's shape, so either can start sending more at any time. It queues while disconnected, reconnects with backoff, and `remote.close(exitCode)` flushes before exit (called from `showStatsAndExit`). -`remote.registerOption(program)` adds the flag through a Commander `preAction` hook, so it covers every command including the mounted `api`/`docs` subcommands and the standalone `boat/*` bins. +`--ws` itself is declared outside remote, as a `BaseOption` in `src/commands/options/` — see below — and its hook calls `remote.attach()`. + +## Shared CLI options (`src/commands/options/`) + +A flag that belongs to a run rather than to one command lives here, one `BaseOption` subclass per flag: `flags`, `description`, an optional `collect` for a repeatable value, and an `apply()` that runs after parsing. `register(program)` declares the flag on the program root and hooks `preAction`, so one registration covers every command, the mounted `api`/`docs` subcommands and the standalone `boat/*` bins alike, and the flag can sit anywhere on the line. + +Each bin registers the options it offers — `wsOption.register(program)`, `knowledgeOption.register(program)` — so a boat carries only the flags that mean something for it. + +`apply()` hands the value to whoever owns it and nothing else: `--ws` to `remote.attach()`, `--knowledge` to config, which holds the run's inputs. Nothing downstream learns that a command line exists. ## Command Line Usage @@ -593,7 +601,7 @@ explorbot plan /login authentication # plan with focus on authentication ```bash explorbot drill # drill all components on page explorbot drill /components --max-components 10 # limit to 10 components -explorbot drill /login --knowledge /login # save to knowledge file +explorbot drill /login --save-knowledge /login # save to knowledge file ``` ### Show resolved configuration: diff --git a/bin/explorbot-cli.ts b/bin/explorbot-cli.ts index 63732f3b..dce2253b 100755 --- a/bin/explorbot-cli.ts +++ b/bin/explorbot-cli.ts @@ -10,6 +10,7 @@ import React from 'react'; import { flushTelemetry } from '../src/ai/provider.js'; import { App } from '../src/components/App.js'; import { StatusPane } from '../src/components/StatusPane.js'; +import { knowledgeOption, wsOption } from '../src/commands/options/index.js'; import { ConfigParser, EXPLORBOT_ENV_VARS, PROVIDERS } from '../src/config.js'; import { ExplorBot, type ExplorBotOptions } from '../src/explorbot.js'; import { remote } from '../src/remote.js'; @@ -28,7 +29,8 @@ const pkgPath = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../p const pkgVersion = JSON.parse(fs.readFileSync(pkgPath, 'utf-8')).version as string; program.name(cli).description('AI-powered web exploration tool').version(pkgVersion, '-V, --version'); -remote.registerOption(program); +wsOption.register(program); +knowledgeOption.register(program); process.on('uncaughtException', async (error) => { tag('error').log(`Uncaught exception: ${error instanceof Error ? `${error.message}\n${error.stack}` : String(error)}`); @@ -701,7 +703,7 @@ addCommonOptions(program.command('navigate ').description('Navigate to a UR }); addCommonOptions( - program.command('drill ').alias('driller').description('Drill all components on a page to learn interactions').option('--knowledge ', 'Save learned interactions to knowledge file at this URL path').option('--max-components ', 'Maximum number of components to drill') + program.command('drill ').alias('driller').description('Drill all components on a page to learn interactions').option('--save-knowledge ', 'Save learned interactions to knowledge file at this URL path').option('--max-components ', 'Maximum number of components to drill') ).action(async (url, options) => { try { const explorBot = new ExplorBot(buildExplorBotOptions(url, options)); @@ -710,7 +712,7 @@ addCommonOptions( await explorBot.visit(url); const plan = await explorBot.agentDriller().drill({ - knowledgePath: options.knowledge, + knowledgePath: options.saveKnowledge, maxComponents: Number.parseInt(options.maxComponents || '30', 10), interactive: false, }); diff --git a/boat/api-tester/bin/apibot-cli.ts b/boat/api-tester/bin/apibot-cli.ts index 28e91987..9bd61e62 100755 --- a/boat/api-tester/bin/apibot-cli.ts +++ b/boat/api-tester/bin/apibot-cli.ts @@ -1,7 +1,8 @@ #!/usr/bin/env bun -import { remote } from '../../../src/remote.ts'; +import { knowledgeOption, wsOption } from '../../../src/commands/options/index.ts'; import { createApiCommands } from '../src/cli.ts'; const program = createApiCommands('apibot'); -remote.registerOption(program); +wsOption.register(program); +knowledgeOption.register(program); program.parse(); diff --git a/boat/api-tester/src/ai/chief.ts b/boat/api-tester/src/ai/chief.ts index 897a218d..0cd962ed 100644 --- a/boat/api-tester/src/ai/chief.ts +++ b/boat/api-tester/src/ai/chief.ts @@ -3,6 +3,7 @@ import { z } from 'zod'; import { Conversation } from '../../../../src/ai/conversation.ts'; import { WithSessionDedup } from '../../../../src/ai/planner/session-dedup.ts'; import type { AIProvider } from '../../../../src/ai/provider.ts'; +import type { KnowledgeTracker } from '../../../../src/knowledge-tracker.ts'; import { Observability } from '../../../../src/observability.ts'; import { Plan, Test } from '../../../../src/test-plan.ts'; import { createDebug, tag } from '../../../../src/utils/logger.ts'; @@ -33,17 +34,19 @@ export class Chief extends ChiefBase { private provider: AIProvider; private config: ApibotConfig; private apiClient: ApiClient | null; + private knowledgeTracker?: KnowledgeTracker; currentPlan: Plan | null = null; private lastStyleName = ''; MIN_TASKS = 3; MAX_TASKS = 10; - constructor(provider: AIProvider, config: ApibotConfig, apiClient?: ApiClient | null) { + constructor(provider: AIProvider, config: ApibotConfig, apiClient?: ApiClient | null, knowledgeTracker?: KnowledgeTracker) { super(); this.provider = provider; this.config = config; this.apiClient = apiClient || null; + this.knowledgeTracker = knowledgeTracker; } async plan(endpoint: string, opts?: { style?: string; specDefinition?: string }): Promise { @@ -56,6 +59,9 @@ export class Chief extends ChiefBase { const sampleData = await this.collectSampleData(endpoint); const conversation = this.buildConversation(endpoint, opts?.style, sampleData); + const knowledge = this.knowledgeTracker?.renderEndpointKnowledge(endpoint); + if (knowledge) conversation.addUserText(knowledge); + if (opts?.specDefinition) { conversation.addUserText(dedent` diff --git a/boat/api-tester/src/ai/curler.ts b/boat/api-tester/src/ai/curler.ts index a5ca9c82..71f6a1db 100644 --- a/boat/api-tester/src/ai/curler.ts +++ b/boat/api-tester/src/ai/curler.ts @@ -2,6 +2,7 @@ import dedent from 'dedent'; import { z } from 'zod'; import type { AIProvider } from '../../../../src/ai/provider.ts'; import type { RequestStore } from '../../../../src/api/request-store.ts'; +import type { KnowledgeTracker } from '../../../../src/knowledge-tracker.ts'; import type { Reporter } from '../../../../src/reporter.ts'; import { type Test, TestResult } from '../../../../src/test-plan.ts'; import { createDebug, tag } from '../../../../src/utils/logger.ts'; @@ -18,12 +19,14 @@ export class Curler { private apiClient: ApiClient; private requestState: RequestStore; private reporter: Reporter; + private knowledgeTracker?: KnowledgeTracker; - constructor(provider: AIProvider, apiClient: ApiClient, requestState: RequestStore, reporter: Reporter) { + constructor(provider: AIProvider, apiClient: ApiClient, requestState: RequestStore, reporter: Reporter, knowledgeTracker?: KnowledgeTracker) { this.provider = provider; this.apiClient = apiClient; this.requestState = requestState; this.reporter = reporter; + this.knowledgeTracker = knowledgeTracker; } async test(test: Test, opts?: { specDefinition?: string; baseEndpoint?: string; searchSpec?: (query: string) => string }): Promise<{ success: boolean }> { @@ -37,6 +40,9 @@ export class Curler { const conversation = this.provider.startConversation(this.buildSystemPrompt(), 'curler', this.provider.getAgenticModel('curler')); const tools = createCurlerTools(this.apiClient, this.requestState, test, opts?.searchSpec); + const knowledge = test.startUrl && this.knowledgeTracker?.renderEndpointKnowledge(test.startUrl); + if (knowledge) conversation.addUserText(knowledge); + const initialPrompt = this.buildTestPrompt(test, opts?.specDefinition, opts?.baseEndpoint); conversation.addUserText(initialPrompt); diff --git a/boat/api-tester/src/apibot.ts b/boat/api-tester/src/apibot.ts index 06096d93..f6a0f0d3 100644 --- a/boat/api-tester/src/apibot.ts +++ b/boat/api-tester/src/apibot.ts @@ -3,6 +3,7 @@ import path from 'node:path'; import { AIProvider } from '../../../src/ai/provider.ts'; import { RequestStore } from '../../../src/api/request-store.ts'; import { extractEndpointDefinition, loadSpec, searchEndpoints, validateSpecs } from '../../../src/api/spec-reader.ts'; +import { KnowledgeTracker } from '../../../src/knowledge-tracker.ts'; import { Reporter } from '../../../src/reporter.ts'; import { Plan } from '../../../src/test-plan.ts'; import { setVerboseMode, tag } from '../../../src/utils/logger.ts'; @@ -20,6 +21,7 @@ export class ApiBot { private apiClient!: ApiClient; private requestState!: RequestStore; private reporter!: Reporter; + private knowledgeTracker!: KnowledgeTracker; private options: ApibotOptions; private apiSpec: any; @@ -33,7 +35,7 @@ export class ApiBot { } async start(): Promise { - this.config = await this.configParser.loadConfig({ config: this.options.config, path: this.options.path, endpoint: this.options.endpoint }); + this.config = await this.configParser.loadConfig(this.options); this.provider = new AIProvider(this.config.ai); await this.provider.validateConnection(); @@ -47,6 +49,7 @@ export class ApiBot { this.configParser.ensureDirectory(outputDir); this.requestState = new RequestStore(outputDir); this.reporter = new Reporter(this.config.reporter); + this.knowledgeTracker = new KnowledgeTracker({ knowledgeDir: this.configParser.getKnowledgeDir() }); validateSpecs(this.config.api.spec); this.apiSpec = await loadSpec(this.config.api.spec!, outputDir); @@ -84,21 +87,22 @@ export class ApiBot { await this.apiClient?.teardown(); } - createAgent(factory: (deps: { ai: AIProvider; config: ApibotConfig; apiClient: ApiClient; requestState: RequestStore }) => T): T { + createAgent(factory: (deps: { ai: AIProvider; config: ApibotConfig; apiClient: ApiClient; requestState: RequestStore; knowledge: KnowledgeTracker }) => T): T { return factory({ ai: this.provider, config: this.config, apiClient: this.apiClient, requestState: this.requestState, + knowledge: this.knowledgeTracker, }); } agentChief(): Chief { - return (this.agents.chief ||= this.createAgent(({ ai, config, apiClient }) => new Chief(ai, config, apiClient))); + return (this.agents.chief ||= this.createAgent(({ ai, config, apiClient, knowledge }) => new Chief(ai, config, apiClient, knowledge))); } agentCurler(): Curler { - return (this.agents.curler ||= this.createAgent(({ ai, apiClient, requestState }) => new Curler(ai, apiClient, requestState, this.reporter))); + return (this.agents.curler ||= this.createAgent(({ ai, apiClient, requestState, knowledge }) => new Curler(ai, apiClient, requestState, this.reporter, knowledge))); } async plan(target: string, opts: { style?: string; fresh?: boolean } = {}): Promise { @@ -200,6 +204,8 @@ interface ApibotOptions { config?: string; path?: string; endpoint?: string; + baseEndpoint?: string; + spec?: string; } export type { ApibotOptions }; diff --git a/boat/api-tester/src/cli.ts b/boat/api-tester/src/cli.ts index 1ac4c01b..b41b339f 100644 --- a/boat/api-tester/src/cli.ts +++ b/boat/api-tester/src/cli.ts @@ -13,11 +13,19 @@ function buildOptions(options: any): ApibotOptions { verbose: options.verbose || options.debug, config: options.config, path: options.path, + baseEndpoint: options.endpoint, + spec: options.spec, }; } function addCommonOptions(cmd: Command): Command { - return cmd.option('-v, --verbose', 'Enable verbose logging').option('--debug', 'Enable debug logging').option('-c, --config ', 'Path to configuration file').option('-p, --path ', 'Working directory path'); + return cmd + .option('-v, --verbose', 'Enable verbose logging') + .option('--debug', 'Enable debug logging') + .option('-c, --config ', 'Path to configuration file') + .option('-p, --path ', 'Working directory path') + .option('--endpoint ', 'Base API endpoint to test (env: EXPLORBOT_URL)') + .option('--spec ', 'OpenAPI spec file or URL (env: EXPLORBOT_API_SPEC)'); } function selectTests(tests: any[], index?: string): any[] { @@ -90,8 +98,10 @@ export function createApiCommands(name = 'api'): Command { .action(async (endpoint, options) => { const parser = ApibotConfigParser.getInstance(); const [site] = listSites(); + const runOptions = buildOptions(options); + runOptions.endpoint = endpoint || site?.url; try { - const config = await parser.loadConfig({ config: options.config, path: options.path, endpoint: endpoint || site?.url }); + const config = await parser.loadConfig(runOptions); console.log(ConfigCommand.render(config, { configPath: parser.getConfigPath(), root: parser.getProjectRoot(), json: options.json })); } catch (error) { console.error(error instanceof Error ? error.message : 'Unknown error'); diff --git a/boat/api-tester/src/config.ts b/boat/api-tester/src/config.ts index 15bcc19a..7328e844 100644 --- a/boat/api-tester/src/config.ts +++ b/boat/api-tester/src/config.ts @@ -45,7 +45,7 @@ export class ApibotConfigParser { Object.assign(process.env, parseEnv(readFileSync(resolved, 'utf8'))); } - async loadConfig(options?: { config?: string; path?: string; endpoint?: string }): Promise { + async loadConfig(options?: { config?: string; path?: string; endpoint?: string; baseEndpoint?: string; spec?: string }): Promise { if (this.config && !options?.config && !options?.path) return this.config; const originalCwd = process.cwd(); @@ -55,6 +55,7 @@ export class ApibotConfigParser { ApibotConfigParser.loadEnv(globalEnvPath()); ApibotConfigParser.loadEnv('.env'); + this.applyRunOptions(options); const resolvedPath = options?.config || this.findConfigFile(); if (!resolvedPath) { @@ -82,6 +83,8 @@ export class ApibotConfigParser { } this.config = this.mergeWithDefaults(loadedConfig); + this.applyEnvSpec(this.config.api); + if (options?.baseEndpoint) this.config.api.baseEndpoint = options.baseEndpoint.replace(/\/$/, ''); await resolveConfigModels(this.config.ai); this.configPath = resolvedPath; this.site = null; @@ -126,6 +129,11 @@ export class ApibotConfigParser { const resolved = resolveSiteTarget(endpoint, this.site.url); if (resolved.baseUrl !== this.site.url) return endpoint; + + const basePath = new URL(this.getConfig().api.baseEndpoint).pathname.replace(/\/$/, ''); + if (!basePath) return resolved.path; + if (resolved.path === basePath) return '/'; + if (resolved.path.startsWith(`${basePath}/`)) return resolved.path.slice(basePath.length); return resolved.path; } @@ -148,13 +156,27 @@ export class ApibotConfigParser { } } + private applyRunOptions(options?: { baseEndpoint?: string; spec?: string }): void { + if (options?.baseEndpoint) process.env.EXPLORBOT_URL = options.baseEndpoint; + if (options?.spec) process.env.EXPLORBOT_API_SPEC = options.spec; + } + + private applyEnvSpec(api: ApiConfig): void { + if (!process.env.EXPLORBOT_API_SPEC) return; + api.spec = [process.env.EXPLORBOT_API_SPEC]; + } + private enterGlobalMode(config: ApibotConfig, endpoint?: string): void { const site = resolveSiteTarget(endpoint); this.site = registerSite(site.baseUrl); + let baseEndpoint = site.baseUrl; + const envUrl = process.env.EXPLORBOT_URL; + if (envUrl && URL.parse(envUrl)?.origin === site.baseUrl) baseEndpoint = envUrl.replace(/\/$/, ''); + config.dirs = { output: 'output', knowledge: 'knowledge' }; - config.api = { ...config.api, baseEndpoint: site.baseUrl }; - if (process.env.EXPLORBOT_API_SPEC) config.api.spec = [process.env.EXPLORBOT_API_SPEC]; + config.api = { ...config.api, baseEndpoint }; + materializeKnowledge(this.site.dir); } private async loadEnvConfig(): Promise { @@ -169,16 +191,14 @@ export class ApibotConfigParser { const baseEndpoint = process.env.EXPLORBOT_URL; if (!baseEndpoint) { - throw new Error('No API endpoint to test. Set EXPLORBOT_URL to the API base endpoint'); + throw new Error('No API endpoint to test. Pass --endpoint or set EXPLORBOT_URL to the API base endpoint'); } const outputRoot = resolveOutputRoot(); materializeKnowledge(outputRoot); const api: ApiConfig = { baseEndpoint }; - if (process.env.EXPLORBOT_API_SPEC) { - api.spec = [process.env.EXPLORBOT_API_SPEC]; - } + this.applyEnvSpec(api); let model: any; if (provider && modelSpec) model = await createModel(provider, modelSpec); @@ -246,7 +266,7 @@ export class ApibotConfigParser { } private mergeWithDefaults(config: Partial): ApibotConfig { - return this.deepMerge({ dirs: { output: 'output' } }, config); + return this.deepMerge({ dirs: { output: 'output' }, api: {} }, config); } private deepMerge(target: any, source: any): any { diff --git a/boat/doc-collector/bin/doc-collector-cli.ts b/boat/doc-collector/bin/doc-collector-cli.ts index 5d2b6bf0..616b15ce 100644 --- a/boat/doc-collector/bin/doc-collector-cli.ts +++ b/boat/doc-collector/bin/doc-collector-cli.ts @@ -1,7 +1,8 @@ #!/usr/bin/env bun -import { remote } from '../../../src/remote.ts'; +import { knowledgeOption, wsOption } from '../../../src/commands/options/index.ts'; import { createDocsCommands } from '../src/cli.ts'; const program = createDocsCommands('doc-collector'); -remote.registerOption(program); +wsOption.register(program); +knowledgeOption.register(program); program.parse(); diff --git a/boat/doc-collector/src/cli.ts b/boat/doc-collector/src/cli.ts index b7f6d92d..7b4b441d 100644 --- a/boat/doc-collector/src/cli.ts +++ b/boat/doc-collector/src/cli.ts @@ -16,6 +16,7 @@ function buildOptions(options: any): DocbotOptions { incognito: options.incognito, session: options.session, docsConfig: options.docsConfig, + baseUrl: options.url, }; } @@ -26,6 +27,7 @@ function addCommonOptions(cmd: Command): Command { .option('-c, --config ', 'Path to explorbot configuration file') .option('--docs-config ', 'Path to doc collector configuration file') .option('-p, --path ', 'Working directory path') + .option('--url ', 'Base URL of the site, when the path argument is relative (env: EXPLORBOT_URL)') .option('-s, --show', 'Show browser window') .option('--headless', 'Run browser in headless mode') .option('--incognito', 'Run without recording experiences') @@ -74,7 +76,7 @@ export function createDocsCommands(name = 'docs'): Command { .action(async (url, options) => { setQuietMode(!isVerboseMode()); try { - console.log(await ConfigCommand.summary({ config: options.config, path: options.path, url, json: options.json })); + console.log(await ConfigCommand.summary({ config: options.config, path: options.path, url: url || options.url, json: options.json })); } catch (error) { console.error(error instanceof Error ? error.message : 'Unknown error'); process.exit(1); diff --git a/boat/doc-collector/src/docbot.ts b/boat/doc-collector/src/docbot.ts index f48f880e..6029cab6 100644 --- a/boat/doc-collector/src/docbot.ts +++ b/boat/doc-collector/src/docbot.ts @@ -24,7 +24,7 @@ class DocBot { constructor(options: DocbotOptions = {}) { this.options = options; - const baseUrl = this.extractAbsoluteBaseUrl(options.startUrl || '/'); + const baseUrl = this.extractAbsoluteBaseUrl(options.startUrl || '/') || options.baseUrl; this.explorBot = new ExplorBot({ baseUrl, verbose: options.verbose, diff --git a/boat/prima/bin/prima-cli.ts b/boat/prima/bin/prima-cli.ts index adc10a73..ffed76a6 100755 --- a/boat/prima/bin/prima-cli.ts +++ b/boat/prima/bin/prima-cli.ts @@ -1,5 +1,7 @@ #!/usr/bin/env bun +import { knowledgeOption } from '../../../src/commands/options/index.ts'; import { createPrimaCommands } from '../src/cli.ts'; const program = createPrimaCommands('prima'); +knowledgeOption.register(program); program.parse(); diff --git a/boat/prima/src/cli.ts b/boat/prima/src/cli.ts index ed3ff6c5..16490f12 100644 --- a/boat/prima/src/cli.ts +++ b/boat/prima/src/cli.ts @@ -109,6 +109,7 @@ function addCommonOptions(cmd: Command): Command { .option('--ephemeral', 'Keep no state between runs; applies to config-free runs, where output goes to a temp directory') .option('--framework ', 'Not active yet: framework the reported code targets, codeceptjs or playwright') .option('--url ', 'Page to open when the session has no page yet') + .option('--spec ', 'Docbot application spec directory or index.md to read as page knowledge') .option('--endpoint ', 'Websocket endpoint of a browser server to attach to, skipping discovery') .option('--pw-session ', 'Title of the playwright-cli session to attach to') .addHelpText('after', `\n${sessionHelp}`); @@ -119,6 +120,7 @@ function primaFor(options: any): Prima { if (options.ephemeral) process.env.EXPLORBOT_EPHEMERAL = '1'; if (options.model) process.env.EXPLORBOT_AI_MODEL = options.model; if (options.visionModel) process.env.EXPLORBOT_VISION_MODEL = options.visionModel; + if (options.spec) process.env.EXPLORBOT_SPEC = options.spec; return new Prima(buildOptions(options)); } diff --git a/docs/api-testing/basics.md b/docs/api-testing/basics.md index b362f629..25018373 100644 --- a/docs/api-testing/basics.md +++ b/docs/api-testing/basics.md @@ -56,6 +56,21 @@ api: { A matching `teardown` hook runs after all tests finish — use it to clean up data. +### Without a config file + +Chief and Curler need three things: where the API is, what its spec says, and how to authenticate. Pass all three on the command line and no config file is needed: + +```bash +npx explorbot api plan /users \ + --endpoint https://api.example.com/v1 \ + --spec ./openapi.yaml \ + --knowledge 'Send X-Api-Key: ${env.API_KEY} on every request' +``` + +`--endpoint` and `--spec` each have an environment twin — `EXPLORBOT_URL` and `EXPLORBOT_API_SPEC` — and the flag wins when both are set. `--knowledge` adds to the facts `EXPLORBOT_KNOWLEDGE` and `EXPLORBOT_KNOWLEDGE_FILE` bring in rather than replacing them. Configure your models once with `npx explorbot init --global` and every run stores its plans and requests per host under `~/.explorbot/sites/<host>/`, so a later `api test` against the same API picks up where the last one left off. Knowledge given on the command line lasts for the run; `api know` is what writes it down. + +`--endpoint` keeps its path prefix: given `https://api.example.com/v1`, steps stay relative (`/users`) and Curler sends them to `https://api.example.com/v1/users`. `api test`, which takes a plan file rather than an endpoint, reads it from the flag or the variable. + ### A dedicated API project If you don't have a web `explorbot.config.js`, run `npx explorbot api init`. It asks for your base endpoint, spec, and a one-line description of the API, then writes a standalone `apibot.config.ts` (with an `ai` and `api` section) plus `output/` and `knowledge/` directories. When both files exist, `apibot.config.*` takes precedence over `explorbot.config.*`. diff --git a/docs/api-testing/planning.md b/docs/api-testing/planning.md index 2a10f32a..90f21add 100644 --- a/docs/api-testing/planning.md +++ b/docs/api-testing/planning.md @@ -21,7 +21,16 @@ endpoint: "/users" CRUD for users. Admin role required for writes. IDs are UUIDs. ``` -Chief loads knowledge matching the endpoint it's planning. Running `know` again on the same endpoint appends to the file. See [knowledge](../workflow/knowledge.md) for how matching and files work. +Chief loads knowledge matching the endpoint it's planning, and Curler loads it again for the endpoint it's testing, so auth headers and payload rules reach the requests themselves. Running `know` again on the same endpoint appends to the file. See [knowledge](../workflow/knowledge.md) for how matching and files work. + +For a fact that should not be stored — a token, a one-off fixture — pass `--knowledge` instead. It applies to the run only: + +```bash +npx explorbot api explore /users --knowledge '--- +endpoint: /users/* +--- +Send X-Api-Key: ${env.API_KEY} on every request' +``` ## Choose a planning style diff --git a/docs/reference/commands.md b/docs/reference/commands.md index 7b0fe905..f596b086 100644 --- a/docs/reference/commands.md +++ b/docs/reference/commands.md @@ -33,7 +33,7 @@ Inside the TUI, use the matching slash command: `/explore`, `/research`, `/plan` | Generate test plan | `npx explorbot plan <path>` | `/plan [--focus <feature>]` | Writes plan markdown | | List saved plans | `npx explorbot plans [plan]` | `/plans [plan]` | Show plans and their tests | | Navigate to a URL | `npx explorbot navigate <url>` | `/navigate <target>` | Reachability probe + session capture | -| Drill page components | `npx explorbot drill <url>` | `/drill [--knowledge <path>] [--max-components <n>]` | Learn interactions | +| Drill page components | `npx explorbot drill <url>` | `/drill [--save-knowledge <path>] [--max-components <n>]` | Learn interactions | | Execute plan tests | `npx explorbot test <planfile> [index]` | `/test [scenario\|number\|*]` | Run scenarios | | Re-run generated tests | `npx explorbot rerun <file> [index]` | `/rerun <file> [index]` | With AI auto-healing | | List generated tests | `npx explorbot runs [file]` | `/runs [file]` | Index + dry-run | @@ -69,6 +69,16 @@ Every CLI command that drives a browser accepts these options (`start`, `explore | `--incognito` | Run without recording experiences | | `--session [file]` | Save/restore browser session (cookies, localStorage) from file | +### `--knowledge` + +Passes facts to the run without creating a file in `knowledge/`. Plain text applies everywhere; add frontmatter to scope it to a page or an API endpoint. Repeat the flag for several facts. See [Knowledge](../workflow/knowledge.md#per-session-knowledge). + +```bash +npx explorbot explore /pay --knowledge 'Test card 4111 1111 1111 1111, any future expiry' +``` + +Like `--ws`, it is a program-level option rather than a per-command one: it works on every command — including `api`, `docs` and `prima` — and can go anywhere on the line. It is listed under `npx explorbot --help` rather than in each command's own help. + ### `--session` Saves browser state (cookies, localStorage, sessionStorage) to a JSON file. The next run restores the session, so you skip login and setup steps. @@ -104,6 +114,7 @@ EXPLORBOT_AI_PROVIDER=openrouter \ | `EXPLORBOT_EPHEMERAL` | Keep no state between runs — output goes to a fresh temp directory instead of the site dir | | `EXPLORBOT_KNOWLEDGE` | Inline knowledge text, applied to every page | | `EXPLORBOT_KNOWLEDGE_FILE` | Path to a knowledge markdown file | +| `EXPLORBOT_SPEC` | Docbot application spec directory or index.md, used as page knowledge | | `EXPLORBOT_API_SPEC` | OpenAPI spec path for the API boat | | `EXPLORBOT_NO_BANNER` | Suppress the startup banner, for machine-readable output | <!-- END env --> @@ -439,18 +450,18 @@ Drill all components on a page to learn interactions. # CLI npx explorbot drill /components npx explorbot drill /components --max-components 10 -npx explorbot drill /login --knowledge /login +npx explorbot drill /login --save-knowledge /login ``` ``` # TUI /drill -/drill --knowledge /login --max-components 10 +/drill --save-knowledge /login --max-components 10 ``` | Option | Description | |---|---| -| `--knowledge <path>` | Save learned interactions to a knowledge file at this URL path | +| `--save-knowledge <path>` | Save learned interactions to a knowledge file at this URL path | | `--max-components <count>` | Maximum number of components to drill | ## Test Rerun @@ -588,9 +599,15 @@ Crawl pages and generate a documentation spec with `Purpose`, `User Can`, and `U ```bash npx explorbot docs collect /users/sign_in npx explorbot docs collect /docs/openapi#tag/project-analytics-tags --max-pages 20 +npx explorbot docs collect /dashboard --url https://app.example.com npx explorbot docs collect https://teleportal.ua/ua/serials/stb/kod --path explorbot-testing --show --session --max-pages 20 ``` +| Option | Description | +|---|---| +| `--url <url>` | Base URL of the site, for a relative path argument. Same as `EXPLORBOT_URL`; an absolute path argument carries its own | +| `--max-pages <count>` | Stop after documenting this many pages | + Output is written to: - `output/docs/spec.md` @@ -711,6 +728,7 @@ Every command takes these: | `-i, --instance <name>` | Which prima-owned browser to talk to; parallel work needs one each | | `--session [file]` | Cookies and storage persisted across processes; ignored while attached, since the attached session keeps its own | | `--url <url>` | Page to open when the session has no page yet | +| `--spec <path>` | A Docbot application spec directory or its `index.md`, read as page knowledge. Same as `EXPLORBOT_SPEC` / `PRIMA_CLI_SPEC` | | `--ephemeral` | Keep no state between runs. Applies to config-free runs only — with a config file the output directory comes from the config | | `--framework <name>` | Parsed but not active yet; reported code is CodeceptJS whatever you pass | | `-c, --config <path>`, `-p, --path <path>` | As on every other Explorbot command | @@ -760,6 +778,8 @@ Prima follows the same [configuration ladder](#environment-variables) as every o EXPLORBOT_AI_PROVIDER=groq npx explorbot prima go https://app.example.com ``` +The three inputs a run needs beyond the model come from flags or the environment, so no file has to exist: `--url` / `PRIMA_CLI_URL` for the site, `--spec` / `PRIMA_CLI_SPEC` for collected documentation, and `--knowledge` / `PRIMA_CLI_KNOWLEDGE` for facts such as credentials. Every `EXPLORBOT_*` variable has a `PRIMA_CLI_*` twin that prima reads first. + `pw` still works when no model is usable at all; commands that need one say so and point at the fallback. ## Plan Management diff --git a/docs/workflow/agentic-usage.md b/docs/workflow/agentic-usage.md index c699a3f4..521288d0 100644 --- a/docs/workflow/agentic-usage.md +++ b/docs/workflow/agentic-usage.md @@ -56,6 +56,7 @@ No `init`, no config file, no project directory, no model IDs to look up. These | `EXPLORBOT_EPHEMERAL` | no | Keep no state between runs — output goes to a fresh temp directory instead of the site dir | | `EXPLORBOT_KNOWLEDGE` | no | Inline knowledge text, applied to every page | | `EXPLORBOT_KNOWLEDGE_FILE` | no | Path to a knowledge markdown file | +| `EXPLORBOT_SPEC` | no | Docbot application spec directory or index.md, used as page knowledge | | `EXPLORBOT_API_SPEC` | no | OpenAPI spec path for the API boat | | `EXPLORBOT_NO_BANNER` | no | Suppress the startup banner, for machine-readable output | <!-- END env --> @@ -123,6 +124,12 @@ EXPLORBOT_AI_PROVIDER=openrouter \ EXPLORBOT_KNOWLEDGE_FILE=./checkout-knowledge.md npx explorbot explore /checkout ``` +Both variables work in config-free runs and in runs on the global configuration, where what they carry is written into the site's knowledge directory for that run — rewritten on the next run, and removed by a run that sets neither variable. Facts worth keeping belong in `learn` or `know`. The `--knowledge` flag does the same thing as an argument, works with a project config as well, and writes nothing, so prefer it when one command needs one fact: + +```bash +npx explorbot explore /checkout --knowledge 'Use the sandbox card 4111 1111 1111 1111' +``` + ### What this mode changes Config-free runs leave no trace in the working directory: @@ -212,9 +219,11 @@ The same variables drive API testing and doc collection. EXPLORBOT_URL=https://api.example.com \ EXPLORBOT_API_SPEC=./openapi.yaml \ EXPLORBOT_AI_PROVIDER=openrouter \ - npx explorbot api explore + npx explorbot api explore /users ``` +The API boat also takes those two as flags, so one line carries the whole run: `npx explorbot api explore /users --endpoint https://api.example.com --spec ./openapi.yaml`. + ```bash EXPLORBOT_AI_PROVIDER=openrouter \ npx explorbot docs collect https://app.example.com/dashboard --max-pages 20 @@ -222,7 +231,7 @@ EXPLORBOT_AI_PROVIDER=openrouter \ `docs collect` takes its base URL from the absolute path argument, so `EXPLORBOT_URL` is optional there. -Knowledge written by `EXPLORBOT_KNOWLEDGE` carries `endpoint: '*'` frontmatter alongside `url: '*'`, matching the convention `api init` and `api know` use. The API boat does not read knowledge at runtime yet; the frontmatter is there for when it does, and the web side ignores it. +Knowledge written by `EXPLORBOT_KNOWLEDGE` carries `endpoint: '*'` frontmatter alongside `url: '*'`, matching the convention `api init` and `api know` use, so one variable reaches both boats. ## See Also diff --git a/docs/workflow/knowledge.md b/docs/workflow/knowledge.md index 7d699057..1d0daa73 100644 --- a/docs/workflow/knowledge.md +++ b/docs/workflow/knowledge.md @@ -45,7 +45,48 @@ While exploring, use the `/learn` command. ### API Testing -[API testing](../api-testing/basics.md) shares the same `knowledge/` directory. `npx explorbot api know <endpoint> "<description>"` adds endpoint-scoped notes, stored with an `endpoint:` frontmatter field instead of `url:`. +[API testing](../api-testing/basics.md) shares the same `knowledge/` directory. `npx explorbot api know <endpoint> "<description>"` adds endpoint-scoped notes, stored with an `endpoint:` frontmatter field instead of `url:`. Chief reads them when planning an endpoint and Curler reads them when running its tests, so auth headers and payload rules reach both. + +## Per-Session Knowledge + +`--knowledge` passes facts to a single run. Nothing is written to `knowledge/`, so credentials and one-off test data stay out of the repository. + +```bash +npx explorbot explore /pay --knowledge 'My credit card is 4111 1111 1111 1111' +``` + +Plain text applies to every page. Add frontmatter to scope it, with the same URL patterns knowledge files use: + +```bash +npx explorbot explore / --knowledge '--- +url: /pay +--- +Use the sandbox card 4111 1111 1111 1111 with any future expiry' +``` + +Repeat the flag for several facts: + +```bash +npx explorbot explore / \ + --knowledge '--- +url: /login +--- +Log in as admin@example.com / secret123' \ + --knowledge 'Dismiss the cookie banner before anything else' +``` + +Everything a knowledge file supports works here: `${env.VAR}` interpolation, and page automation fields such as `wait` and `waitForElement`. + +The flag works on every command of `explorbot`, `explorbot api`, `explorbot docs` and `prima`, and can go anywhere on the line. Scope API knowledge with `endpoint:` instead of `url:`: + +```bash +npx explorbot api explore /orders --knowledge '--- +endpoint: /orders/* +--- +Send X-Api-Key: ${env.API_KEY} on every request' +``` + +For runs driven from the environment — config-free, or on the global configuration — `EXPLORBOT_KNOWLEDGE` does the same job for the length of one run. See [Agentic usage](./agentic-usage.md). ## URL Patterns diff --git a/src/api/spec-reader.ts b/src/api/spec-reader.ts index 0ef46ba1..2b279a9f 100644 --- a/src/api/spec-reader.ts +++ b/src/api/spec-reader.ts @@ -6,7 +6,7 @@ import { tag } from '../utils/logger.ts'; export function validateSpecs(specs?: string[]): void { if (!specs?.length) { - throw new Error('API spec is required. Set api.spec in your config file.'); + throw new Error('API spec is required. Pass --spec, set EXPLORBOT_API_SPEC, or set api.spec in your config file.'); } } diff --git a/src/commands/config-command.ts b/src/commands/config-command.ts index 22008039..786fbac6 100644 --- a/src/commands/config-command.ts +++ b/src/commands/config-command.ts @@ -37,7 +37,7 @@ export class ConfigCommand extends BaseCommand { const dirs: Record<string, string> = {}; if (options.root) { for (const [name, dir] of Object.entries({ output: 'output', ...config.dirs })) { - dirs[name] = path.join(options.root, dir); + dirs[name] = path.resolve(options.root, dir); } } diff --git a/src/commands/drill-command.ts b/src/commands/drill-command.ts index a2328de9..3808fe6f 100644 --- a/src/commands/drill-command.ts +++ b/src/commands/drill-command.ts @@ -26,7 +26,7 @@ export class DrillCommand extends BaseCommand { } private parseKnowledgeArg(args: string): string | undefined { - const match = args.match(/--knowledge\s+(\S+)/); + const match = args.match(/--save-knowledge\s+(\S+)/); return match ? match[1] : undefined; } diff --git a/src/commands/options/base-option.ts b/src/commands/options/base-option.ts new file mode 100644 index 00000000..e102694d --- /dev/null +++ b/src/commands/options/base-option.ts @@ -0,0 +1,18 @@ +import type { Command } from 'commander'; + +export abstract class BaseOption { + abstract flags: string; + abstract description: string; + collect?: (value: string, previous: any) => any; + + register(program: Command): void { + if (this.collect) program.option(this.flags, this.description, this.collect); + if (!this.collect) program.option(this.flags, this.description); + + program.hook('preAction', (_thisCommand, actionCommand) => { + this.apply(actionCommand.optsWithGlobals(), actionCommand); + }); + } + + protected abstract apply(options: Record<string, any>, command: Command): void; +} diff --git a/src/commands/options/index.ts b/src/commands/options/index.ts new file mode 100644 index 00000000..2f3e2949 --- /dev/null +++ b/src/commands/options/index.ts @@ -0,0 +1,7 @@ +import { KnowledgeOption } from './knowledge-option.js'; +import { WsOption } from './ws-option.js'; + +export { BaseOption } from './base-option.js'; + +export const knowledgeOption = new KnowledgeOption(); +export const wsOption = new WsOption(); diff --git a/src/commands/options/knowledge-option.ts b/src/commands/options/knowledge-option.ts new file mode 100644 index 00000000..4941bf0c --- /dev/null +++ b/src/commands/options/knowledge-option.ts @@ -0,0 +1,12 @@ +import { setOptionKnowledge } from '../../config.js'; +import { BaseOption } from './base-option.js'; + +export class KnowledgeOption extends BaseOption { + flags = '--knowledge <text>'; + description = 'Knowledge for this run only, not saved to disk. Markdown text; add url: or endpoint: frontmatter to scope it, otherwise it applies everywhere. Repeatable'; + collect = (value: string, previous: string[] = []) => [...previous, value]; + + protected apply(options: Record<string, any>): void { + setOptionKnowledge(options.knowledge || []); + } +} diff --git a/src/commands/options/ws-option.ts b/src/commands/options/ws-option.ts new file mode 100644 index 00000000..7e990a16 --- /dev/null +++ b/src/commands/options/ws-option.ts @@ -0,0 +1,24 @@ +import type { Command } from 'commander'; +import { remote } from '../../remote.js'; +import { BaseOption } from './base-option.js'; + +export class WsOption extends BaseOption { + flags = '--ws <url>'; + description = 'Stream this run to a remote UI over WebSocket'; + + protected apply(options: Record<string, any>, command: Command): void { + const url = options.ws || process.env.EXPLORBOT_WS_URL; + if (!url) return; + remote.attach(String(url), commandPath(command)); + } +} + +function commandPath(command: Command): string { + const parts: string[] = []; + let node: Command | null = command; + while (node) { + parts.unshift(node.name()); + node = node.parent; + } + return parts.slice(1).join(' ') || parts.join(' '); +} diff --git a/src/config.ts b/src/config.ts index fc960c13..a584d7ad 100644 --- a/src/config.ts +++ b/src/config.ts @@ -20,6 +20,7 @@ export const PROVIDERS: Record<string, ProviderInfo> = { }; let cachedOutputRoot: string | null = null; +let knowledgeFromOptions: string[] = []; interface PlaywrightConfig { browser: 'chromium' | 'firefox' | 'webkit'; @@ -271,6 +272,7 @@ export const EXPLORBOT_ENV_VARS: EnvVar[] = [ { name: 'EXPLORBOT_EPHEMERAL', description: 'Keep no state between runs — output goes to a fresh temp directory instead of the site dir' }, { name: 'EXPLORBOT_KNOWLEDGE', description: 'Inline knowledge text, applied to every page' }, { name: 'EXPLORBOT_KNOWLEDGE_FILE', description: 'Path to a knowledge markdown file' }, + { name: 'EXPLORBOT_SPEC', description: 'Docbot application spec directory or index.md, used as page knowledge' }, { name: 'EXPLORBOT_API_SPEC', description: 'OpenAPI spec path for the API boat' }, { name: 'EXPLORBOT_NO_BANNER', description: 'Suppress the startup banner, for machine-readable output' }, ]; @@ -404,6 +406,8 @@ export class ConfigParser { this.enterGlobalMode(this.config, target); } + this.applyEnvSpec(this.config); + // Restore original directory after successful config load if (options?.path && originalCwd !== process.cwd()) { process.chdir(originalCwd); @@ -554,10 +558,18 @@ export class ConfigParser { config.dirs = { knowledge: 'knowledge', experience: 'experience', output: 'output' }; config.playwright = { ...config.playwright, browser: config.playwright?.browser || 'chromium', url: site.baseUrl }; + materializeKnowledge(this.site.dir); log(`Global mode: ${site.baseUrl} stored in ${this.site.dir}`); } + private applyEnvSpec(config: ExplorbotConfig): void { + const spec = process.env.EXPLORBOT_SPEC; + if (!spec) return; + if (!config.dirs) config.dirs = { knowledge: 'knowledge', experience: 'experience', output: 'output' }; + config.dirs.spec = spec; + } + private async buildEnvConfig(baseUrl: string | undefined, outputRoot: string): Promise<ExplorbotConfig> { const provider = process.env.EXPLORBOT_AI_PROVIDER; const modelSpec = process.env.EXPLORBOT_AI_MODEL; @@ -842,6 +854,14 @@ export function resolveStateRoot(baseUrl: string, ephemeral?: boolean): string { return registerSite(url.origin).dir; } +export function setOptionKnowledge(values: string[]): void { + knowledgeFromOptions = values; +} + +export function optionKnowledge(): string[] { + return knowledgeFromOptions; +} + export function materializeKnowledge(outputRoot: string): void { const inline = process.env.EXPLORBOT_KNOWLEDGE; const knowledgeFile = process.env.EXPLORBOT_KNOWLEDGE_FILE; diff --git a/src/explorbot.ts b/src/explorbot.ts index 5421ce7f..580b71c3 100644 --- a/src/explorbot.ts +++ b/src/explorbot.ts @@ -166,7 +166,7 @@ export class ExplorBot { } knowledgeTracker(): KnowledgeTracker { - return (this._knowledgeTracker ||= new KnowledgeTracker(this.options.applicationSpec)); + return (this._knowledgeTracker ||= new KnowledgeTracker({ applicationSpec: this.options.applicationSpec })); } experienceTracker(): ExperienceTracker { diff --git a/src/knowledge-tracker.ts b/src/knowledge-tracker.ts index 71be2f9f..12a56e0a 100644 --- a/src/knowledge-tracker.ts +++ b/src/knowledge-tracker.ts @@ -4,13 +4,14 @@ import dedent from 'dedent'; import matter from 'gray-matter'; import { ActionResult } from './action-result.js'; import { ApplicationSpec } from './application-spec.ts'; -import { ConfigParser } from './config.js'; +import { ConfigParser, optionKnowledge } from './config.js'; import { getCliName } from './utils/cli-name.ts'; import { createDebug, pluralize, tag } from './utils/logger.js'; import { loadMarkdownFiles } from './utils/markdown-files.js'; import { mdq } from './utils/markdown-query.js'; import { isSecretName, registerSecret } from './utils/secrets.js'; import { slugify } from './utils/strings.js'; +import { extractStatePath, matchesUrl } from './utils/url-matcher.js'; const debugLog = createDebug('explorbot:knowledge-tracker'); @@ -24,23 +25,33 @@ export interface Knowledge { export class KnowledgeTracker { private knowledgeDir: string; private knowledgeFiles: Knowledge[] = []; + private sessionKnowledge: Knowledge[] = []; private isLoaded = false; private applicationSpec?: ApplicationSpec; - constructor(applicationSpecPath?: string) { - const configParser = ConfigParser.getInstance(); - const config = configParser.getConfig(); - this.knowledgeDir = configParser.resolveProjectDir(config.dirs?.knowledge || 'knowledge'); + constructor(options: KnowledgeTrackerOptions = {}) { + let knowledgeDir = options.knowledgeDir; + let specPath = options.applicationSpec; + + if (!knowledgeDir) { + const configParser = ConfigParser.getInstance(); + const config = configParser.getConfig(); + knowledgeDir = configParser.resolveProjectDir(config.dirs?.knowledge || 'knowledge'); + specPath ||= config.dirs?.spec; + } + + this.knowledgeDir = knowledgeDir; if (!existsSync(this.knowledgeDir)) { mkdirSync(this.knowledgeDir, { recursive: true }); } - const specPath = applicationSpecPath || config.dirs?.spec; if (specPath) { this.applicationSpec = new ApplicationSpec(specPath); tag('info').log(`Loaded application spec with ${this.applicationSpec.pageCount} documented pages`); } + + this.sessionKnowledge = this.parseSessionKnowledge(options.knowledge ?? optionKnowledge()); } private loadKnowledgeFiles(): void { @@ -63,28 +74,24 @@ export class KnowledgeTracker { getRelevantKnowledge(state: ActionResult): Knowledge[] { this.loadKnowledgeFiles(); - return this.knowledgeFiles.filter((knowledge) => { + return this.allKnowledge().filter((knowledge) => { return state.isMatchedBy(knowledge); }); } - renderRelevantKnowledge(state: ActionResult): string { - const knowledgeFiles = this.getRelevantKnowledge(state); - if (knowledgeFiles.length === 0) return ''; + getEndpointKnowledge(endpoint: string): Knowledge[] { + this.loadKnowledgeFiles(); + const path = extractStatePath(endpoint); - const knowledgeContent = knowledgeFiles - .map((k) => k.content) - .filter((k) => !!k) - .join('\n\n'); + return this.allKnowledge().filter((knowledge) => knowledge.endpoint && matchesUrl(knowledge.endpoint, path)); + } - tag('operation').log(`Found ${knowledgeFiles.length} relevant knowledge ${pluralize(knowledgeFiles.length, 'file')}`); - return dedent` - <knowledge> - Here is relevant knowledge for this page: + renderRelevantKnowledge(state: ActionResult): string { + return this.renderKnowledge(this.getRelevantKnowledge(state), 'page'); + } - ${knowledgeContent} - </knowledge> - `; + renderEndpointKnowledge(endpoint: string): string { + return this.renderKnowledge(this.getEndpointKnowledge(endpoint), 'endpoint'); } renderRelevantContext(state: ActionResult): string { @@ -242,4 +249,51 @@ export class KnowledgeTracker { return result; } + + private allKnowledge(): Knowledge[] { + return [...this.knowledgeFiles, ...this.sessionKnowledge]; + } + + private renderKnowledge(knowledgeFiles: Knowledge[], scope: string): string { + if (knowledgeFiles.length === 0) return ''; + + const knowledgeContent = knowledgeFiles + .map((k) => k.content) + .filter((k) => !!k) + .join('\n\n'); + + tag('operation').log(`Found ${knowledgeFiles.length} relevant knowledge ${pluralize(knowledgeFiles.length, 'file')}`); + return dedent` + <knowledge> + Here is relevant knowledge for this ${scope}: + + ${knowledgeContent} + </knowledge> + `; + } + + private parseSessionKnowledge(entries?: string[]): Knowledge[] { + if (!entries?.length) return []; + + return entries.map((entry, index) => { + const parsed = matter(entry); + const knowledge: Knowledge = { + filePath: `--knowledge #${index + 1}`, + url: parsed.data.url || parsed.data.path || '*', + content: this.interpolateVars(parsed.content.trim()), + ...parsed.data, + }; + + if (!parsed.data.url && !parsed.data.path && !parsed.data.endpoint) knowledge.endpoint = '*'; + debugLog(`Session knowledge for ${knowledge.url}`); + + return knowledge; + }); + } +} + +export interface KnowledgeTrackerOptions { + applicationSpec?: string; + knowledge?: string[]; + knowledgeDir?: string; } diff --git a/src/remote.ts b/src/remote.ts index 2f256c9e..9ed001da 100644 --- a/src/remote.ts +++ b/src/remote.ts @@ -1,4 +1,3 @@ -import type { Command } from 'commander'; import stripAnsi from 'strip-ansi'; import { type ActivityEntry, addActivityListener } from './activity.ts'; import { executionController } from './execution-controller.ts'; @@ -30,15 +29,6 @@ export class Remote implements LogDestination { private askCounter = 0; private lastActivity: string | null = null; - registerOption(program: Command): void { - program.option('--ws <url>', 'Stream this run to a remote UI over WebSocket'); - program.hook('preAction', (_thisCommand, actionCommand) => { - const url = actionCommand.optsWithGlobals().ws || process.env.EXPLORBOT_WS_URL; - if (!url) return; - this.attach(String(url), this.commandPath(actionCommand)); - }); - } - attach(url: string, command: string): void { if (this.url) return; this.url = url; @@ -225,16 +215,6 @@ export class Remote implements LogDestination { if (typeof failure.message === 'string') return failure.message; return undefined; } - - private commandPath(command: Command): string { - const parts: string[] = []; - let node: Command | null = command; - while (node) { - parts.unshift(node.name()); - node = node.parent; - } - return parts.slice(1).join(' ') || parts.join(' '); - } } export const remote = new Remote(); diff --git a/tests/unit/apibot-config.test.ts b/tests/unit/apibot-config.test.ts index 9dbe4431..8f068b9a 100644 --- a/tests/unit/apibot-config.test.ts +++ b/tests/unit/apibot-config.test.ts @@ -1,5 +1,5 @@ import { afterEach, beforeEach, describe, expect, it } from 'bun:test'; -import { mkdtempSync, rmSync } from 'node:fs'; +import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { ApibotConfigParser } from '../../boat/api-tester/src/config.ts'; @@ -65,6 +65,16 @@ describe('ApibotConfigParser environment fallback', () => { expect(config.api.spec).toEqual(['./openapi.yaml']); }); + it('overrides the config file endpoint and spec with the command options', async () => { + const configPath = join(outputRoot, 'apibot.config.js'); + writeFileSync(configPath, "export default { ai: { model: { modelId: 'test-model' } }, api: { baseEndpoint: 'https://staging.example.com/v1', spec: ['./staging.yaml'] } };\n", 'utf8'); + + const config = await parser.loadConfig({ config: configPath, baseEndpoint: 'https://api.example.com/v2/', spec: './openapi.yaml' }); + + expect(config.api.baseEndpoint).toBe('https://api.example.com/v2'); + expect(config.api.spec).toEqual(['./openapi.yaml']); + }); + it('throws when EXPLORBOT_URL is unset', async () => { process.env.EXPLORBOT_AI_MODEL = 'openrouter/openai/gpt-oss-120b'; await expect(parser.loadConfig()).rejects.toThrow(/EXPLORBOT_URL/); diff --git a/tests/unit/global-config.test.ts b/tests/unit/global-config.test.ts index e884501a..85555782 100644 --- a/tests/unit/global-config.test.ts +++ b/tests/unit/global-config.test.ts @@ -10,7 +10,7 @@ import InitWizard from '../../src/components/InitWizard.tsx'; import { ConfigParser, PROVIDERS } from '../../src/config.ts'; import { listSites, registerSite, resolveSiteTarget, siteFolderName } from '../../src/global-config.ts'; -const ENV_KEYS = ['EXPLORBOT_AI_PROVIDER', 'EXPLORBOT_AI_MODEL', 'EXPLORBOT_URL', 'EXPLORBOT_OUTPUT', 'GLOBAL_ONLY_KEY', 'SHARED_KEY']; +const ENV_KEYS = ['EXPLORBOT_AI_PROVIDER', 'EXPLORBOT_AI_MODEL', 'EXPLORBOT_URL', 'EXPLORBOT_OUTPUT', 'EXPLORBOT_SPEC', 'EXPLORBOT_API_SPEC', 'EXPLORBOT_KNOWLEDGE', 'EXPLORBOT_KNOWLEDGE_FILE', 'GLOBAL_ONLY_KEY', 'SHARED_KEY']; const GLOBAL_CONFIG = "export default { ai: { model: { modelId: 'global-model' } } };\n"; let home: string; @@ -47,6 +47,14 @@ function siteDir(folder: string): string { return join(home, '.explorbot', 'sites', folder); } +function resetApibotParser(): ApibotConfigParser { + const parser = ApibotConfigParser.getInstance(); + (parser as any).config = null; + (parser as any).configPath = null; + (parser as any).site = null; + return parser; +} + beforeEach(() => { savedEnv = { HOME: process.env.HOME }; for (const key of ENV_KEYS) { @@ -263,6 +271,26 @@ describe('global mode', () => { expect(parser.getOutputDir()).toBe(join(siteDir('app.example.com'), 'output')); }); + it('writes environment knowledge into the site knowledge directory', async () => { + const parser = ConfigParser.getInstance(); + writeGlobalConfig(); + process.env.EXPLORBOT_KNOWLEDGE = 'Log in as admin@example.com / secret123'; + + await parser.loadConfig({ path: workDir, from: 'https://app.example.com' }); + + expect(readFileSync(join(siteDir('app.example.com'), 'knowledge', 'global.md'), 'utf8')).toContain('Log in as admin@example.com / secret123'); + }); + + it('takes the application spec from the environment', async () => { + const parser = ConfigParser.getInstance(); + writeGlobalConfig(); + process.env.EXPLORBOT_SPEC = join(workDir, 'docs'); + + const config = await parser.loadConfig({ path: workDir, from: 'https://app.example.com' }); + + expect(config.dirs?.spec).toBe(join(workDir, 'docs')); + }); + it('falls back to a URL pinned in the global config when the command passes none', async () => { const parser = ConfigParser.getInstance(); writeGlobalConfig("export default { web: { url: 'https://pinned.example.com' }, ai: { model: { modelId: 'global-model' } } };\n"); @@ -322,10 +350,7 @@ describe('global mode', () => { describe('global mode in the API boat', () => { it('derives the base endpoint and site folder from the endpoint host', async () => { writeGlobalConfig(); - const parser = ApibotConfigParser.getInstance(); - (parser as any).config = null; - (parser as any).configPath = null; - (parser as any).site = null; + const parser = resetApibotParser(); const config = await parser.loadConfig({ path: workDir, endpoint: 'https://api.example.com/users' }); @@ -334,9 +359,46 @@ describe('global mode in the API boat', () => { expect(parser.getKnowledgeDir()).toBe(join(siteDir('api.example.com'), 'knowledge')); expect(parser.resolveEndpointPath('https://api.example.com/users')).toBe('/users'); - (parser as any).config = null; - (parser as any).configPath = null; - (parser as any).site = null; + resetApibotParser(); + }); + + it('keeps the path prefix of the endpoint option in the base endpoint', async () => { + writeGlobalConfig(); + const parser = resetApibotParser(); + + const config = await parser.loadConfig({ path: workDir, baseEndpoint: 'https://api.example.com/v2/team' }); + + expect(config.api.baseEndpoint).toBe('https://api.example.com/v2/team'); + expect(parser.getOutputDir()).toBe(join(siteDir('api.example.com'), 'output')); + expect(parser.resolveEndpointPath('/users')).toBe('/users'); + expect(parser.resolveEndpointPath('https://api.example.com/v2/team/users')).toBe('/users'); + expect(parser.resolveEndpointPath('https://api.example.com/v2/team')).toBe('/'); + + resetApibotParser(); + }); + + it('takes the spec from the command option over the config file', async () => { + writeGlobalConfig("export default { ai: { model: { modelId: 'global-model' } }, api: { spec: ['from-config.yaml'] } };\n"); + const parser = resetApibotParser(); + + const config = await parser.loadConfig({ path: workDir, baseEndpoint: 'https://api.example.com', spec: 'from-option.yaml' }); + + expect(config.api.spec).toEqual(['from-option.yaml']); + expect(process.env.EXPLORBOT_API_SPEC).toBe('from-option.yaml'); + + resetApibotParser(); + }); + + it('writes environment knowledge into the site knowledge directory', async () => { + writeGlobalConfig(); + process.env.EXPLORBOT_KNOWLEDGE = 'Send a bearer token with every request'; + const parser = resetApibotParser(); + + await parser.loadConfig({ path: workDir, baseEndpoint: 'https://api.example.com' }); + + expect(readFileSync(join(siteDir('api.example.com'), 'knowledge', 'global.md'), 'utf8')).toContain('Send a bearer token with every request'); + + resetApibotParser(); }); }); diff --git a/tests/unit/knowledge-tracker.test.ts b/tests/unit/knowledge-tracker.test.ts index 0551a34d..369439d5 100644 --- a/tests/unit/knowledge-tracker.test.ts +++ b/tests/unit/knowledge-tracker.test.ts @@ -1,10 +1,12 @@ import { afterEach, beforeEach, describe, expect, it } from 'bun:test'; -import { existsSync, rmSync } from 'node:fs'; +import { existsSync, readdirSync, rmSync } from 'node:fs'; import { mkdirSync, writeFileSync } from 'node:fs'; +import { Command } from 'commander'; import matter from 'gray-matter'; import { ActionResult } from '../../src/action-result.js'; import { APPLICATION_SPEC_FORMAT, APPLICATION_SPEC_VERSION } from '../../src/application-spec-contract.ts'; import { ConfigParser } from '../../src/config'; +import { knowledgeOption } from '../../src/commands/options/index'; import { KnowledgeTracker } from '../../src/knowledge-tracker'; import { clearRegisteredSecrets, redactSecrets } from '../../src/utils/secrets'; @@ -76,7 +78,7 @@ describe('KnowledgeTracker', () => { }), 'utf8' ); - const tracker = new KnowledgeTracker(applicationSpecDir); + const tracker = new KnowledgeTracker({ applicationSpec: applicationSpecDir }); const state = new ActionResult({ url: '/login', html: '<html></html>' }); const rendered = tracker.renderRelevantContext(state); @@ -261,4 +263,76 @@ describe('KnowledgeTracker', () => { expect(matched[0].content).not.toContain('first entry'); }); }); + + describe('session knowledge', () => { + it('applies knowledge without frontmatter to every page and endpoint', () => { + const tracker = new KnowledgeTracker({ knowledge: ['My credit card is 4111 1111 1111 1111'] }); + + expect(tracker.renderRelevantKnowledge(new ActionResult({ url: '/pay' }))).toContain('4111 1111 1111 1111'); + expect(tracker.renderRelevantKnowledge(new ActionResult({ url: '/anywhere-else' }))).toContain('4111 1111 1111 1111'); + expect(tracker.renderEndpointKnowledge('/payments')).toContain('4111 1111 1111 1111'); + }); + + it('scopes knowledge with url frontmatter to matching pages only', () => { + const tracker = new KnowledgeTracker({ knowledge: [matter.stringify('Card expires 12/30', { url: '/pay' })] }); + + expect(tracker.renderRelevantKnowledge(new ActionResult({ url: '/pay' }))).toContain('Card expires 12/30'); + expect(tracker.renderRelevantKnowledge(new ActionResult({ url: '/dashboard' }))).toBe(''); + expect(tracker.renderEndpointKnowledge('/pay')).toBe(''); + }); + + it('scopes knowledge with endpoint frontmatter to matching endpoints', () => { + const tracker = new KnowledgeTracker({ knowledge: [matter.stringify('Send X-Token header', { endpoint: '/users/*' })] }); + + expect(tracker.renderEndpointKnowledge('/users/42')).toContain('Send X-Token header'); + expect(tracker.renderEndpointKnowledge('/orders')).toBe(''); + }); + + it('keeps several entries independent', () => { + const tracker = new KnowledgeTracker({ + knowledge: [matter.stringify('Login as admin', { url: '/login' }), matter.stringify('Use the sandbox card', { url: '/pay' })], + }); + + const rendered = tracker.renderRelevantKnowledge(new ActionResult({ url: '/pay' })); + expect(rendered).toContain('Use the sandbox card'); + expect(rendered).not.toContain('Login as admin'); + }); + + it('exposes frontmatter hints through state parameters', () => { + const tracker = new KnowledgeTracker({ knowledge: [matter.stringify('Slow page', { url: '/reports', wait: 3000 })] }); + + expect(tracker.getStateParameters(new ActionResult({ url: '/reports' }), ['wait'])).toEqual({ wait: 3000 }); + }); + + it('interpolates environment variables', () => { + process.env.EXPLORBOT_TEST_TOKEN = 'abc123'; + const tracker = new KnowledgeTracker({ knowledge: ['Token is ${env.EXPLORBOT_TEST_TOKEN}'] }); + + expect(tracker.renderRelevantKnowledge(new ActionResult({ url: '/any' }))).toContain('Token is abc123'); + Reflect.deleteProperty(process.env, 'EXPLORBOT_TEST_TOKEN'); + }); + + it('reaches the tracker from a --knowledge flag anywhere on the command line', () => { + const program = new Command(); + knowledgeOption.register(program); + const rendered: string[] = []; + program.command('explore <path>').action(() => { + rendered.push(new KnowledgeTracker().renderRelevantKnowledge(new ActionResult({ url: '/pay' }))); + }); + + program.parse(['explore', '/', '--knowledge', matter.stringify('Use the sandbox card', { url: '/pay' })], { from: 'user' }); + program.parse(['explore', '/'], { from: 'user' }); + + expect(rendered[0]).toContain('Use the sandbox card'); + expect(rendered[1]).toBe(''); + }); + + it('never writes session knowledge to the knowledge directory', () => { + const tracker = new KnowledgeTracker({ knowledge: ['Temporary fact'] }); + tracker.renderRelevantKnowledge(new ActionResult({ url: '/any' })); + + expect(tracker.listAllKnowledge()).toHaveLength(0); + expect(readdirSync(knowledgeDir)).toHaveLength(0); + }); + }); }); diff --git a/tests/unit/remote.test.ts b/tests/unit/remote.test.ts index d53fcbb1..39a27ece 100644 --- a/tests/unit/remote.test.ts +++ b/tests/unit/remote.test.ts @@ -1,8 +1,10 @@ import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; import type { ServerWebSocket } from 'bun'; +import { Command } from 'commander'; import Action from '../../src/action.ts'; import { saveResearch } from '../../src/ai/researcher/cache.ts'; import { SessionAnalyst } from '../../src/ai/session-analyst.ts'; +import { wsOption } from '../../src/commands/options/index.ts'; import { isInteractive } from '../../src/ai/task-agent.ts'; import { ConfigParser } from '../../src/config.ts'; import { executionController } from '../../src/execution-controller.ts'; @@ -61,6 +63,17 @@ afterEach(async () => { }); describe('remote', () => { + test('the --ws option attaches the run and announces the command it names', async () => { + const program = new Command(); + wsOption.register(program); + const boat = program.command('api'); + boat.command('plan <endpoint>').action(() => {}); + + program.parse(['api', 'plan', '/users', '--ws', url()], { from: 'user' }); + + expect(await waitFor(frameOf('hello'))).toMatchObject({ command: 'api plan', pid: process.pid }); + }); + test('announces the run, and frames sent before the socket opens are flushed in order', async () => { remote.attach(url(), 'explore'); remote.send('activity', { message: 'first' });