Skip to content

Add hosted Factory lifecycle telemetry #197

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
khaliqgant merged 2 commits into from
Jul 23, 2026
Merged

Add hosted Factory lifecycle telemetry #197

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
30e65e8
feat(hosted): report durable lifecycle telemetry
khaliqgant Jul 23, 2026
e2f56dc
fix(hosted): dedupe clarification telemetry
khaliqgant Jul 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .agentworkforce/agents/factory-feature-guardian/manifest-contract.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,7 @@ describe('Factory feature manifest contract', () => {
const expected: Record<string, string> = {
'.': 'createFactory()',
'./observability': '@agent-relay/factory/observability',
'./telemetry': '@agent-relay/factory/telemetry',
'./testing': '@agent-relay/factory/testing',
'./writeback': 'LinearWriteback / GithubWriteback / SlackWriteback',
'./node': 'createFactoryNodeDefinition()',
Expand Down
22 changes: 18 additions & 4 deletions .agentworkforce/features/manifest.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
version: '1.1'
updated: '2026-07-21'
updated: '2026-07-23'
catalog:
category_count: 25
feature_count: 302
feature_count: 304
tier_counts:
1: 44
2: 123
1: 45
2: 124
3: 11
4: 51
5: 58
Expand Down Expand Up @@ -1454,6 +1454,13 @@ categories:
location: src/observability/, src/index.ts, package.json
verify_tier: 1

- id: api-telemetry-contract
name: Worker-Safe Telemetry Contract API
api: '@agent-relay/factory/telemetry'
description: Export only the Factory-owned event schemas, creator, and reporter types without pulling filesystem-backed outbox or instance identity modules into Worker consumers
location: src/telemetry.ts, src/observability/events.ts, src/ports/observability.ts, package.json
verify_tier: 1

- id: api-hosted-control-plane
name: Hosted Factory API
api: '@agent-relay/factory/hosted'
Expand Down Expand Up @@ -1494,6 +1501,13 @@ categories:
location: src/hosted/orchestrator.ts
verify_tier: 2

- id: hosted-durable-run-telemetry
name: Durable Hosted Run Telemetry
api: hostedFactoryRunId() / HostedFactoryPorts.reporter
description: Persist one deterministic run identity per issue and report lifecycle, agent, clarification, writeback, success, and failure events only after their fenced hosted state writes
location: src/hosted/orchestrator.ts, src/hosted/types.ts, src/telemetry.ts
verify_tier: 2

- id: hosted-merge-gate-writeback
name: Hosted Merge Gate and Idempotent Writeback
api: HostedFactoryMergeGate / HostedFactoryWriteback
Expand Down
11 changes: 7 additions & 4 deletions .agentworkforce/features/verify/procedures.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -688,7 +688,7 @@ mkdir "$TMP/consumer" && cd "$TMP/consumer"
npm init -y >/dev/null
npm install --ignore-scripts "$TMP"/*.tgz >/dev/null
node --input-type=module <<'NODE'
for (const subpath of ['', '/observability', '/testing', '/writeback', '/featuremap', '/hosted', '/environments']) {
for (const subpath of ['', '/observability', '/telemetry', '/testing', '/writeback', '/featuremap', '/hosted', '/environments']) {
const mod = await import(`@agent-relay/factory${subpath}`)
if (Object.keys(mod).length === 0) throw new Error(`empty export ${subpath || '/'}`)
}
Expand All @@ -698,7 +698,7 @@ NODE
```

Back in the checkout, run the focused tests named by each API entry. Assert root
types and all eight package export keys resolve, hosted remains worker-safe,
types and all nine JavaScript package entrypoints resolve, hosted remains worker-safe,
feature-map validation includes procedure routing, dependency/worktree ports are
exported, and fake clients support a hermetic consumer. Clean only `$TMP`.

Expand All @@ -718,8 +718,11 @@ npm run verify:e2e
```

Assert reconciliation precedes discovery; scope, dedupe, batch capacity, triage,
deterministic invocation IDs, clarification, spawn, pushed completion, polling
completion, merge gate, and every provider writeback transition. Repeat each
deterministic run and invocation IDs, clarification, spawn, pushed completion,
polling completion, merge gate, and every provider writeback transition. Require
the exact persisted lifecycle event order, no second start or spawn event on a
duplicate sweep, correct succeeded/failed terminal status, and unchanged
dispatch/state when a custom reporter rejects. Repeat each
external success immediately before a simulated fenced save and require the
same invocation or idempotency key after takeover. Two owners must yield one
live epoch; expired owners cannot save. Run the same matrix through in-memory
Expand Down
10 changes: 9 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -434,12 +434,20 @@ import {
const state = new DurableObjectHostedFactoryStateStore(durableObjectState.storage)
const factory = createHostedFactory(
{ workspaceId, ownerId: isolateId, config },
{ state, discovery, fleet, completions, mergeGate, writeback },
{ state, discovery, fleet, completions, mergeGate, writeback, reporter },
)

await factory.runOnce() // invoke from cron/alarm and safe webhook wakeups
```

The optional `reporter` implements the canonical `FactoryEventReporter`.
Hosted runs persist a deterministic run ID per workspace and issue, then emit
lifecycle events only after the corresponding fenced state write. Worker hosts
that only need the Factory-owned wire schema, event creator, and reporter types
should import `@agent-relay/factory/telemetry`; unlike the broader
`@agent-relay/factory/observability` surface, it does not export the
filesystem-backed outbox or instance identity helpers.

The Durable Object adapter stores each workspace independently and performs
lease claims plus lifecycle writes in storage transactions. Every mutation is
checked against the current owner and monotonically increasing lease epoch; an
Expand Down
4 changes: 4 additions & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@
"types": "./dist/observability/index.d.ts",
"import": "./dist/observability/index.js"
},
"./telemetry": {
"types": "./dist/telemetry.d.ts",
"import": "./dist/telemetry.js"
},
Comment on lines +41 to +44

@coderabbitai coderabbitai Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Confirm telemetry is a build entry and that dist emission is covered by a test.
fd -t f 'tsconfig*.json' | xargs -I{} sh -c 'echo "== {} =="; cat "{}"'
fd -a 'telemetry.ts' src
rg -nP 'telemetry' -g '*.mjs' -g '*.cjs' -g '*.json' -g '*.ts' --glob '!**/node_modules/**' -C2 | rg -i 'entry|build|dist|rollup|esbuild|tsup'
fd 'dist-entrypoints.test.ts' src -x cat -n

Repository: AgentWorkforce/factory

Length of output: 647

🏁 Script executed:

#!/bin/bash
set -u

echo "== package exports/files/build scripts =="
python3 - <<'PY'
import json, pathlib
p=pathlib.Path('package.json')
data=json.loads(p.read_text())
for key in ['main','type','files','exports','scripts']:
    if key in data:
        print(f'-- {key} --')
        print(json.dumps(data[key], indent=2, sort_keys=True))
PY

echo "== telemetry export and nearby package fields =="
sed -n '1,90p' package.json | cat -n

echo "== files mentioning dist emission tests/config =="
rg -n "dist-entrypoints|outDir|emitDeclaration|declaration|exports|telemetry|src/telemetry|dist/" -g '*.js' -g '*.mjs' -g '*.cjs' -g '*.ts' -g '*.json' -g '*.md' --glob '!**/node_modules/**' .

echo "== locate config/build/test files =="
git ls-files | rg '(^package\.json$|(^|/)dist-entrypoints\.test\.(ts|js|mjs)$|(^|/)rollup|(^|/)tsup|(^|/)vite|(^|/)webpack|(^|/)tsconfig.*\.json$|(^|/)src/telemetry\.(ts|js)$|test/.*dist|vitest|jest)' | sort

Repository: AgentWorkforce/factory

Length of output: 34825

🏁 Script executed:

#!/bin/bash
set -u

echo "== tsconfig.build.json =="
cat -n tsconfig.build.json

echo "== telemetry.ts top exports =="
sed -n '1,120p' src/telemetry.ts | cat -n

echo "== dist-entrypoints.test.ts current imports =="
cat -n src/__tests__/dist-entrypoints.test.ts | sed -n '1,220p'

Repository: AgentWorkforce/factory

Length of output: 3684

Add dist/telemetry.js to the published dist entrypoint test.

tsconfig.build.json includes src/**/* and emits declarations to dist, so ./telemetry can resolve at consumer time, but src/__tests__/dist-entrypoints.test.ts only imports the other top-level dist exports. If dist/telemetry.js or dist/telemetry.d.ts is not emitted/published, that export will fail at runtime/compile-time for consumers.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@package.json` around lines 41 - 44, Update the top-level dist entrypoint
coverage in dist-entrypoints.test.ts to import and validate the telemetry
export, including its runtime JavaScript and declaration output. Use the
existing test pattern for other dist exports and ensure the test catches missing
dist/telemetry.js or dist/telemetry.d.ts artifacts.

"./testing": {
"types": "./dist/testing/index.d.ts",
"import": "./dist/testing/index.js"
Expand Down
33 changes: 33 additions & 0 deletions scripts/packed-e2e-scenario.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import {
createHostedFactory,
InMemoryHostedFactoryStateStore,
} from '@agent-relay/factory/hosted'
import { FactoryCloudEventInputV1Schema } from '@agent-relay/factory/telemetry'

const checks = ['public-exports-import']
const workspaceId = 'factory-e2e-workspace'
Expand Down Expand Up @@ -53,6 +54,7 @@ const state = new InMemoryHostedFactoryStateStore({ now: () => nowMs })
const spawned = []
const invocationStatus = new Map()
const writebacks = { clarification: [], dispatched: [], completed: [] }
const telemetryEvents = []
const ports = {
discovery: { discoverReady: async () => [structuredClone(issue), structuredClone(issue)] },
state,
Expand All @@ -72,6 +74,15 @@ const ports = {
dispatched: async (input) => writebacks.dispatched.push(structuredClone(input)),
completed: async (input) => writebacks.completed.push(structuredClone(input)),
},
reporter: {
report: async (event) => telemetryEvents.push(FactoryCloudEventInputV1Schema.parse(event)),
flush: async () => ({
delivered: telemetryEvents.length,
pending: 0,
attempts: telemetryEvents.length,
stoppedReason: 'empty',
}),
},
now: () => new Date(nowMs),
}
const factory = createHostedFactory({
Expand All @@ -87,14 +98,28 @@ assert.equal(dispatched.discovered, 1, 'duplicate discovery entries must collaps
assert.equal(spawned.length, 2, 'single scope must dispatch implementer plus reviewer')
assert.equal(new Set(spawned.map(({ invocationId }) => invocationId)).size, 2)
assert.equal(writebacks.dispatched.length, 1)
assert.deepEqual(telemetryEvents.map(({ type }) => type), [
'run.started',
'run.phase_changed',
'agent.planned',
'agent.planned',
'agent.spawned',
'agent.spawned',
'run.phase_changed',
'writeback.applied',
])
assert.equal(new Set(telemetryEvents.map(({ runId }) => runId)).size, 1)
checks.push('discover-triage-dispatch-writeback')
checks.push('hosted-telemetry-dispatch-lifecycle')

const eventCountAfterDispatch = telemetryEvents.length
const duplicateSweep = await factory.runOnce()
assert.equal(spawned.length, 2, 'repeat discovery must not duplicate fleet spawns')
assert.ok(duplicateSweep.skipped.some(({ issueKey, reason }) =>
issueKey === issue.key && reason === 'already running'
))
assert.equal(writebacks.dispatched.length, 1, 'dispatch writeback must be idempotent')
assert.equal(telemetryEvents.length, eventCountAfterDispatch, 'repeat discovery must not duplicate lifecycle events')
checks.push('at-least-once-deduplication')

nowMs += 5_000
Expand All @@ -112,6 +137,14 @@ const terminal = await state.getIssue(workspaceId, issue.uuid)
assert.equal(terminal?.phase, 'complete')
assert.equal(terminal?.mergeGate?.status, 'ready')
assert.equal(writebacks.completed.length, 1)
assert.deepEqual(telemetryEvents.slice(eventCountAfterDispatch).map(({ type }) => type), [
'agent.exited',
'agent.exited',
'run.phase_changed',
'writeback.applied',
'run.succeeded',
])
assert.equal(telemetryEvents.at(-1)?.status, 'succeeded')
checks.push('completion-merge-gate-writeback')

const replayed = await factory.ingestCompletion({
Expand Down
1 change: 1 addition & 0 deletions src/hosted/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ export {
createHostedFactory,
HostedFactoryLoop,
hostedFactoryInvocationId,
hostedFactoryRunId,
} from './orchestrator.js'
export {
DurableObjectHostedFactoryStateStore,
Expand Down
Loading