Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,15 @@ const TOUR_WORKFLOW = { path: 'tour.json' }
const OTHER_WORKFLOW = { path: 'other.json' }
const INTRO_PREVIEW_MS = 500
const OFFLINE_GRACE_MS = 20_000
const ACCEPT_DEADLINE_MS = 15_000

const mocks = vi.hoisted(() => ({
canRunWorkflows: { value: true },
showSubscriptionDialog: vi.fn(),
workflowStatus: { value: new Map<unknown, string>() },
executionErrors: { hasNodeError: false, hasPromptError: false },
activeWorkflow: { value: null as unknown },
queuedJobs: { value: {} as Record<string, { workflow?: unknown }> },
linearMode: { value: false },
vueNodesEnabled: true,
steps: [] as CoachStep[],
Expand All @@ -45,29 +47,33 @@ vi.mock('@/composables/billing/useBillingContext', () => ({
})
}))

// Each factory runs on the first dynamic import, which lands mid-test for
// whichever test runs first. Seed the new ref from the holder so that test's
// setup survives instead of being discarded.
vi.mock('@/stores/executionStore', async () => {
const { shallowRef } = await import('vue')
mocks.workflowStatus = shallowRef(new Map<unknown, string>())
mocks.workflowStatus = shallowRef(new Map(mocks.workflowStatus.value))
mocks.queuedJobs = shallowRef(mocks.queuedJobs.value)
return {
useExecutionStore: () => ({
getWorkflowStatus: (workflow: unknown) =>
mocks.workflowStatus.value.get(workflow)
mocks.workflowStatus.value.get(workflow),
get queuedJobs() {
return mocks.queuedJobs.value
}
})
}
})

vi.mock('@/stores/executionErrorStore', async () => {
const { reactive } = await import('vue')
mocks.executionErrors = reactive({
hasNodeError: false,
hasPromptError: false
})
mocks.executionErrors = reactive({ ...mocks.executionErrors })
return { useExecutionErrorStore: () => mocks.executionErrors }
})

vi.mock('@/platform/workflow/management/stores/workflowStore', async () => {
const { shallowRef } = await import('vue')
mocks.activeWorkflow = shallowRef(null as unknown)
mocks.activeWorkflow = shallowRef(mocks.activeWorkflow.value)
return {
useWorkflowStore: () => ({
get activeWorkflow() {
Expand All @@ -79,7 +85,7 @@ vi.mock('@/platform/workflow/management/stores/workflowStore', async () => {

vi.mock('@/renderer/core/canvas/canvasStore', async () => {
const { shallowRef } = await import('vue')
mocks.linearMode = shallowRef(false)
mocks.linearMode = shallowRef(mocks.linearMode.value)
return {
useCanvasStore: () => ({
get linearMode() {
Expand Down Expand Up @@ -190,6 +196,18 @@ const EVERY_ENDING: { named: string; ending: TourEnding }[] = [
...UNFINISHED_ENDINGS
]

/** The queue storing a job, which is what acceptance actually looks like. */
function acceptRun(workflow: unknown) {
mocks.queuedJobs.value = { 'job-1': { workflow } }

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER — this write leaks into every later test; the deadline test only passes by that leak.

beforeEach resets every mock except mocks.queuedJobs, and TOUR_WORKFLOW is module-level, so after the first acceptRun() every subsequent test runs with tourRunAccepted === true and startAcceptDeadline() early-returns — the deadline is suppressed, not disarmed. On a clean export of this head:

vitest run src/renderer/extensions/firstRunTour/tour/useFirstRunTourController.test.ts \
  -t 'leaves an accepted run past the acceptance deadline alone'
# FAIL: expected 'generating', received 'failed'

while the full file passes 54/54. So the suite is order-dependent (.only, sharding, or a reorder breaks CI), and that test's own requirement — "a job that answered must never be cut short" — is not currently verified at all.

Note from trialing the obvious fix: resetting mocks.queuedJobs.value = {} in beforeEach plus restoring the status disarm makes the full file pass, but the isolated -t run still failed — the status watcher never saw the finishRun write in isolation, which points at an additional wiring subtlety between vi.resetModules() in freshController and the hoisted mock refs. Please validate whatever fix lands with the isolated run, not just the file run.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up — I chased the "wiring subtlety" to ground. Three distinct things, each demonstrated by a run on a clean export of this head, no speculation:

1. The order-dependence is wider than one test. Adding only mocks.queuedJobs.value = {} to beforeEach (controller untouched) fails four tests in the full file:

  • leaves an accepted run past the acceptance deadline alone
  • keeps waiting when the socket comes back
  • leaves a run that is still reporting alone
  • lets a recovered socket cancel every retry that preceded it

All four assert that a run with a 'running' status survives past 15s; today they pass only because this acceptRun write leaks a matching queuedJobs entry into them.

2. The isolated-run failure has a second, pre-existing cause. The store-mock factories run on the first dynamic import — which is mid-test for whichever test runs first — and each factory replaces the hoisted holder, discarding whatever the test already wrote to it. A probe inside the workflowStore factory during an isolated run:

[probe] workflowStore factory RUN; discarding activeWorkflow holder with value= { path: 'tour.json' }

After that, beginTour reads null, tourWorkflow stays null, and every getWorkflowStatus(tourWorkflow.value) lookup key-misses. That's why the pre-PR test says so when the run produced nothing also fails in isolation (expected 'idle' to be 'failed') — this trap predates your PR and hits the whole file under -t/.only. (The engine mock is immune for exactly the right reason: reactive(mocks.engine) wraps the same object.)

3. A combination validated green both ways (54/54 full file, plus isolated runs of the deadline test, the D2 test, and the socket-return test):

  • controller: restore if (status !== undefined) stopAcceptDeadline() alongside the acceptance watch
  • beforeEach: add mocks.queuedJobs.value = {}
  • factories: seed instead of discard — mocks.activeWorkflow = shallowRef(mocks.activeWorkflow.value), mocks.queuedJobs = shallowRef(mocks.queuedJobs.value), mocks.workflowStatus = shallowRef(new Map(mocks.workflowStatus.value)), mocks.linearMode = shallowRef(mocks.linearMode.value), mocks.executionErrors = reactive({ ...mocks.executionErrors })

Item 2 is pre-existing, so splitting it into its own PR is reasonable — the blockers here hang on item 1 plus the controller line.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarification — my last comment buried a dependency and it's causing confusion. Plain version.

There are three fixes. They do different jobs:

  1. Controller: put back if (status !== undefined) stopAcceptDeadline() next to the acceptance watch. This fixes the behavior bug.
  2. Test file: add mocks.queuedJobs.value = {} to beforeEach. This makes the four leak-shielded tests honest.
  3. Test file: make each mock factory carry the holder's value into the new ref instead of throwing it away: mocks.activeWorkflow = shallowRef(mocks.activeWorkflow.value) — same pattern for workflowStatus, queuedJobs, linearMode, executionErrors. This — and only this — fixes isolated runs.

What each step gets you (every row verified by running it):

Applied vitest run <file> vitest run <file> -t '<one test>'
nothing (head as-is) 54/54, but 4 tests pass only via leaked state deadline test fails
1 + 2 54/54, honest still fails
1 + 2 + 3 54/54 passes

The sentence I should have written the first time: fixes 1+2 alone will never make an isolated run pass. Only fix 3 does that. The isolated failure without fix 3 is not caused by anything in this PR. The mock factories run in the middle of whichever test executes first, and they throw away values that test already wrote (probe output: the factory discarded activeWorkflow = { path: 'tour.json' } mid-test). In a full run the "first test" is a starting-section test that doesn't care. In an isolated run, your test is the first test, so it always gets hit.

So pick one of two paths and validate accordingly:

  • Fix 3 goes in this PR: run both commands above. Both must be green.
  • Fix 3 goes to a follow-up: validate with the full-file run only. Isolated runs stay red until the follow-up lands, and that red is the pre-existing trap — not your change, and not a signal about fixes 1+2.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 2e0ba75, all three fixes including the factory seeding, so this validates both ways.

reproduced your control first: adding only the queuedJobs reset failed precisely the four tests you named, nothing else.

after the full fix, full file is 54/54 and the isolated runs pass, including the deadline test, says so when the run produced nothing, keeps waiting when the socket comes back, leaves a run that is still reporting alone, lets a recovered socket cancel every retry that preceded it, lets the offline grace outlive the acceptance deadline, and both acceptance tests.

also checked the tests are honest now rather than passing some other way: reverting only the controller line re-fails exactly those four.

thanks for chasing the factory behavior to ground, the mid-test holder replacement is not something I would have found from the failure alone.

return nextTick()
}

/** The queue letting a job go, which `resetExecutionState` does silently. */
function removeRun() {
mocks.queuedJobs.value = {}
return nextTick()
}

function finishRun(workflow: unknown, status: string) {
mocks.workflowStatus.value = new Map(mocks.workflowStatus.value).set(
workflow,
Expand Down Expand Up @@ -230,6 +248,7 @@ describe('useFirstRunTourController', () => {
beforeEach(() => {
mocks.canRunWorkflows = ref(true)
mocks.workflowStatus.value = new Map()
mocks.queuedJobs.value = {}
mocks.executionErrors.hasNodeError = false
mocks.executionErrors.hasPromptError = false
mocks.activeWorkflow.value = null
Expand Down Expand Up @@ -348,14 +367,84 @@ describe('useFirstRunTourController', () => {
})

describe('a run behind a dropped socket', () => {
/**
* A run the queue accepted: the click reports `generating`, then the
* backend answers with a status. The acknowledgement matters — an
* unacknowledged submission is a refusal, and is covered separately below.
*/
async function generatingRun() {
await tourOnRunStep()
mountRunButton('queue-button', () => {}).click()
expect(mocks.runState.value).toBe('generating')
await finishRun(TOUR_WORKFLOW, 'running')
const { api } = await import('@/scripts/api')
return api
}

it('stops promising a result the queue never accepted', async () => {
await tourOnRunStep()
mountRunButton('queue-button', () => {}).click()
expect(mocks.runState.value).toBe('generating')

// No status ever arrives. A refused submission gets no prompt_id, and
// account preconditions - sign-in, subscription, credits - are kept out
// of the error stores on purpose, so nothing else can report this.
await vi.advanceTimersByTimeAsync(ACCEPT_DEADLINE_MS)

expect(
mocks.runState.value,
'a paid user out of credits is refused silently; the card must not promise a result forever'
).toBe('failed')
})

it('leaves a run accepted but still waiting for a machine alone', async () => {
// Cloud accepts the job and reports "Waiting for a machine" — it is in
// `initializingJobIds` with NO workflow status until a worker picks it
// up, which routinely outlasts the deadline. Keying on status instead of
// acceptance would fail this healthy run and tell the user to run again,
// prompting a duplicate paid submission.
await tourOnRunStep()
mountRunButton('queue-button', () => {}).click()
await acceptRun(TOUR_WORKFLOW)

await vi.advanceTimersByTimeAsync(ACCEPT_DEADLINE_MS * 4)

expect(
mocks.runState.value,
'an accepted job with no status yet is queued, not refused'
).toBe('generating')
})

it('lets the offline grace outlive the acceptance deadline', async () => {
// The grace is 20s and the acceptance deadline 15s. A drop before the
// first status must still get the full grace: acceptance arrives on the
// queuePrompt response, not the socket, so it disarms this deadline even
// while the connection is down.
const { api } = await import('@/scripts/api')
await tourOnRunStep()
mountRunButton('queue-button', () => {}).click()
await acceptRun(TOUR_WORKFLOW)

api.dispatchCustomEvent('reconnecting')
await vi.advanceTimersByTimeAsync(OFFLINE_GRACE_MS - 1)

expect(
mocks.runState.value,
'the 15s acceptance deadline must not cut the 20s grace short'
).toBe('generating')
})

it('leaves an accepted run past the acceptance deadline alone', async () => {
await generatingRun()

await vi.advanceTimersByTimeAsync(ACCEPT_DEADLINE_MS * 4)

expect(
mocks.runState.value,
'the deadline is on acceptance, not on the run: a job that answered must never be cut short'
).toBe('generating')
})

it('stops promising a result once the socket stays gone', async () => {
const api = await generatingRun()

Expand Down Expand Up @@ -423,6 +512,110 @@ describe('useFirstRunTourController', () => {
'the grace timer must not clobber an outcome that arrived before it fired'
).toBe('succeeded')
})

it('stops promising a result once the queue lets go of its job', async () => {
await tourOnRunStep()
mountRunButton('queue-button', () => {}).click()
await acceptRun(TOUR_WORKFLOW)

await removeRun()

expect(
mocks.runState.value,
'an accepted job that leaves without an outcome leaves the card waiting on a result nobody will send'
).toBe('failed')
})

it('stops promising a result when a running job is dropped mid-run', async () => {
// `handleServiceLevelError` ("Job has stagnated") is the live path: it
// drops the job and records a prompt error but never touches
// `workflowStatus`, so the `running` from `handleExecutionStart`
// outlives the run and no status change reports the end.
//
// Deliberately not the mid-run credits path — #15161 made
// `handleAccountPreconditionError` clear the status, so that one ends
// via the `undefined`-after-`running` branch without this watcher.
await tourOnRunStep()
mountRunButton('queue-button', () => {}).click()
await acceptRun(TOUR_WORKFLOW)
await finishRun(TOUR_WORKFLOW, 'running')

await removeRun()

expect(
mocks.runState.value,
'a run cut short for credits keeps its running status, so losing the job is the only signal left'
).toBe('failed')
})

it('keeps a completed run that drops out of the queue as it finishes', async () => {
await tourOnRunStep()
mountRunButton('queue-button', () => {}).click()
await acceptRun(TOUR_WORKFLOW)

// `handleExecutionSuccess` reports the outcome and drops the job in one
// tick, so both land before either watcher runs.
void finishRun(TOUR_WORKFLOW, 'completed')
await removeRun()

expect(
mocks.runState.value,
'every healthy run leaves the queue when it finishes; failing those would fail every run'
).toBe('succeeded')
})

it('keeps a failed run that leaves the queue after reporting', async () => {
await tourOnRunStep()
mountRunButton('queue-button', () => {}).click()
await acceptRun(TOUR_WORKFLOW)
await finishRun(TOUR_WORKFLOW, 'failed')

await removeRun()

expect(
mocks.runState.value,
'a reported outcome is the last word; losing the job afterwards says nothing new'
).toBe('failed')
})

// Pins the transition gate on the status watcher. The stagnation path
// leaves `running` in `workflowStatus` forever, and that source
// re-evaluates whenever the map is replaced for *any* workflow. Without
// the gate the stale `running` is re-read and the card goes back to
// promising a result it has already given up on.
it('stays failed when an unrelated workflow churns the status map', async () => {
await tourOnRunStep()
mountRunButton('queue-button', () => {}).click()
await acceptRun(TOUR_WORKFLOW)
await finishRun(TOUR_WORKFLOW, 'running')

await removeRun()
expect(mocks.runState.value).toBe('failed')

await finishRun(OTHER_WORKFLOW, 'running')

expect(
mocks.runState.value,
'another workflow starting is not this run coming back from the dead'
).toBe('failed')
})

// The other half of the stagnation path: the prompt error it records must
// still be able to end the run while the stale `running` sits there.
it('gives up on a stagnated job that leaves an error and a stale status', async () => {
await tourOnRunStep()
mountRunButton('queue-button', () => {}).click()
await acceptRun(TOUR_WORKFLOW)
await finishRun(TOUR_WORKFLOW, 'running')

mocks.executionErrors.hasPromptError = true
await removeRun()

expect(
mocks.runState.value,
'a stagnated run reports an error and abandons the job; the status it leaves behind is not news'
).toBe('failed')
})
})

describe('run outcome', () => {
Expand Down
Loading
Loading