-
Notifications
You must be signed in to change notification settings - Fork 167
[plan][runtime] Do not retry or persist interrupted chat calls on cancellation #1071
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
ad45bc9
c30ba89
5803bb1
5d5cf62
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,161 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package org.apache.flink.agents.plan.actions; | ||
|
|
||
| import org.apache.flink.agents.api.agents.Agent; | ||
| import org.apache.flink.agents.api.agents.AgentExecutionOptions; | ||
| import org.apache.flink.agents.api.chat.model.BaseChatModelSetup; | ||
| import org.apache.flink.agents.api.configuration.ReadableConfiguration; | ||
| import org.apache.flink.agents.api.context.RunnerContext; | ||
| import org.apache.flink.agents.api.metrics.FlinkAgentsMetricGroup; | ||
| import org.apache.flink.agents.api.resource.ResourceType; | ||
| import org.junit.jupiter.api.AfterEach; | ||
| import org.junit.jupiter.api.Test; | ||
|
|
||
| import java.util.List; | ||
| import java.util.Map; | ||
| import java.util.UUID; | ||
|
|
||
| import static org.junit.jupiter.api.Assertions.assertThrows; | ||
| import static org.junit.jupiter.api.Assertions.assertTrue; | ||
| import static org.mockito.ArgumentMatchers.any; | ||
| import static org.mockito.Mockito.mock; | ||
| import static org.mockito.Mockito.times; | ||
| import static org.mockito.Mockito.verify; | ||
| import static org.mockito.Mockito.when; | ||
|
|
||
| /** Tests for {@link ChatModelInvoker}. */ | ||
| class ChatModelInvokerTest { | ||
|
|
||
| @AfterEach | ||
| void clearInterruptStatus() { | ||
| // Prevents a leftover interrupt flag (e.g. if the assertion below the interruption test | ||
| // ever fails before consuming it) from failing an unrelated later test's real Thread.sleep | ||
| // backoff with a spurious InterruptedException. | ||
| Thread.interrupted(); | ||
| } | ||
|
|
||
| @Test | ||
| void testChatWithRetriesDoesNotRetryOnInterruption() throws Exception { | ||
| RunnerContext ctx = mock(RunnerContext.class); | ||
| BaseChatModelSetup chatModel = mock(BaseChatModelSetup.class); | ||
| ReadableConfiguration config = mock(ReadableConfiguration.class); | ||
| when(ctx.getConfig()).thenReturn(config); | ||
| when(config.get(AgentExecutionOptions.CHAT_ASYNC)).thenReturn(false); | ||
| when(ctx.getResource("test-model", ResourceType.CHAT_MODEL)).thenReturn(chatModel); | ||
| when(ctx.getActionMetricGroup()).thenReturn(mock(FlinkAgentsMetricGroup.class)); | ||
| when(ctx.durableExecute(any())).thenThrow(new InterruptedException("cancelled")); | ||
|
|
||
| // Clear any interrupt status left over from a previous test before asserting on it below. | ||
| Thread.interrupted(); | ||
|
|
||
| assertThrows( | ||
| InterruptedException.class, | ||
| () -> | ||
| ChatModelInvoker.chatWithRetries( | ||
| UUID.randomUUID(), | ||
| "test-model", | ||
| "durable-call-id", | ||
| List.of(), | ||
| Map.of(), | ||
| null, | ||
| ctx, | ||
| Agent.ErrorHandlingStrategy.RETRY, | ||
| 3, | ||
| 0)); | ||
|
|
||
| // Only the first attempt should have run: retry backoff must not consume more attempts | ||
| // after a cancellation interrupts the call. | ||
| verify(ctx, times(1)).durableExecute(any()); | ||
| assertTrue(Thread.interrupted(), "interrupt status should be restored on the thread"); | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit:
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed — assertTrue only consumes the flag when it passes, so a failing assertion here would leave it set on the JUnit thread. Added an @AfterEach in both this test class and RunnerContextImplDurableExecuteTest to clear it unconditionally, in c30ba89. |
||
| } | ||
|
|
||
| @Test | ||
| void testChatWithRetriesRestoresInterruptFlagFromRetryBackoffSleep() throws Exception { | ||
| RunnerContext ctx = mock(RunnerContext.class); | ||
| BaseChatModelSetup chatModel = mock(BaseChatModelSetup.class); | ||
| ReadableConfiguration config = mock(ReadableConfiguration.class); | ||
| when(ctx.getConfig()).thenReturn(config); | ||
| when(config.get(AgentExecutionOptions.CHAT_ASYNC)).thenReturn(false); | ||
| when(ctx.getResource("test-model", ResourceType.CHAT_MODEL)).thenReturn(chatModel); | ||
| when(ctx.getActionMetricGroup()).thenReturn(mock(FlinkAgentsMetricGroup.class)); | ||
| // The interrupt fires from within the retry backoff's Thread.sleep, not from the call | ||
| // itself: set the flag first so Thread.sleep throws immediately, at no wall-clock cost. | ||
| when(ctx.durableExecute(any())) | ||
| .thenAnswer( | ||
| invocation -> { | ||
| Thread.currentThread().interrupt(); | ||
| throw new RuntimeException("transient failure"); | ||
| }); | ||
|
|
||
| assertThrows( | ||
| InterruptedException.class, | ||
| () -> | ||
| ChatModelInvoker.chatWithRetries( | ||
| UUID.randomUUID(), | ||
| "test-model", | ||
| "durable-call-id", | ||
| List.of(), | ||
| Map.of(), | ||
| null, | ||
| ctx, | ||
| Agent.ErrorHandlingStrategy.RETRY, | ||
| 1, | ||
| 1)); | ||
|
|
||
| // Only the first attempt should have run: the sleep before the retry throws before a | ||
| // second call is made. | ||
| verify(ctx, times(1)).durableExecute(any()); | ||
| // This is the only assertion that distinguishes the fix from the pre-fix code: the | ||
| // InterruptedException/times(1) shape above passes either way, since Thread.sleep still | ||
| // aborts the retry loop on both. Only the restored flag proves the backoff sleep's catch | ||
| // block restores it instead of leaving it cleared. | ||
| assertTrue(Thread.interrupted(), "interrupt status should be restored on the thread"); | ||
| } | ||
|
|
||
| @Test | ||
| void testChatWithRetriesRetriesOnOrdinaryFailure() throws Exception { | ||
| RunnerContext ctx = mock(RunnerContext.class); | ||
| BaseChatModelSetup chatModel = mock(BaseChatModelSetup.class); | ||
| ReadableConfiguration config = mock(ReadableConfiguration.class); | ||
| when(ctx.getConfig()).thenReturn(config); | ||
| when(config.get(AgentExecutionOptions.CHAT_ASYNC)).thenReturn(false); | ||
| when(ctx.getResource("test-model", ResourceType.CHAT_MODEL)).thenReturn(chatModel); | ||
| when(ctx.getActionMetricGroup()).thenReturn(mock(FlinkAgentsMetricGroup.class)); | ||
| when(ctx.durableExecute(any())).thenThrow(new RuntimeException("transient failure")); | ||
|
|
||
| assertThrows( | ||
| ChatModelInvoker.ChatAttemptFailed.class, | ||
| () -> | ||
| ChatModelInvoker.chatWithRetries( | ||
| UUID.randomUUID(), | ||
| "test-model", | ||
| "durable-call-id", | ||
| List.of(), | ||
| Map.of(), | ||
| null, | ||
| ctx, | ||
| Agent.ErrorHandlingStrategy.RETRY, | ||
| 2, | ||
| 0)); | ||
|
|
||
| // An ordinary failure must still consume the full retry budget (initial attempt + 2 | ||
| // retries), confirming the interruption fix doesn't disturb normal retry behavior. | ||
| verify(ctx, times(3)).durableExecute(any()); | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -576,6 +576,12 @@ protected <T> T durableExecuteCompletionOnly( | |
| Exception exception = null; | ||
| try { | ||
| result = executionCallable.call(); | ||
| } catch (InterruptedException e) { | ||
| // A cancellation signal, not a genuine call failure: leave the durable slot | ||
| // unfinished so recovery re-executes or reconciles the call instead of replaying a | ||
| // stale interruption as a completed success or failure. | ||
| Thread.currentThread().interrupt(); | ||
| throw e; | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Traced it through — matches what you described. It's real but needs a RoutingStrategy that does I/O, which nothing in-tree does today, so it's about the extension point rather than this PR's own code. Given the issue is scoped to the chat/tool call paths, I'd rather keep this PR to what it already touches and file a follow-up for the routing-resolver + IGNORE interaction rather than pull it in here — let me know if you'd rather it go in this PR instead.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed, leave it out of this PR. I would skip the follow-up issue too. #1042 already has the same fix. It adds #1042 is also what makes this path reachable. It is still open though, so this only holds if it lands as it stands. Does that look right to you?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That reasoning matches what I see in #1042: |
||
| } catch (Exception e) { | ||
| exception = e; | ||
| } | ||
|
|
@@ -939,6 +945,12 @@ protected <T> T executeAndFinalizeCurrentCall( | |
| Exception exception = null; | ||
| try { | ||
| result = callSupplier.call(); | ||
| } catch (InterruptedException e) { | ||
| // A cancellation signal, not a genuine call failure: leave the pending call | ||
| // unfinalized so recovery re-executes or reconciles it instead of replaying a stale | ||
| // interruption as a completed success or failure. | ||
| Thread.currentThread().interrupt(); | ||
| throw e; | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Tool calls run through the two methods you patched, so this rethrow reaches them too, but the tool path still finishes normally after a cancel.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed, same shape in both executeSequentially and executeParallel, and it predates this PR. Tool-call cancellation handling looks like its own piece of work (the response still gets sent and the action still finishes), not a natural extension of the chat-retry fix here. I'd lean toward a separate issue/PR for it rather than scope-creeping this one — open to doing it here instead if you'd prefer to keep it together.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed, a separate issue is right. Nothing open covers tool-call cancellation, so it will not duplicate anything. One thing that might be worth adding to it. Because the action returns normally,
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good catch, worth folding in. Traced it: ActionExecutionOperator persists the action as completed at line ~491 (durableExecManager.maybePersistTaskResult) right after the tool-call catch swallows the interruption, and on recovery line ~437 (actionState.isCompleted()) skips re-execution and replays the stale output. Confirms it's the same class of problem as #1070's problem 2, just reached through the action-state path instead of the durable-slot path since the chat path's raw rethrow at line 482 escapes before the persist call. I'll fold this into the follow-up issue's description when I file it. |
||
| } catch (Exception e) { | ||
| exception = e; | ||
| } | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thread.sleepon line 203 sits inside thiscatch (Exception e)block, so a cancel during the backoff wait throws from in here rather than from the call above. A catch block is not covered by its own sibling catch, andThread.sleepclears the interrupt status when it throws. The call still stops, so this is only about the flag, but on this one path it ends up cleared rather than restored.RETRY_WAIT_INTERVALdefaults to 1, so under RETRY there is a one second window on every retry.Something like this, if useful:
Worth restoring the flag there too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Confirmed, good catch — the sleep sits in the sibling
catch (Exception e)block, not theInterruptedExceptionhandler above it, so it wasn't covered by that fix. Wrapped it the same way (restore flag, rethrow) in c30ba89.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The new block has no test. Both tests pass
retryWaitIntervalSec = 0(lines 80 and 112), so the guard on line 202 skips lines 203 to 208.A cheap deterministic test, in case it helps. If the stub sets the flag before it throws,
Thread.sleepthrows at once and costs no wall time:with
RETRY,numRetries = 1,retryWaitIntervalSec = 1.One catch.
assertThrowsand thetimes(1)verify pass on the old code too, so onlyassertTrue(Thread.interrupted())proves the fix. Is a test here worth adding?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added, in 5803bb1. Set the interrupt flag from within the mocked
durableExecutecall itself soThread.sleepthrows immediately at no wall-clock cost, with RETRY / numRetries=1 / retryWaitIntervalSec=1 as you suggested. Verified it's the flag assertion alone that distinguishes the fix — temporarily restored the pre-c30ba894 sleep block and confirmed this new test fails there (times(1)/InterruptedException-thrown still pass either way) while the two existing tests stay green.