Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,12 @@ public ChatMessage call() throws Exception {
}
return new ChatAttemptResult(
model, chatModel, response, actualRetryCount, totalWaitTimeSec);
} catch (InterruptedException e) {
// A cancellation signal, not a model failure: restore the interrupt status and
// propagate immediately so task shutdown isn't delayed by retry backoff or an
// extra model call, regardless of the configured error-handling strategy.
Thread.currentThread().interrupt();
throw e;
} catch (Exception e) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thread.sleep on line 203 sits inside this catch (Exception e) block, so a cancel during the backoff wait throws from in here rather than from the call above. A catch block is not covered by its own sibling catch, and Thread.sleep clears the interrupt status when it throws. The call still stops, so this is only about the flag, but on this one path it ends up cleared rather than restored. RETRY_WAIT_INTERVAL defaults to 1, so under RETRY there is a one second window on every retry.

Something like this, if useful:

try {
    Thread.sleep(currentWaitSec * 1000L);
} catch (InterruptedException ie) {
    Thread.currentThread().interrupt();
    throw ie;
}

Worth restoring the flag there too?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, good catch — the sleep sits in the sibling catch (Exception e) block, not the InterruptedException handler above it, so it wasn't covered by that fix. Wrapped it the same way (restore flag, rethrow) in c30ba89.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new block has no test. Both tests pass retryWaitIntervalSec = 0 (lines 80 and 112), so the guard on line 202 skips lines 203 to 208.

A cheap deterministic test, in case it helps. If the stub sets the flag before it throws, Thread.sleep throws at once and costs no wall time:

when(ctx.durableExecute(any()))
        .thenAnswer(
                inv -> {
                    Thread.currentThread().interrupt();
                    throw new RuntimeException("transient failure");
                });

with RETRY, numRetries = 1, retryWaitIntervalSec = 1.

One catch. assertThrows and the times(1) verify pass on the old code too, so only assertTrue(Thread.interrupted()) proves the fix. Is a test here worth adding?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added, in 5803bb1. Set the interrupt flag from within the mocked durableExecute call itself so Thread.sleep throws immediately at no wall-clock cost, with RETRY / numRetries=1 / retryWaitIntervalSec=1 as you suggested. Verified it's the flag assertion alone that distinguishes the fix — temporarily restored the pre-c30ba894 sleep block and confirmed this new test fails there (times(1)/InterruptedException-thrown still pass either way) while the two existing tests stay green.

if (strategy == Agent.ErrorHandlingStrategy.RETRY && attempt < numRetries) {
actualRetryCount = attempt + 1;
Expand All @@ -194,7 +200,12 @@ public ChatMessage call() throws Exception {
numRetries,
currentWaitSec);
if (currentWaitSec > 0) {
Thread.sleep(currentWaitSec * 1000L);
try {
Thread.sleep(currentWaitSec * 1000L);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
throw ie;
}
totalWaitTimeSec += currentWaitSec;
}
continue;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.flink.agents.plan.actions;

import org.apache.flink.agents.api.agents.Agent;
import org.apache.flink.agents.api.agents.AgentExecutionOptions;
import org.apache.flink.agents.api.chat.model.BaseChatModelSetup;
import org.apache.flink.agents.api.configuration.ReadableConfiguration;
import org.apache.flink.agents.api.context.RunnerContext;
import org.apache.flink.agents.api.metrics.FlinkAgentsMetricGroup;
import org.apache.flink.agents.api.resource.ResourceType;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;

import java.util.List;
import java.util.Map;
import java.util.UUID;

import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;

/** Tests for {@link ChatModelInvoker}. */
class ChatModelInvokerTest {

@AfterEach
void clearInterruptStatus() {
// Prevents a leftover interrupt flag (e.g. if the assertion below the interruption test
// ever fails before consuming it) from failing an unrelated later test's real Thread.sleep
// backoff with a spurious InterruptedException.
Thread.interrupted();
}

@Test
void testChatWithRetriesDoesNotRetryOnInterruption() throws Exception {
RunnerContext ctx = mock(RunnerContext.class);
BaseChatModelSetup chatModel = mock(BaseChatModelSetup.class);
ReadableConfiguration config = mock(ReadableConfiguration.class);
when(ctx.getConfig()).thenReturn(config);
when(config.get(AgentExecutionOptions.CHAT_ASYNC)).thenReturn(false);
when(ctx.getResource("test-model", ResourceType.CHAT_MODEL)).thenReturn(chatModel);
when(ctx.getActionMetricGroup()).thenReturn(mock(FlinkAgentsMetricGroup.class));
when(ctx.durableExecute(any())).thenThrow(new InterruptedException("cancelled"));

// Clear any interrupt status left over from a previous test before asserting on it below.
Thread.interrupted();

assertThrows(
InterruptedException.class,
() ->
ChatModelInvoker.chatWithRetries(
UUID.randomUUID(),
"test-model",
"durable-call-id",
List.of(),
Map.of(),
null,
ctx,
Agent.ErrorHandlingStrategy.RETRY,
3,
0));

// Only the first attempt should have run: retry backoff must not consume more attempts
// after a cancellation interrupts the call.
verify(ctx, times(1)).durableExecute(any());
assertTrue(Thread.interrupted(), "interrupt status should be restored on the thread");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Thread.interrupted() on this line clears the flag, but it only runs if the verify above it passes. If that verify ever fails, the flag stays set on the JUnit thread. ChatModelActionRetryTest.chatRetriesWithExponentialBackoff is in this same package and drives a real one second backoff, so it can then fail with an unrelated InterruptedException and send someone chasing the wrong test. RunnerContextImplDurableExecuteTest:91 and :115 have the same shape. Would an @AfterEach calling Thread.interrupted() be worth adding, so the cleanup runs either way?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed — assertTrue only consumes the flag when it passes, so a failing assertion here would leave it set on the JUnit thread. Added an @AfterEach in both this test class and RunnerContextImplDurableExecuteTest to clear it unconditionally, in c30ba89.

}

@Test
void testChatWithRetriesRestoresInterruptFlagFromRetryBackoffSleep() throws Exception {
RunnerContext ctx = mock(RunnerContext.class);
BaseChatModelSetup chatModel = mock(BaseChatModelSetup.class);
ReadableConfiguration config = mock(ReadableConfiguration.class);
when(ctx.getConfig()).thenReturn(config);
when(config.get(AgentExecutionOptions.CHAT_ASYNC)).thenReturn(false);
when(ctx.getResource("test-model", ResourceType.CHAT_MODEL)).thenReturn(chatModel);
when(ctx.getActionMetricGroup()).thenReturn(mock(FlinkAgentsMetricGroup.class));
// The interrupt fires from within the retry backoff's Thread.sleep, not from the call
// itself: set the flag first so Thread.sleep throws immediately, at no wall-clock cost.
when(ctx.durableExecute(any()))
.thenAnswer(
invocation -> {
Thread.currentThread().interrupt();
throw new RuntimeException("transient failure");
});

assertThrows(
InterruptedException.class,
() ->
ChatModelInvoker.chatWithRetries(
UUID.randomUUID(),
"test-model",
"durable-call-id",
List.of(),
Map.of(),
null,
ctx,
Agent.ErrorHandlingStrategy.RETRY,
1,
1));

// Only the first attempt should have run: the sleep before the retry throws before a
// second call is made.
verify(ctx, times(1)).durableExecute(any());
// This is the only assertion that distinguishes the fix from the pre-fix code: the
// InterruptedException/times(1) shape above passes either way, since Thread.sleep still
// aborts the retry loop on both. Only the restored flag proves the backoff sleep's catch
// block restores it instead of leaving it cleared.
assertTrue(Thread.interrupted(), "interrupt status should be restored on the thread");
}

@Test
void testChatWithRetriesRetriesOnOrdinaryFailure() throws Exception {
RunnerContext ctx = mock(RunnerContext.class);
BaseChatModelSetup chatModel = mock(BaseChatModelSetup.class);
ReadableConfiguration config = mock(ReadableConfiguration.class);
when(ctx.getConfig()).thenReturn(config);
when(config.get(AgentExecutionOptions.CHAT_ASYNC)).thenReturn(false);
when(ctx.getResource("test-model", ResourceType.CHAT_MODEL)).thenReturn(chatModel);
when(ctx.getActionMetricGroup()).thenReturn(mock(FlinkAgentsMetricGroup.class));
when(ctx.durableExecute(any())).thenThrow(new RuntimeException("transient failure"));

assertThrows(
ChatModelInvoker.ChatAttemptFailed.class,
() ->
ChatModelInvoker.chatWithRetries(
UUID.randomUUID(),
"test-model",
"durable-call-id",
List.of(),
Map.of(),
null,
ctx,
Agent.ErrorHandlingStrategy.RETRY,
2,
0));

// An ordinary failure must still consume the full retry budget (initial attempt + 2
// retries), confirming the interruption fix doesn't disturb normal retry behavior.
verify(ctx, times(3)).durableExecute(any());
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -576,6 +576,12 @@ protected <T> T durableExecuteCompletionOnly(
Exception exception = null;
try {
result = executionCallable.call();
} catch (InterruptedException e) {
// A cancellation signal, not a genuine call failure: leave the durable slot
// unfinished so recovery re-executes or reconciles the call instead of replaying a
// stale interruption as a completed success or failure.
Thread.currentThread().interrupt();
throw e;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ModelRoutingResolver.java:97 runs the routing strategy through ctx.durableExecute, so an interrupt there reaches this rethrow. ChatModelAction.java:609 then catches it, and under IGNORE line 621 returns normally. I could not trigger this with in-tree code. It needs a MODEL_ROUTER resource, and the only routing strategy in the repo today does no I/O. But RoutingStrategy is a user extension point, and the comment on line 612 already expects strategies that do I/O. The issue's first bullet asks for cancellation to propagate under IGNORE too. Should that catch let InterruptedException through before the IGNORE check?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Traced it through — matches what you described. It's real but needs a RoutingStrategy that does I/O, which nothing in-tree does today, so it's about the extension point rather than this PR's own code. Given the issue is scoped to the chat/tool call paths, I'd rather keep this PR to what it already touches and file a follow-up for the routing-resolver + IGNORE interaction rather than pull it in here — let me know if you'd rather it go in this PR instead.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, leave it out of this PR. I would skip the follow-up issue too. #1042 already has the same fix. It adds ModelRoutingResolver.isCancellation(...) and calls it in ChatModelAction.processChatRequest just before the IGNORE check.

#1042 is also what makes this path reachable. Strategies.llm(...) runs a judge chat call through chatWithRetries, so the strategy that does I/O arrives with its own guard.

It is still open though, so this only holds if it lands as it stands. Does that look right to you?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That reasoning matches what I see in #1042: ModelRoutingResolver.isCancellation(...) gets added and called from ChatModelAction.processChatRequest right before the IGNORE check, and Strategies.llm(...)'s judge chat call through chatWithRetries is what would make this path reachable in the first place. Agreed on skipping a separate follow-up issue, conditional on #1042 landing as-is — I'll keep an eye on it and flag here (or file the issue myself) if it merges in a materially different shape.

} catch (Exception e) {
exception = e;
}
Expand Down Expand Up @@ -939,6 +945,12 @@ protected <T> T executeAndFinalizeCurrentCall(
Exception exception = null;
try {
result = callSupplier.call();
} catch (InterruptedException e) {
// A cancellation signal, not a genuine call failure: leave the pending call
// unfinalized so recovery re-executes or reconciles it instead of replaying a stale
// interruption as a completed success or failure.
Thread.currentThread().interrupt();
throw e;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tool calls run through the two methods you patched, so this rethrow reaches them too, but the tool path still finishes normally after a cancel. ToolCallAction.java:256 catches the InterruptedException, line 257 records it as a tool error, and the loop moves on to the next tool. Line 85 then sends the ToolResponseEvent anyway, which drives another chat call, and the action is persisted as finished. executeParallel has the same shape at line 209. That catch predates this PR. Is the tool path meant to be in scope here, or is it worth a separate issue?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, same shape in both executeSequentially and executeParallel, and it predates this PR. Tool-call cancellation handling looks like its own piece of work (the response still gets sent and the action still finishes), not a natural extension of the chat-retry fix here. I'd lean toward a separate issue/PR for it rather than scope-creeping this one — open to doing it here instead if you'd prefer to keep it together.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, a separate issue is right. Nothing open covers tool-call cancellation, so it will not duplicate anything.

One thing that might be worth adding to it. Because the action returns normally, ActionExecutionOperator persists it as completed (line 491), and on recovery line 437 replays the output events without running the call again. That is problem 2 in #1070, carried by the action state rather than the durable slot. The chat path escapes it because the raw InterruptedException is rethrown on line 482, before the persist. Worth folding in?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, worth folding in. Traced it: ActionExecutionOperator persists the action as completed at line ~491 (durableExecManager.maybePersistTaskResult) right after the tool-call catch swallows the interruption, and on recovery line ~437 (actionState.isCompleted()) skips re-execution and replays the stale output. Confirms it's the same class of problem as #1070's problem 2, just reached through the action-state path instead of the durable-slot path since the chat path's raw rethrow at line 482 escapes before the persist call. I'll fold this into the follow-up issue's description when I file it.

} catch (Exception e) {
exception = e;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
import org.apache.flink.agents.runtime.actionstate.CallResult;
import org.apache.flink.agents.runtime.metrics.FlinkAgentsMetricGroupImpl;
import org.apache.flink.runtime.metrics.groups.UnregisteredMetricGroups;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

Expand Down Expand Up @@ -53,6 +54,13 @@ void setUp() {
lastPersistedState = null;
}

@AfterEach
void clearInterruptStatus() {
// Prevents a leftover interrupt flag (e.g. if an assertion below one of the interruption
// tests ever fails before consuming it) from leaking into an unrelated later test.
Thread.interrupted();
}

@Test
void testDurableExecuteLegacyCall() throws Exception {
RunnerContextImpl context = createContext(new ActionState(null));
Expand All @@ -72,6 +80,53 @@ void testDurableExecuteLegacyCall() throws Exception {
assertSame(context.getDurableExecutionContext().getActionState(), lastPersistedState);
}

@Test
void testDurableExecuteCompletionOnlyDoesNotPersistInterruption() {
RunnerContextImpl context = createContext(new ActionState(null));
TestDurableCallable<String> callable =
new TestDurableCallable<>(
"legacy-call",
String.class,
() -> {
throw new InterruptedException("cancelled");
});

// Clear any interrupt status left over from a previous test before asserting on it below.
Thread.interrupted();

assertThrows(InterruptedException.class, () -> context.durableExecute(callable));

assertTrue(Thread.interrupted(), "interrupt status should be restored on the thread");
// A cancellation must not be finalized as a durable success or failure: the slot stays
// unfinished so recovery re-executes the call instead of replaying a stale interruption.
assertEquals(0, persistCallCount.get());
assertEquals(0, context.getDurableExecutionContext().getActionState().getCallResultCount());
}

@Test
void testDurableExecuteCompletionOnlyReExecutesPendingSlotDoesNotPersistInterruption() {
ActionState actionState = new ActionState(null);
actionState.addCallResult(CallResult.pending("tool-call", ""));
RunnerContextImpl context = createContext(actionState);
TestDurableCallable<String> callable =
new TestDurableCallable<>(
"tool-call",
String.class,
() -> {
throw new InterruptedException("cancelled");
});

Thread.interrupted();

assertThrows(InterruptedException.class, () -> context.durableExecute(callable));

assertTrue(Thread.interrupted(), "interrupt status should be restored on the thread");
assertEquals(0, persistCallCount.get());
CallResult pending =
context.getDurableExecutionContext().getActionState().getCallResults().get(0);
assertTrue(pending.isPending(), "interrupted pending slot should remain unfinalized");
}

@Test
void testDurableExecuteReconcilableSuccessCall() throws Exception {
RunnerContextImpl context = createContext(new ActionState(null));
Expand Down
Loading