Skip to content

Breaking: reject lost-lock renewal and incomplete lock acquisition - #580

Open
niemyjski wants to merge 19 commits into
fix/in-memory-cache-immutable-entriesfrom
feature/reject-lost-lock-renewal
Open

niemyjski wants to merge 19 commits into
fix/in-memory-cache-immutable-entriesfrom
feature/reject-lost-lock-renewal

Conversation

@niemyjski

@niemyjski niemyjski commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

Renewal previously reported success after ownership was lost, and multi-resource acquisition could return an incomplete set. This change rejects lost cache-backed renewal, requires every requested lease, and releases partial acquisitions. Stacked on #570; #586 contains required follow-up corrections that are not yet in this PR's head.

Breaking runtime changes: CacheLockProvider.RenewAsync throws LockException for missing, expired, released, or replaced ownership; supplied renewal durations below 5 ms throw ArgumentOutOfRangeException before cache mutation. Scoped renewal requires the held lock's scoped ILock.Resource and LockId. Multi-resource acquisition is all-or-nothing: unsuccessful sets return null from TryAcquireAsync or throw LockAcquisitionTimeoutException from AcquireAsync. Work-item lease loss cancels the linked handler token and prevents automatic completion even when the handler swallows cancellation.

Validation at 64a4fa6a68e42beff1d583c6763bc29035bf690a: Hosted Release build and tests passed: 2,362 passed / 22 skipped / 0 failed. Local Release build, 163 conditional-cache tests, 47 lock tests, changed-file whitespace verification and diff checks passed. PR Build and push Build succeeded. The existing ASPIRE010 warning remains.

Before merge/release: incorporate #586 through the reviewed stack merge, then rerun affected combined validation on the resulting head. Current #580 still delays lease-loss cancellation behind queue renewal and overstates queue-renewal protection for the default EmptyLock; acquisition-latency handling also requires #586. Historical combined Redis execution and binary API compatibility passed for the October 23 revisions below; neither was rerun after the October 27 parent correction. Durable provider test registrations remain a separate release prerequisite; isolated registrations are not provider-main/CI coverage. The sample MessagePack advisory is addressed by the parent dependency update; no NU1902 is reported in this head’s CI log. Next-release and documented cache-performance acceptance remain separate release decisions; green CI does not close them.

Verification, merge sequence and migration notes

Exact revisions

Feedback disposition

The newest-lease-after-slow-renewal finding is satisfied here: acquisition checks every member after renewal, and TryAcquireAsync_AfterSlowRenewal_ReturnsOnlyUnexpiredLeases covers the expiry boundary and replacement owner. The immediate-cancellation and exclusive-lock documentation findings remain open until #586 is incorporated. The Redis execution finding is satisfied by actual discovery and passing runtime results against exact combined binaries. Durable provider-main/CI registration remains a separate release gate; no permanent registration is inferred from the isolated test checkout.

Safe integration sequence

  1. Preserve the completed combined core/API/Redis evidence below. Before the provider release, land durable attributed lock-contract registrations (Redis Bump Microsoft.Extensions.Hosting from 3.1.1 to 3.1.2 #191) and the six additional shared cache contracts in direct/scoped fixtures, then verify discovery in provider CI.
  2. After stack-merge authorization, merge reviewed Reject expired acquisitions and cancel lost work-item leases #586 into this branch without copying its changes or flattening the stack; rerun affected checks on the resulting exact head and update feedback dispositions.
  3. Land the validated Fix in-memory cache update races and memory accounting #570 parent before this PR lands on main. Retarget Breaking: reject lost-lock renewal and incomplete lock acquisition #580 to main after the parent lands, refresh main, inspect the resulting diff and verify the final candidate. If parent or child revisions change, invalidate affected evidence and repeat those checks.
  4. Release only after durable provider registrations and the retained next-release/performance decisions are complete. No main merge or release is performed by this validation.

Consumer migration / release note

Observe renewal failures in an awaited task; stop cooperative protected work on LockException. An expired handle cannot revive ownership: reacquire and revalidate the work. Replace timer or async void renewal patterns that leave exceptions unobserved. Use renewal durations of at least 5 ms and use scoped identifiers from the held lock. Handle failed multi-resource acquisition as failure of the whole set. Partial cleanup attempts every release; failures are logged and unreleased leases expire naturally. Empty and throttling locks have no-op renewal and provide no renewable exclusive ownership. Override GetWorkItemLockAsync with a shared exclusive lock when work-item exclusion is required. Locks are not destination-enforced fencing, and cancellation cannot revoke writes already dispatched.

Historical October 23 combined validation (separate from this PR head)

Combined core 70533664e7dfccf011fae4ddb5ef41c95dc68d63, tree e3052914a241d1116fae694969d80b028fe2d912, includes #570 d2c24efd, #580 53ce936b, #582 3907f54a, #586 fffdf32f, and #587 143f24fe, with main/API baseline be10785c.

  • Core: 2,484 passed / 22 skipped / 0 failed; 68 selected critical regression executions passed within the full suite.
  • Redis 8.10.2: 778 passed / 18 skipped / 0 failed. Actual discovery found 796 cases, including 26 lock methods / 30 lock cases and 12 added cache cases. All 42 required cases executed and passed; none skipped. Provider-output core and TestHarness SHA256 values match the combined build outputs.
  • APICompat 10.0.402: Foundatio net8.0/net10.0 and TestHarness net10.0 passed against main, including parameter-name checks, without suppressions.
  • Redis test snapshot be092d17 combines main 06668733, registration PR Bump Microsoft.Extensions.Hosting from 3.1.1 to 3.1.2 #191 ae79c527, and test-only cache registrations. These isolated registrations remain unpublished; provider main/CI is not claimed to discover them automatically.
  • The ten-provider source workspace build passed with two ASPIRE010 warnings after an isolated RabbitMQ sample MessagePack 2.5.303 overlay. Unmodified provider main was not green; the provider owner must ship that dependency correction and verify its own build/audit. No non-Redis runtime suites ran.

The combined tree is an integration-validation artifact, not a change to this PR branch. Its passing tests do not resolve child-only source findings before #586 is actually integrated. This refresh reran the local and hosted checks listed above. No combined tree, provider runtime suite, API compatibility check or benchmark was rerun by this refresh.

October 27 parent evidence and retained limits

The parent owner reported a corrected combined core run with 2,493 passed / 22 skipped / 0 failed at tree 83733043e2a0b4cca7fca9b9a2a27ca4273f53d8. An initial run failed the existing timing-sensitive ScopedInMemoryCacheClientTests.ListAddAsync_WithExpiration_SetsExpirationCorrectly; the affected fixtures and full rerun passed. This separate evidence does not establish current provider runtime or API compatibility. The October 25 custom-sizing benchmark with cloning disabled recorded 3.28x time / 37x allocation at 1,000 entries and 6.70x / 318x at 10,000; performance acceptance remains open.

Limits and reproduction

The hosted build retains ASPIRE010. The parent pins sample MessagePack 2.5.303; the current CI log has no NU1902 warning. This is not a claim that every repository dependency is advisory-free. Skips remain skips. The historical combined run establishes binary compatibility and Redis runtime verification only for its listed revisions, not the current head. No benchmark improvement or non-Redis runtime coverage is claimed.

dotnet build Foundatio.slnx -c Release --disable-build-servers -m:1 -p:UseSharedCompilation=false
dotnet test --solution Foundatio.slnx -c Release --no-build
git diff --check

Use Foundatio.All.slnx for combined source-workspace validation. The ordinary push workflow successfully published nine CI preview packages, version 13.0.6-preview.reject-lost-lock-renewal.0.5, to GitHub Packages and Feedz; release publication was skipped.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-28T04:04:15.902452Z 64a4fa6 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@niemyjski niemyjski changed the title Reject renewal of expired or lost cache locks Reject lost-lock renewal and incomplete lock acquisition Oct 6, 2026
…tries' into HEAD

# Conflicts:
#	.agents/skills/foundatio/SKILL.md
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@niemyjski

Copy link
Copy Markdown
Member Author

Merged the updated #570 (83b1bed6 + latest main) into this branch as 88fbc8fa. Only SKILL.md conflicted; I kept this PR's lock-renewal gotcha and #570's updated in-memory expiry gotcha. Merged stack: Release build 0 errors, 2,301 passed / 22 skipped / 0 failed. Foundatio.Redis cache and lock tests against this source on single-node Redis: 752 passed / 0 failed.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

…tries' into feature/reject-lost-lock-renewal
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

…tries' into feature/reject-lost-lock-renewal
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

…ider errors, document breaking changes

WorkItemJob no longer completes an entry when the work item lock was lost, even if the handler returns early or swallows the cancellation. Lock-set operations rethrow the definite LockException and log the provider errors they drop. Raises the 250ms lease in the multi-resource renewal tests, restores the InMemoryLockTests member order, and adds a breaking-changes callout to the locks guide.
…st lock

- Attempt every renew/release in a lock set and prefer provider errors over LockException
- Track acquired locks by position instead of name suffix and log cleanup failures on the null-return path
- Cancel WorkItemContext.CancellationToken when automatic progress renewal loses the lock
- Add regression tests and update lock/job docs
@niemyjski niemyjski changed the title Reject lost-lock renewal and incomplete lock acquisition Breaking: reject lost-lock renewal and incomplete lock acquisition Oct 7, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 562ba954e5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…' into feature/reject-lost-lock-renewal

# Conflicts:
#	.agents/skills/foundatio/SKILL.md
#	docs/guide/jobs.md
#	docs/guide/locks.md
#	src/Foundatio/Jobs/WorkItemJob/WorkItemJob.cs
#	src/Foundatio/Lock/DisposableLockCollection.cs
#	src/Foundatio/Lock/ILockProvider.cs
#	src/Foundatio/Lock/LockSetOperations.cs
#	tests/Foundatio.Tests/Jobs/WorkItemJobTests.cs
#	tests/Foundatio.Tests/Locks/InMemoryLockTests.cs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e33ffb746b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0bc0f584a8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/Foundatio/Lock/ILockProvider.cs Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a68d22e522

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/guide/locks.md

Choose an interval shorter than the lease duration, leaving time for cache latency and retries. A pause can still let the lease expire before renewal detects the loss, and cancellation cannot revoke an operation already dispatched to another system.

`WorkItemJob` applies this pattern when a handler sets `AutoRenewLockOnProgress`. Each `ReportProgressAsync` call renews the queue entry and the work item lock. Renewal errors are logged and progress reporting continues. If the work item lock is lost (`LockException`), `WorkItemContext.CancellationToken` is cancelled, and the job abandons the queue entry for retry instead of completing it, whether the handler throws, returns early, or swallows the cancellation, and even if the job itself is cancelled afterward. A handler that ignores the token keeps running unprotected until it returns. A failed queue entry renewal is only logged: queues don't report a definite loss, and while the work item lock is held a redelivered entry can't be processed concurrently.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Qualify queue-renewal safety without an exclusive lock

This safety claim does not hold for the default WorkItemHandlerBase: GetWorkItemLockAsync returns Disposable.EmptyLock, whose renewal is a no-op, so a failed queue-entry renewal can allow redelivery and concurrent processing while the original handler continues. Please state that this protection applies only when the handler overrides lock acquisition with a genuinely exclusive work-item lock; otherwise the new guidance can lead consumers to overlook duplicate execution.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9760d3a8c3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


try
{
await Task.WhenAll(queueEntryRenewal, workItemLockRenewal).AnyContext();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Signal lease loss before awaiting queue renewal

When workItemLockRenewal quickly faults with LockException but a remote queue provider's renewal is slow or stuck, Task.WhenAll prevents leaseLoss.SignalAsync() from running until that unrelated queue operation finishes. Concurrent work already started by the handler therefore continues without cancellation even though another worker may own the lease, potentially indefinitely because queue renewal has no cancellation token; signal the definite lease loss as soon as the work-item renewal settles, while awaiting/logging the queue task separately.

AGENTS.md reference: AGENTS.md:L97-L97

Useful? React with 👍 / 👎.

…tries' into feature/pr580-oct27-refresh

# Conflicts:
#	.agents/skills/foundatio/SKILL.md

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant