Use a Mutex on Windows so an abandoned package journal lock can be recovered

[xwipeoutx]

⚠️ Does this change require a corresponding Server Change? No — the named lock manager is internal to Calamari. See "Risks" below for a public-API note and a mixed-version note.

Related to LEV-1901 Stop Calamari hanging on clean packages when process is killed

Problem

On Windows, SystemSemaphoreManager.Acquire backed the package journal lock with a named Semaphore. A Semaphore has no notion of ownership, so when the process holding it goes away without running the Releaser — killed mid-ApplyRetention, Tentacle restart, OOM — the count is never restored.

The catch (AbandonedMutexException) already in that method could therefore never fire on Windows, because a semaphore cannot be abandoned. Any other process that already holds a handle keeps the kernel object (and its zeroed count) alive, so the unbounded WaitOne() waits forever.

The visible symptom is a deployment that hangs during package cleanup or registration, with nothing in the logs but a single verbose Another process is using the package journal.

Fix

1. Use a named Mutex on every platform

An abandoned mutex is signalled by the kernel and handed to the next waiter, so the existing AbandonedMutexException handler now does the job it was written for and Acquire() recovers. Recovery is logged as a warning, since the previous holder died inside the critical section and whatever it was protecting may be inconsistent. The separate Windows (Semaphore) and *nix (Mutex) code paths collapse into one; the only platform-specific step left is the ACL below.

Deliberately preserved:

• The Everyone full-control ACL, ported from SemaphoreSecurity to MutexSecurity/MutexAccessRule and still Windows-only. Without it a Global\ mutex gets a default DACL, and a lock taken by a Tentacle running as a service could be unopenable by Calamari under a different account.
• The Polly creation retry from Add retry to Semaphore acquisition #1414, and the 3-second-then-log wait behaviour.

This also corrects a naming inconsistency introduced in #1414: the Windows path created the semaphore under the unprefixed name while building a Global\ prefixed name only for the ACL. Two Calamari processes in different sessions therefore took different locks and could write the journal concurrently. The lock is now consistently Global\-prefixed on both platforms.

2. Own the Mutex on a dedicated thread

A Mutex is thread-affine: it can only be released by the thread that acquired it. Acquire returns an IDisposable, and callers may dispose it from a different thread than the one that called Acquire — most obviously any async method that awaits in between. The old *nix path already had this problem; switching Windows to a Mutex would have spread it.

Acquire() therefore starts a background thread that performs the real WaitOne()/ReleaseMutex(), and Acquire()/Dispose() just signal to and from it. The returned IDisposable can be handed to, and disposed from, any thread. Disposing twice is a no-op. Acquisition failures are rethrown on the caller's thread with the original stack. A failure during release is logged rather than crashing the process, and leaves the mutex abandoned for the next waiter to recover.

The owner thread, the two ManualResetEventSlims and the double-dispose guard are all held by the Releaser itself, so the teardown lives in the type that does the disposing. The acquisition-failure path in Acquire() reuses Releaser.Dispose() rather than repeating the cleanup.

3. Rename Semaphore to NamedLock

Nothing here is a semaphore any more, so the names no longer say one:

Before	After
ISemaphoreFactory	INamedLockManager
SystemSemaphoreManager	MutexBasedNamedLockManager
Calamari.Common.Features.Processes.Semaphores	Calamari.Common.Features.Processes.NamedLocks
SemaphoreFixtureBase	NamedLockFixtureBase

Fields, parameters and the SemaphoreName constants follow. The lock name strings are deliberately unchanged ("Octopus.Calamari.DeploymentJournal", nameof(WindowsX509CertificateStore), nameof(PackageJournal)) so the rename adds no new mixed-version exclusion gap beyond the Global\ one above.

Tests

Three tests in NamedLockFixtureBase, so they run on both the Windows and *nix fixtures:

• AcquireRecoversWhenTheHolderIsAbandoned — a thread takes a raw named Mutex under the same Global\ name and exits without releasing it, modelling an external process dying mid-critical-section. Acquire() must return within 15s. Fails on main on Windows (times out) and passes with this branch.
• ReleasingFromADifferentThreadThanAcquiredSucceeds — acquires on one thread, disposes on another, then reacquires to prove the lock was really released. Fails without the owner-thread change (ApplicationException from ReleaseMutex).
• DisposingTheReleaserTwiceIsANoOp — documents the IDisposable contract. Worth being honest about this one: it passes with and without the Interlocked guard, because ManualResetEventSlim.Set() on a disposed event does not throw and Join() on a finished thread returns immediately. Double disposal was already benign; the guard is belt-and-braces, and this test pins the contract rather than covering a demonstrated failure.

The first two were checked red-then-green by running the fixture at the relevant commits. The package-retention and journal fixtures were also run and pass, and the full solution builds clean.

Risks

• ISemaphoreFactory was public API on the Octopus.Calamari.Common package. Renaming it to INamedLockManager is a source and binary break for anything outside this repo that referenced it. We decided to rename outright and accept the break rather than carry an [Obsolete] shim — worth a second opinion if anyone knows of an external consumer.
• Wider blast radius than the package journal. MutexBasedNamedLockManager is the only INamedLockManager, so every caller gets the new behaviour: DeploymentJournal, ScriptExecutor (CalamariSynchronizeProcess), ApplicationDirectory (extraction directory), and WindowsX509CertificateStore. All of them use using around a synchronous block, so none should notice, but worth a look from the owners.
• Mixed-version window. Because the Windows lock name changes from unprefixed to Global\-prefixed, an old and a new Calamari running concurrently on the same machine will not exclude each other. This is transient and no worse than the cross-session gap the old code already had, but it is worth knowing during a rollout.
• One thread per held lock. Each Acquire() spins up a background thread that lives until Dispose(). Locks are short-lived and never held concurrently in large numbers, so this is negligible, but a leaked Releaser now also leaks a thread (background, so it will not block process exit).
• Nested acquisition self-deadlocks, and on *nix this is a behaviour change. Each Acquire() waits on its own owner thread, so a caller that nests two Acquire() calls for the same name on one thread blocks forever with no timeout. On Windows this matches the old Semaphore(1,1). On *nix the old code took the Mutex on the caller's thread, and a Mutex is re-entrant for its owning thread, so nested acquisition used to succeed there. Every caller was checked (PackageJournal, DeploymentJournal, ScriptExecutor, ApplicationDirectory, WindowsX509CertificateStore) and none nests today.

Not addressed

The unbounded WaitOne() itself. A live process that holds the lock for a long time will still stall others indefinitely — and ApplyRetention holds the lock across every package file deletion, each of which retries for up to a minute. Abandonment now recovers; a slow holder does not. Candidates for a follow-up are a bounded wait that fails the step with a clear message, and moving file deletion outside the critical section.

🤖 Generated with Claude Code

[APErebus]

We should rename this class (and interface) as we are no longer using Semaphores

[APErebus]

This logic should exist in the Releaser, where the disposal is actually occurring

[xwipeoutx]

I've refactored this somewhat, changing "releaser" into a more readable "tracked thread". I think it's much nicer, good call.

[LukeButters]

I think this is where we create (new) up the mutex rather than acquire it. The name is confusing.

[APErebus]

I could be wrong, but I think if the mutex is abandoned, you can take over it (acquire it, if you will)

[LukeButters]

acquire? I think "create makes more sense"

[LukeButters]

I guess calamari is a throw away process so it wont matter, but we probably should try finally each one of these.

[xwipeoutx]

I threw fable at this, because I'm not super well versed in what can be thrown here. Turns out nothing that we haven't already guarded against, I'm comfortable with this as-is

[APErebus]

yeah, that's fine

[LukeButters]

I think adding some verbose logging into where we get up to would be handy to diagnose issues where we see calamari seemingly hang, I think one before and after mutex create. And one just before the attempt to WaitOne. That ill let us know does it reach creation, could it create it, could it get past user stuff?

Oh maybe also a "mutex acquired" one would be good.

[APErebus]

I'm comfortable with this change. Thank you!

[APErebus]

I could be wrong, but I think if the mutex is abandoned, you can take over it (acquire it, if you will)

[APErebus]

yeah, that's fine