improve lease code - #911
Open
pjfanning wants to merge 9 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
H9. Premature LeaseAcquired Reply Before Lease Is Granted
File: lease-kubernetes/.../LeaseActor.scala:172-177
On conflict retry, LeaseAcquired is sent to the caller before the retry update succeeds. granted.set(true) is never called.
H10. No Heartbeat Retry — Immediate Lease Loss on Transient Failure
File: lease-kubernetes/.../LeaseActor.scala:201-206
A single heartbeat failure immediately releases the lease (with 120s TTL budget available for retries). Contains a FIXME acknowledging this.
M9 — Connection leak on timeout: Replaced Future.firstCompletedOf with a Promise-based approach. Both the response and timeout write to the same promise via trySuccess/tryFailure. When the timeout fires first, it calls response.foreach(_.discardEntityBytes()) to consume the response entity if one arrives later, preventing the HTTP connection from leaking. The promise-based approach avoids the race condition where firstCompletedOf abandons the slower future without cleanup.
L6 — DateTimeFormatter on every heartbeat: Added UTC_FORMATTER as a val in the companion object (NativeKubernetesApiImpl.scala:43) that caches RFC3339MICRO_FORMATTER.withZone(ZoneId.of("UTC")). The currentTimeRFC3339 method now uses this cached formatter instead of calling .withZone on every heartbeat.
L7 — Silent Acquire drop: In LeaseActor.scala:247, the whenUnhandled Acquire handler now sends InvalidRequest("Tried to acquire a lease while previous acquire/release still in progress") to the sender, matching the behavior of the Release handler. Previously it logged but never replied, leaving the caller hanging.
Also deals with: FIXME deal with failure from releasing the the lock, currently handled in whenUnhandled but could retry to remove