Skip to content

fix: defer migration when a Node's ExternalIP is transiently missing - #1

Open
zzzz465 wants to merge 4 commits into
v1.57.0-pdo-767-fleet-pressurefrom
fix/pdo-1305-transient-external-ip
Open

zzzz465 wants to merge 4 commits into
v1.57.0-pdo-767-fleet-pressurefrom
fix/pdo-1305-transient-external-ip

Conversation

@zzzz465

@zzzz465 zzzz465 commented Nov 2, 2026 •

Copy link
Copy Markdown

Background

  • Azure AKS cloud-node-manager can drop the ExternalIP from a Node's status.addresses for one of its 5-minute reconcile cycles
    • Hostname and InternalIP stay, and the same ExternalIP comes back 5 minutes later
    • The public IP keeps working during that time
  • The migration controller treats this as an address mismatch and marks every post-Ready GameServer on that Node as Unhealthy
    • As a result, every match running on the Node is terminated
  • A GameServer scheduled onto the Node while the ExternalIP is missing gets the InternalIP as its address
    • It keeps the InternalIP after the ExternalIP comes back, because the InternalIP is still one of the Node's addresses

Changes

  • Defer the migration check while the ExternalIP is transiently missing
    • Condition: the pod's nodeName is unchanged, the GameServer's address type is ExternalIP (based on status.addresses), and the Node has no ExternalIP
    • Record the first time this is seen per GameServer in memory, and re-check with EnqueueAfter for 15 minutes
    • If the ExternalIP is still missing after 15 minutes, fall back to the existing behavior
    • Clear the record when the pod is no longer a migration candidate or is deleted
  • Fix GameServers that were given an InternalIP while the ExternalIP was missing
    • Condition: the GameServer's address type is InternalIP or InternalDNS, and the Node has an ExternalIP or ExternalDNS
    • Before Ready: re-apply the address
    • Ready or Reserved: move to Unhealthy so it is recreated
    • Allocated: keep it, to preserve the running session

Reasonably and others added 2 commits November 2, 2026 15:55
- Azure cloud-node-manager can drop ExternalIP from Node.status.addresses for one
  5 minute reconcile cycle while the public IP keeps working
- When a GameServer on the same Node was given that ExternalIP and the Node has no
  ExternalIP, wait 15 minutes before treating it as migrated
- When a GameServer was given an internal address while the Node had no external
  address and the Node has one again, re-apply the address before Ready and move
  Ready/Reserved GameServers to Unhealthy, keeping Allocated ones
- ExternalIP changes to a different value and NodeName changes are handled as before

Refs: PDO-1305
@zzzz465
zzzz465 marked this pull request as ready for review November 2, 2026 07:09
@zzzz465
zzzz465 requested a balanced review from Copilot November 2, 2026 07:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Controller image builds fail with the existing build context, the image omits dependency distribution artifacts, and migration handling can misclassify servers or mishandle grace periods.

7 open issues
What changed in this PR

This PR aims to prevent transient Node ExternalIP omissions from terminating running GameServers, while correcting internal addresses when external connectivity returns.

Changes:

  • Adds a 15-minute migration grace period with per-Pod tracking and delayed retries.
  • Handles restored external addresses according to GameServer state, with tests for deferral, expiry, and recovery.
  • Switches the controller image to a multi-stage source build and expands the image workflow to include it.
File Description
pkg/​gameservers/​migration.go Adds grace-period tracking and external-address recovery handling.
pkg/​gameservers/​migration_test.go Tests missing and restored addresses, state transitions, and grace-period timing.
cmd/​controller/​Dockerfile Builds the controller from source in a multi-stage image.
.github/​workflows/​build-images.yml Builds and publishes allocator and controller images through a matrix.

Review effort: Balanced

Comment thread cmd/controller/Dockerfile Resolved
Comment thread cmd/controller/Dockerfile Resolved
Comment thread pkg/gameservers/migration.go Resolved
Comment thread pkg/gameservers/migration.go Resolved
Comment thread pkg/gameservers/migration.go Resolved
Comment thread pkg/gameservers/migration.go Resolved
Comment thread pkg/gameservers/migration.go Resolved
@mnthe

mnthe commented Nov 3, 2026

Copy link
Copy Markdown

엣지케이스나 상태관리 찌꺼기 피드백들 좀 있네요, 체크해보면 좋을 것 같아요

그리고 언헬시 노드에 Pod을 받는게 아니라 ExtenralIP 없으면 안받고있다가 풀리면 받는게 나을 것 같긴 한데, 주소가 있는걸로 판단해서 어려운거죠?

- Generate LICENSES and dependencies-src.tgz in the builder stage of the
  allocator and controller Dockerfiles, as the prebuilt-binary images did
- Allow the two build scripts through .dockerignore
- Build the allocator and controller images from the repository root in
  build/Makefile, forwarding VERSION and GO_BUILD_TAGS
- Resolve GameServer and Node address types with address() precedence
  and IP validity, so equal InternalIP/ExternalIP values or invalid
  ExternalIP entries do not trigger repeated recreation
- Treat a Node as restored only when address() would now pick an
  external address
- Clear the grace period record only after a successful update, on
  Pod not-found, and on Node updates that bring the ExternalIP back
- Key the record by Pod UID so a replacement Pod gets its own grace period
@github-actions github-actions Bot added size/l and removed size/m labels Nov 3, 2026
@zzzz465

zzzz465 commented Nov 3, 2026 •

Copy link
Copy Markdown
Author

엣지케이스나 상태관리 찌꺼기 피드백들 좀 있네요, 체크해보면 좋을 것 같아요

Copilot 리뷰 기반으로 수정했습니다.

그리고 언헬시 노드에 Pod을 받는게 아니라 ExtenralIP 없으면 안받고있다가 풀리면 받는게 나을 것 같긴 한데, 주소가 있는걸로 판단해서 어려운거죠?

문제점은 node.status.address 만 읽는 것이라 public IP 가 없는 노드, 일시적으로 IP 가 빠진 노드를 구분하지 못해서 그렇습니다. GameServer CR 에 custom status 를 추가해서 기존에 IP 가 있었는지 여부를 기록하면 되긴 해요.

버그 수정 이외에 추가적인 분기를 만드는건 원본 소스에서 기능이 달라지는 것이라 검토를 좀 더 하면 좋겠습니다

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants