Skip to content

bug: failed sync controller prevents subsequent startup retries #190

Open

Description

@tolgayakar

Describe the bug

When a sync controller fails during startup, it is removed from DynamicMultiClusterManager.runnables, but its entry remains in the sync manager's syncCancels map.

Subsequent calls to ensureSyncController() for the same PublishedResource UID/resourceVersion find this stale entry and return an empty result and nil error without creating a replacement controller.

The controller has stopped, but the sync manager still considers it present.

Steps To Reproduce

  1. Run the agent with an enabled PublishedResource and verify that an annotation update on an existing object in kcp propagates to the service cluster.
  2. Restart the agent with a temporary proxy that returns HTTP 503 for LIST/WATCH requests to that resource in the service cluster. Keep API discovery and other API requests available.
  3. Keep those requests blocked for longer than the controller's cache sync timeout. In this test, requests were blocked for 180 seconds.
  4. Observe Controller failed to start with a cache sync timeout error.
  5. Restore API access without restarting the agent or modifying the PublishedResource.
  6. Update the annotation on the same object in kcp.
  7. Observe that the update does not propagate to the service cluster and no replacement sync controller is created. Confirm that the PublishedResource's resourceVersion is unchanged.
  8. Restart the agent and observe that the pending annotation update propagates.

Expected Behaviour

When a sync controller fails during startup, the sync manager should clear the failed controller's registration and retry startup with backoff.

Once API access is restored, synchronization should resume without restarting the agent or modifying the PublishedResource.

Additional Context

Related: #182 concerns old controllers remaining active after PublishedResource updates. This report concerns failed controllers preventing subsequent startup retries.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug.
    on Oct 7, 2026
  2. Sarthak-Shreshtha01 commented on Oct 8, 2026

    @Sarthak-Shreshtha01

    I now save the sync controller's cancel function only after it has started successfully, so a failed start no longer blocks the next retry. I added a unit test for this. (PR: #191)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions