Skip to content

Swift SDK: IdentityResolverSignIntegrationTests flakes on CI due to real-login-keychain writes #4026

Open

Description

@Claudius-Maginificent

Expected Behavior

IdentityResolverSignIntegrationTests should pass reliably in CI regardless of the runner's keychain state.

Current Behavior

Two tests in IdentityResolverSignIntegrationTests intermittently fail on the self-hosted macOS CI runner (latte-ssh) with a keychain error thrown from WalletStorage.swift:95:

  • testResolverRejectsWrongPathBeforeSigning — keychainError(-60008)
  • testResolverSignsForConsistentBreadcrumb — keychainError(-60005)

Both tests call storage.storeMnemonic(mnemonic, for: walletId), which writes a real BIP-39 mnemonic to the host macOS login keychain via SecItemAdd — no kSecUseDataProtectionKeychain, no test-specific access group, no mock storage. On a headless/SSH CI session the login keychain write can fail nondeterministically. Two different OSStatus codes surfacing across the same run (-60008 vs -60005) is the signature of nondeterministic securityd state, not a deterministic logic error.

Confirmed as an environment flake, not a code defect:

  • Same tests pass on v4.1-dev runs on the same runner pool.
  • A clean rerun of the exact same CI job (run 28791155484, job 85376353259) passed with zero code changes (Swift SDK build / Swift SDK build + tests completed green in 9m19s).
  • The test file was introduced by feat: complete dashpay in platform wallet and swift example app #3841 (already on v4.1-dev) and is unrelated to whatever PR happens to trigger the run.

Possible Solution

Isolate these tests from the real login keychain, e.g.:

  • Use kSecUseDataProtectionKeychain: true with a dedicated test access group, or
  • Inject an in-memory/mock WalletStorage for these tests, or
  • throw XCTSkip(...) when the keychain is unavailable in a headless session, so CI success stops depending on runner keychain state.

Steps to Reproduce (for bugs)

  1. Run the Swift SDK CI job (Swift SDK build / Swift SDK build + tests) on the latte-ssh self-hosted runner enough times.
  2. Occasionally IdentityResolverSignIntegrationTests.testResolverRejectsWrongPathBeforeSigning and/or testResolverSignsForConsistentBreadcrumb fail with a keychainError thrown from WalletStorage.swift:95, while an immediate rerun with no code change passes.

Context

This flake blocked PR #3692's CI checks for reasons unrelated to that PR's own diff, costing review/merge time. Any other PR that happens to run the Swift SDK job on this runner is equally exposed.

Your Environment

🤖 Co-authored by Claudius the Magnificent AI Agent

Activity

  1. Claudius-Maginificent commented on Jul 10, 2026

    @Claudius-Maginificent
    CollaboratorAuthor

    Related infra flake (different root cause): the team also hit a second, unrelated macOS-keychain failure mode on mac-runner-2 — SecKeychainCopyDomainDefault fails with "A default keychain could not be found" and the job dies in ~16s, before any code compiles. That's the runner's latte-ssh session having no default keychain at all after a reboot (pure infra/session-state issue), not this test writing to the login keychain. Per team Slack (2026-07-10), vivek.sharma applied a fix and a Swift SDK job on mac-runner-2 got past that immediate failure and reached cargo build.

    Noting here so the two keychain-flavored CI flakes aren't conflated during triage — this issue's fix (isolate WalletStorage tests from the real login keychain) is still needed independently of the runner-level fix.

  2. Claudius-Maginificent commented on Jul 10, 2026

    @Claudius-Maginificent
    CollaboratorAuthor

    Update — likely same root cause as the mac-runner-2 infra fix, not a separate code defect.

    Checked the actual OSStatus meanings:

    • -60005 = errAuthorizationDenied (an Authorization Services request was denied)
    • -60008 = "unable to obtain authorization for this operation" — same family

    Both are authorization/session-context failures, not a logic bug in the test or in WalletStorage. That fits the same underlying disease as the mac-runner-2 "no default keychain" flake reported in Slack today: a headless/SSH session with no valid security session has nothing to authorize SecItemAdd against. Different API call (SecItemAdd vs SecKeychainCopyDomainDefault), same broken-session root cause.

    Since the runner's keychain/session was patched today (2026-07-10, per team Slack — vivek.sharma), this specific flake may disappear on its own. Sampled the last 50 "Tests" workflow runs and found zero recurrences of keychainError -60005/-60008 in that window, but that predates/overlaps the fix so it isn't conclusive either way.

    Recommendation: leave this issue open but deprioritize the code-side fix (mock/isolate WalletStorage for tests) pending observation — if the flake doesn't recur over the next couple weeks of CI runs post-fix, this can likely be closed as infra-caused rather than requiring a test-isolation change. If it does recur despite a healthy runner keychain, that's stronger evidence of a real code-level gap and the original proposed fix should proceed.

  3. lklimek commented on Jul 10, 2026

    @lklimek
    Contributor

    Most likely this is infra issue. Closing for now, reopen if it reappears.

  4. lklimek commented on Jul 10, 2026

    @lklimek
    Contributor

    as latte suggested:

    we might actually need to not rely on the keyring, otherwise we have to manually unlock it every time the thing reboots

    Reopening :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions