Skip to content

Coordinate password login admission - #2601

Merged
niemyjski merged 19 commits into
mainfrom
feature/shared-auth-service
Oct 13, 2026
Merged

niemyjski merged 19 commits into
mainfrom
feature/shared-auth-service

Conversation

@niemyjski

@niemyjski niemyjski commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Share password-login admission between interactive and Basic authentication, preserving API-key authentication and recovery behavior. Use configured test services and control completion order while requests overlap at the admission boundary. Handler tests live in ApiKeyAuthenticationHandlerTests.

Validation: 71/71 focused tests passed; all eight Basic cases passed again without retries. A separate localhost Redis probe passed account/IP limits and failure accounting across two independent clients, 80 rounds, and 65 requests per round. All hosted checks passed for 1d2dddd29: Build. AuthService coverage is 100% lines and 94.2% branches.

Known limitation: in-memory concurrent writes remain tracked in #2655 and are accepted temporarily; the Foundatio upgrade stays separate. The pinned Redis provider uses atomic conditional operations. Redis-backed deployments do not require that upgrade for this change.

Performance: uncontended account-plus-IP admission uses four cache calls and completion four more. Conditional writes retry up to eight times. Basic requests poll every 50 ms for up to two seconds, with at most 100 waiters per service instance; provider I/O retains its own timeout. These are source-derived bounds, not measured throughput. Shared-IP budgets aggregate clients and may deny sustained bursts.

Rollout: temporary login counters reset on first deployment; mixed versions use separate budgets until rollout completes.

Verification details

Focused validation covers AuthServiceTests, ApiKeyAuthenticationHandlerTests, and HttpExtensionsTests. The Basic fixture tests overlapping admission with controlled completions; it does not claim simultaneous in-memory writes are fixed. Local Redis 8.6 used the installed Foundatio.Redis 13.0.4 package and production serializer. Its older-Redis Lua fallback was inspected, not runtime-tested. Live production configuration was not inspected; the checked-in Kubernetes defaults select Redis.

@niemyjski

Copy link
Copy Markdown
Member Author

@ejsmith could you review this specific cache-counter behavior before we simplify AuthService? I would prefer ordinary counters over retaining the bounded-marker workaround.

With Foundatio 13.0.4, .NET 10 on macOS arm64, five isolated runs of 10,000 InMemoryCacheClient.IncrementAsync(key, 1) calls at concurrency 16 produced:

Run Expected Actual
1 10,000 9,405
2 10,000 9,329
3 10,000 9,663
4 10,000 9,594
5 10,000 9,539

This reproduction uses only an in-memory cache, without application services or network access. The decompiled implementation reads and mutates the same existing CacheEntry inside the ConcurrentDictionary.AddOrUpdate callback. Concurrent callbacks can read the same value and overwrite each other's increments; that appears to explain the observed results. Redis counter behavior is not implicated by this reproduction.

Exact reproduction

Create a .NET 10 console project with <PackageReference Include="Foundatio" Version="13.0.4" /> and implicit usings enabled, then run:

using Foundatio.Caching;

using var cache = new InMemoryCacheClient();
const int incrementCount = 10000;
for (int run = 0; run < 5; run++)
{
    string key = $"counter-{run}";
    await cache.SetAsync(key, 0L);
    await Parallel.ForEachAsync(Enumerable.Range(0, incrementCount),
        new ParallelOptions { MaxDegreeOfParallelism = 16 },
        async (attempt, cancellationToken) => await cache.IncrementAsync(key, 1));
    long actual = (await cache.GetAsync<long>(key)).Value;
    Console.WriteLine($"Run {run + 1}: expected {incrementCount}, actual {actual}");
    if (actual != incrementCount)
        Environment.ExitCode = 1;
}

This PR remains draft because simply replacing the markers with the current in-memory counter implementation would lose concurrent failures. The dependency should provide the atomic counter behavior so authentication can stay simple.

@ejsmith

ejsmith commented Sep 29, 2026

Copy link
Copy Markdown
Member

Reviewing 3cd7d986e: this PR is broader and more complicated than the Basic-auth password-guessing fix requires. Several changes are worthwhile, but I recommend splitting them up.

Roughly 80% of the additions are tests, and security regression coverage is justified. The concern is the production scope and the custom throttling machinery.

  • Keep: one shared password-attempt limiter used by normal login and Basic auth, consistent email normalization, account/IP limits, existing recovery integration, and compatibility with Basic API-key credentials.
  • Separate: OAuth authorization-code redemption, outbound address filtering, source-map changes, webhook validation/logging, and broader inactive-account enforcement. Those deserve their own focused security changes.
  • Simplify: the five account markers, fifteen IP markers, GUID values, captured failure snapshots, and conditional cleanup. That machinery works around the Foundatio counter bug, yet still leaves a parallel-guessing gap.

For the concurrency concern, a local reproduction using the actual Basic-auth handler with delayed repository responses admitted 100 requests before completion. It evaluated 99 incorrect passwords, and the final correct guess authenticated even after new requests were blocked. Admission needs to account for password checks already underway; the existing tests establish eventual blocking after failures complete.

The Foundatio counter bug is real and was also reproduced locally, so simply replacing the markers with today's counters would be incorrect. Fix and test that underlying primitive separately, then use a small limiter with explicit handling of in-flight verification. Atomic counters alone do not solve the admission race.

There is necessary complexity around concurrency, successful requests, and recovery. However, requiring successful authentication to perform no cache writes or distributed locking makes the design harder. That optimization should have a measured need, and successful requests should not consume failure quota.

I recommend narrowing this PR to the limiter, its two authentication callers, recovery wiring, and targeted regression tests. That should make the change substantially smaller and easier to establish as correct.

@niemyjski
niemyjski force-pushed the feature/shared-auth-service branch 2 times, most recently from f3425cb to d1d1c99 Compare October 2, 2026 03:18
@niemyjski niemyjski changed the title Normalize authentication through a shared AuthService Coordinate password login admission Oct 2, 2026
@niemyjski
niemyjski requested a review from ejsmith October 4, 2026 23:58
@niemyjski niemyjski self-assigned this Oct 4, 2026
@niemyjski
niemyjski added this pull request to stack #2642 October 5, 2026 02:11
@niemyjski
niemyjski marked this pull request as ready for review October 5, 2026 02:11
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-13T01:28:56.186939Z 4bc67e6 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: da2f75b130

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

}

private Task RemoveFailuresAsync(IEnumerable<KeyValuePair<string, string>> failures)
=> Task.WhenAll(failures.Select(failure => _cache.RemoveIfEqualAsync(failure.Key, failure.Value)));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Require atomic compare-and-delete for admission slots

When concurrent successful logins observed the same prior failure, both call this removal; if a third login reserves that slot between the provider's comparison and deletion, the stale remover can delete the third login's reservation. The checked-in Foundatio 13.0.4 stack still has unresolved conditional-operation concurrency (upstream #570), while the new tests exercise only InMemoryCacheClient, so the five-user/fifteen-IP bounds are not established for the deployed provider. Use a provider-level atomic compare-and-delete primitive (or an equivalent lock) before relying on these entries for login admission.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 474a079a83

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

{
string normalizedEmailAddress = emailAddress.Trim().ToLowerInvariant();

return Enumerable.Range(0, UserFailureLimit).Select(index => $"user:{normalizedEmailAddress}:attempts:{expiresUtc.Ticks}:{index}").ToArray();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve admission limits across rolling deployments

Keep the new slots interoperable with the previous user:{email}:attempts counter during migration. The production workflow in .github/workflows/build.yaml performs a normal Helm upgrade, and k8s/exceptionless/templates/api.yaml uses the default rolling Deployment strategy, so old and new API pods coexist while writing disjoint user and IP keys; during every rollout, a client can obtain both versions' independent budgets and roughly double the intended password checks.

Useful? React with 👍 / 👎.

Comment thread src/Exceptionless.AppHost/Program.cs Resolved
@niemyjski
niemyjski marked this pull request as draft October 12, 2026 23:37
@niemyjski
niemyjski force-pushed the feature/shared-auth-service branch from f258e6b to 84b34ad Compare October 12, 2026 23:44
@niemyjski
niemyjski marked this pull request as ready for review October 13, 2026 00:44
@niemyjski
niemyjski force-pushed the feature/shared-auth-service branch from f541c2e to 69d4f36 Compare October 13, 2026 01:05
@github-actions

Copy link
Copy Markdown

Code Coverage

Package Line Rate Branch Rate Complexity Health
Exceptionless.AppHost 23% 23% 128 ❌
Exceptionless.Core 77% 69% 10916 ✔
Exceptionless.Insulation 51% 43% 370 ➖
Exceptionless.Web 86% 71% 9177 ✔
Summary 80% (28079 / 35061) 69% (13885 / 20168) 20591 ✔

@niemyjski
niemyjski merged commit f689155 into main Oct 13, 2026
22 checks passed
@niemyjski
niemyjski deleted the feature/shared-auth-service branch October 13, 2026 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants