Repository navigation
Make a failure and the build that produced it visible in the logs - #1399
Merged
Merged
Conversation
A pod answered sign-in with a Postgres 42703 for a column a migration had already renamed -- which only the code from before that migration asks for -- while the operator had built and pushed the version after it. Three things had to be true for that to take three rounds to diagnose. Nothing recorded the build. Every log event now carries Build, and the first line of every boot names it alongside the environment and where searchable logs are going, so "is this pod running the code I think it is" is answerable from one line. The searchable copy went nowhere. Serilog:Seq:ServerUrl shipped as http://localhost:5341, which every deployment inherits and which in a container resolves to itself. The batched sink retries against nothing and the operator reads an empty Seq as silence. The base file now ships blank -- console-only, and it says so -- and the local convenience moves to appsettings.Development.json. Failures thrown outside the MediatR pipeline reached no sink at all. GlobalExceptionFilter logged nothing on the grounds that RequestFailureObserver had, but that only sees exceptions escaping a handler; anything from a controller, model binding or an auth filter was a masked 500 with no line anywhere. The filter now records what the observer marked as unrecorded, at the same severity. Also refuses to start when DbUp discovers no migration scripts. They travel as content files beside the assembly, and if that copy is ever missing PerformUpgrade reports success having applied nothing, which presents exactly as this incident did.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A pod answered
/api/auth/sign-inwith42703: column t.owner_user_id does not exist— a column #1379 renamed, which only pre-#1379 code asks for — while the latest build had been pushed. Diagnosing it took three rounds because of three separate gaps, each fixed here.Nothing recorded which build a pod was running. Every event now carries
Build(fromAssemblyInformationalVersion, which the SDK stamps with the commit sha), and the first line of each boot names it, the environment, and the log destination.The searchable copy went nowhere.
Serilog:Seq:ServerUrlshipped ashttp://localhost:5341; every deployment inherits it, and in a container that resolves to itself. The batched sink retries against nothing and an empty Seq reads as silence. The base file now ships blank — console-only, and it says so at startup — with the zero-config local Seq moved toappsettings.Development.json. Also addsReadFrom.Configuration+Serilog:MinimumLevel, so levels are tunable without a rebuild.Failures thrown outside the MediatR pipeline reached no sink.
GlobalExceptionFilterlogged nothing, reasoning thatRequestFailureObserveralready had — but that only sees exceptions escaping a handler. Anything from a controller, model binding or an auth filter was a masked 500 with no line on any sink. The observer now marks what it records; the filter records what nothing else did, at the same severity.DbUp no longer "succeeds" having found nothing. Scripts travel as content files beside the assembly; if that copy is ever missing,
PerformUpgradereturns success having applied nothing and the app serves an unmigrated database — which presents exactly as this incident did.Note on embedding the migrations
Adding
<EmbeddedResource>alongside the existing content copy — the obvious hardening — would re-run all 124 migrations on any live database. DbUp journals by name, and the two providers name the same file differently:MigrationDiscoveryTestsnow fails if anyone adds a second script source, and the dead embedded provider (which matched nothing, and is what made the change look free) is removed.Test plan
FailuresAreAlwaysLoggedTests(6) — mutation-verified: filter stops logging → 3 red; observer stops marking → 1 redMigrationDiscoveryTests(3) — reded on the real duplicate-name hazard before it was revertedSeqSinkSettingsTestsupdated to pin both halves: base ships blank, Development ships localhost