feat(mux-elasticsearch): opt-in schema-noise demo mode for outgoing ES body#234
Merged
Merged
Conversation
This was referenced Jun 4, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an env-gated “demo mode” in the mux-elasticsearch sample to intentionally add volatility to the outgoing Elasticsearch request body (to exercise Keploy schema-based request-body noise detection), and also adds a new aerospike-tls sample with record/replay helper scripts plus CI lint coverage.
Changes:
mux-elasticsearch: addSTAMP_CREATED_AT-gated request-body stamping (created_at) and disable ES HTTP keep-alives in demo mode.aerospike-tls: add a full Aerospike CE sample (service + docker-compose + Keploy config) and scripts to record/replay multiple deterministic test-sets.- CI: include
aerospike-tlsin thegolangci-lintworkflow matrix.
Reviewed changes
Copilot reviewed 14 out of 15 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| mux-elasticsearch/go.mod | Promotes direct deps (go-elasticsearch/v8, gorilla/mux) out of indirect list. |
| mux-elasticsearch/app.go | Adds env-gated schema-noise demo mode (created_at stamping + keep-alive disabling). |
| aerospike-tls/scripts/script-1.sh | Script entrypoint for recording/replaying CRUD test-set-0. |
| aerospike-tls/scripts/script-2.sh | Script entrypoint for recording/replaying /parallel test-set-1. |
| aerospike-tls/scripts/script-3.sh | Script entrypoint for recording/replaying /multiclient + /freshclient test-set-2. |
| aerospike-tls/scripts/common.sh | Shared script helpers for bringing up deps, recording, normalizing, noise insertion, and replay. |
| aerospike-tls/README.md | Documents the Aerospike sample layout, endpoints, and script-based record/replay flow. |
| aerospike-tls/main.go | Implements the Aerospike-backed HTTP service and concurrency/determinism helpers. |
| aerospike-tls/keploy.yml | Keploy CLI configuration for running the Aerospike sample under record/test. |
| aerospike-tls/go.sum | Adds module checksums for the new Aerospike sample. |
| aerospike-tls/go.mod | Declares the Aerospike sample module and dependencies. |
| aerospike-tls/Dockerfile | Container build for the Aerospike sample binary (distroless runtime). |
| aerospike-tls/docker-compose.yml | Compose file for Aerospike CE + the sample service. |
| aerospike-tls/aerospike-conf/aerospike.conf | Aerospike CE config for clear-text service port 3000. |
| .github/workflows/golangci-lint.yml | Adds aerospike-tls to lint matrix so it’s included in CI. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
gouravkrosx
approved these changes
Jun 4, 2026
…S body Adds an env-gated (STAMP_CREATED_AT) demo mode to the mux-elasticsearch sample so it can exercise keploy's new schema-based request-body noise detection for HTTP mocks (keploy/keploy#4234, issue #4233): - createDocument stamps a server-side created_at on the indexed document, so the OUTGOING Elasticsearch request body carries a volatile field that drifts between a keploy recording and each replay -- exactly what --schema-noise-detection detects and persists as req_body_noise: { body.created_at: [] } on the ES mock. - The ES client disables HTTP keep-alives in demo mode so each outgoing call closes its connection and keploy can finalize the mock (a pooled keep-alive connection is otherwise buffered until shutdown and dropped). Both behaviours are OFF by default, so the app's normal behaviour and the committed keploy recordings are unchanged; only the schema-noise-detection CI pipeline sets STAMP_CREATED_AT=1. go.mod: promote the two directly-imported modules out of the indirect block (go mod tidy). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Aditya-eddy
force-pushed
the
feat/mux-elasticsearch-schema-noise
branch
from
d2df29c to
7faf0c0
Compare
Aditya-eddy
added a commit
to keploy/keploy
that referenced
this pull request
Jun 4, 2026
Two fixes for the failing schema_noise_detection_linux job: 1. The script is sourced into GitHub's `bash -e` step shell. It does its own error handling (FAILURES counter + explicit exit) and runs commands that return non-zero by design (pkill with no match, grep with no match, startup curl retries), so the first such command aborted the run under errexit. Disable errexit in the script (set +e) and keep pipefail. 2. The job checked out samples-go@main, which does not yet carry the mux-elasticsearch schema-noise demo mode (env STAMP_CREATED_AT) -- so the app produced no drifting ES request body and no outgoing mock. Point the samples-go checkout at the PR branch feat/mux-elasticsearch-schema-noise (keploy/samples-go#234) until it merges; revert to main afterwards. Validated locally end-to-end by sourcing the script into `bash -e` with the patched app and Elasticsearch 8.14.3: all three phases pass (rc=0). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Aditya-eddy
added a commit
to keploy/keploy
that referenced
this pull request
Jun 4, 2026
Two fixes for the failing schema_noise_detection_linux job: 1. The script is sourced into GitHub's `bash -e` step shell. It does its own error handling (FAILURES counter + explicit exit) and runs commands that return non-zero by design (pkill with no match, grep with no match, startup curl retries), so the first such command aborted the run under errexit. Disable errexit in the script (set +e) and keep pipefail. 2. The job checked out samples-go@main, which does not yet carry the mux-elasticsearch schema-noise demo mode (env STAMP_CREATED_AT) -- so the app produced no drifting ES request body and no outgoing mock. Point the samples-go checkout at the PR branch feat/mux-elasticsearch-schema-noise (keploy/samples-go#234) until it merges; revert to main afterwards. Validated locally end-to-end by sourcing the script into `bash -e` with the patched app and Elasticsearch 8.14.3: all three phases pass (rc=0). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Aditya Sharma <aditya282003@gmail.com>
Aditya-eddy
added a commit
to keploy/keploy
that referenced
this pull request
Jun 4, 2026
) * ci: end-to-end pipeline for schema-based req-body noise detection Adds a Linux CI job that exercises the schema-noise-detection feature (#4233, #4234) end-to-end against the real keploy/samples-go mux-elasticsearch app (run with STAMP_CREATED_AT=1 so its outgoing ES request body drifts on body.created_at). The job builds keploy from this branch, runs Elasticsearch as a service, checks out samples-go, then records once and replays three ways: Phase A (control) -- replay without the flag writes NO req_body_noise. Phase B (detect) -- --schema-noise-detection persists req_body_noise: { body.created_at: [] } on the ES mock. Phase C (strict) -- with test.schemaNoiseStrict, a drift on a NON-noise field (content, induced by tampering the recorded mock) is rejected by the matcher. - .github/workflows/schema_noise_detection_linux.yml - .github/workflows/test_workflow_scripts/golang/mux_elasticsearch_schema_noise/golang-linux.sh Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Aditya Sharma <aditya282003@gmail.com> * ci(schema-noise): fix failing pipeline (errexit + samples-go branch) Two fixes for the failing schema_noise_detection_linux job: 1. The script is sourced into GitHub's `bash -e` step shell. It does its own error handling (FAILURES counter + explicit exit) and runs commands that return non-zero by design (pkill with no match, grep with no match, startup curl retries), so the first such command aborted the run under errexit. Disable errexit in the script (set +e) and keep pipefail. 2. The job checked out samples-go@main, which does not yet carry the mux-elasticsearch schema-noise demo mode (env STAMP_CREATED_AT) -- so the app produced no drifting ES request body and no outgoing mock. Point the samples-go checkout at the PR branch feat/mux-elasticsearch-schema-noise (keploy/samples-go#234) until it merges; revert to main afterwards. Validated locally end-to-end by sourcing the script into `bash -e` with the patched app and Elasticsearch 8.14.3: all three phases pass (rc=0). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Aditya Sharma <aditya282003@gmail.com> --------- Signed-off-by: Aditya Sharma <aditya282003@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Aditya-eddy
added a commit
to keploy/keploy
that referenced
this pull request
Jun 4, 2026
* feat: schema-based request-body noise detection for HTTP mocks Add an opt-in (--schemaNoiseDetection) detector that, during auto-replay mock matching, diffs the recorded mock request body against the actual replayed request body, records the drifting field paths as field-path noise (req_body_noise) on the mock, and persists them back to the mock YAML so future matches ignore them. - pkg/matcher/risk.go: new exported ChangedJSONFieldPaths reusing the existing collectJSON/diffMaps testcase machinery; excludes already-known paths and obfuscator-redacted fields. - pkg/agent/proxy/integrations/http: computeReqBodyNoise (JSON + form bodies) wired into h.match, carried out via updateMock. - pkg/models/http.go: HTTPReq.ReqBodyNoise (+ BSON shadow structs). - pkg/models/mock.go: MockState.ReqBodyNoise carrier + DeepCopy of the map. - pkg/agent/proxy/mockmanager.go: attach ReqBodyNoise on all flagMockAsUsed MockState literals. - pkg/platform/yaml/mockdb/db.go: merge ReqBodyNoise onto retained mocks in UpdateMocks. - config/cli/instrument plumbing for the flag; default off => byte-identical match path to today. Closes #4233 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address Copilot review on schema-noise-detection - cli: make --schema-noise-detection the canonical kebab-case flag with a schemaNoiseDetection camelCase alias (matches the schema-match precedent); ValidateFlags reads the kebab-case name so --help and validation agree. - match.go: formReqBodyNoise now feeds the obfuscator checker the full raw "key=value" segment (split on raw bytes, not url.ParseQuery), matching how Mock.Noise is evaluated for form bodies in formBodiesMatchModuloNoise — a value-only check never matched the key-anchored formKeyNoiseRegex, so obfuscated form fields were wrongly re-flagged as schema noise. - mock_test.go: fix stale copy-pasted doc comment on TestDeepCopyPreservesReqBodyNoise. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: compare form req-body values element-wise in formReqBodyNoise strings.Join(..., ",") is lossy for multi-value form fields — embedded commas and repeated-key cardinality differences (["a","bc"] vs ["a,bc"]) collapse to the same joined string, missing or falsely reporting drift. Compare occurrences element-wise (order-sensitive), mirroring formBodiesMatchModuloNoise. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: make TestMergeReqBodyNoise actually detect aliased slices The previous test mutated an empty slice via append, which always reallocates — so it passed even if mergeReqBodyNoise reused the input slice. Use non-empty slices and mutate by index, and assert both the detected and the existing entry's slices are independently copied. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(replay): strict request-body matching for schema-noise mocks On the replay path (SchemaNoiseStrict), an HTTP mock that already carries learned req_body_noise is matched strictly: every request-body field must match except the learned-noise paths, so a drift on a non-noise field rejects the mock instead of being served leniently. The resulting "expected mock not consumed" mismatch now fails the testcase rather than being ignored when the response coincidentally matches — so request-only drift is caught even when the dependency response is deterministic. Auto-replay leaves the flag off so it can still learn noise leniently. Adds SchemaNoiseStrict to config.Test and OutgoingOptions, threads it through runner/replay instrument options and decode -> h.match, adds filterStrictNoiseMatches (reusing detectReqBodyNoise to find non-noise drift). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Aditya Sharma <aditya282003@gmail.com> * ci: end-to-end pipeline for schema-based req-body noise detection (#4243) * ci: end-to-end pipeline for schema-based req-body noise detection Adds a Linux CI job that exercises the schema-noise-detection feature (#4233, #4234) end-to-end against the real keploy/samples-go mux-elasticsearch app (run with STAMP_CREATED_AT=1 so its outgoing ES request body drifts on body.created_at). The job builds keploy from this branch, runs Elasticsearch as a service, checks out samples-go, then records once and replays three ways: Phase A (control) -- replay without the flag writes NO req_body_noise. Phase B (detect) -- --schema-noise-detection persists req_body_noise: { body.created_at: [] } on the ES mock. Phase C (strict) -- with test.schemaNoiseStrict, a drift on a NON-noise field (content, induced by tampering the recorded mock) is rejected by the matcher. - .github/workflows/schema_noise_detection_linux.yml - .github/workflows/test_workflow_scripts/golang/mux_elasticsearch_schema_noise/golang-linux.sh Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Aditya Sharma <aditya282003@gmail.com> * ci(schema-noise): fix failing pipeline (errexit + samples-go branch) Two fixes for the failing schema_noise_detection_linux job: 1. The script is sourced into GitHub's `bash -e` step shell. It does its own error handling (FAILURES counter + explicit exit) and runs commands that return non-zero by design (pkill with no match, grep with no match, startup curl retries), so the first such command aborted the run under errexit. Disable errexit in the script (set +e) and keep pipefail. 2. The job checked out samples-go@main, which does not yet carry the mux-elasticsearch schema-noise demo mode (env STAMP_CREATED_AT) -- so the app produced no drifting ES request body and no outgoing mock. Point the samples-go checkout at the PR branch feat/mux-elasticsearch-schema-noise (keploy/samples-go#234) until it merges; revert to main afterwards. Validated locally end-to-end by sourcing the script into `bash -e` with the patched app and Elasticsearch 8.14.3: all three phases pass (rc=0). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Aditya Sharma <aditya282003@gmail.com> --------- Signed-off-by: Aditya Sharma <aditya282003@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Signed-off-by: Aditya Sharma <aditya282003@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an env-gated (
STAMP_CREATED_AT) demo mode to themux-elasticsearchsample so it can exercise keploy's new schema-based request-body noise detection for HTTP mocks (keploy/keploy#4234, parent issue keploy/keploy#4233).When enabled:
createDocumentstamps a server-sidecreated_aton the indexed document, so the outgoing Elasticsearch request body (POST /documents/_doc) carries a volatile field that drifts between a keploy recording and each replay — exactly what--schema-noise-detectiondetects and persists asreq_body_noise: { body.created_at: [] }on the ES mock.Backward compatibility
Both behaviours are OFF by default. With
STAMP_CREATED_ATunset the app behaves exactly as before (NewClienthonoursELASTICSEARCH_URLthe same wayNewDefaultClientdid), so the committedkeploy/recordings remain valid. Only the schema-noise-detection CI pipeline setsSTAMP_CREATED_AT=1.go.mod: promote the two directly-imported modules (go-elasticsearch/v8,gorilla/mux) out of the// indirectblock (go mod tidy).Used by
The keploy CI pipeline that records this app, replays with
--schema-noise-detection, and asserts the ES mock gainsreq_body_noise: body.created_at(and that strict matching rejects a non-noise drift).🤖 Generated with Claude Code