feat(infra+processing): distributed event-filter store, dependabot, security scan - #447
Open
Eleora57 wants to merge 2 commits into
Open
feat(infra+processing): distributed event-filter store, dependabot, security scan#447Eleora57 wants to merge 2 commits into
Eleora57 wants to merge 2 commits into
Conversation
…ecurity scan Closes XStreamRollz#351, XStreamRollz#368, XStreamRollz#372. XStreamRollz#351 — processing: EventFilter configs are worker-local — filter changes do not propagate in multi-worker deployments. Refactor EventFilter to delegate persistence + propagation to a pluggable FilterConfigStore (mirrors the LockManager abstraction). Backends: - MemoryFilterConfigStore (default, single-worker / tests) - RedisFilterConfigStore (hash + pub/sub channel with periodic reconcile; proper defensive copies on every read/write) Wiring: - Worker start() picks a backend via EVENT_FILTER_BACKEND=memory|redis and EVENT_FILTER_REDIS_URL (falls back to REDIS_URL). On a failed install the cached startPromise is cleared so subsequent setConfig/clearConfig calls retry without bricking the filter. - Worker graceful shutdown closes the store alongside the lock manager, so dangling pub/sub connections cannot keep the loop alive past drain. - setConfig/clearConfig on EventFilter update the local map synchronously so the hot-path allow() call never awaits the store round trip — a transient Redis outage cannot stall the polling loop. Tests: - MemoryFilterConfigStore + createFilterConfigStore factory - RedisFilterConfigStore with a in-process faked ioredis that exercises cross-instance pub/sub fan-out - Regression for the startPromise-stuck-on-failure bug - Integration test mock extended to provide start/close/setStore so the existing worker startup sequence is exercised end-to-end. XStreamRollz#368 — Dependabot configuration for automated dependency updates. Added .github/dependabot.yml subscribing to the root workspace plus every package (app, api, xstreamroll-sdk, xstreamroll-processing) and to GitHub Actions. Weekly Monday schedule; per-workspace PR caps of 3-5 keep the queue manageable; minor/patch updates are grouped per dependency-type. XStreamRollz#372 — Trivy vulnerability scanning of Docker images + filesystem. Added .github/workflows/security-scan.yml. Two scan targets: - Filesystem (TRIVY fs) — surfaces lockfile CVEs cheaply - Per-image (TRIVY image) — api, app, processing built with BuildKit, scanned for OS package CVEs SARIF uploads land in the GitHub Security tab. The PR-comment job runs on pull_request_target so fork PRs (where standard pull_request events lack the security-events permission) also receive the summary. Scheduled weekly Monday 06:15 UTC, fifteen minutes after the Dependabot run, so newly disclosed CVEs are re-checked promptly.
Contributor
|
@Eleora57 resolve conflicts |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #351, Closes #368, Closes #372.
Summary
This PR resolves all three issues assigned to @Eleora57 in one batch:
EventFilterconfigs are now distributed across worker pods via a pluggableFilterConfigStore(memory default, Redis-backed for clusters). EachsetConfig/clearConfigcall fans out to peer workers in real time through a Redis hash + pub/sub channel, with a periodic reconcile loop for missed messages..github/dependabot.ymlautomating weekly dependency updates for the root workspace, every package (app,api,xstreamroll-sdk,xstreamroll-processing), and GitHub Actions. Per-workspace PR caps of 3-5 keep the queue manageable..github/workflows/security-scan.ymlrunning Trivy against the filesystem and each of the three production Docker images (api,app,xstreamroll-processing). SARIF results land in the GitHub Security tab; a PR-comment summarises critical/high counts. The job listens onpull_request_targetso fork PRs (where the standard event strips write permissions) still receive the summary.Changes
Issue #351 (processing/event-filter)
xstreamroll-processing/src/pipeline/event-filter-store.tsFilterConfigStore+MemoryFilterConfigStore+RedisFilterConfigStore+createFilterConfigStorefactory. Defensive copies on every read/write; reconcile interval as the safety net for missed pub/sub messages.xstreamroll-processing/src/pipeline/event-filter.tsEventFilternow delegates to an injected store.setConfig/clearConfigupdate the in-memory map synchronously so the hot-pathallow(event)call never awaits the store; transient Redis outages cannot stall the polling loop. The cachedstartPromiseis cleared on install failure so subsequent calls retry instead of bricking the filter.xstreamroll-processing/src/pipeline/index.tsxstreamroll-processing/src/config.tsEVENT_FILTER_BACKEND(memory/redis, defaultmemory) andEVENT_FILTER_REDIS_URL(optional, falls back toREDIS_URL).xstreamroll-processing/.env.examplexstreamroll-processing/src/worker.tsinitFilterStore()pick the backend, swap it into the default filter viasetStore(), callawait filter.start()before spinning the registry, and register a graceful-shutdown hook that closes the store.xstreamroll-processing/package.jsonioredis@^5.4.1runtime dependency.xstreamroll-processing/__tests__/event-filter-store.test.tsxstreamroll-processing/__tests__/event-filter-store-redis.test.tsioredis.xstreamroll-processing/__tests__/integration/pipeline.integration.test.tsEventFilterextended with no-opstart/close/setStore/setConfig/clearConfigso the worker startup sequence is exercised end-to-end.Issue #368 (dependabot)
.github/dependabot.ymldependencies/automated. Commits prefixedchore(deps*)per workspace.Issue #372 (security scan)
.github/workflows/security-scan.ymlgithub.ref. Schedules weekly Monday 06:15 UTC.Verification
npx tsc --noEmitis clean.npx jest --runInBandis green: 110 tests pass across 13 suites, including the existingevent-filter.test.ts(unchanged, still uses the synchronousf.setConfig(...)pattern), the newevent-filter-store.test.ts, and the newevent-filter-store-redis.test.tsthat exercises cross-instance pub/sub through a fakedioredisclient.worker.test.tsandpipeline.integration.test.tscontinue to pass without modification beyond the mock extension described above.Backwards compatibility
EVENT_FILTER_BACKENDdefaults tomemory, so existing single-worker deployments see no behaviour change.EVENT_FILTER_REDIS_URLis optional and falls back toREDIS_URLso workers sharing the API's Redis need no extra config.Suggested followups (out of scope, future PRs)
aquasecurity/trivy-action,docker/build-push-action,docker/setup-buildx-action,github/codeql-actionso a tag repoint cannot silently change behaviour. Tracked as a one-line follow-up per action.POST /admin/filters/:streamIdto the worker's existing metrics HTTP server so operators can push filter changes without booting a Redis-cli session. (The plumbing is in place — only the HTTP handler and an auth check are missing.)ioredisclient doesn't double-instrument the API'scache-manager-redis-storeinstance under the same Redis.