Restore WireGuard Reddit access

[protostatis]

Summary

• restore host-level WireGuard as the Reddit access path
• remove Bright Data/residential proxy fallback from the crawler runtime path
• clear proxy envs in deploy so Reddit traffic uses the host VPN
• update deployment docs/examples back to WireGuard

Verification

• uv run pytest tests
• uv run --extra dev ruff check crypto_sentiment_crawler/crawler/fetcher.py
• git diff --check

Production note

WireGuard is enabled and handshaking on EC2, but Reddit is still returning block pages for old.reddit.com across tested Mullvad US relays. This PR restores the VPN setup in code/deploy; it does not resolve Reddit blocking Mullvad egress IPs.

[protostatis]

Sky's Code Review

This PR reverts back to host-level WireGuard VPN for Reddit access, removing the residential proxy (Bright Data) fallback from the crawler runtime. The change is clean across deploy scripts, CI/CD, Docker Compose, and docs. However, there are two notable issues: (1) the diff does not include deploy/setup-wireguard.sh, which is the critical script now invoked in deploy.yml and push-to-ec2.sh — this makes the PR incomplete, since new reviewers can't verify the WireGuard setup logic (especially EC2 metadata/GitHub bypass routes). (2) In .env.example, PROXY_URL is removed entirely, which could confuse someone setting up the project who wants to use a proxy as an optional override.

Verdict: Comment

Comments

• deploy/setup-wireguard.sh is referenced in both deploy.yml and push-to-ec2.sh but is not included in this diff. That script is now on the critical path for production Reddit access and its bypass routes for EC2 metadata/GitHub. Without seeing it, reviewers cannot verify correctness or security of the WireGuard setup.
• .env.example removes both RESIDENTIAL_PROXY and PROXY_URL. While .env.docker.example retains PROXY_URL as optional, newcomers using .env.example as their starting template won't know proxy override is still available. Consider keeping PROXY_URL in .env.example as an empty/commented entry.
• In deploy.yml, the crawler docker run explicitly passes '-e PROXY_URL=' and '-e RESIDENTIAL_PROXY=' to blank them. This is a good defensive pattern — it prevents accidental leakage from a host .env file. Consider adding a brief inline comment explaining the intent.
• The deleted test file (tests/test_fetcher_proxy.py) removed tests for _needs_proxy() and the RESIDENTIAL_PROXY fallback. The proxy fallback removal is correct given the architectural change, but there are no tests validating that PROXY_URL still works when explicitly set. Consider keeping a minimal test for the proxy rotation feature in case someone uses it in the future.
• The PR's own production note confirms WireGuard handshakes but Reddit still returns block pages for Mullvad US relays. This is a good honest caveat — operators should not assume this PR alone fixes Reddit crawling.
• docs/EC2_INSTANCES.md removes the EC2 metadata bypass troubleshooting advice and replaces it with 're-run the setup script.' This is reasonable if setup-wireguard.sh handles DNS/route restoration correctly, but it's worth confirming the script is idempotent.
• docker-compose.yml removes RESIDENTIAL_PROXY env mapping. This is consistent with the change, but note that anyone still using docker-compose directly (not the CI deploy) won't have the explicit env-blanking that deploy.yml adds. They'd need PROXY_URL unset in their .env file.

---

Reviewed by Sky — Unchained Sky engineering agent

[protostatis]

Title renamed from 'Residential Proxy Setup' to 'WireGuard Setup', but the old file content still references proxy concepts in places. Verify the full file body is consistent — the proxy rotation info and comma-separated proxy URL syntax should be removed if WireGuard is now the primary path.

[protostatis]

WG_PRIVATE_KEY is stored as a plain env var here. Consider adding a comment noting that this key will be written to /etc/wireguard/wg0.conf by the setup script, so it persists on disk — operators should ensure the file is mode 600.