From 05e1d389e368e6d4cea5314b513d6e1f5f5aefbb Mon Sep 17 00:00:00 2001
From: Akeen Karkare <akeenkarkare@gmail.com>
Date: Thu, 20 Nov 2025 03:41:52 -0500
Subject: [PATCH 01/11] Fix WAL dump to use operation log instead of raw cache
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

## Problem
After PR #28 removed cache invalidations, the WAL dump was failing with:
- ValidationException: "cannot be converted to a numeric value: True"
- ValidationException: "provided key element does not match the schema"

Root cause: `cache_dumper.py` was trying to write **entire cache entries**
to DynamoDB, but cache has wrapped structures like `{"success": True, "data": [...]}`
that don't match DB schemas.

## Solution
Rewrite `dump_cache_to_db()` to use WAL operation log instead of raw cache:
- Read from `wal_manager.get_entries_since(0)` (correctly structured operations)
- Process each WAL operation type: UPDATE, PUT, DELETE, INCREMENT
- Use `update_item()` for partial updates instead of `put_item()` overwrites
- Track errors per operation, only clear WAL if ALL succeed

## Changes
- **cache_dumper.py**: Complete rewrite of `dump_cache_to_db()`
  - Now replays WAL operations instead of dumping raw cache
  - Proper DynamoDB UpdateExpression for UPDATE operations
  - Graceful error handling per operation
  - Only marks synced and clears WAL on full success

- **CACHE_FIX_PLAN.md**: Updated Phase 2 status to completed

## Testing Plan
1. Complete a daily problem → verify streak persists → restart → verify streak still there
2. Create/accept/complete duels → verify all state transitions work
3. Monitor logs for ValidationException errors → should see ZERO
4. Check WAL stats → entries should clear after successful sync

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
---
 CACHE_FIX_PLAN.md               | 246 ++++++++++++++++++++++++++++++++
 scripts/fastapi/cache_dumper.py | 196 ++++++++++++++++---------
 2 files changed, 375 insertions(+), 67 deletions(-)
 create mode 100644 CACHE_FIX_PLAN.md

diff --git a/CACHE_FIX_PLAN.md b/CACHE_FIX_PLAN.md
new file mode 100644
index 0000000..d9b4c78
--- /dev/null
+++ b/CACHE_FIX_PLAN.md
@@ -0,0 +1,246 @@
+# YeetCode Backend Cache Fix Plan
+
+## Executive Summary
+The backend cache system has fundamental architectural issues causing data loss, inconsistent state, and broken functionality. Multiple bandaid fixes have been applied that treat symptoms rather than root causes.
+
+## Root Causes
+
+### 1. **Excessive Cache Invalidation Pattern**
+**Problem**: Most endpoints invalidate cache immediately after write operations
+**Impact**: Dirty data (uncommitted changes) is deleted before WAL can sync to DynamoDB
+**Evidence**:
+- All duel endpoints call `cache_manager.invalidate_all(CacheType.DUELS)` after operations
+- Daily endpoints invalidate DAILY_PROBLEM and DAILY_COMPLETIONS after completion
+- This triggers warnings: "⚠️ Invalidating X dirty cache entries - data may be lost"
+
+**Affected Endpoints**:
+- `/create-duel` - Line 74: invalidates DUELS
+- `/start-duel` - Line 112: invalidates DUELS
+- `/complete-duel` - Line 130: invalidates DUELS
+- `/reject-duel` - Line 151: invalidates DUELS
+- `/record-duel-submission` - Line 175: invalidates DUELS
+- `/complete-daily-problem` - Lines 107-108: invalidates DAILY_PROBLEM, DAILY_COMPLETIONS
+- `/submit-bounty-solution` - Lines 124-125: invalidates BOUNTIES, BOUNTY_COMPETITIONS
+
+### 2. **Inconsistent Cache-First Implementation**
+**Problem**: Mix of cache-first and invalidate-on-write patterns
+**Impact**: Unpredictable behavior, race conditions
+**Evidence**:
+- `/accept-duel` uses cache-first (PR #27 fix)
+- All other duel endpoints still use invalidate-on-write
+- Group/user operations use cache-first with comments saying "do NOT invalidate"
+
+### 3. **WAL Sync Timing Issues**
+**Problem**: WAL background task runs every 30 seconds, but cache invalidation is immediate
+**Impact**: 29-second window where dirty data can be lost
+**Evidence**:
+- `wal_manager.py` syncs every 30 seconds
+- Cache invalidation happens immediately after writes
+- No guarantee dirty data reaches DB before invalidation
+
+### 4. **Corrupted USERS Cache Structure**
+**Problem**: USERS cache has 23 entries but lookups fail
+**Impact**: All user data endpoints return null
+**Evidence**:
+- Cache stats: `"users": 23`
+- `/users/akeen_exe` returns: `{"xp": null, "easy": null, "medium": null, ...}`
+- Suggests cache_operations.py wrote malformed data structure
+
+### 5. **Streak Reset on Restart**
+**Problem**: Streak resets to 0 after server restart
+**Impact**: Users lose streak progress
+**Root Cause**:
+- USER_DAILY_DATA cache not persisted to DB (no WAL operation)
+- Only exists in memory cache with TTL
+- On restart, cache is empty, DB has no streak data
+
+### 6. **XP Discrepancy Between Leaderboards**
+**Problem**: Group leaderboard shows different XP than university leaderboard
+**Impact**: Users see inconsistent stats
+**Root Cause**:
+- Different leaderboards read from different caches
+- Cache invalidation causes cache misses at different times
+- Some leaderboards read stale DB data, others read fresh cache
+
+## Proposed Solutions
+
+### Phase 1: Stop the Bleeding (URGENT - Deploy ASAP)
+
+#### Fix 1.1: Remove ALL cache invalidations from write endpoints
+**Files to modify**:
+- `routes/duels.py` - Remove lines 74, 112, 130, 151, 175
+- `routes/daily.py` - Remove lines 107-108
+- `routes/bounties.py` - Remove lines 124-125
+
+**Rationale**: Cache-first writes update cache in-place. Invalidation destroys uncommitted changes. Let cache TTL handle expiration.
+
+#### Fix 1.2: Fix USERS cache lookup
+**Files to check**:
+- `cache_operations.py` - `update_user_in_cache()` function
+- Verify users are being added to cache with correct structure
+- Ensure writes preserve the list structure: `{"success": True, "data": [...]}`
+
+#### Fix 1.3: Persist USER_DAILY_DATA to database
+**Files to modify**:
+- `cache_operations.py` - `complete_daily_in_cache()` line 224-228
+- Change from `cache_manager.set()` to `cache_manager.write()` with WAL operation
+- This ensures streak persists across restarts
+
+### Phase 2: Architectural Fixes (Deploy within 24 hours)
+
+#### Fix 2.1: Implement immediate WAL sync for critical operations
+**Files to modify**:
+- `cache_manager.py` - Add `write_immediate()` function
+- Calls `write()` then immediately triggers WAL sync for that entry
+- Use for: daily completion, duel completion, XP awards
+
+#### Fix 2.2: Add cache warming on startup
+**Files to modify**:
+- `main.py` - On startup, load USERS table into cache
+- Prevents cache misses on first requests after restart
+- Ensures consistent data immediately
+
+#### Fix 2.3: Make cache invalidation safer
+**Files to modify**:
+- `cache_manager.py` - `invalidate()` and `invalidate_all()`
+- Dump dirty entries to DB BEFORE deleting them
+- Return error if dump fails (don't invalidate)
+- Add `force=True` parameter for admin operations only
+
+### Phase 3: Long-term Improvements (Deploy within 1 week)
+
+#### Fix 3.1: Unified leaderboard data source
+**Problem**: Multiple leaderboards read from different places
+**Solution**: Create single `/leaderboard/{type}` endpoint that:
+- Always reads from same cache
+- Falls back to DB if cache miss
+- Ensures consistency across all leaderboard views
+
+#### Fix 3.2: Add cache health monitoring
+**Files to create**:
+- `cache_health.py` - Monitor dirty entry count, age
+- Alert if dirty entries > threshold
+- Alert if WAL sync is lagging
+- Expose via `/admin/cache/health` endpoint
+
+#### Fix 3.3: Reduce WAL sync interval
+**Files to modify**:
+- `wal_manager.py` - Reduce from 30s to 5s
+- Or implement adaptive sync (sync more frequently when dirty count is high)
+
+## Implementation Priority
+
+### CRITICAL (Deploy Today):
+1. Remove cache invalidations from duel endpoints (Fix 1.1)
+2. Fix USERS cache lookup bug (Fix 1.2)
+3. Persist USER_DAILY_DATA to DB (Fix 1.3)
+
+### HIGH (Deploy Tomorrow):
+1. Immediate WAL sync for critical ops (Fix 2.1)
+2. Cache warming on startup (Fix 2.2)
+
+### MEDIUM (Deploy This Week):
+1. Safer cache invalidation (Fix 2.3)
+2. Unified leaderboard endpoint (Fix 3.1)
+3. Reduce WAL sync interval (Fix 3.3)
+
+### LOW (Deploy When Possible):
+1. Cache health monitoring (Fix 3.2)
+
+## Testing Plan
+
+### After Phase 1 Deploy:
+1. Complete a daily problem → verify streak increments → restart server → verify streak persists
+2. Create a duel → accept duel → verify both users see updated duel status
+3. Check all leaderboards → verify XP matches across all views
+4. Complete 5 duels rapidly → verify all completions recorded correctly
+5. Monitor logs for "⚠️ Invalidating dirty" warnings → should see ZERO
+
+### After Phase 2 Deploy:
+1. Restart server → verify all data immediately available (cache warming working)
+2. Complete daily → verify XP updates within 1 second (immediate WAL)
+3. Monitor cache stats → verify no dirty entries linger > 5 seconds
+
+## Rollback Plan
+
+If Phase 1 causes issues:
+1. Revert to commit before cache invalidation removal
+2. Manually dump all dirty cache to DB: `POST /cache/dump`
+3. Clear cache: `POST /cache/clear`
+4. Monitor for data loss, restore from DB backups if needed
+
+## Success Criteria
+
+- ✅ Streaks persist across server restarts
+- ✅ Duels can be created, accepted, completed without errors
+- ✅ XP is consistent across all leaderboards
+- ✅ Zero "⚠️ Invalidating dirty" warnings in logs
+- ✅ Cache hit rate > 90% for USERS, DUELS, DAILY_PROBLEM
+- ✅ WAL dirty entry count stays < 5 at all times
+
+---
+
+**Created**: 2025-11-18
+**Status**: DRAFT - Awaiting approval
+**Severity**: CRITICAL - Production data loss occurring
+
+## ADDENDUM: WAL Dump Critical Bug (Discovered 2025-11-20)
+
+### Problem
+The `cache_dumper.py` is trying to write **entire cache entries** to DynamoDB, but cache entries have wrapped structures that don't match DB schemas:
+
+```python
+# What's in cache (wrapped):
+{"success": True, "data": [user1, user2, user3]}
+
+# What cache_dumper tries to write:
+convert_to_dynamodb_format({"success": True, "data": [...]})
+# Results in invalid DynamoDB item!
+```
+
+### Errors Observed:
+1. **Daily table**: `"cannot be converted to a numeric value: True"` 
+   - Cache has `users: {username: True}` (boolean)
+   - DynamoDB might expect numeric values
+
+2. **USERS table**: `"provided key element does not match the schema"`
+   - Cache has wrapped structure `{"success": ..., "data": [...]}`
+   - DynamoDB expects individual user objects with `username` key
+
+### Root Cause:
+The WAL system has TWO write mechanisms:
+1. **WAL operations** (from `cache_operations.py`) - Correctly structured, partial updates ✅
+2. **Cache dumps** (from `cache_dumper.py`) - Dumps raw cache, wrong structure ❌
+
+The cache dump should use the WAL operation log, NOT dump raw cache entries.
+
+### Impact:
+- Cache dumps fail silently
+- Dirty data doesn't reach database
+- On server crash/restart, data lost
+
+### Why Phase 1 Helps:
+By removing cache invalidations, we:
+- Reduce frequency of cache dumps (only triggered on explicit /cache/clear)
+- Reduce dirty entry count (entries sync via normal WAL)
+- Buy time to fix the dump logic properly
+
+### Phase 2 Fix Required:
+1. ✅ **COMPLETED** - Rewrite `cache_dumper.py` to use WAL operation log instead of raw cache
+2. ✅ **COMPLETED** - Add validation before processing (checks for complete WAL entries)
+3. ✅ **COMPLETED** - Add error tracking and graceful failure handling
+4. ⏳ **PENDING** - Test thoroughly with all cache types
+
+**Priority**: HIGH (after Phase 1 deploys)
+**Complexity**: MEDIUM-HIGH
+**Risk**: HIGH if not done carefully
+
+### What Was Fixed:
+The `dump_cache_to_db()` function in `cache_dumper.py` now:
+- Reads from WAL operation log (`wal_manager.get_entries_since(0)`) instead of raw cache entries
+- Processes each WAL operation type correctly: UPDATE, PUT, DELETE, INCREMENT
+- Converts data to proper DynamoDB format based on operation type
+- Uses `update_item()` for UPDATEs (partial updates) instead of `put_item()` (full overwrites)
+- Tracks errors per operation instead of failing entire batch
+- Only clears WAL and marks entries synced if ALL operations succeed
+
diff --git a/scripts/fastapi/cache_dumper.py b/scripts/fastapi/cache_dumper.py
index 92e70e4..67aaa71 100644
--- a/scripts/fastapi/cache_dumper.py
+++ b/scripts/fastapi/cache_dumper.py
@@ -127,104 +127,166 @@ def batch_write_to_dynamodb(table_name: str, items: List[Dict]) -> Dict:
 
 async def dump_cache_to_db() -> Dict:
     """
-    Dump all dirty cache entries to DynamoDB
+    Dump all dirty cache entries to DynamoDB by replaying WAL operations.
+
+    CRITICAL FIX: Instead of dumping raw cache entries (which have wrapped structures
+    like {"success": True, "data": [...]}), we now read from the WAL operation log
+    which has correctly structured operations for DynamoDB updates.
 
     Returns:
         Dictionary with dump statistics
     """
-    info("🗄️ Starting cache dump to DynamoDB...")
+    info("🗄️ Starting cache dump to DynamoDB via WAL operations...")
 
     try:
-        # Get all dirty entries from cache
-        dirty_entries = cache_manager.get_dirty_entries()
+        # Get all WAL entries (these have correct structure for DB writes)
+        wal_entries = wal_manager.get_entries_since(0)
 
-        if not dirty_entries:
-            info("✅ No dirty entries to dump")
-            return {"success": True, "entries": 0, "message": "No dirty entries"}
+        if not wal_entries:
+            info("✅ No WAL entries to sync")
+            return {"success": True, "entries": 0, "message": "No WAL entries"}
 
-        info(f"📦 Found {len(dirty_entries)} dirty entries to dump")
+        info(f"📦 Found {len(wal_entries)} WAL entries to sync to DynamoDB")
 
-        # Group entries by table
-        users_items = []
-        daily_items = []
-        duels_items = []
-        bounties_items = []
+        # Track stats
+        total_synced = 0
+        total_failed = 0
+        errors = []
 
-        for entry in dirty_entries:
-            cache_type = entry.get('cache_type')
+        # Process each WAL operation
+        for entry in wal_entries:
+            operation = entry.get('operation')
+            table = entry.get('table')
+            key = entry.get('key')
             data = entry.get('data')
 
-            if not data:
+            if not all([operation, table, key, data]):
+                warning(f"Skipping incomplete WAL entry: {entry}")
                 continue
 
-            # Convert to DynamoDB format
-            dynamodb_item = convert_to_dynamodb_format(data)
-
-            # CRITICAL FIX: Convert string cache_type to CacheType enum for comparison
-            # cache_type is a string (e.g., "users") from cache key, not a CacheType enum
             try:
-                cache_type_enum = CacheType(cache_type)
-            except ValueError:
-                error(f"Unknown cache type: {cache_type}, skipping entry")
+                if operation == "UPDATE":
+                    # Build UpdateExpression from data
+                    update_expr_parts = []
+                    expr_attr_values = {}
+
+                    for field, value in data.items():
+                        update_expr_parts.append(f"{field} = :{field}")
+                        # Convert to DynamoDB format
+                        if isinstance(value, str):
+                            expr_attr_values[f":{field}"] = {'S': value}
+                        elif isinstance(value, int):
+                            expr_attr_values[f":{field}"] = {'N': str(value)}
+                        elif isinstance(value, float):
+                            expr_attr_values[f":{field}"] = {'N': str(value)}
+                        elif isinstance(value, bool):
+                            expr_attr_values[f":{field}"] = {'BOOL': value}
+                        elif isinstance(value, dict):
+                            expr_attr_values[f":{field}"] = {'M': convert_to_dynamodb_format(value)}
+                        elif isinstance(value, list):
+                            expr_attr_values[f":{field}"] = {'L': [{'S': str(item)} for item in value]}
+
+                    # Convert key to DynamoDB format
+                    dynamodb_key = {}
+                    for k, v in key.items():
+                        if isinstance(v, str):
+                            dynamodb_key[k] = {'S': v}
+                        elif isinstance(v, (int, float)):
+                            dynamodb_key[k] = {'N': str(v)}
+
+                    # Perform update
+                    ddb.update_item(
+                        TableName=table,
+                        Key=dynamodb_key,
+                        UpdateExpression=f"SET {', '.join(update_expr_parts)}",
+                        ExpressionAttributeValues=expr_attr_values
+                    )
+
+                elif operation == "PUT":
+                    # Full item put - merge key and data
+                    item = {**key, **data}
+                    dynamodb_item = convert_to_dynamodb_format(item)
+
+                    ddb.put_item(
+                        TableName=table,
+                        Item=dynamodb_item
+                    )
+
+                elif operation == "DELETE":
+                    # Convert key to DynamoDB format
+                    dynamodb_key = {}
+                    for k, v in key.items():
+                        if isinstance(v, str):
+                            dynamodb_key[k] = {'S': v}
+                        elif isinstance(v, (int, float)):
+                            dynamodb_key[k] = {'N': str(v)}
+
+                    ddb.delete_item(
+                        TableName=table,
+                        Key=dynamodb_key
+                    )
+
+                elif operation == "INCREMENT":
+                    # Build increment expression
+                    update_expr_parts = []
+                    expr_attr_values = {}
+
+                    for field, value in data.items():
+                        update_expr_parts.append(f"{field} = if_not_exists({field}, :zero) + :{field}")
+                        expr_attr_values[f":{field}"] = {'N': str(value)}
+
+                    expr_attr_values[":zero"] = {'N': '0'}
+
+                    # Convert key to DynamoDB format
+                    dynamodb_key = {}
+                    for k, v in key.items():
+                        if isinstance(v, str):
+                            dynamodb_key[k] = {'S': v}
+                        elif isinstance(v, (int, float)):
+                            dynamodb_key[k] = {'N': str(v)}
+
+                    ddb.update_item(
+                        TableName=table,
+                        Key=dynamodb_key,
+                        UpdateExpression=f"SET {', '.join(update_expr_parts)}",
+                        ExpressionAttributeValues=expr_attr_values
+                    )
+
+                total_synced += 1
+
+            except Exception as e:
+                total_failed += 1
+                error_msg = f"Failed to sync WAL entry to {table}: {e}"
+                error(error_msg)
+                errors.append(error_msg)
                 continue
 
-            # Route to appropriate table
-            if cache_type_enum == CacheType.USERS:
-                users_items.append(dynamodb_item)
-            elif cache_type_enum == CacheType.DAILY_PROBLEM or cache_type_enum == CacheType.DAILY_COMPLETIONS:
-                daily_items.append(dynamodb_item)
-            elif cache_type_enum == CacheType.DUELS:
-                duels_items.append(dynamodb_item)
-            elif cache_type_enum == CacheType.BOUNTIES or cache_type_enum == CacheType.BOUNTY_COMPETITIONS:
-                bounties_items.append(dynamodb_item)
-
-        # Batch write to each table
-        results = {}
-
-        if users_items:
-            info(f"💾 Writing {len(users_items)} users to DynamoDB...")
-            results['users'] = batch_write_to_dynamodb(USERS_TABLE, users_items)
-
-        if daily_items:
-            info(f"💾 Writing {len(daily_items)} daily items to DynamoDB...")
-            results['daily'] = batch_write_to_dynamodb(DAILY_TABLE, daily_items)
-
-        if duels_items:
-            info(f"💾 Writing {len(duels_items)} duels to DynamoDB...")
-            results['duels'] = batch_write_to_dynamodb(DUELS_TABLE, duels_items)
-
-        if bounties_items:
-            info(f"💾 Writing {len(bounties_items)} bounties to DynamoDB...")
-            results['bounties'] = batch_write_to_dynamodb(BOUNTIES_TABLE, bounties_items)
-
-        # Check overall success
-        all_successful = all(r.get('success', False) for r in results.values())
-
-        if all_successful:
-            # Mark all entries as synced
+        # Mark success if all synced
+        if total_failed == 0:
+            # Mark all cache entries as synced
+            dirty_entries = cache_manager.get_dirty_entries()
             for entry in dirty_entries:
                 cache_manager.mark_synced(entry['cache_type'], entry.get('identifier', ''))
 
-            # Clear WAL file
+            # Clear WAL file after successful sync
             wal_manager.clear()
 
-            total_written = sum(r.get('written', 0) for r in results.values())
-            info(f"✅ Cache dump complete: {total_written} entries written to DynamoDB")
+            info(f"✅ Cache dump complete: {total_synced} WAL operations synced to DynamoDB")
 
             return {
                 "success": True,
-                "entries": total_written,
-                "results": results
+                "entries": total_synced,
+                "failed": 0
             }
         else:
-            total_failed = sum(r.get('failed', 0) for r in results.values())
-            error(f"⚠️ Cache dump partially failed: {total_failed} entries failed")
+            warning(f"⚠️ Cache dump partially failed: {total_failed}/{len(wal_entries)} operations failed")
 
             return {
                 "success": False,
-                "entries": len(dirty_entries),
+                "entries": len(wal_entries),
+                "synced": total_synced,
                 "failed": total_failed,
-                "results": results
+                "errors": errors
             }
 
     except Exception as e:

From 9dccec71f26af0da08cddb920d176703738eb0de Mon Sep 17 00:00:00 2001
From: Akeen Karkare <akeenkarkare@gmail.com>
Date: Thu, 20 Nov 2025 03:52:18 -0500
Subject: [PATCH 02/11] Fix critical DELETE validation and INCREMENT replay
 bugs
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

## Problem 1: DELETE operations rejected
The validation logic used `if not all([operation, table, key, data])`
which rejected DELETE operations because they don't have a data field.
This caused DELETE operations to be silently skipped, and since
total_failed wasn't incremented, the WAL was cleared anyway, losing
delete operations permanently.

## Problem 2: INCREMENT operations replayed multiple times
Non-idempotent INCREMENT operations were replayed from sequence 0
on every retry, causing double/triple increments. No checkpoint was
persisted, so retries always started from the beginning.

## Solutions

### Fix 1: Per-operation validation
- DELETE: Only require operation, table, key (no data)
- All others: Require operation, table, key, AND data
- Increment total_failed for invalid entries (prevents silent WAL clear)
- Log skipped entries with context for debugging

### Fix 2: Sequence checkpointing
- Added checkpoint file: /tmp/yeetcode/wal.checkpoint
- Tracks last_applied_sequence to avoid replaying operations
- Atomic writes (temp file + rename) for crash safety
- Update checkpoint after each successful DB write
- On restart, resume from last checkpoint + 1

## Changes

### wal_manager.py
- Added _checkpoint_file and _last_applied_sequence state
- Added _load_checkpoint() to restore state on startup
- Added get_last_applied_sequence() and set_last_applied_sequence()
- Checkpoint uses atomic write (temp file + os.replace)
- WAL stats now include last_applied_sequence

### cache_dumper.py
- Load checkpoint at start: last_applied = wal_manager.get_last_applied_sequence()
- Fetch only unapplied entries: wal_manager.get_entries_since(last_applied + 1)
- Per-operation validation (DELETE doesn't require data)
- Update checkpoint after each successful operation
- Don't update checkpoint on failure (will retry next time)

## Why This Matters

**Without checkpointing:**
- User completes daily → +200 XP (via INCREMENT)
- Sync partially fails, retries
- Replays from sequence 0 → +200 XP again
- User gets 400 XP instead of 200

**With checkpointing:**
- Checkpoint tracks sequence 0 was applied
- Retry starts from sequence 1
- User gets correct 200 XP

**Without DELETE validation:**
- User rejects duel → DELETE operation created
- Sync runs, rejects DELETE (missing data field)
- WAL cleared anyway
- Duel stays in DB forever

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
---
 scripts/fastapi/cache_dumper.py | 39 ++++++++++++++++-----
 scripts/fastapi/wal_manager.py  | 60 ++++++++++++++++++++++++++++++---
 2 files changed, 87 insertions(+), 12 deletions(-)

diff --git a/scripts/fastapi/cache_dumper.py b/scripts/fastapi/cache_dumper.py
index 67aaa71..962d0ef 100644
--- a/scripts/fastapi/cache_dumper.py
+++ b/scripts/fastapi/cache_dumper.py
@@ -139,14 +139,18 @@ async def dump_cache_to_db() -> Dict:
     info("🗄️ Starting cache dump to DynamoDB via WAL operations...")
 
     try:
-        # Get all WAL entries (these have correct structure for DB writes)
-        wal_entries = wal_manager.get_entries_since(0)
+        # Get checkpoint to avoid replaying already-applied entries
+        last_applied = wal_manager.get_last_applied_sequence()
+        info(f"📍 Checkpoint: last_applied_sequence = {last_applied}")
+
+        # Get WAL entries since last checkpoint (+ 1 to get next unapplied entry)
+        wal_entries = wal_manager.get_entries_since(last_applied + 1)
 
         if not wal_entries:
-            info("✅ No WAL entries to sync")
-            return {"success": True, "entries": 0, "message": "No WAL entries"}
+            info("✅ No new WAL entries to sync")
+            return {"success": True, "entries": 0, "message": "No new WAL entries"}
 
-        info(f"📦 Found {len(wal_entries)} WAL entries to sync to DynamoDB")
+        info(f"📦 Found {len(wal_entries)} new WAL entries to sync to DynamoDB")
 
         # Track stats
         total_synced = 0
@@ -160,9 +164,22 @@ async def dump_cache_to_db() -> Dict:
             key = entry.get('key')
             data = entry.get('data')
 
-            if not all([operation, table, key, data]):
-                warning(f"Skipping incomplete WAL entry: {entry}")
-                continue
+            # Validate based on operation type
+            # DELETE operations don't require data, all others do
+            if operation == "DELETE":
+                if not all([operation, table, key]):
+                    total_failed += 1
+                    error_msg = f"Skipping incomplete DELETE entry (missing operation/table/key): {entry}"
+                    warning(error_msg)
+                    errors.append(error_msg)
+                    continue
+            else:
+                if not all([operation, table, key, data]):
+                    total_failed += 1
+                    error_msg = f"Skipping incomplete {operation} entry (missing required fields): {entry}"
+                    warning(error_msg)
+                    errors.append(error_msg)
+                    continue
 
             try:
                 if operation == "UPDATE":
@@ -254,11 +271,17 @@ async def dump_cache_to_db() -> Dict:
 
                 total_synced += 1
 
+                # Update checkpoint after successful operation to avoid replay
+                entry_sequence = entry.get('sequence', -1)
+                if entry_sequence >= 0:
+                    wal_manager.set_last_applied_sequence(entry_sequence)
+
             except Exception as e:
                 total_failed += 1
                 error_msg = f"Failed to sync WAL entry to {table}: {e}"
                 error(error_msg)
                 errors.append(error_msg)
+                # Don't update checkpoint on failure - will retry this entry next time
                 continue
 
         # Mark success if all synced
diff --git a/scripts/fastapi/wal_manager.py b/scripts/fastapi/wal_manager.py
index 8731a51..7b52c6d 100644
--- a/scripts/fastapi/wal_manager.py
+++ b/scripts/fastapi/wal_manager.py
@@ -43,14 +43,17 @@ def __init__(self, wal_file_path: str = None):
             wal_file_path = str(wal_dir / "wal.log")
 
         self._wal_file = wal_file_path
+        self._checkpoint_file = wal_file_path.replace('.log', '.checkpoint')
         self._lock = threading.RLock()
         self._sequence = 0
+        self._last_applied_sequence = -1
         self._file_handle = None
 
-        # Initialize WAL file
+        # Initialize WAL file and checkpoint
         self._init_wal_file()
+        self._load_checkpoint()
 
-        info(f"📝 WAL Manager initialized: {self._wal_file}")
+        info(f"📝 WAL Manager initialized: {self._wal_file} (checkpoint: {self._last_applied_sequence})")
 
     def _init_wal_file(self):
         """Initialize WAL file if it doesn't exist"""
@@ -222,6 +225,53 @@ def get_entries_since(self, sequence: int) -> List[Dict]:
                 error(f"Failed to get WAL entries: {e}")
                 return []
 
+    def _load_checkpoint(self) -> None:
+        """Load last applied sequence from checkpoint file"""
+        try:
+            if os.path.exists(self._checkpoint_file):
+                with open(self._checkpoint_file, 'r') as f:
+                    data = json.load(f)
+                    self._last_applied_sequence = data.get('last_applied_sequence', -1)
+                    info(f"📍 Loaded checkpoint: last_applied_sequence = {self._last_applied_sequence}")
+            else:
+                info("📍 No checkpoint file found, starting from sequence -1")
+        except Exception as e:
+            error(f"Failed to load checkpoint: {e}, starting from -1")
+            self._last_applied_sequence = -1
+
+    def get_last_applied_sequence(self) -> int:
+        """Get the sequence number of the last successfully applied WAL entry"""
+        with self._lock:
+            return self._last_applied_sequence
+
+    def set_last_applied_sequence(self, sequence: int) -> bool:
+        """
+        Update the last applied sequence checkpoint (atomic write)
+
+        Args:
+            sequence: Sequence number of last successfully applied entry
+
+        Returns:
+            True if successful, False otherwise
+        """
+        with self._lock:
+            try:
+                # Write to temp file first, then atomic rename
+                temp_file = self._checkpoint_file + '.tmp'
+                with open(temp_file, 'w') as f:
+                    json.dump({'last_applied_sequence': sequence}, f)
+                    f.flush()
+                    os.fsync(f.fileno())
+
+                # Atomic rename (overwrites existing checkpoint)
+                os.replace(temp_file, self._checkpoint_file)
+
+                self._last_applied_sequence = sequence
+                return True
+            except Exception as e:
+                error(f"Failed to save checkpoint: {e}")
+                return False
+
     def get_stats(self) -> Dict:
         """Get WAL statistics"""
         with self._lock:
@@ -230,7 +280,8 @@ def get_stats(self) -> Dict:
                     return {
                         "exists": False,
                         "entries": 0,
-                        "size_bytes": 0
+                        "size_bytes": 0,
+                        "last_applied_sequence": self._last_applied_sequence
                     }
 
                 size = os.path.getsize(self._wal_file)
@@ -240,7 +291,8 @@ def get_stats(self) -> Dict:
                     "entries": self._sequence,
                     "size_bytes": size,
                     "size_kb": round(size / 1024, 2),
-                    "path": self._wal_file
+                    "path": self._wal_file,
+                    "last_applied_sequence": self._last_applied_sequence
                 }
 
             except Exception as e:

From f5dcd79b8cfac3bb8102fdceb18a42a33e73a97a Mon Sep 17 00:00:00 2001
From: Akeen Karkare <akeenkarkare@gmail.com>
Date: Thu, 20 Nov 2025 03:52:50 -0500
Subject: [PATCH 03/11] Document DELETE validation and INCREMENT replay fixes
 in plan

---
 CACHE_FIX_PLAN.md | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/CACHE_FIX_PLAN.md b/CACHE_FIX_PLAN.md
index d9b4c78..6c3b8c7 100644
--- a/CACHE_FIX_PLAN.md
+++ b/CACHE_FIX_PLAN.md
@@ -236,11 +236,34 @@ By removing cache invalidations, we:
 **Risk**: HIGH if not done carefully
 
 ### What Was Fixed:
+
+**Initial Fix (Commit 1):**
 The `dump_cache_to_db()` function in `cache_dumper.py` now:
-- Reads from WAL operation log (`wal_manager.get_entries_since(0)`) instead of raw cache entries
+- Reads from WAL operation log (`wal_manager.get_entries_since()`) instead of raw cache entries
 - Processes each WAL operation type correctly: UPDATE, PUT, DELETE, INCREMENT
 - Converts data to proper DynamoDB format based on operation type
 - Uses `update_item()` for UPDATEs (partial updates) instead of `put_item()` (full overwrites)
 - Tracks errors per operation instead of failing entire batch
 - Only clears WAL and marks entries synced if ALL operations succeed
 
+**Critical Follow-up Fixes (Commit 2):**
+
+1. **DELETE Validation Bug**:
+   - Problem: Validation used `if not all([operation, table, key, data])` which rejected DELETEs (no data field)
+   - Impact: DELETE operations silently skipped, WAL cleared anyway, deletes lost permanently
+   - Fix: Per-operation validation (DELETE only needs operation/table/key)
+   - Fix: Increment `total_failed` for invalid entries to prevent silent WAL clearing
+
+2. **INCREMENT Replay Bug**:
+   - Problem: Non-idempotent INCREMENT operations replayed from sequence 0 on every retry
+   - Impact: Users got 2x-3x XP/streak increments after partial sync failures
+   - Fix: Added checkpoint file tracking `last_applied_sequence`
+   - Fix: Resume from checkpoint + 1, update checkpoint after each successful write
+   - Implementation: Atomic checkpoint writes (temp file + `os.replace`) for crash safety
+
+**New WAL Manager Features:**
+- `get_last_applied_sequence()` - Returns checkpoint value
+- `set_last_applied_sequence(seq)` - Atomically updates checkpoint
+- Checkpoint file: `/tmp/yeetcode/wal.checkpoint`
+- Survives crashes and restarts
+

From 37d22359caf5133ef6ba082d99009ec09f406ec8 Mon Sep 17 00:00:00 2001
From: Akeen Karkare <akeenkarkare@gmail.com>
Date: Thu, 20 Nov 2025 03:59:34 -0500
Subject: [PATCH 04/11] Fix list type conversion to preserve nested types
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

## Problem
The list conversion in UPDATE operations was coercing all list items
to strings with `[{'S': str(item)} for item in value]`. This corrupted:
- Numeric values: [1, 2, 3] → [{'S': '1'}, {'S': '2'}, {'S': '3'}]
- Booleans: [true, false] → [{'S': 'True'}, {'S': 'False'}]
- Nested dicts/lists: Lost structure entirely

Additionally, bool check came after int check, causing booleans to be
treated as integers (since bool is subclass of int in Python).

## Solution
Created `convert_value_to_dynamodb(value)` recursive helper that:
- Checks bool BEFORE int (critical for correct type detection)
- Recursively handles nested lists and dicts
- Preserves proper DynamoDB type keys for all types
- Reused by both `convert_to_dynamodb_format()` and UPDATE operation

## Changes
- Added `convert_value_to_dynamodb()` helper function
- Simplified UPDATE operation to use helper instead of inline conversion
- Fixed type check order: bool before int
- Added recursive list/dict handling with proper type preservation

## Example Fix
Before:
```python
# Input: {"tags": ["easy", "array"], "completed": [true, false], "scores": [100, 95]}
# Output: {"tags": [{'S': 'easy'}, {'S': 'array'}],
#          "completed": [{'S': 'True'}, {'S': 'False'}],
#          "scores": [{'S': '100'}, {'S': '95'}]}
```

After:
```python
# Input: {"tags": ["easy", "array"], "completed": [true, false], "scores": [100, 95]}
# Output: {"tags": [{'S': 'easy'}, {'S': 'array'}],
#          "completed": [{'BOOL': true}, {'BOOL': false}],
#          "scores": [{'N': '100'}, {'N': '95'}]}
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
---
 scripts/fastapi/cache_dumper.py | 73 +++++++++++++++++----------------
 1 file changed, 37 insertions(+), 36 deletions(-)

diff --git a/scripts/fastapi/cache_dumper.py b/scripts/fastapi/cache_dumper.py
index 962d0ef..86f4160 100644
--- a/scripts/fastapi/cache_dumper.py
+++ b/scripts/fastapi/cache_dumper.py
@@ -28,6 +28,40 @@ def chunks(lst: List, n: int):
         yield lst[i:i + n]
 
 
+def convert_value_to_dynamodb(value):
+    """
+    Convert a single Python value to DynamoDB format (recursive helper)
+
+    Args:
+        value: Python value of any type
+
+    Returns:
+        DynamoDB formatted value with type key
+    """
+    if isinstance(value, bool):
+        # IMPORTANT: Check bool before int (bool is subclass of int in Python)
+        return {'BOOL': value}
+    elif isinstance(value, str):
+        return {'S': value}
+    elif isinstance(value, int):
+        return {'N': str(value)}
+    elif isinstance(value, float):
+        return {'N': str(value)}
+    elif isinstance(value, dict):
+        return {'M': convert_to_dynamodb_format(value)}
+    elif isinstance(value, list):
+        # Recursively convert each item in the list
+        dynamodb_list = []
+        for item in value:
+            dynamodb_list.append(convert_value_to_dynamodb(item))
+        return {'L': dynamodb_list}
+    elif value is None:
+        return {'NULL': True}
+    else:
+        # Fallback: convert to string
+        return {'S': str(value)}
+
+
 def convert_to_dynamodb_format(data: Dict) -> Dict:
     """
     Convert Python dict to DynamoDB format
@@ -41,29 +75,7 @@ def convert_to_dynamodb_format(data: Dict) -> Dict:
     dynamodb_item = {}
 
     for key, value in data.items():
-        if isinstance(value, str):
-            dynamodb_item[key] = {'S': value}
-        elif isinstance(value, int):
-            dynamodb_item[key] = {'N': str(value)}
-        elif isinstance(value, float):
-            dynamodb_item[key] = {'N': str(value)}
-        elif isinstance(value, bool):
-            dynamodb_item[key] = {'BOOL': value}
-        elif isinstance(value, dict):
-            dynamodb_item[key] = {'M': convert_to_dynamodb_format(value)}
-        elif isinstance(value, list):
-            # Convert list to DynamoDB List
-            dynamodb_list = []
-            for item in value:
-                if isinstance(item, str):
-                    dynamodb_list.append({'S': item})
-                elif isinstance(item, (int, float)):
-                    dynamodb_list.append({'N': str(item)})
-                elif isinstance(item, dict):
-                    dynamodb_list.append({'M': convert_to_dynamodb_format(item)})
-            dynamodb_item[key] = {'L': dynamodb_list}
-        elif value is None:
-            dynamodb_item[key] = {'NULL': True}
+        dynamodb_item[key] = convert_value_to_dynamodb(value)
 
     return dynamodb_item
 
@@ -189,19 +201,8 @@ async def dump_cache_to_db() -> Dict:
 
                     for field, value in data.items():
                         update_expr_parts.append(f"{field} = :{field}")
-                        # Convert to DynamoDB format
-                        if isinstance(value, str):
-                            expr_attr_values[f":{field}"] = {'S': value}
-                        elif isinstance(value, int):
-                            expr_attr_values[f":{field}"] = {'N': str(value)}
-                        elif isinstance(value, float):
-                            expr_attr_values[f":{field}"] = {'N': str(value)}
-                        elif isinstance(value, bool):
-                            expr_attr_values[f":{field}"] = {'BOOL': value}
-                        elif isinstance(value, dict):
-                            expr_attr_values[f":{field}"] = {'M': convert_to_dynamodb_format(value)}
-                        elif isinstance(value, list):
-                            expr_attr_values[f":{field}"] = {'L': [{'S': str(item)} for item in value]}
+                        # Convert to DynamoDB format using helper to preserve types
+                        expr_attr_values[f":{field}"] = convert_value_to_dynamodb(value)
 
                     # Convert key to DynamoDB format
                     dynamodb_key = {}

From 35b0a8b77a361cd8968cec1523c94373260cfda6 Mon Sep 17 00:00:00 2001
From: Akeen Karkare <akeenkarkare@gmail.com>
Date: Thu, 20 Nov 2025 03:59:58 -0500
Subject: [PATCH 05/11] Document list type conversion fix in plan

---
 CACHE_FIX_PLAN.md | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/CACHE_FIX_PLAN.md b/CACHE_FIX_PLAN.md
index 6c3b8c7..7a1eceb 100644
--- a/CACHE_FIX_PLAN.md
+++ b/CACHE_FIX_PLAN.md
@@ -267,3 +267,18 @@ The `dump_cache_to_db()` function in `cache_dumper.py` now:
 - Checkpoint file: `/tmp/yeetcode/wal.checkpoint`
 - Survives crashes and restarts
 
+**Additional Type Conversion Fix (Commit 3):**
+
+3. **List Type Corruption Bug**:
+   - Problem: List conversion used `[{'S': str(item)} for item in value]` which coerced all items to strings
+   - Impact:
+     - Numeric lists corrupted: `[1, 2, 3]` → `[{'S': '1'}, {'S': '2'}, {'S': '3'}]`
+     - Boolean lists corrupted: `[true, false]` → `[{'S': 'True'}, {'S': 'False'}]`
+     - Nested structures lost entirely
+     - Bool checked after int (bool is subclass of int in Python), causing bools treated as ints
+   - Fix: Created `convert_value_to_dynamodb(value)` recursive helper
+     - Checks bool BEFORE int (critical for correct type detection)
+     - Recursively handles nested lists and dicts with proper type preservation
+     - Reused by both `convert_to_dynamodb_format()` and UPDATE operations
+   - Impact: Proper type preservation for all DynamoDB updates
+

From a9836002bd8b2f7cd02299965708ffe97a3df1ca Mon Sep 17 00:00:00 2001
From: Akeen Karkare <akeenkarkare@gmail.com>
Date: Thu, 20 Nov 2025 17:49:54 -0500
Subject: [PATCH 06/11] Add cache debug endpoint and fix duel ID mismatch
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

## Problem 1: No visibility into cache state
Without visibility into cache contents, WAL status, and dirty entries,
debugging production issues requires SSH access and manual inspection.

## Problem 2: Duel lookup failing
`get_duel_endpoint` uses `duel.get('id')` but DynamoDB schema uses
`duelId` as the primary key, causing all duel lookups to fail.

## Solutions

### Cache Debug Endpoint
Added `/admin/cache/status?api_key=...` endpoint that returns:
- Cache stats (size, entries per type, hit rate)
- WAL stats (entries, checkpoint, file size)
- Dirty entries count and summary (without exposing user data)
- Cache keys by type (for debugging key structure issues)

Also added `/admin/cache/dump?api_key=...` to manually trigger dumps.

### Duel ID Fix
Changed `routes/duels.py:193` from:
```python
if duel.get('id') == duel_id:
```

To:
```python
if duel.get('duelId') == duel_id:
```

This matches the DynamoDB schema and cache_operations.py which all
use `duelId` as the primary key.

## Usage

Access cache debug dashboard:
```
GET http://209.182.234.33:6969/admin/cache/status?api_key=YOUR_KEY
```

Response includes:
- `cache`: Stats from cache_manager.get_stats()
- `wal`: Stats from wal_manager.get_stats() with checkpoint info
- `dirty_entries`: Count and sample of unsync entries
- `cache_keys_by_type`: Keys grouped by cache type for debugging

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
---
 scripts/fastapi/routes/admin.py | 77 +++++++++++++++++++++++++++++++++
 scripts/fastapi/routes/duels.py |  3 +-
 2 files changed, 79 insertions(+), 1 deletion(-)

diff --git a/scripts/fastapi/routes/admin.py b/scripts/fastapi/routes/admin.py
index d69903d..1080770 100644
--- a/scripts/fastapi/routes/admin.py
+++ b/scripts/fastapi/routes/admin.py
@@ -153,3 +153,80 @@ async def get_log_content(
         return {"success": True, "content": content, "path": log_path}
     except Exception as error:
         return {"success": False, "error": str(error)}
+
+
+@router.get("/cache/status")
+async def get_cache_status(
+    api_key: str = Depends(verify_api_key_query)
+):
+    """Get comprehensive cache status including all entries and WAL stats
+
+    Access via: /admin/cache/status?api_key=YOUR_API_KEY
+
+    Returns:
+        - Cache stats (size, hit rate, entries per type)
+        - WAL stats (entries, checkpoint, file size)
+        - Sample of cache entries (keys only, for privacy)
+        - Dirty entries count
+    """
+    try:
+        from cache_manager import cache_manager
+        from wal_manager import wal_manager
+
+        # Get cache stats
+        cache_stats = cache_manager.get_stats()
+
+        # Get WAL stats
+        wal_stats = wal_manager.get_stats()
+
+        # Get dirty entries info (without exposing data)
+        dirty_entries = cache_manager.get_dirty_entries()
+        dirty_summary = []
+        for entry in dirty_entries:
+            dirty_summary.append({
+                "cache_type": entry.get('cache_type'),
+                "identifier": entry.get('identifier', '(no identifier)'),
+                "timestamp": entry.get('timestamp'),
+                "dirty": entry.get('dirty')
+            })
+
+        # Get cache keys by type (for debugging)
+        cache_keys_by_type = {}
+        for cache_type in ["users", "duels", "bounties", "daily_problem", "daily_completions", "user_daily_data"]:
+            keys = [k for k in cache_manager._cache.keys() if k.startswith(f"{cache_type}:")]
+            cache_keys_by_type[cache_type] = {
+                "count": len(keys),
+                "sample_keys": keys[:5]  # Only show first 5 for privacy
+            }
+
+        return {
+            "success": True,
+            "data": {
+                "cache": cache_stats,
+                "wal": wal_stats,
+                "dirty_entries": {
+                    "count": len(dirty_entries),
+                    "entries": dirty_summary[:10]  # Only show first 10
+                },
+                "cache_keys_by_type": cache_keys_by_type,
+                "timestamp": datetime.utcnow().isoformat()
+            }
+        }
+    except Exception as error:
+        return {"success": False, "error": str(error)}
+
+
+@router.get("/cache/dump")
+async def trigger_cache_dump(
+    api_key: str = Depends(verify_api_key_query)
+):
+    """Manually trigger cache dump to DynamoDB
+
+    Access via: /admin/cache/dump?api_key=YOUR_API_KEY
+    """
+    try:
+        from cache_dumper import dump_cache_to_db
+        result = await dump_cache_to_db()
+        return {"success": True, "data": result}
+    except Exception as error:
+        return {"success": False, "error": str(error)}
diff --git a/scripts/fastapi/routes/duels.py b/scripts/fastapi/routes/duels.py
index 501de75..93a0c19 100644
--- a/scripts/fastapi/routes/duels.py
+++ b/scripts/fastapi/routes/duels.py
@@ -190,7 +190,8 @@ async def get_duel_endpoint(
         cached_duels = cache_manager.get(CacheType.DUELS)
         if cached_duels:
             for duel in cached_duels.get('data', []):
-                if duel.get('id') == duel_id:
+                # FIXED: Use 'duelId' not 'id' to match DynamoDB schema
+                if duel.get('duelId') == duel_id:
                     return {"success": True, "data": duel}
         
         # Fallback to database

From 2b6ce841f802276f3fbd5f3f08f9b2ef9e50fd1c Mon Sep 17 00:00:00 2001
From: Akeen Karkare <akeenkarkare@gmail.com>
Date: Thu, 20 Nov 2025 18:00:55 -0500
Subject: [PATCH 07/11] Address CodeRabbit review feedback on admin endpoints
 and WAL dump
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Fixes all issues identified in CodeRabbit review:

1. Fixed dirty entry field reference (admin.py:190)
   - Changed from entry.get('dirty') to entry.get('last_synced')
   - 'dirty' field doesn't exist in dirty_entries dict

2. Added encapsulation TODO (admin.py:195)
   - Documented that direct _cache access needs proper encapsulation
   - Will create public method in future PR

3. Changed cache dump to POST method (admin.py:221)
   - GET methods should not perform write operations
   - Dump operation modifies WAL checkpoint state

4. Removed double-wrapping (admin.py:233)
   - dump_cache_to_db() already returns {"success": ..., ...}
   - Don't wrap again - causes nested response structure

5. Added unknown operation handling (cache_dumper.py:273-279)
   - Explicitly handle unknown WAL operation types
   - Increment total_failed and log warning
   - Previously would silently count as successful

Updated CACHE_FIX_PLAN.md to document these fixes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
---
 CACHE_FIX_PLAN.md               | 18 ++++++++++++++++++
 scripts/fastapi/cache_dumper.py |  8 ++++++++
 scripts/fastapi/routes/admin.py | 11 +++++++----
 3 files changed, 33 insertions(+), 4 deletions(-)

diff --git a/CACHE_FIX_PLAN.md b/CACHE_FIX_PLAN.md
index 7a1eceb..ed31e79 100644
--- a/CACHE_FIX_PLAN.md
+++ b/CACHE_FIX_PLAN.md
@@ -282,3 +282,21 @@ The `dump_cache_to_db()` function in `cache_dumper.py` now:
      - Reused by both `convert_to_dynamodb_format()` and UPDATE operations
    - Impact: Proper type preservation for all DynamoDB updates
 
+**Admin Endpoints and CodeRabbit Fixes (Commit 4-5):**
+
+4. **Cache Debug Endpoints** (`routes/admin.py`):
+   - Added GET `/admin/cache/status?api_key=...` endpoint returning:
+     - Cache stats (size, entries per type, hit rate)
+     - WAL stats (entries, checkpoint, file size)
+     - Dirty entries count and summary (without sensitive data)
+     - Cache keys by type for debugging
+   - Added POST `/admin/cache/dump?api_key=...` endpoint for manual cache sync
+   - Fixed duel ID mismatch in `routes/duels.py`: `duel.get('id')` → `duel.get('duelId')`
+
+5. **CodeRabbit Review Fixes** (`routes/admin.py`, `cache_dumper.py`):
+   - Fixed dirty entry field: `entry.get('dirty')` → `entry.get('last_synced')`
+   - Added TODO comment for `_cache` encapsulation issue
+   - Changed `/cache/dump` from GET to POST (write operations should use POST)
+   - Removed double-wrapping of dump response
+   - Added explicit handling for unknown WAL operation types (prevents silent failures)
+
diff --git a/scripts/fastapi/cache_dumper.py b/scripts/fastapi/cache_dumper.py
index 86f4160..8efbb9f 100644
--- a/scripts/fastapi/cache_dumper.py
+++ b/scripts/fastapi/cache_dumper.py
@@ -270,6 +270,14 @@ async def dump_cache_to_db() -> Dict:
                         ExpressionAttributeValues=expr_attr_values
                     )
 
+                else:
+                    # Unknown operation type - fail explicitly
+                    total_failed += 1
+                    error_msg = f"Unknown WAL operation type '{operation}' for table {table}, key {key}"
+                    warning(error_msg)
+                    errors.append(error_msg)
+                    continue
+
                 total_synced += 1
 
                 # Update checkpoint after successful operation to avoid replay
diff --git a/scripts/fastapi/routes/admin.py b/scripts/fastapi/routes/admin.py
index 1080770..0416c11 100644
--- a/scripts/fastapi/routes/admin.py
+++ b/scripts/fastapi/routes/admin.py
@@ -187,10 +187,12 @@ async def get_cache_status(
                 "cache_type": entry.get('cache_type'),
                 "identifier": entry.get('identifier', '(no identifier)'),
                 "timestamp": entry.get('timestamp'),
-                "dirty": entry.get('dirty')
+                "last_synced": entry.get('last_synced')
             })
 
         # Get cache keys by type (for debugging)
+        # Note: Accessing _cache directly for admin debugging only
+        # TODO: Add public method to CacheManager for proper encapsulation
         cache_keys_by_type = {}
         for cache_type in ["users", "duels", "bounties", "daily_problem", "daily_completions", "user_daily_data"]:
             keys = [k for k in cache_manager._cache.keys() if k.startswith(f"{cache_type}:")]
@@ -216,17 +218,18 @@ async def get_cache_status(
         return {"success": False, "error": str(error)}
 
 
-@router.get("/cache/dump")
+@router.post("/cache/dump")
 async def trigger_cache_dump(
     api_key: str = Depends(verify_api_key_query)
 ):
     """Manually trigger cache dump to DynamoDB
 
-    Access via: /admin/cache/dump?api_key=YOUR_API_KEY
+    Access via: POST /admin/cache/dump?api_key=YOUR_API_KEY
     """
     try:
         from cache_dumper import dump_cache_to_db
         result = await dump_cache_to_db()
-        return {"success": True, "data": result}
+        # dump_cache_to_db already returns {"success": ..., ...}, don't double-wrap
+        return result
     except Exception as error:
         return {"success": False, "error": str(error)}

From 8abe72be9bd44cfbd56ab00e63476ad751007ce2 Mon Sep 17 00:00:00 2001
From: Akeen Karkare <akeenkarkare@gmail.com>
Date: Thu, 20 Nov 2025 18:15:59 -0500
Subject: [PATCH 08/11] Fix critical checkpoint skip and race condition bugs in
 WAL dump
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Addresses CodeRabbit's critical findings on data loss scenarios:

1. Checkpoint Skip Bug (cache_dumper.py)
   Problem: Failed WAL entries permanently lost if later entries succeed
   - Example: seq 1 ✅ checkpoint=1, seq 2 ❌, seq 3 ✅ checkpoint=3
   - Next run starts from seq 4, so seq 2 never retried

   Fix: Track first_failed_sequence throughout processing
   - Only update checkpoint if NO failures occurred
   - Ensures checkpoint never advances past failed entry
   - Failed entries retried on next dump cycle

2. WAL Race Condition (cache_dumper.py, wal_manager.py)
   Problem: Concurrent writes lost during dump
   - Dump snapshots WAL at start
   - New writes append after snapshot
   - Old clear() deleted ALL entries including new ones
   - New writes permanently lost

   Fix: Added wal_manager.clear_up_to(max_sequence)
   - Only clears WAL up to last applied sequence
   - Keeps entries with seq > max_sequence (concurrent writes)
   - Next dump picks them up via get_entries_since()

   Fix: Removed mark_synced() calls from dump logic
   - Cache dirty flags now advisory only (monitoring)
   - WAL checkpoint is source of truth for persistence

3. Additional Safety
   - Deprecated wal_manager.clear() with warning comment
   - Added checkpoint value to dump response for debugging
   - Added extensive comments explaining race prevention

Impact: Prevents silent data loss under both failure and concurrent write scenarios

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
---
 CACHE_FIX_PLAN.md               | 28 ++++++++++++++++++++
 scripts/fastapi/cache_dumper.py | 47 +++++++++++++++++++++++++--------
 scripts/fastapi/wal_manager.py  | 47 +++++++++++++++++++++++++++++++++
 3 files changed, 111 insertions(+), 11 deletions(-)

diff --git a/CACHE_FIX_PLAN.md b/CACHE_FIX_PLAN.md
index ed31e79..d98d3fd 100644
--- a/CACHE_FIX_PLAN.md
+++ b/CACHE_FIX_PLAN.md
@@ -300,3 +300,31 @@ The `dump_cache_to_db()` function in `cache_dumper.py` now:
    - Removed double-wrapping of dump response
    - Added explicit handling for unknown WAL operation types (prevents silent failures)
 
+**Critical WAL Checkpoint and Race Condition Fixes (Commit 6):**
+
+6. **Checkpoint Skip Bug** (`cache_dumper.py`):
+   - Problem: Failed WAL entries could be skipped forever if later entries succeeded
+     - Example: seq 1 succeeds → checkpoint = 1, seq 2 fails, seq 3 succeeds → checkpoint = 3
+     - Next run starts from seq 4, so seq 2 is never retried (silent data loss)
+   - Impact: Permanent data loss for any operation that fails but is followed by successes
+   - Fix: Track `first_failed_sequence` throughout processing
+     - Only update checkpoint if NO failures have occurred (`first_failed_sequence is None`)
+     - This ensures checkpoint never advances past a failed entry
+     - Failed entries are retried on next dump cycle
+
+7. **WAL Race Condition** (`cache_dumper.py`, `wal_manager.py`):
+   - Problem: Concurrent writes during dump could be marked synced but never applied to DynamoDB
+     - Dump snapshots WAL entries at start
+     - New writes append to WAL after snapshot
+     - Old code called `wal_manager.clear()` at end, clearing ALL entries including new ones
+     - Old code also called `mark_synced()` on ALL dirty cache entries
+     - New writes lost permanently
+   - Impact: Data loss under concurrent load (production scenario)
+   - Fix: Added `wal_manager.clear_up_to(max_sequence)` method
+     - Only clears WAL entries up to last successfully applied sequence
+     - Keeps entries with sequence > max_sequence (concurrent writes)
+     - Next dump picks them up via `get_entries_since(last_applied + 1)`
+   - Fix: Removed `mark_synced()` calls from dump logic
+     - Cache dirty flags now purely advisory (for monitoring)
+     - WAL checkpoint is source of truth for persistence
+
diff --git a/scripts/fastapi/cache_dumper.py b/scripts/fastapi/cache_dumper.py
index 8efbb9f..5b9b3f9 100644
--- a/scripts/fastapi/cache_dumper.py
+++ b/scripts/fastapi/cache_dumper.py
@@ -168,6 +168,7 @@ async def dump_cache_to_db() -> Dict:
         total_synced = 0
         total_failed = 0
         errors = []
+        first_failed_sequence = None  # Track first failure to stop checkpoint advancement
 
         # Process each WAL operation
         for entry in wal_entries:
@@ -176,6 +177,9 @@ async def dump_cache_to_db() -> Dict:
             key = entry.get('key')
             data = entry.get('data')
 
+            # Track sequence for checkpoint management
+            entry_sequence = entry.get('sequence', -1)
+
             # Validate based on operation type
             # DELETE operations don't require data, all others do
             if operation == "DELETE":
@@ -184,6 +188,9 @@ async def dump_cache_to_db() -> Dict:
                     error_msg = f"Skipping incomplete DELETE entry (missing operation/table/key): {entry}"
                     warning(error_msg)
                     errors.append(error_msg)
+                    # Track first failed sequence to prevent checkpoint skip
+                    if first_failed_sequence is None:
+                        first_failed_sequence = entry_sequence
                     continue
             else:
                 if not all([operation, table, key, data]):
@@ -191,6 +198,9 @@ async def dump_cache_to_db() -> Dict:
                     error_msg = f"Skipping incomplete {operation} entry (missing required fields): {entry}"
                     warning(error_msg)
                     errors.append(error_msg)
+                    # Track first failed sequence to prevent checkpoint skip
+                    if first_failed_sequence is None:
+                        first_failed_sequence = entry_sequence
                     continue
 
             try:
@@ -276,13 +286,16 @@ async def dump_cache_to_db() -> Dict:
                     error_msg = f"Unknown WAL operation type '{operation}' for table {table}, key {key}"
                     warning(error_msg)
                     errors.append(error_msg)
+                    # Track first failed sequence to prevent checkpoint skip
+                    if first_failed_sequence is None:
+                        first_failed_sequence = entry_sequence
                     continue
 
                 total_synced += 1
 
-                # Update checkpoint after successful operation to avoid replay
-                entry_sequence = entry.get('sequence', -1)
-                if entry_sequence >= 0:
+                # Update checkpoint after successful operation ONLY if no prior failures
+                # This prevents checkpoint from skipping failed entries
+                if entry_sequence >= 0 and first_failed_sequence is None:
                     wal_manager.set_last_applied_sequence(entry_sequence)
 
             except Exception as e:
@@ -290,25 +303,37 @@ async def dump_cache_to_db() -> Dict:
                 error_msg = f"Failed to sync WAL entry to {table}: {e}"
                 error(error_msg)
                 errors.append(error_msg)
+                # Track first failed sequence to prevent checkpoint skip
+                if first_failed_sequence is None:
+                    first_failed_sequence = entry_sequence
                 # Don't update checkpoint on failure - will retry this entry next time
                 continue
 
         # Mark success if all synced
         if total_failed == 0:
-            # Mark all cache entries as synced
-            dirty_entries = cache_manager.get_dirty_entries()
-            for entry in dirty_entries:
-                cache_manager.mark_synced(entry['cache_type'], entry.get('identifier', ''))
+            # CRITICAL FIX: Clear WAL entries ONLY up to the last successfully applied sequence
+            # This prevents race condition where new writes after our snapshot are lost
+
+            # Get the highest sequence we successfully applied
+            last_synced_sequence = wal_manager.get_last_applied_sequence()
+
+            # Clear WAL entries up to last_synced_sequence, keeping any concurrent writes
+            # that occurred after our snapshot (sequence > last_synced_sequence)
+            wal_manager.clear_up_to(last_synced_sequence)
 
-            # Clear WAL file after successful sync
-            wal_manager.clear()
+            # Note: We intentionally DON'T call cache_manager.mark_synced() here because:
+            # - Cache entries don't track their WAL sequence number
+            # - We can't safely determine which cache entries map to synced WAL entries
+            # - Dirty flags in cache are eventually consistent (background task marks them synced)
+            # - The WAL checkpoint is our source of truth for what's been persisted
 
-            info(f"✅ Cache dump complete: {total_synced} WAL operations synced to DynamoDB")
+            info(f"✅ Cache dump complete: {total_synced} WAL operations synced to DynamoDB (checkpoint: {last_synced_sequence})")
 
             return {
                 "success": True,
                 "entries": total_synced,
-                "failed": 0
+                "failed": 0,
+                "checkpoint": last_synced_sequence
             }
         else:
             warning(f"⚠️ Cache dump partially failed: {total_failed}/{len(wal_entries)} operations failed")
diff --git a/scripts/fastapi/wal_manager.py b/scripts/fastapi/wal_manager.py
index 7b52c6d..2c0021b 100644
--- a/scripts/fastapi/wal_manager.py
+++ b/scripts/fastapi/wal_manager.py
@@ -176,6 +176,9 @@ def clear(self) -> bool:
         """
         Clear the WAL file after successful cache dump
 
+        WARNING: This resets sequence to 0, breaking checkpoint-based replay!
+        Use clear_up_to(sequence) instead for production code.
+
         Returns:
             True if successful, False otherwise
         """
@@ -193,6 +196,50 @@ def clear(self) -> bool:
                 error(f"Failed to clear WAL: {e}")
                 return False
 
+    def clear_up_to(self, max_sequence: int) -> bool:
+        """
+        Clear WAL entries up to and including max_sequence, keeping later entries
+
+        This is the correct way to clear WAL after partial sync without breaking
+        the checkpoint system or losing concurrent writes.
+
+        Args:
+            max_sequence: Clear all entries with sequence <= this value
+
+        Returns:
+            True if successful, False otherwise
+        """
+        with self._lock:
+            try:
+                # Read all entries
+                entries_to_keep = []
+                with open(self._wal_file, 'r') as f:
+                    for line in f:
+                        line = line.strip()
+                        if not line:
+                            continue
+                        try:
+                            entry = json.loads(line)
+                            # Keep entries with sequence > max_sequence
+                            if entry.get('sequence', 0) > max_sequence:
+                                entries_to_keep.append(entry)
+                        except json.JSONDecodeError:
+                            continue
+
+                # Rewrite WAL file with only entries to keep
+                with open(self._wal_file, 'w') as f:
+                    for entry in entries_to_keep:
+                        f.write(json.dumps(entry) + '\n')
+                    f.flush()
+                    os.fsync(f.fileno())
+
+                info(f"🧹 WAL file cleared up to sequence {max_sequence}, kept {len(entries_to_keep)} entries")
+                return True
+
+            except Exception as e:
+                error(f"Failed to clear WAL up to sequence {max_sequence}: {e}")
+                return False
+
     def get_entries_since(self, sequence: int) -> List[Dict]:
         """
         Get all WAL entries since a specific sequence number

From 5fd690e7730d037fafcee0804c3ed0485057cb7d Mon Sep 17 00:00:00 2001
From: Akeen Karkare <akeenkarkare@gmail.com>
Date: Thu, 20 Nov 2025 18:32:39 -0500
Subject: [PATCH 09/11] Fix INCREMENT double-apply bug by stopping on first
 failure
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Critical fix for CodeRabbit finding on checkpoint advancement:

Problem: Previous fix tracked first_failed_sequence but continued
processing later entries. This caused double-apply of INCREMENTs:

Example scenario:
- Seq 1: ✅ UPDATE → checkpoint = 1
- Seq 2: ❌ validation failure → first_failed_sequence = 2
- Seq 3: ✅ INCREMENT xp +100 → total_synced++, checkpoint stays 1
- Next run: starts from seq 2
  - Seq 2: fails again
  - Seq 3: INCREMENT +100 REPLAYED → user gets +200 XP total!

Root cause: Checkpoint stayed at 1, but seq 3 was already applied.
Next replay from checkpoint+1 re-applied seq 3.

Fix: STOP processing on first failure (break loop)
- Process entries in order until first failure
- Update checkpoint after EACH successful operation
- On any failure: break immediately
- Leave failed entry + all later entries for next run
- Next run starts from failed sequence, processes in order

Guarantees:
✅ No checkpoint skips (failed entries always retried)
✅ No double-applies (later entries not processed until earlier succeeds)
✅ Contiguous checkpoint (always points to last successfully applied)
✅ Idempotency for INCREMENT operations preserved

Changes:
- Removed first_failed_sequence tracking (no longer needed)
- Changed all validation/error continues to breaks
- Update checkpoint after every success (safe with break-on-failure)
- Improved error messages with sequence numbers

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
---
 CACHE_FIX_PLAN.md               | 21 +++++++++------
 scripts/fastapi/cache_dumper.py | 45 ++++++++++++++-------------------
 2 files changed, 32 insertions(+), 34 deletions(-)

diff --git a/CACHE_FIX_PLAN.md b/CACHE_FIX_PLAN.md
index d98d3fd..bceadda 100644
--- a/CACHE_FIX_PLAN.md
+++ b/CACHE_FIX_PLAN.md
@@ -302,15 +302,20 @@ The `dump_cache_to_db()` function in `cache_dumper.py` now:
 
 **Critical WAL Checkpoint and Race Condition Fixes (Commit 6):**
 
-6. **Checkpoint Skip Bug** (`cache_dumper.py`):
-   - Problem: Failed WAL entries could be skipped forever if later entries succeeded
-     - Example: seq 1 succeeds → checkpoint = 1, seq 2 fails, seq 3 succeeds → checkpoint = 3
+6. **Checkpoint Skip and Double-Apply Bug** (`cache_dumper.py`):
+   - Problem 1: Failed WAL entries could be skipped forever if later entries succeeded
+     - Example: seq 1 ✅ → checkpoint = 1, seq 2 ❌, seq 3 ✅ → checkpoint = 3
      - Next run starts from seq 4, so seq 2 is never retried (silent data loss)
-   - Impact: Permanent data loss for any operation that fails but is followed by successes
-   - Fix: Track `first_failed_sequence` throughout processing
-     - Only update checkpoint if NO failures have occurred (`first_failed_sequence is None`)
-     - This ensures checkpoint never advances past a failed entry
-     - Failed entries are retried on next dump cycle
+   - Problem 2: Later successful entries (especially INCREMENTs) could be double-applied
+     - Example: seq 1 ✅ → checkpoint = 1, seq 2 ❌, seq 3 INCREMENT ✅ (but checkpoint still 1)
+     - Next run: seq 2 fails again, seq 3 replayed → INCREMENT applied twice!
+   - Impact: Both permanent data loss AND data corruption (double XP/streak)
+   - Fix: STOP processing on first failure (break loop)
+     - Process entries in order until first failure
+     - Update checkpoint after each successful operation
+     - On failure: break loop, leave failed entry and all later entries for next run
+     - Next run starts from failed sequence, retries in order
+     - Guarantees: No skips, no double-applies, contiguous checkpoint
 
 7. **WAL Race Condition** (`cache_dumper.py`, `wal_manager.py`):
    - Problem: Concurrent writes during dump could be marked synced but never applied to DynamoDB
diff --git a/scripts/fastapi/cache_dumper.py b/scripts/fastapi/cache_dumper.py
index 5b9b3f9..e8012bf 100644
--- a/scripts/fastapi/cache_dumper.py
+++ b/scripts/fastapi/cache_dumper.py
@@ -168,9 +168,11 @@ async def dump_cache_to_db() -> Dict:
         total_synced = 0
         total_failed = 0
         errors = []
-        first_failed_sequence = None  # Track first failure to stop checkpoint advancement
 
-        # Process each WAL operation
+        # Process each WAL operation until first failure
+        # CRITICAL: We must stop on first failure to prevent double-applying later INCREMENTs
+        # Example: seq 1 ✅, seq 2 ❌, seq 3 INCREMENT ✅ (checkpoint=1)
+        # Next run: seq 2 fails again, seq 3 replayed → INCREMENT applied twice!
         for entry in wal_entries:
             operation = entry.get('operation')
             table = entry.get('table')
@@ -185,23 +187,19 @@ async def dump_cache_to_db() -> Dict:
             if operation == "DELETE":
                 if not all([operation, table, key]):
                     total_failed += 1
-                    error_msg = f"Skipping incomplete DELETE entry (missing operation/table/key): {entry}"
+                    error_msg = f"Incomplete DELETE entry at sequence {entry_sequence}: {entry}"
                     warning(error_msg)
                     errors.append(error_msg)
-                    # Track first failed sequence to prevent checkpoint skip
-                    if first_failed_sequence is None:
-                        first_failed_sequence = entry_sequence
-                    continue
+                    # STOP processing to prevent replaying later entries (especially INCREMENTs)
+                    break
             else:
                 if not all([operation, table, key, data]):
                     total_failed += 1
-                    error_msg = f"Skipping incomplete {operation} entry (missing required fields): {entry}"
+                    error_msg = f"Incomplete {operation} entry at sequence {entry_sequence}: {entry}"
                     warning(error_msg)
                     errors.append(error_msg)
-                    # Track first failed sequence to prevent checkpoint skip
-                    if first_failed_sequence is None:
-                        first_failed_sequence = entry_sequence
-                    continue
+                    # STOP processing to prevent replaying later entries (especially INCREMENTs)
+                    break
 
             try:
                 if operation == "UPDATE":
@@ -283,31 +281,26 @@ async def dump_cache_to_db() -> Dict:
                 else:
                     # Unknown operation type - fail explicitly
                     total_failed += 1
-                    error_msg = f"Unknown WAL operation type '{operation}' for table {table}, key {key}"
+                    error_msg = f"Unknown WAL operation type '{operation}' at sequence {entry_sequence}"
                     warning(error_msg)
                     errors.append(error_msg)
-                    # Track first failed sequence to prevent checkpoint skip
-                    if first_failed_sequence is None:
-                        first_failed_sequence = entry_sequence
-                    continue
+                    # STOP processing to prevent replaying later entries (especially INCREMENTs)
+                    break
 
                 total_synced += 1
 
-                # Update checkpoint after successful operation ONLY if no prior failures
-                # This prevents checkpoint from skipping failed entries
-                if entry_sequence >= 0 and first_failed_sequence is None:
+                # Update checkpoint after each successful operation
+                # This is safe because we STOP on first failure (break above)
+                if entry_sequence >= 0:
                     wal_manager.set_last_applied_sequence(entry_sequence)
 
             except Exception as e:
                 total_failed += 1
-                error_msg = f"Failed to sync WAL entry to {table}: {e}"
+                error_msg = f"Failed to sync WAL entry at sequence {entry_sequence} to {table}: {e}"
                 error(error_msg)
                 errors.append(error_msg)
-                # Track first failed sequence to prevent checkpoint skip
-                if first_failed_sequence is None:
-                    first_failed_sequence = entry_sequence
-                # Don't update checkpoint on failure - will retry this entry next time
-                continue
+                # STOP processing to prevent replaying later entries (especially INCREMENTs)
+                break
 
         # Mark success if all synced
         if total_failed == 0:

From 1cb2a125aaaf7bdb08a194ea53ea3ddaf446f1d8 Mon Sep 17 00:00:00 2001
From: Akeen Karkare <akeenkarkare@gmail.com>
Date: Thu, 20 Nov 2025 18:44:55 -0500
Subject: [PATCH 10/11] Add cache viewer tab to log viewer HTML
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Implements the user's request to view cache status alongside logs in the admin panel.

Features:
- Tab-based UI: Toggle between "📜 Logs" and "💾 Cache Status" views
- Cache Statistics: Total entries, hit rate, users, duels, bounties, daily problems
- WAL Status: WAL entries, file size, last applied sequence, file path
- Dirty Entries: Shows uncommitted cache changes with timestamps
- Cache Keys by Type: Sample keys for each cache type with counts
- Actions:
  - 🔄 Refresh Cache Status: Reload cache data from server
  - 💾 Trigger Cache Dump: Manually sync dirty entries to DynamoDB

Integration:
- Fetches from GET /admin/cache/status?api_key=...
- Triggers POST /admin/cache/dump?api_key=...
- Uses same API key from URL as log viewer
- Responsive grid layout with VS Code dark theme styling

User experience:
- Auto-hides log controls when viewing cache tab
- Visual indicators for dirty entries (red border)
- Success/error alerts for cache dump operations
- Real-time refresh after manual dump

Access: http://209.182.234.33:6969/admin/logs?api_key=YOUR_API_KEY

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
---
 scripts/fastapi/static/log_viewer.html | 348 ++++++++++++++++++++++++-
 1 file changed, 347 insertions(+), 1 deletion(-)

diff --git a/scripts/fastapi/static/log_viewer.html b/scripts/fastapi/static/log_viewer.html
index 7c9450f..9118d7c 100644
--- a/scripts/fastapi/static/log_viewer.html
+++ b/scripts/fastapi/static/log_viewer.html
@@ -320,6 +320,130 @@
         .filter-info strong {
             color: #4ec9b0;
         }
+
+        .tab-button {
+            background: #3e3e42;
+            color: #d4d4d4;
+            border: none;
+            padding: 12px 20px;
+            border-radius: 6px;
+            cursor: pointer;
+            font-family: inherit;
+            font-size: 14px;
+            font-weight: 600;
+            transition: all 0.2s;
+        }
+
+        .tab-button:hover {
+            background: #4e4e52;
+        }
+
+        .tab-button.active {
+            background: #0e639c;
+            color: white;
+        }
+
+        #cacheViewer {
+            display: none;
+        }
+
+        #cacheViewer.active {
+            display: block;
+        }
+
+        #logViewer.active {
+            display: block;
+        }
+
+        #logViewer {
+            display: none;
+        }
+
+        .cache-section {
+            background: #252526;
+            padding: 20px;
+            border-radius: 8px;
+            margin-bottom: 20px;
+            border: 1px solid #3e3e42;
+        }
+
+        .cache-section h2 {
+            color: #4ec9b0;
+            font-size: 18px;
+            margin-bottom: 15px;
+            display: flex;
+            align-items: center;
+            gap: 10px;
+        }
+
+        .cache-grid {
+            display: grid;
+            grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
+            gap: 15px;
+        }
+
+        .cache-item {
+            background: #1e1e1e;
+            padding: 15px;
+            border-radius: 6px;
+            border: 1px solid #3e3e42;
+        }
+
+        .cache-item-label {
+            color: #858585;
+            font-size: 11px;
+            text-transform: uppercase;
+            margin-bottom: 8px;
+        }
+
+        .cache-item-value {
+            color: #d4d4d4;
+            font-size: 16px;
+            font-weight: 600;
+        }
+
+        .cache-item-value.highlight {
+            color: #4ec9b0;
+            font-size: 24px;
+        }
+
+        .cache-keys-list {
+            max-height: 300px;
+            overflow-y: auto;
+            background: #1e1e1e;
+            padding: 10px;
+            border-radius: 4px;
+            border: 1px solid #3e3e42;
+        }
+
+        .cache-key {
+            padding: 6px 10px;
+            margin-bottom: 4px;
+            background: #2d2d30;
+            border-radius: 3px;
+            font-size: 12px;
+            font-family: 'SF Mono', monospace;
+            color: #9cdcfe;
+        }
+
+        .dirty-entry {
+            background: #3e2723;
+            border-left: 3px solid #f48771;
+            padding: 10px;
+            margin-bottom: 8px;
+            border-radius: 4px;
+        }
+
+        .dirty-entry-header {
+            color: #f48771;
+            font-weight: 600;
+            margin-bottom: 5px;
+        }
+
+        .dirty-entry-info {
+            color: #858585;
+            font-size: 11px;
+        }
     </style>
 </head>
 <body>
@@ -327,7 +451,12 @@
         <header>
             <h1>📊 YeetCode FastAPI Log Viewer</h1>
 
-            <div class="controls">
+            <div class="tab-buttons" style="margin-bottom: 15px; display: flex; gap: 10px;">
+                <button id="logsTab" class="tab-button active" style="flex: 1;">📜 Logs</button>
+                <button id="cacheTab" class="tab-button" style="flex: 1;">💾 Cache Status</button>
+            </div>
+
+            <div class="controls" id="logControls">
                 <div class="control-group">
                     <label>Search / Filter</label>
                     <input type="text" id="searchInput" placeholder="Search logs...">
@@ -410,6 +539,8 @@ <h1>📊 YeetCode FastAPI Log Viewer</h1>
             </div>
         </header>
 
+        <!-- Log Viewer Section -->
+        <div id="logViewer" class="active">
         <div class="stats" id="stats" style="display: none;">
             <div class="stat-item">
                 <span class="stat-label">Total Lines</span>
@@ -452,6 +583,51 @@ <h1>📊 YeetCode FastAPI Log Viewer</h1>
                 ⏳ Loading logs from server...
             </div>
         </div>
+        </div>
+        <!-- End Log Viewer -->
+
+        <!-- Cache Viewer Section -->
+        <div id="cacheViewer">
+            <div class="cache-section">
+                <h2>📊 Cache Statistics</h2>
+                <div class="cache-grid" id="cacheStatsGrid">
+                    <div class="cache-item">
+                        <div class="cache-item-label">Loading...</div>
+                        <div class="cache-item-value">⏳</div>
+                    </div>
+                </div>
+            </div>
+
+            <div class="cache-section">
+                <h2>🗄️ WAL Status</h2>
+                <div class="cache-grid" id="walStatsGrid">
+                    <div class="cache-item">
+                        <div class="cache-item-label">Loading...</div>
+                        <div class="cache-item-value">⏳</div>
+                    </div>
+                </div>
+            </div>
+
+            <div class="cache-section">
+                <h2>⚠️ Dirty Entries</h2>
+                <div id="dirtyEntriesContainer">
+                    <p style="color: #858585;">Loading...</p>
+                </div>
+            </div>
+
+            <div class="cache-section">
+                <h2>🔑 Cache Keys by Type</h2>
+                <div id="cacheKeysContainer">
+                    <p style="color: #858585;">Loading...</p>
+                </div>
+            </div>
+
+            <div class="button-group">
+                <button id="refreshCacheBtn">🔄 Refresh Cache Status</button>
+                <button id="dumpCacheBtn" class="secondary">💾 Trigger Cache Dump</button>
+            </div>
+        </div>
+        <!-- End Cache Viewer -->
     </div>
 
     <script>
@@ -933,6 +1109,176 @@ <h1>📊 YeetCode FastAPI Log Viewer</h1>
             reverseSortBtn.textContent = isReversed ? '⬇️ Latest First' : '⬆️ Oldest First';
             filterLogs();
         });
+
+        // ===== Cache Viewer Functionality =====
+
+        // Tab switching
+        const logsTab = document.getElementById('logsTab');
+        const cacheTab = document.getElementById('cacheTab');
+        const logViewer = document.getElementById('logViewer');
+        const cacheViewer = document.getElementById('cacheViewer');
+        const logControls = document.getElementById('logControls');
+
+        logsTab.addEventListener('click', () => {
+            logsTab.classList.add('active');
+            cacheTab.classList.remove('active');
+            logViewer.classList.add('active');
+            cacheViewer.classList.remove('active');
+            logControls.style.display = 'grid';
+        });
+
+        cacheTab.addEventListener('click', () => {
+            cacheTab.classList.add('active');
+            logsTab.classList.remove('active');
+            cacheViewer.classList.add('active');
+            logViewer.classList.remove('active');
+            logControls.style.display = 'none';
+            loadCacheStatus();
+        });
+
+        // Load cache status from API
+        async function loadCacheStatus() {
+            try {
+                const response = await fetch(`/admin/cache/status?api_key=${apiKey}`);
+                const data = await response.json();
+
+                if (!data.success) {
+                    throw new Error(data.error || 'Failed to load cache status');
+                }
+
+                renderCacheStats(data.data);
+            } catch (error) {
+                console.error('Failed to load cache status:', error);
+                document.getElementById('cacheStatsGrid').innerHTML =
+                    `<div class="cache-item"><div class="cache-item-label">Error</div><div class="cache-item-value" style="color: #f48771;">${error.message}</div></div>`;
+            }
+        }
+
+        // Render cache statistics
+        function renderCacheStats(data) {
+            const cache = data.cache;
+            const wal = data.wal;
+            const dirtyEntries = data.dirty_entries;
+            const cacheKeys = data.cache_keys_by_type;
+
+            // Cache Stats
+            const cacheStatsHTML = `
+                <div class="cache-item">
+                    <div class="cache-item-label">Total Entries</div>
+                    <div class="cache-item-value highlight">${cache.total_entries || 0}</div>
+                </div>
+                <div class="cache-item">
+                    <div class="cache-item-label">Hit Rate</div>
+                    <div class="cache-item-value">${cache.hit_rate ? (cache.hit_rate * 100).toFixed(1) + '%' : 'N/A'}</div>
+                </div>
+                <div class="cache-item">
+                    <div class="cache-item-label">Users</div>
+                    <div class="cache-item-value">${cache.users || 0}</div>
+                </div>
+                <div class="cache-item">
+                    <div class="cache-item-label">Duels</div>
+                    <div class="cache-item-value">${cache.duels || 0}</div>
+                </div>
+                <div class="cache-item">
+                    <div class="cache-item-label">Bounties</div>
+                    <div class="cache-item-value">${cache.bounties || 0}</div>
+                </div>
+                <div class="cache-item">
+                    <div class="cache-item-label">Daily Problems</div>
+                    <div class="cache-item-value">${cache.daily_problem || 0}</div>
+                </div>
+            `;
+            document.getElementById('cacheStatsGrid').innerHTML = cacheStatsHTML;
+
+            // WAL Stats
+            const walStatsHTML = `
+                <div class="cache-item">
+                    <div class="cache-item-label">WAL Entries</div>
+                    <div class="cache-item-value highlight">${wal.entries || 0}</div>
+                </div>
+                <div class="cache-item">
+                    <div class="cache-item-label">File Size</div>
+                    <div class="cache-item-value">${wal.size_kb ? wal.size_kb.toFixed(2) + ' KB' : 'N/A'}</div>
+                </div>
+                <div class="cache-item">
+                    <div class="cache-item-label">Last Applied Sequence</div>
+                    <div class="cache-item-value">${wal.last_applied_sequence ?? 'N/A'}</div>
+                </div>
+                <div class="cache-item">
+                    <div class="cache-item-label">WAL Path</div>
+                    <div class="cache-item-value" style="font-size: 11px; word-break: break-all;">${wal.path || 'N/A'}</div>
+                </div>
+            `;
+            document.getElementById('walStatsGrid').innerHTML = walStatsHTML;
+
+            // Dirty Entries
+            if (dirtyEntries.count === 0) {
+                document.getElementById('dirtyEntriesContainer').innerHTML =
+                    '<p style="color: #4ec9b0;">✅ No dirty entries - all changes synced to DynamoDB</p>';
+            } else {
+                const dirtyHTML = dirtyEntries.entries.map(entry => `
+                    <div class="dirty-entry">
+                        <div class="dirty-entry-header">${entry.cache_type}: ${entry.identifier || '(no identifier)'}</div>
+                        <div class="dirty-entry-info">
+                            Timestamp: ${new Date(entry.timestamp * 1000).toLocaleString()} |
+                            Last Synced: ${entry.last_synced ? new Date(entry.last_synced * 1000).toLocaleString() : 'Never'}
+                        </div>
+                    </div>
+                `).join('');
+                document.getElementById('dirtyEntriesContainer').innerHTML =
+                    `<p style="color: #dcdcaa; margin-bottom: 10px;">⚠️ ${dirtyEntries.count} dirty entries (showing first 10)</p>` + dirtyHTML;
+            }
+
+            // Cache Keys by Type
+            let keysHTML = '';
+            for (const [type, info] of Object.entries(cacheKeys)) {
+                keysHTML += `
+                    <div class="cache-section" style="margin-top: 10px;">
+                        <h3 style="color: #9cdcfe; font-size: 14px; margin-bottom: 10px;">${type} (${info.count} entries)</h3>
+                        <div class="cache-keys-list">
+                            ${info.sample_keys.length > 0
+                                ? info.sample_keys.map(key => `<div class="cache-key">${key}</div>`).join('')
+                                : '<p style="color: #858585;">No keys</p>'}
+                            ${info.count > 5 ? `<p style="color: #858585; margin-top: 10px; font-size: 11px;">... and ${info.count - 5} more</p>` : ''}
+                        </div>
+                    </div>
+                `;
+            }
+            document.getElementById('cacheKeysContainer').innerHTML = keysHTML;
+        }
+
+        // Refresh cache button
+        document.getElementById('refreshCacheBtn').addEventListener('click', loadCacheStatus);
+
+        // Dump cache button
+        document.getElementById('dumpCacheBtn').addEventListener('click', async () => {
+            if (!confirm('Trigger manual cache dump to DynamoDB? This will sync all dirty entries.')) {
+                return;
+            }
+
+            const btn = document.getElementById('dumpCacheBtn');
+            btn.disabled = true;
+            btn.textContent = '⏳ Dumping...';
+
+            try {
+                const response = await fetch(`/admin/cache/dump?api_key=${apiKey}`, {
+                    method: 'POST'
+                });
+                const data = await response.json();
+
+                if (data.success) {
+                    alert(`✅ Cache dump successful!\n\nSynced: ${data.entries} entries\nCheckpoint: ${data.checkpoint ?? 'N/A'}`);
+                    loadCacheStatus(); // Refresh after dump
+                } else {
+                    alert(`❌ Cache dump failed:\n\n${data.error || 'Unknown error'}\n\nSynced: ${data.synced || 0}/${data.entries || 0}`);
+                }
+            } catch (error) {
+                alert(`❌ Cache dump failed: ${error.message}`);
+            } finally {
+                btn.disabled = false;
+                btn.textContent = '💾 Trigger Cache Dump';
+            }
+        });
     </script>
 </body>
 </html>

From 3b36ff8889265f80215663ce404e134be4678026 Mon Sep 17 00:00:00 2001
From: Akeen Karkare <61316992+akeenkarkare@users.noreply.github.com>
Date: Thu, 20 Nov 2025 18:57:43 -0500
Subject: [PATCH 11/11] Delete CACHE_FIX_PLAN.md

---
 CACHE_FIX_PLAN.md | 335 ----------------------------------------------
 1 file changed, 335 deletions(-)
 delete mode 100644 CACHE_FIX_PLAN.md

diff --git a/CACHE_FIX_PLAN.md b/CACHE_FIX_PLAN.md
deleted file mode 100644
index bceadda..0000000
--- a/CACHE_FIX_PLAN.md
+++ /dev/null
@@ -1,335 +0,0 @@
-# YeetCode Backend Cache Fix Plan
-
-## Executive Summary
-The backend cache system has fundamental architectural issues causing data loss, inconsistent state, and broken functionality. Multiple bandaid fixes have been applied that treat symptoms rather than root causes.
-
-## Root Causes
-
-### 1. **Excessive Cache Invalidation Pattern**
-**Problem**: Most endpoints invalidate cache immediately after write operations
-**Impact**: Dirty data (uncommitted changes) is deleted before WAL can sync to DynamoDB
-**Evidence**:
-- All duel endpoints call `cache_manager.invalidate_all(CacheType.DUELS)` after operations
-- Daily endpoints invalidate DAILY_PROBLEM and DAILY_COMPLETIONS after completion
-- This triggers warnings: "⚠️ Invalidating X dirty cache entries - data may be lost"
-
-**Affected Endpoints**:
-- `/create-duel` - Line 74: invalidates DUELS
-- `/start-duel` - Line 112: invalidates DUELS
-- `/complete-duel` - Line 130: invalidates DUELS
-- `/reject-duel` - Line 151: invalidates DUELS
-- `/record-duel-submission` - Line 175: invalidates DUELS
-- `/complete-daily-problem` - Lines 107-108: invalidates DAILY_PROBLEM, DAILY_COMPLETIONS
-- `/submit-bounty-solution` - Lines 124-125: invalidates BOUNTIES, BOUNTY_COMPETITIONS
-
-### 2. **Inconsistent Cache-First Implementation**
-**Problem**: Mix of cache-first and invalidate-on-write patterns
-**Impact**: Unpredictable behavior, race conditions
-**Evidence**:
-- `/accept-duel` uses cache-first (PR #27 fix)
-- All other duel endpoints still use invalidate-on-write
-- Group/user operations use cache-first with comments saying "do NOT invalidate"
-
-### 3. **WAL Sync Timing Issues**
-**Problem**: WAL background task runs every 30 seconds, but cache invalidation is immediate
-**Impact**: 29-second window where dirty data can be lost
-**Evidence**:
-- `wal_manager.py` syncs every 30 seconds
-- Cache invalidation happens immediately after writes
-- No guarantee dirty data reaches DB before invalidation
-
-### 4. **Corrupted USERS Cache Structure**
-**Problem**: USERS cache has 23 entries but lookups fail
-**Impact**: All user data endpoints return null
-**Evidence**:
-- Cache stats: `"users": 23`
-- `/users/akeen_exe` returns: `{"xp": null, "easy": null, "medium": null, ...}`
-- Suggests cache_operations.py wrote malformed data structure
-
-### 5. **Streak Reset on Restart**
-**Problem**: Streak resets to 0 after server restart
-**Impact**: Users lose streak progress
-**Root Cause**:
-- USER_DAILY_DATA cache not persisted to DB (no WAL operation)
-- Only exists in memory cache with TTL
-- On restart, cache is empty, DB has no streak data
-
-### 6. **XP Discrepancy Between Leaderboards**
-**Problem**: Group leaderboard shows different XP than university leaderboard
-**Impact**: Users see inconsistent stats
-**Root Cause**:
-- Different leaderboards read from different caches
-- Cache invalidation causes cache misses at different times
-- Some leaderboards read stale DB data, others read fresh cache
-
-## Proposed Solutions
-
-### Phase 1: Stop the Bleeding (URGENT - Deploy ASAP)
-
-#### Fix 1.1: Remove ALL cache invalidations from write endpoints
-**Files to modify**:
-- `routes/duels.py` - Remove lines 74, 112, 130, 151, 175
-- `routes/daily.py` - Remove lines 107-108
-- `routes/bounties.py` - Remove lines 124-125
-
-**Rationale**: Cache-first writes update cache in-place. Invalidation destroys uncommitted changes. Let cache TTL handle expiration.
-
-#### Fix 1.2: Fix USERS cache lookup
-**Files to check**:
-- `cache_operations.py` - `update_user_in_cache()` function
-- Verify users are being added to cache with correct structure
-- Ensure writes preserve the list structure: `{"success": True, "data": [...]}`
-
-#### Fix 1.3: Persist USER_DAILY_DATA to database
-**Files to modify**:
-- `cache_operations.py` - `complete_daily_in_cache()` line 224-228
-- Change from `cache_manager.set()` to `cache_manager.write()` with WAL operation
-- This ensures streak persists across restarts
-
-### Phase 2: Architectural Fixes (Deploy within 24 hours)
-
-#### Fix 2.1: Implement immediate WAL sync for critical operations
-**Files to modify**:
-- `cache_manager.py` - Add `write_immediate()` function
-- Calls `write()` then immediately triggers WAL sync for that entry
-- Use for: daily completion, duel completion, XP awards
-
-#### Fix 2.2: Add cache warming on startup
-**Files to modify**:
-- `main.py` - On startup, load USERS table into cache
-- Prevents cache misses on first requests after restart
-- Ensures consistent data immediately
-
-#### Fix 2.3: Make cache invalidation safer
-**Files to modify**:
-- `cache_manager.py` - `invalidate()` and `invalidate_all()`
-- Dump dirty entries to DB BEFORE deleting them
-- Return error if dump fails (don't invalidate)
-- Add `force=True` parameter for admin operations only
-
-### Phase 3: Long-term Improvements (Deploy within 1 week)
-
-#### Fix 3.1: Unified leaderboard data source
-**Problem**: Multiple leaderboards read from different places
-**Solution**: Create single `/leaderboard/{type}` endpoint that:
-- Always reads from same cache
-- Falls back to DB if cache miss
-- Ensures consistency across all leaderboard views
-
-#### Fix 3.2: Add cache health monitoring
-**Files to create**:
-- `cache_health.py` - Monitor dirty entry count, age
-- Alert if dirty entries > threshold
-- Alert if WAL sync is lagging
-- Expose via `/admin/cache/health` endpoint
-
-#### Fix 3.3: Reduce WAL sync interval
-**Files to modify**:
-- `wal_manager.py` - Reduce from 30s to 5s
-- Or implement adaptive sync (sync more frequently when dirty count is high)
-
-## Implementation Priority
-
-### CRITICAL (Deploy Today):
-1. Remove cache invalidations from duel endpoints (Fix 1.1)
-2. Fix USERS cache lookup bug (Fix 1.2)
-3. Persist USER_DAILY_DATA to DB (Fix 1.3)
-
-### HIGH (Deploy Tomorrow):
-1. Immediate WAL sync for critical ops (Fix 2.1)
-2. Cache warming on startup (Fix 2.2)
-
-### MEDIUM (Deploy This Week):
-1. Safer cache invalidation (Fix 2.3)
-2. Unified leaderboard endpoint (Fix 3.1)
-3. Reduce WAL sync interval (Fix 3.3)
-
-### LOW (Deploy When Possible):
-1. Cache health monitoring (Fix 3.2)
-
-## Testing Plan
-
-### After Phase 1 Deploy:
-1. Complete a daily problem → verify streak increments → restart server → verify streak persists
-2. Create a duel → accept duel → verify both users see updated duel status
-3. Check all leaderboards → verify XP matches across all views
-4. Complete 5 duels rapidly → verify all completions recorded correctly
-5. Monitor logs for "⚠️ Invalidating dirty" warnings → should see ZERO
-
-### After Phase 2 Deploy:
-1. Restart server → verify all data immediately available (cache warming working)
-2. Complete daily → verify XP updates within 1 second (immediate WAL)
-3. Monitor cache stats → verify no dirty entries linger > 5 seconds
-
-## Rollback Plan
-
-If Phase 1 causes issues:
-1. Revert to commit before cache invalidation removal
-2. Manually dump all dirty cache to DB: `POST /cache/dump`
-3. Clear cache: `POST /cache/clear`
-4. Monitor for data loss, restore from DB backups if needed
-
-## Success Criteria
-
-- ✅ Streaks persist across server restarts
-- ✅ Duels can be created, accepted, completed without errors
-- ✅ XP is consistent across all leaderboards
-- ✅ Zero "⚠️ Invalidating dirty" warnings in logs
-- ✅ Cache hit rate > 90% for USERS, DUELS, DAILY_PROBLEM
-- ✅ WAL dirty entry count stays < 5 at all times
-
----
-
-**Created**: 2025-11-18
-**Status**: DRAFT - Awaiting approval
-**Severity**: CRITICAL - Production data loss occurring
-
-## ADDENDUM: WAL Dump Critical Bug (Discovered 2025-11-20)
-
-### Problem
-The `cache_dumper.py` is trying to write **entire cache entries** to DynamoDB, but cache entries have wrapped structures that don't match DB schemas:
-
-```python
-# What's in cache (wrapped):
-{"success": True, "data": [user1, user2, user3]}
-
-# What cache_dumper tries to write:
-convert_to_dynamodb_format({"success": True, "data": [...]})
-# Results in invalid DynamoDB item!
-```
-
-### Errors Observed:
-1. **Daily table**: `"cannot be converted to a numeric value: True"` 
-   - Cache has `users: {username: True}` (boolean)
-   - DynamoDB might expect numeric values
-
-2. **USERS table**: `"provided key element does not match the schema"`
-   - Cache has wrapped structure `{"success": ..., "data": [...]}`
-   - DynamoDB expects individual user objects with `username` key
-
-### Root Cause:
-The WAL system has TWO write mechanisms:
-1. **WAL operations** (from `cache_operations.py`) - Correctly structured, partial updates ✅
-2. **Cache dumps** (from `cache_dumper.py`) - Dumps raw cache, wrong structure ❌
-
-The cache dump should use the WAL operation log, NOT dump raw cache entries.
-
-### Impact:
-- Cache dumps fail silently
-- Dirty data doesn't reach database
-- On server crash/restart, data lost
-
-### Why Phase 1 Helps:
-By removing cache invalidations, we:
-- Reduce frequency of cache dumps (only triggered on explicit /cache/clear)
-- Reduce dirty entry count (entries sync via normal WAL)
-- Buy time to fix the dump logic properly
-
-### Phase 2 Fix Required:
-1. ✅ **COMPLETED** - Rewrite `cache_dumper.py` to use WAL operation log instead of raw cache
-2. ✅ **COMPLETED** - Add validation before processing (checks for complete WAL entries)
-3. ✅ **COMPLETED** - Add error tracking and graceful failure handling
-4. ⏳ **PENDING** - Test thoroughly with all cache types
-
-**Priority**: HIGH (after Phase 1 deploys)
-**Complexity**: MEDIUM-HIGH
-**Risk**: HIGH if not done carefully
-
-### What Was Fixed:
-
-**Initial Fix (Commit 1):**
-The `dump_cache_to_db()` function in `cache_dumper.py` now:
-- Reads from WAL operation log (`wal_manager.get_entries_since()`) instead of raw cache entries
-- Processes each WAL operation type correctly: UPDATE, PUT, DELETE, INCREMENT
-- Converts data to proper DynamoDB format based on operation type
-- Uses `update_item()` for UPDATEs (partial updates) instead of `put_item()` (full overwrites)
-- Tracks errors per operation instead of failing entire batch
-- Only clears WAL and marks entries synced if ALL operations succeed
-
-**Critical Follow-up Fixes (Commit 2):**
-
-1. **DELETE Validation Bug**:
-   - Problem: Validation used `if not all([operation, table, key, data])` which rejected DELETEs (no data field)
-   - Impact: DELETE operations silently skipped, WAL cleared anyway, deletes lost permanently
-   - Fix: Per-operation validation (DELETE only needs operation/table/key)
-   - Fix: Increment `total_failed` for invalid entries to prevent silent WAL clearing
-
-2. **INCREMENT Replay Bug**:
-   - Problem: Non-idempotent INCREMENT operations replayed from sequence 0 on every retry
-   - Impact: Users got 2x-3x XP/streak increments after partial sync failures
-   - Fix: Added checkpoint file tracking `last_applied_sequence`
-   - Fix: Resume from checkpoint + 1, update checkpoint after each successful write
-   - Implementation: Atomic checkpoint writes (temp file + `os.replace`) for crash safety
-
-**New WAL Manager Features:**
-- `get_last_applied_sequence()` - Returns checkpoint value
-- `set_last_applied_sequence(seq)` - Atomically updates checkpoint
-- Checkpoint file: `/tmp/yeetcode/wal.checkpoint`
-- Survives crashes and restarts
-
-**Additional Type Conversion Fix (Commit 3):**
-
-3. **List Type Corruption Bug**:
-   - Problem: List conversion used `[{'S': str(item)} for item in value]` which coerced all items to strings
-   - Impact:
-     - Numeric lists corrupted: `[1, 2, 3]` → `[{'S': '1'}, {'S': '2'}, {'S': '3'}]`
-     - Boolean lists corrupted: `[true, false]` → `[{'S': 'True'}, {'S': 'False'}]`
-     - Nested structures lost entirely
-     - Bool checked after int (bool is subclass of int in Python), causing bools treated as ints
-   - Fix: Created `convert_value_to_dynamodb(value)` recursive helper
-     - Checks bool BEFORE int (critical for correct type detection)
-     - Recursively handles nested lists and dicts with proper type preservation
-     - Reused by both `convert_to_dynamodb_format()` and UPDATE operations
-   - Impact: Proper type preservation for all DynamoDB updates
-
-**Admin Endpoints and CodeRabbit Fixes (Commit 4-5):**
-
-4. **Cache Debug Endpoints** (`routes/admin.py`):
-   - Added GET `/admin/cache/status?api_key=...` endpoint returning:
-     - Cache stats (size, entries per type, hit rate)
-     - WAL stats (entries, checkpoint, file size)
-     - Dirty entries count and summary (without sensitive data)
-     - Cache keys by type for debugging
-   - Added POST `/admin/cache/dump?api_key=...` endpoint for manual cache sync
-   - Fixed duel ID mismatch in `routes/duels.py`: `duel.get('id')` → `duel.get('duelId')`
-
-5. **CodeRabbit Review Fixes** (`routes/admin.py`, `cache_dumper.py`):
-   - Fixed dirty entry field: `entry.get('dirty')` → `entry.get('last_synced')`
-   - Added TODO comment for `_cache` encapsulation issue
-   - Changed `/cache/dump` from GET to POST (write operations should use POST)
-   - Removed double-wrapping of dump response
-   - Added explicit handling for unknown WAL operation types (prevents silent failures)
-
-**Critical WAL Checkpoint and Race Condition Fixes (Commit 6):**
-
-6. **Checkpoint Skip and Double-Apply Bug** (`cache_dumper.py`):
-   - Problem 1: Failed WAL entries could be skipped forever if later entries succeeded
-     - Example: seq 1 ✅ → checkpoint = 1, seq 2 ❌, seq 3 ✅ → checkpoint = 3
-     - Next run starts from seq 4, so seq 2 is never retried (silent data loss)
-   - Problem 2: Later successful entries (especially INCREMENTs) could be double-applied
-     - Example: seq 1 ✅ → checkpoint = 1, seq 2 ❌, seq 3 INCREMENT ✅ (but checkpoint still 1)
-     - Next run: seq 2 fails again, seq 3 replayed → INCREMENT applied twice!
-   - Impact: Both permanent data loss AND data corruption (double XP/streak)
-   - Fix: STOP processing on first failure (break loop)
-     - Process entries in order until first failure
-     - Update checkpoint after each successful operation
-     - On failure: break loop, leave failed entry and all later entries for next run
-     - Next run starts from failed sequence, retries in order
-     - Guarantees: No skips, no double-applies, contiguous checkpoint
-
-7. **WAL Race Condition** (`cache_dumper.py`, `wal_manager.py`):
-   - Problem: Concurrent writes during dump could be marked synced but never applied to DynamoDB
-     - Dump snapshots WAL entries at start
-     - New writes append to WAL after snapshot
-     - Old code called `wal_manager.clear()` at end, clearing ALL entries including new ones
-     - Old code also called `mark_synced()` on ALL dirty cache entries
-     - New writes lost permanently
-   - Impact: Data loss under concurrent load (production scenario)
-   - Fix: Added `wal_manager.clear_up_to(max_sequence)` method
-     - Only clears WAL entries up to last successfully applied sequence
-     - Keeps entries with sequence > max_sequence (concurrent writes)
-     - Next dump picks them up via `get_entries_since(last_applied + 1)`
-   - Fix: Removed `mark_synced()` calls from dump logic
-     - Cache dirty flags now purely advisory (for monitoring)
-     - WAL checkpoint is source of truth for persistence
-