Performance: Bound memory use for large backup and snapshot operations

[bcherrington]

Current gap

Memory-oriented tests exist for portions of the CLI and benchmark tooling, but representative large backup and snapshot flows do not yet have a current end-to-end peak-memory budget.

Acceptance criteria

• Profile representative large backup, snapshot listing, search, and diff operations.
• Record peak memory and identify material retention or eager-loading hotspots.
• Stream or batch large collections where needed.
• Add correctness tests and stable memory regression signals.
• Document representative datasets, commands, and supported budgets.

[github-actions]

🔍 Issue Triage Analysis

Priority Level: HIGH 🔴

• Impact: High - Memory optimization directly affects user experience with large datasets
• Urgency: Medium - Performance optimization is critical for v1.0.0 release readiness
• User Impact: Significant - Large backup operations may fail or cause system instability without optimization

Issue Type: Performance Enhancement ⚡

• Category: Performance optimization
• Subcategory: Memory usage optimization
• Area: Backup operations

Complexity Estimate: MEDIUM-HIGH 🧩

• Technical Complexity: Medium-High - Requires profiling, analysis, and careful optimization
• Implementation Effort: ~3-5 days for a senior developer
• Risk Level: Medium - Performance changes require thorough testing to avoid regressions

Analysis Summary 📊

Context: This issue is part of the Performance Optimization milestone for v1.0.0 release preparation. The codebase already has performance monitoring infrastructure in place.

Current State:

• ✅ Performance profiling framework exists (src/TimeLocker/performance/profiler.py)
• ✅ Memory monitoring capabilities implemented
• ✅ Operation tracking and metrics collection available
• ✅ Benchmark suite exists (src/TimeLocker/performance/benchmarks.py)

Key Areas for Investigation:

1. File Selection Operations - FileSelection.estimate_backup_size() and get_effective_paths()
2. Backup Manager - Memory usage during execute_backup_with_retry() operations
3. Repository Operations - Large file handling in backup target processing

Suggested Labels 🏷️

• ✅ enhancement (already applied)
• ✅ area:performance (already applied)
• 🆕 priority:high - Due to v1.0.0 release dependency
• 🆕 complexity:medium-high - Technical complexity assessment
• 🆕 milestone:v1.0.0 - Release milestone tracking

Team/Component Assignment 👥

• Primary Team: Core/Performance team
• Component: Backup operations, file selection algorithms
• Expertise Required: Python performance optimization, memory profiling, backup systems

Dependencies & Blockers 🔗

• Prerequisites: None identified - can proceed immediately
• Related Issues:
  • Issue Performance: Profile and optimize file selection algorithms #18 (Profile and optimize file selection algorithms) - Should be completed first
  • Issue Performance: Measure and bound backup progress-reporting overhead #19 (Improve backup progress reporting efficiency) - Complementary optimization
• Blocking: No other issues are blocked by this

Reproducibility Assessment ✅

• Reproduction Steps: Clear - "large backup operations"
• Test Data Needed: Large dataset scenarios (>1GB, >10k files)
• Environment: Any system with limited memory

Acceptance Criteria Review ✅

• Well-defined: Yes
• Measurable: Yes - "Peak memory reduced on representative large backup scenarios"
• Testable: Yes - "No regressions in performance or correctness; tests added"

Implementation Recommendations 🛠️

1. Phase 1: Profiling & Analysis (1-2 days)

   • Use existing profiling infrastructure to identify memory hotspots
   • Create large dataset test scenarios
   • Establish baseline memory usage metrics

2. Phase 2: Optimization (2-3 days)

   • Optimize identified hotspots (likely in file selection and backup processing)
   • Implement memory-efficient algorithms for large file sets
   • Consider streaming/chunked processing approaches

3. Phase 3: Validation (1 day)

   • Add comprehensive tests for memory usage
   • Validate no performance regressions
   • Update benchmarks

Next Steps 📋

1. Immediate Actions:

   • Assign to performance team member
   • Add suggested labels
   • Link to milestone:v1.0.0

2. Before Starting Work:

   • Complete Issue Performance: Profile and optimize file selection algorithms #18 (file selection profiling) first
   • Define specific "large backup scenarios" for testing
   • Set target memory reduction goals (e.g., 30% reduction)

3. Success Criteria:

   • Measurable memory usage reduction on large datasets
   • No performance regressions in existing benchmarks
   • Comprehensive test coverage for memory optimization

Risk Assessment ⚠️

• Low Risk: Existing performance infrastructure reduces implementation risk
• Medium Risk: Performance optimizations require careful testing
• Mitigation: Comprehensive benchmarking and regression testing

---

Recommendation: Proceed with HIGH priority. This is a well-defined, implementable enhancement that's critical for v1.0.0 release quality. The existing performance infrastructure significantly reduces implementation risk.

[bcherrington]

Reconciled on 2026-07-18. Retained as current performance debt with acceptance based on reproducible evidence rather than a deleted report.