Skip to content

Add histogram and optical flow comparison methods to video_describer's extract_significant_frames #10

Description

@chigwell

User Story
As a video analysis user,
I want multiple frame comparison methods in extract_significant_frames
so that semantic changes between visually similar frames are captured more effectively.

Background
The current grayscale pixel difference method in VideoDescriber.extract_significant_frames (lines 33-49 of video_describer.py) only detects absolute pixel value changes. This fails to capture meaningful content changes when frames have similar brightness distributions but different color patterns or object movements (e.g., a red object turning blue while maintaining luminance). The existing threshold-based approach also lacks context-aware comparison capabilities.

Acceptance Criteria

  • Modify frame_story/video_describer.py to implement:
    • Histogram comparison using OpenCV's cv2.compareHist with correlation method
    • Dense optical flow analysis via cv2.calcOpticalFlowFarneback
  • Add method parameter to VideoDescriber.__init__ with options:
    ["grayscale" (default), "histogram", "optical_flow"]
  • Update frame significance logic to:
    • For histogram mode: Compare HSV histograms using 16-bin quantization
    • For optical flow: Calculate magnitude thresholding on flow vectors
  • Validate through unit tests in tests/test_video_describer.py:
    1. Ensure all methods detect frame changes in test videos with:
      • Same composition but different color schemes (must fail with grayscale)
      • Static background with moving foreground objects
    2. Maintain ≤5% false positive rate compared to current implementation
  • Keep backward compatibility by preserving default grayscale behavior
  • Add configuration parameters for histogram/optical flow thresholds in VideoDescriber initialization

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions