Skip to content

PROJECT_SUMMARY.md

Latest commit

 

History

History
69 lines (50 loc) · 4.89 KB

PROJECT_SUMMARY.md

File metadata and controls

69 lines (50 loc) · 4.89 KB
Raw

SWIFT: Speech & Waveform Investigative Forensic Toolkit

Deepfake Audio Detection & Real-Time Forensic Dashboard

Welcome to the SWIFT project! This document serves as a comprehensive onboarding guide to get you up to speed on the architecture, datasets, preprocessing pipelines, and the machine learning methodology we've built so far.

1. Project Overview

SWIFT is an end-to-end system designed to distinguish between authentic human speech and high-fidelity synthetic AI voices (e.g., ElevenLabs, Amazon Polly, OpenAI TTS). It features a Python backend (handling feature extraction and inference) and a real-time web dashboard (allowing users to stream microphone audio or analyze pre-recorded .wav samples).

2. The Dataset (Training & Validation)

Initially, we faced severe "Dataset Shift" and overfitting because the model was trained on a tiny subset of data. We resolved this by building a robust composite dataset script (train_robust_model.py) that fetches and processes nearly 600 unique audio samples:

  • Commercial AI Voices: Sourced from Hugging Face (garystafford/deepfake-audio-detection). We specifically target deepfakes generated by state-of-the-art TTS models like ElevenLabs, Hume, and Amazon Polly.
  • Authentic Human Voices: Sourced from the same dataset as well as LibriSpeech to provide a clean baseline.
  • Frontend Test Samples: We forcefully inject the specific LibriSpeech and MMS-TTS test samples used in the UI (public/samples/) into the training mix via upweighting to ensure the model perfectly adapts to the frontend testing domain.

3. Preprocessing & Feature Extraction

Standard Mel-spectrograms compress high frequencies to mimic human hearing, which unfortunately hides the microscopic synthetic artifacts left by AI generators. To counter this, we use a custom 6-Channel Forensic Spectrogram Extractor:

  1. LFCC (Linear Frequency Cepstral Coefficients): Uses a linear scale to preserve high-frequency AI artifacts.
  2. F0 Pitch Contour: Tracks unnatural, robotic pitch perfection.
  3. Formant Tracks (F1-F3): Maps the resonant frequencies of the vocal tract (AI often lacks physical acoustic properties).
  4. Harmonic-to-Noise Ratio (HNR): Detects synthetic smoothness.
  5. Spectral Flux / Jitter: Measures frame-to-frame variations.
  6. Instantaneous Phase: Captures phase inconsistencies introduced by AI vocoders (like HiFi-GAN or VITS).

All audio is normalized, resampled to 16 kHz, and padded/truncated to exactly 40,000 samples (~2.5 seconds) before being passed through the 6-channel extractor, producing a (6, 224, 224) tensor.

4. Model Architecture (PhysioSpecNet)

Our custom neural network, PhysioSpecNet, is designed to process the 6-channel forensic tensor:

  • Cross-Channel Attention Module: The model doesn't just treat the 6 channels independently. It uses a custom attention mechanism where the 5 physiological channels dynamically attend to and modulate the primary LFCC channel, highlighting inconsistencies between acoustic features and spectral data.
  • EfficientNet-B0 Backbone: The attention-modulated tensor is fed into an EfficientNet-B0 backbone. We use this state-of-the-art vision model because it excels at finding complex visual patterns (like artifacts in spectrograms) while remaining computationally lightweight enough for real-time streaming.
  • Classification Head: Global Average Pooling followed by dense layers with a 0.5 dropout rate, outputting binary logits (Real vs. Fake).

5. Training Methodology & Robustness

To prevent the model from memorizing the data and predicting everything as "Fake" (a bug we recently fixed), we enforce strict ML methodologies:

  • Data Augmentation (SpecAugment): We heavily multiply our dataset by applying random time and frequency masking directly onto the 6-channel spectrogram tensors. This forces the model to learn universal deepfake features rather than relying on a single artifact.
  • Stratified 80/20 Split: The model is strictly evaluated on an unseen 20% validation set during training.
  • Optimizer & LR Scheduler: AdamW optimizer combined with a CosineAnnealingLR scheduler to smoothly navigate the loss landscape.
  • Early Stopping: Checkpoints are saved purely based on Validation Loss improvements.

Current Performance: The robust pipeline (best_physiospecnet.pth) achieved 99.73% Accuracy and a 0.54% Equal Error Rate (EER) on the unseen validation set, with a natural classification threshold of 0.5.

6. How to Run the Project

  • Start the Server & Dashboard: 
    python server.py 8000
    Then open http://localhost:8000 in your browser.
  • Retrain the Model: 
    python train_robust_model.py
    This will automatically fetch the datasets, apply SpecAugment, run 15 epochs with early stopping, and overwrite checkpoints/best_physiospecnet.pth with the new weights.