Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
2a9db84
Merge pull request #99 from HanMoonSub/main
SEOYUNJE Aug 27, 2026
28ef149
Merge pull request #141 from HanMoonSub/main
SEOYUNJE Sep 29, 2026
cb85a4a
Merge pull request #143 from HanMoonSub/main
SEOYUNJE Sep 29, 2026
f9b075a
Merge branch 'main' into develop
SEOYUNJE Sep 29, 2026
36fde9a
🖼️ Add Branch-Ablation Pareto plot images
SEOYUNJE Oct 3, 2026
c3e8d59
📝 Add Input Size/Parameters/FLOPs table and embed ablation plots
SEOYUNJE Oct 3, 2026
5dd4283
📝 Add SIA, CORE, IID to Reference table (#11-13)
SEOYUNJE Oct 7, 2026
e733a59
📝 Simplify Model Zoo table and remove stale W&B experiment links
SEOYUNJE Oct 7, 2026
d804ada
✨ Add Variant badges (No/Low/High/⭐ Dual Branch) to Pretrained Models…
SEOYUNJE Oct 7, 2026
b28c4a2
📝 Add Project Structure section to README (EN/KR/JP)
SEOYUNJE Oct 13, 2026
3e1a2a3
📝 Replace Model Performance plots with comparison tables
SEOYUNJE Oct 13, 2026
3936513
📝 Replace header image with Params vs Acc graph in MS_EffGCViT
SEOYUNJE Oct 13, 2026
46851b9
✨ Add ablation model support (eff, low/high_eff_gcvit) and “both” bra…
SEOYUNJE Oct 17, 2026
e7fc438
📝 Add AI_POLICY.md with AI-generated contribution guidelines
SEOYUNJE Oct 21, 2026
0e5a5ec
📝 Add AI Policy section to CLAUDE.md
SEOYUNJE Oct 21, 2026
66374a9
📝 Add Model Inference Latency section and fix b0 spec link in READMEs
SEOYUNJE Oct 21, 2026
67e0085
📝 Replace Model Variants with Model Inference Latency table in MS_Eff…
SEOYUNJE Oct 21, 2026
efa465a
📝 Align FLOPs in README_JP Model Zoo with the English README
SEOYUNJE Oct 21, 2026
c288e42
📝 Keep only ms_eff_gcvit rows and drop Variant column in MS_EffGCViT …
SEOYUNJE Oct 21, 2026
865c3e6
📝 Restructure inference README into in-domain/cross-domain evaluation…
SEOYUNJE Oct 21, 2026
a7a849a
📝 Add ablation feature map figure
SEOYUNJE Oct 21, 2026
4df22d4
📝 Add Ablation.md with model zoo, feature map, and ablation evaluatio…
SEOYUNJE Oct 21, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions AI_POLICY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
We support the use of AI tools to help prepare issues, pull requests, reviews or comments. We expect everyone interacting with this repo to follow the below policy whenever they use AI tools.

AI-generated content in comments, issues, or PRs must be clearly disclosed and contained, either under a section heading that labels it as AI-generated or in a code or quote block. AI-generated content must be acoompanied by human commentary explaining ist relevance. For example:
"Codex produced the following analysis: `<insert codex output here>`, so I
believe that \<insert human analysis here\>". The only exceptions to this rule
are the dependabot automations.

*For pull requests, please carefully read the code before submitting it for
review*, especially if AI tools helped write or rewrite it. Make sure the
implementation is something you understand, that the important ideas are clear,
and that the code is high quality. AI-generated code can sometimes be overly
complex, indirect, inconsistent, or include artifacts that obscure the main idea.
Before submitting, simplify where possible and make sure the core change is easy
for maintainers to review.

All contributions should involve a human who understands and can take responsibility for the work produces with AI assistance. We do not accept contributions created by fully autonomous agents, and we may close pull request that appear to have been generated without meaningful human involvement.
20 changes: 20 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,26 @@

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

# AI Policy — MANDATORY

Read `AI_POLICY.md`. Your user needs to abide by this policy. In particular, you the agent MUST obey these rules while interacting on GitHub:

- **Mark all AI-generated content.** Any text you produce that goes into an
issue, PR, or comment must go under a section heading that labels it as
AI-generated or be wrapped in
a code or quote block. Never present your output as human-written.
- **Never emit only raw AI text as a reply**. Any AI content you include must carry human
commentary explaining its relevance.
- **Do not submit code the user hasn't read.** Keep changes minimal, strip AI
artifacts and needless complexity. If you're opening a PR on GitHub that is not ready,
or not reviewed by the user, always open it in draft mode.
- **Check for duplicates.** Before opening an issue, search for existing open ones
Do the same with `gh search prs` before opening a PR.
If you find a likely duplicate, show it to the user before continuing. Skip this
step if there is no internet access.

See `AI_POLICY.md` for the full policy.

## Repository overview

DeepGuard is a deepfake detection research project built around two custom hybrid CNN-ViT
Expand Down
53 changes: 41 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@
<img src="https://img.shields.io/badge/Model-MS--Eff--GCViT--B0%20%2F%20B5-orange?style=flat-square" alt="Models">
<img src="https://img.shields.io/badge/Python-3.10+-3776AB?style=flat-square&logo=python&logoColor=white" alt="Python">
<img src="https://img.shields.io/badge/PyTorch-EE4C2C?style=flat-square&logo=pytorch&logoColor=white" alt="PyTorch">
<img src="https://img.shields.io/badge/W%26B-Recording-FFBE00?style=flat-square&logo=weightsandbiases&logoColor=white" alt="W&B">
</p>

<p align="center">
Expand All @@ -44,12 +43,14 @@

## 📌 Contents

- [🗂 Project Structure](#-project-structure) - Directory layout of the package, web app, and docs
- [🐳 Docker Quick Start](#-docker-quick-start) - Run the full stack (MySQL, Redis, FastAPI, Celery, React) with Docker Compose
- [🤗 Try It Live: Hugging Face Spaces](#-try-it-live-hugging-face-spaces) - No-install browser demos for image, video, and XAI deepfake detection
- [📚 DeepFake Video BenchMark Datasets](#-deepfake-video-benchmark-datasets) — Overview of Celeb-DF-v2, FF++, and KoDF datasets used for training.
- [⚙️ Data Preparation](#data-preparation) — Efficient face detection and landmark extraction pipeline using YOLOv8
- [🏗 Model Architecture](#-model-architecture) — Detailed look into our hybrid CNN-ViT (MS-EffViT & MS-EffGCViT) designs.
- [🧬 Model Zoo](#-model-zoo) — Comparison of model variants, parameter counts, and computational complexity (FLOPs).
- [🕐 Model Inference Latency](#-model-inference-latency) - Fast (b0) vs Pro (b5) inference speed on a single Tesla T4
- [🚀 Training](#-training) - Step-by-step training scrips with Goolge Colab and W&B experiment tracking
- [📈 Model Evaluation](#-model-evaluation) - Benchmarking results
- [💻 Model Usage](#-model-usage) - Load pretrained models via `pip install deepguard` or straight from the Hugging Face Hub
Expand All @@ -60,6 +61,29 @@
- [📝 Reference](#-reference) - Libraries, datasets, and prior work this project builds on
- [⚖️ License](#-license) - MIT license

## 🗂 Project Structure

```
DeepGuard/
├── deepguard/ # Core package (models, layers, data pipeline, optimizer, scheduler, utils)
├── preprocess/ # Face detection, cropping & landmark extraction pipeline
├── inference/ # Image / video prediction
├── explainability/ # Dual-branch CAM-based XAI
├── App/ # FastAPI backend (excluded from PyPI package)
├── client/ # React frontend
├── docker/ # MySQL / Redis Docker images and configs
├── tutorials/ # Colab notebooks (prediction & XAI visualization)
├── docs/ # README assets (architectures, ablation, benchmarks, samples, XAI results)
├── .github/ # CI workflows and repository settings
├── docker-compose.yml # Full stack (MySQL, Redis, FastAPI, Celery, React)
├── train_eff_vit.py # Training entrypoint for MS-EffViT
├── train_eff_gcvit.py # Training entrypoint for MS-EffGCViT
├── pyproject.toml # Package build config
├── requirements.txt # Dev environment dependencies
├── README.md, README_KR.md, README_JP.md
└── LICENSE
```

## 🐳 Docker Quick Start

Run the full stack (MySQL + Redis + FastAPI + Celery + React) with Docker Compose — no local Python/Node setup required.
Expand Down Expand Up @@ -223,10 +247,19 @@ We utilizes two distinct types of self-attention to capture both long-range and

## 🧬 Model Zoo

| Model | Resolution | # Total Params(M) | # Backbone(M) | # L-ViT(M) | # H-ViT(M) | FLOPs (G) | Model Config |
| ----- | ---------- | -------------- | ----------- |------------- | ------------- | -------------- | ------- |
| ⚡ ms_eff_gcvit_b0 | 224 X 224 | 8.7 | 3.6(41.4%) | 1.7(19.5%) | 3.3(37.9%) | 0.87 | [spec](deepguard/config/ms_eff_gcvit_b5/celeb_df_v2.yaml) |
| 🔥 ms_eff_gcvit_b5 | 384 X 384 | 50.3 | 27.3(54.3%) | 6.6(13.1%) | 16.1(32.0%) | 13.64 | [spec](deepguard/config/ms_eff_gcvit_b5/celeb_df_v2.yaml) |
| Model | Resolution | # Params(M) | # FLOPS(G) | Model Config |
| ----- | ---------- | -------------- | ----------- | ------- |
| ⚡ ms_eff_gcvit_b0 | 224 X 224 | 8.70 | 0.85 | [spec](deepguard/config/ms_eff_gcvit_b0/celeb_df_v2.yaml) |
| 🔥 ms_eff_gcvit_b5 | 384 X 384 | 50.34 | 13.43 | [spec](deepguard/config/ms_eff_gcvit_b5/celeb_df_v2.yaml) |

## 🕐 Model Inference Latency

Latency is measured with batch size 1 on a single Tesla T4 and video (num_frames=20).

| Model | Mode | Target Deployment | Latency (s) | Ratio |
| ----- | ---- | ----------------- | ----------- | ----- |
| ⚡ ms_eff_gcvit_b0 | Fast Mode | Edge / on-device deployment | 0.051 | 1.00× |
| 🔥 ms_eff_gcvit_b5 | Pro Mode | Enterprise deployment | 0.479 | 9.41× |

## 🚀 Training

Expand All @@ -240,13 +273,6 @@ pip install -r requirements.txt

We provide training scripts for both `ms_eff_vit` and `ms_eff_gcvit`. We recommend using **Google Colab** for free GPU access and **Weightes & Biases(W&B)** for experiment tracking

#### 📊 Weight & Biases Experiments

* **ms_eff_vit_b0:** [Celeb-DF-v2 🚀](https://wandb.ai/origin6165/ms_eff_vit_b0_celeb_df_v2) | [FaceForensics++ 🚀](https://wandb.ai/origin6165/ms_eff_vit_b0_ff++) | [KoDF 🚀](https://wandb.ai/origin6165/ms_eff_vit_b0_kodf)
* **ms_eff_vit_b5:** [Celeb-DF-v2 🚀](https://wandb.ai/origin6165/ms_eff_vit_b5_celeb_df_v2) | [FaceForensics++ 🚀](https://wandb.ai/origin6165/ms_eff_vit_b5_ff++) | [KoDF 🚀](https://wandb.ai/origin6165/ms_eff_vit_b5_kodf)
* **ms_eff_gcvit_b0:** [Celeb-DF-v2 🚀](https://wandb.ai/origin6165/ms_eff_gcvit_b0_celeb_df_v2) | [FaceForensics++ 🚀](https://wandb.ai/origin6165/ms_eff_gcvit_b0_ff++) | [KoDF 🚀](https://wandb.ai/origin6165/ms_eff_gcvit_b0_kodf)
* **ms_eff_gcvit_b5:** [Celeb-DF-v2 🚀](https://wandb.ai/origin6165/ms_eff_gcvit_b5_celeb_df_v2) | [FaceForensics++ 🚀](https://wandb.ai/origin6165/ms_eff_gcvit_b5_ff++) | [KoDF 🚀](https://wandb.ai/origin6165/ms_eff_gcvit_b5_kodf)

```python
!python -m train_eff_vit \ # train_eff_gcvit
--root-dir DATA_ROOT \
Expand Down Expand Up @@ -601,6 +627,9 @@ The jupyter notebooks themselves can be found under the tutorials folder in the
| 8 | [`Pytorch-Grad-Cam`](https://github.com/jacobgil/pytorch-grad-cam) | Advanced Visual Explanations for PyTorch Models |
| 9 | [`DeepfakeBench`](https://github.com/SCLBD/DeepfakeBench) | A Comprehensive Benchmark of Deepfake Detection by SCLBD |
| 10 | [`Effort-AIGI-Detection`](https://github.com/YZY-stack/Effort-AIGI-Detection) | Official Implementation of Effort (ICML 2025 Oral) by YZY-stack |
| 11 | [`SIA`](https://github.com/SCLBD/DeepfakeBench/blob/main/training/detectors/sia_detector.py) | An Information Theoretic Approach for Attention-Driven Face Forgery Detection (ECCV 2022), implemented in DeepfakeBench |
| 12 | [`CORE`](https://github.com/niyunsheng/CORE) | Official Implementation of CORE: COnsistent REpresentation Learning for Face Forgery Detection (CVPRW 2022) by niyunsheng |
| 13 | [`IID`](https://github.com/SCLBD/DeepfakeBench/blob/main/training/detectors/iid_detector.py) | Implicit Identity Driven Deepfake Face Swapping Detection (CVPR 2023), implemented in DeepfakeBench |

## ⚖️ License

Expand Down
53 changes: 41 additions & 12 deletions README_JP.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@
<img src="https://img.shields.io/badge/Model-MS--Eff--GCViT--B0%20%2F%20B5-orange?style=flat-square" alt="Models">
<img src="https://img.shields.io/badge/Python-3.10+-3776AB?style=flat-square&logo=python&logoColor=white" alt="Python">
<img src="https://img.shields.io/badge/PyTorch-EE4C2C?style=flat-square&logo=pytorch&logoColor=white" alt="PyTorch">
<img src="https://img.shields.io/badge/W%26B-Recording-FFBE00?style=flat-square&logo=weightsandbiases&logoColor=white" alt="W&B">
</p>

<p align="center">
Expand All @@ -44,12 +43,14 @@

## 📌 目次

- [🗂 プロジェクト構成](#-プロジェクト構成) - ディレクトリ構成の概要 (パッケージ、Webアプリ、ドキュメント)
- [🐳 Docker Quick Start](#-docker-quick-start) - Docker Compose でフルスタック(MySQL, Redis, FastAPI, Celery, React)を起動
- [🤗 今すぐ体験: Hugging Face Spaces](#-今すぐ体験-hugging-face-spaces) - インストール不要、ブラウザだけで試せる画像/動画/XAI ディープフェイク検出デモ
- [📚 ディープフェイク動画ベンチマークデータセット](#-ディープフェイク動画ベンチマークデータセット) — 学習に使用した Celeb-DF-v2, FF++, KoDF データセットの概要。
- [⚙️ データ準備](#データ準備) — YOLOv8 を用いた効率的な顔検出およびランドマーク抽出パイプライン。
- [🏗 モデルアーキテクチャ](#-モデルアーキテクチャ) — ハイブリッド CNN-ViT (MS-EffViT & MS-EffGCViT) 設計の詳細。
- [🧬 Model Zoo](#-model-zoo) — モデルバリアントごとのパラメータ数および演算量(FLOPs)の比較。
- [🕐 Model Inference Latency](#-model-inference-latency) — Tesla T4 1基における Fast(b0)と Pro(b5)の推論速度の比較。
- [🚀 学習](#-学習) - Google Colab および W&B を活用した段階的な学習スクリプト。
- [📈 モデル評価](#-モデル評価) - ベンチマーク結果。
- [💻 モデルの使い方](#-モデルの使い方) - `pip install deepguard` または Hugging Face Hub から学習済みモデルをロード
Expand All @@ -62,6 +63,29 @@

---

## 🗂 プロジェクト構成

```
DeepGuard/
├── deepguard/ # コアパッケージ (モデル、レイヤー、データパイプライン、オプティマイザ、スケジューラ、ユーティリティ)
├── preprocess/ # 顔検出、クロップ、ランドマーク抽出パイプライン
├── inference/ # 画像 / 動画の予測
├── explainability/ # デュアルブランチ CAM ベース XAI
├── App/ # FastAPI バックエンド (PyPI パッケージから除外)
├── client/ # React フロントエンド
├── docker/ # MySQL / Redis Docker イメージと設定
├── tutorials/ # Colab ノートブック (予測と XAI 可視化)
├── docs/ # README 用アセット (アーキテクチャ、ablation、ベンチマーク、サンプル、XAI 結果)
├── .github/ # CI ワークフローとリポジトリ設定
├── docker-compose.yml # フルスタック (MySQL, Redis, FastAPI, Celery, React)
├── train_eff_vit.py # MS-EffViT 学習エントリポイント
├── train_eff_gcvit.py # MS-EffGCViT 学習エントリポイント
├── pyproject.toml # パッケージビルド設定
├── requirements.txt # 開発環境の依存関係
├── README.md, README_KR.md, README_JP.md
└── LICENSE
```

## 🐳 Docker Quick Start

Docker Compose でフルスタック(MySQL + Redis + FastAPI + Celery + React)を起動します — ローカルに Python/Node 環境を用意する必要はありません。
Expand Down Expand Up @@ -221,10 +245,19 @@ DATA_ROOT/

## 🧬 Model Zoo

| モデル名 | 解像度 | 総パラメータ(M) | バックボーン(M) | L-ViT(M) | H-ViT(M) | 演算量(FLOPs, G) | 設定ファイル |
| ----- | ---------- | -------------- | ----------- |------------- | ------------- | -------------- | ------- |
| ⚡ ms_eff_gcvit_b0 | 224 X 224 | 8.7 | 3.6(41.4%) | 1.7(19.5%) | 3.3(37.9%) | 0.87 | [spec](deepguard/config/ms_eff_gcvit_b5/celeb_df_v2.yaml) |
| 🔥 ms_eff_gcvit_b5 | 384 X 384 | 50.3 | 27.3(54.3%) | 6.6(13.1%) | 16.1(32.0%) | 13.64 | [spec](deepguard/config/ms_eff_gcvit_b5/celeb_df_v2.yaml) |
| モデル名 | 解像度 | パラメータ(M) | 演算量(GFLOPS) | 設定ファイル |
| ----- | ---------- | -------------- | ----------- | ------- |
| ⚡ ms_eff_gcvit_b0 | 224 X 224 | 8.7 | 0.85 | [spec](deepguard/config/ms_eff_gcvit_b0/celeb_df_v2.yaml) |
| 🔥 ms_eff_gcvit_b5 | 384 X 384 | 50.3 | 13.43 | [spec](deepguard/config/ms_eff_gcvit_b5/celeb_df_v2.yaml) |

## 🕐 Model Inference Latency

推論速度は、Tesla T4 1基でバッチサイズ 1、動画(num_frames=20)を対象に測定しています。

| モデル名 | モード | 展開対象 | 推論速度(s) | 比率 |
| ----- | ---- | -------- | ----------- | ----- |
| ⚡ ms_eff_gcvit_b0 | Fast Mode | エッジ / オンデバイス展開向け | 0.051 | 1.00× |
| 🔥 ms_eff_gcvit_b5 | Pro Mode | エンタープライズ展開向け | 0.479 | 9.41× |

## 🚀 学習

Expand All @@ -238,13 +271,6 @@ pip install -r requirements.txt

`ms_eff_vit` と `ms_eff_gcvit` の両方について学習スクリプトを提供します。無料の GPU 環境として **Google Colab** を、実験の記録およびトラッキングとして **Weights & Biases(W&B)** の使用を推奨します。

#### 📊 Weight & Biases 実験結果

* **ms_eff_vit_b0:** [Celeb-DF-v2 🚀](https://wandb.ai/origin6165/ms_eff_vit_b0_celeb_df_v2) | [FaceForensics++ 🚀](https://wandb.ai/origin6165/ms_eff_vit_b0_ff++) | [KoDF 🚀](https://wandb.ai/origin6165/ms_eff_vit_b0_kodf)
* **ms_eff_vit_b5:** [Celeb-DF-v2 🚀](https://wandb.ai/origin6165/ms_eff_vit_b5_celeb_df_v2) | [FaceForensics++ 🚀](https://wandb.ai/origin6165/ms_eff_vit_b5_ff++) | [KoDF 🚀](https://wandb.ai/origin6165/ms_eff_vit_b5_kodf)
* **ms_eff_gcvit_b0:** [Celeb-DF-v2 🚀](https://wandb.ai/origin6165/ms_eff_gcvit_b0_celeb_df_v2) | [FaceForensics++ 🚀](https://wandb.ai/origin6165/ms_eff_gcvit_b0_ff++) | [KoDF 🚀](https://wandb.ai/origin6165/ms_eff_gcvit_b0_kodf)
* **ms_eff_gcvit_b5:** [Celeb-DF-v2 🚀](https://wandb.ai/origin6165/ms_eff_gcvit_b5_celeb_df_v2) | [FaceForensics++ 🚀](https://wandb.ai/origin6165/ms_eff_gcvit_b5_ff++) | [KoDF 🚀](https://wandb.ai/origin6165/ms_eff_gcvit_b5_kodf)

```python
!python -m train_eff_vit \ # または train_eff_gcvit
--root-dir DATA_ROOT \
Expand Down Expand Up @@ -593,6 +619,9 @@ Jupyterノートブックは、gitリポジトリのtutorialsフォルダ内に
| 8 | [`Pytorch-Grad-Cam`](https://github.com/jacobgil/pytorch-grad-cam) | PyTorch モデルのための高度な視覚的説明ツール |
| 9 | [`DeepfakeBench`](https://github.com/SCLBD/DeepfakeBench) | SCLBD による包括的なディープフェイク検出ベンチマーク |
| 10 | [`Effort-AIGI-Detection`](https://github.com/YZY-stack/Effort-AIGI-Detection) | YZY-stack による Effort(ICML 2025 Oral)の公式実装 |
| 11 | [`SIA`](https://github.com/SCLBD/DeepfakeBench/blob/main/training/detectors/sia_detector.py) | An Information Theoretic Approach for Attention-Driven Face Forgery Detection(ECCV 2022)、DeepfakeBench 内の実装 |
| 12 | [`CORE`](https://github.com/niyunsheng/CORE) | niyunsheng による CORE: COnsistent REpresentation Learning for Face Forgery Detection(CVPRW 2022)の公式実装 |
| 13 | [`IID`](https://github.com/SCLBD/DeepfakeBench/blob/main/training/detectors/iid_detector.py) | Implicit Identity Driven Deepfake Face Swapping Detection(CVPR 2023)、DeepfakeBench 内の実装 |

## ⚖️ ライセンス

Expand Down
Loading
Loading