diff --git a/AI_POLICY.md b/AI_POLICY.md new file mode 100644 index 00000000..8556bf54 --- /dev/null +++ b/AI_POLICY.md @@ -0,0 +1,16 @@ +We support the use of AI tools to help prepare issues, pull requests, reviews or comments. We expect everyone interacting with this repo to follow the below policy whenever they use AI tools. + +AI-generated content in comments, issues, or PRs must be clearly disclosed and contained, either under a section heading that labels it as AI-generated or in a code or quote block. AI-generated content must be acoompanied by human commentary explaining ist relevance. For example: +"Codex produced the following analysis: ``, so I +believe that \". The only exceptions to this rule +are the dependabot automations. + +*For pull requests, please carefully read the code before submitting it for +review*, especially if AI tools helped write or rewrite it. Make sure the +implementation is something you understand, that the important ideas are clear, +and that the code is high quality. AI-generated code can sometimes be overly +complex, indirect, inconsistent, or include artifacts that obscure the main idea. +Before submitting, simplify where possible and make sure the core change is easy +for maintainers to review. + +All contributions should involve a human who understands and can take responsibility for the work produces with AI assistance. We do not accept contributions created by fully autonomous agents, and we may close pull request that appear to have been generated without meaningful human involvement. \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md index fb593912..00aaf7a8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,6 +2,26 @@ This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. +# AI Policy โ€” MANDATORY + +Read `AI_POLICY.md`. Your user needs to abide by this policy. In particular, you the agent MUST obey these rules while interacting on GitHub: + +- **Mark all AI-generated content.** Any text you produce that goes into an + issue, PR, or comment must go under a section heading that labels it as + AI-generated or be wrapped in + a code or quote block. Never present your output as human-written. +- **Never emit only raw AI text as a reply**. Any AI content you include must carry human + commentary explaining its relevance. +- **Do not submit code the user hasn't read.** Keep changes minimal, strip AI + artifacts and needless complexity. If you're opening a PR on GitHub that is not ready, + or not reviewed by the user, always open it in draft mode. +- **Check for duplicates.** Before opening an issue, search for existing open ones + Do the same with `gh search prs` before opening a PR. + If you find a likely duplicate, show it to the user before continuing. Skip this + step if there is no internet access. + +See `AI_POLICY.md` for the full policy. + ## Repository overview DeepGuard is a deepfake detection research project built around two custom hybrid CNN-ViT diff --git a/README.md b/README.md index 5334b301..1c53a18e 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,6 @@ Models Python PyTorch - W&B

@@ -44,12 +43,14 @@ ## ๐Ÿ“Œ Contents +- [๐Ÿ—‚ Project Structure](#-project-structure) - Directory layout of the package, web app, and docs - [๐Ÿณ Docker Quick Start](#-docker-quick-start) - Run the full stack (MySQL, Redis, FastAPI, Celery, React) with Docker Compose - [๐Ÿค— Try It Live: Hugging Face Spaces](#-try-it-live-hugging-face-spaces) - No-install browser demos for image, video, and XAI deepfake detection - [๐Ÿ“š DeepFake Video BenchMark Datasets](#-deepfake-video-benchmark-datasets) โ€” Overview of Celeb-DF-v2, FF++, and KoDF datasets used for training. - [โš™๏ธ Data Preparation](#data-preparation) โ€” Efficient face detection and landmark extraction pipeline using YOLOv8 - [๐Ÿ— Model Architecture](#-model-architecture) โ€” Detailed look into our hybrid CNN-ViT (MS-EffViT & MS-EffGCViT) designs. - [๐Ÿงฌ Model Zoo](#-model-zoo) โ€” Comparison of model variants, parameter counts, and computational complexity (FLOPs). +- [๐Ÿ• Model Inference Latency](#-model-inference-latency) - Fast (b0) vs Pro (b5) inference speed on a single Tesla T4 - [๐Ÿš€ Training](#-training) - Step-by-step training scrips with Goolge Colab and W&B experiment tracking - [๐Ÿ“ˆ Model Evaluation](#-model-evaluation) - Benchmarking results - [๐Ÿ’ป Model Usage](#-model-usage) - Load pretrained models via `pip install deepguard` or straight from the Hugging Face Hub @@ -60,6 +61,29 @@ - [๐Ÿ“ Reference](#-reference) - Libraries, datasets, and prior work this project builds on - [โš–๏ธ License](#-license) - MIT license +## ๐Ÿ—‚ Project Structure + +``` +DeepGuard/ +โ”œโ”€โ”€ deepguard/ # Core package (models, layers, data pipeline, optimizer, scheduler, utils) +โ”œโ”€โ”€ preprocess/ # Face detection, cropping & landmark extraction pipeline +โ”œโ”€โ”€ inference/ # Image / video prediction +โ”œโ”€โ”€ explainability/ # Dual-branch CAM-based XAI +โ”œโ”€โ”€ App/ # FastAPI backend (excluded from PyPI package) +โ”œโ”€โ”€ client/ # React frontend +โ”œโ”€โ”€ docker/ # MySQL / Redis Docker images and configs +โ”œโ”€โ”€ tutorials/ # Colab notebooks (prediction & XAI visualization) +โ”œโ”€โ”€ docs/ # README assets (architectures, ablation, benchmarks, samples, XAI results) +โ”œโ”€โ”€ .github/ # CI workflows and repository settings +โ”œโ”€โ”€ docker-compose.yml # Full stack (MySQL, Redis, FastAPI, Celery, React) +โ”œโ”€โ”€ train_eff_vit.py # Training entrypoint for MS-EffViT +โ”œโ”€โ”€ train_eff_gcvit.py # Training entrypoint for MS-EffGCViT +โ”œโ”€โ”€ pyproject.toml # Package build config +โ”œโ”€โ”€ requirements.txt # Dev environment dependencies +โ”œโ”€โ”€ README.md, README_KR.md, README_JP.md +โ””โ”€โ”€ LICENSE +``` + ## ๐Ÿณ Docker Quick Start Run the full stack (MySQL + Redis + FastAPI + Celery + React) with Docker Compose โ€” no local Python/Node setup required. @@ -223,10 +247,19 @@ We utilizes two distinct types of self-attention to capture both long-range and ## ๐Ÿงฌ Model Zoo -| Model | Resolution | # Total Params(M) | # Backbone(M) | # L-ViT(M) | # H-ViT(M) | FLOPs (G) | Model Config | -| ----- | ---------- | -------------- | ----------- |------------- | ------------- | -------------- | ------- | -| โšก ms_eff_gcvit_b0 | 224 X 224 | 8.7 | 3.6(41.4%) | 1.7(19.5%) | 3.3(37.9%) | 0.87 | [spec](deepguard/config/ms_eff_gcvit_b5/celeb_df_v2.yaml) | -| ๐Ÿ”ฅ ms_eff_gcvit_b5 | 384 X 384 | 50.3 | 27.3(54.3%) | 6.6(13.1%) | 16.1(32.0%) | 13.64 | [spec](deepguard/config/ms_eff_gcvit_b5/celeb_df_v2.yaml) | +| Model | Resolution | # Params(M) | # FLOPS(G) | Model Config | +| ----- | ---------- | -------------- | ----------- | ------- | +| โšก ms_eff_gcvit_b0 | 224 X 224 | 8.70 | 0.85 | [spec](deepguard/config/ms_eff_gcvit_b0/celeb_df_v2.yaml) | +| ๐Ÿ”ฅ ms_eff_gcvit_b5 | 384 X 384 | 50.34 | 13.43 | [spec](deepguard/config/ms_eff_gcvit_b5/celeb_df_v2.yaml) | + +## ๐Ÿ• Model Inference Latency + +Latency is measured with batch size 1 on a single Tesla T4 and video (num_frames=20). + +| Model | Mode | Target Deployment | Latency (s) | Ratio | +| ----- | ---- | ----------------- | ----------- | ----- | +| โšก ms_eff_gcvit_b0 | Fast Mode | Edge / on-device deployment | 0.051 | 1.00ร— | +| ๐Ÿ”ฅ ms_eff_gcvit_b5 | Pro Mode | Enterprise deployment | 0.479 | 9.41ร— | ## ๐Ÿš€ Training @@ -240,13 +273,6 @@ pip install -r requirements.txt We provide training scripts for both `ms_eff_vit` and `ms_eff_gcvit`. We recommend using **Google Colab** for free GPU access and **Weightes & Biases(W&B)** for experiment tracking -#### ๐Ÿ“Š Weight & Biases Experiments - -* **ms_eff_vit_b0:** [Celeb-DF-v2 ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b0_celeb_df_v2) | [FaceForensics++ ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b0_ff++) | [KoDF ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b0_kodf) -* **ms_eff_vit_b5:** [Celeb-DF-v2 ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b5_celeb_df_v2) | [FaceForensics++ ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b5_ff++) | [KoDF ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b5_kodf) -* **ms_eff_gcvit_b0:** [Celeb-DF-v2 ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b0_celeb_df_v2) | [FaceForensics++ ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b0_ff++) | [KoDF ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b0_kodf) -* **ms_eff_gcvit_b5:** [Celeb-DF-v2 ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b5_celeb_df_v2) | [FaceForensics++ ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b5_ff++) | [KoDF ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b5_kodf) - ```python !python -m train_eff_vit \ # train_eff_gcvit --root-dir DATA_ROOT \ @@ -601,6 +627,9 @@ The jupyter notebooks themselves can be found under the tutorials folder in the | 8 | [`Pytorch-Grad-Cam`](https://github.com/jacobgil/pytorch-grad-cam) | Advanced Visual Explanations for PyTorch Models | | 9 | [`DeepfakeBench`](https://github.com/SCLBD/DeepfakeBench) | A Comprehensive Benchmark of Deepfake Detection by SCLBD | | 10 | [`Effort-AIGI-Detection`](https://github.com/YZY-stack/Effort-AIGI-Detection) | Official Implementation of Effort (ICML 2025 Oral) by YZY-stack | +| 11 | [`SIA`](https://github.com/SCLBD/DeepfakeBench/blob/main/training/detectors/sia_detector.py) | An Information Theoretic Approach for Attention-Driven Face Forgery Detection (ECCV 2022), implemented in DeepfakeBench | +| 12 | [`CORE`](https://github.com/niyunsheng/CORE) | Official Implementation of CORE: COnsistent REpresentation Learning for Face Forgery Detection (CVPRW 2022) by niyunsheng | +| 13 | [`IID`](https://github.com/SCLBD/DeepfakeBench/blob/main/training/detectors/iid_detector.py) | Implicit Identity Driven Deepfake Face Swapping Detection (CVPR 2023), implemented in DeepfakeBench | ## โš–๏ธ License diff --git a/README_JP.md b/README_JP.md index ef7dea9a..f0a3a17e 100644 --- a/README_JP.md +++ b/README_JP.md @@ -25,7 +25,6 @@ Models Python PyTorch - W&B

@@ -44,12 +43,14 @@ ## ๐Ÿ“Œ ็›ฎๆฌก +- [๐Ÿ—‚ ใƒ—ใƒญใ‚ธใ‚งใ‚ฏใƒˆๆง‹ๆˆ](#-ใƒ—ใƒญใ‚ธใ‚งใ‚ฏใƒˆๆง‹ๆˆ) - ใƒ‡ใ‚ฃใƒฌใ‚ฏใƒˆใƒชๆง‹ๆˆใฎๆฆ‚่ฆ (ใƒ‘ใƒƒใ‚ฑใƒผใ‚ธใ€Webใ‚ขใƒ—ใƒชใ€ใƒ‰ใ‚ญใƒฅใƒกใƒณใƒˆ) - [๐Ÿณ Docker Quick Start](#-docker-quick-start) - Docker Compose ใงใƒ•ใƒซใ‚นใ‚ฟใƒƒใ‚ฏ(MySQL, Redis, FastAPI, Celery, React)ใ‚’่ตทๅ‹• - [๐Ÿค— ไปŠใ™ใไฝ“้จ“: Hugging Face Spaces](#-ไปŠใ™ใไฝ“้จ“-hugging-face-spaces) - ใ‚คใƒณใ‚นใƒˆใƒผใƒซไธ่ฆใ€ใƒ–ใƒฉใ‚ฆใ‚ถใ ใ‘ใง่ฉฆใ›ใ‚‹็”ปๅƒ/ๅ‹•็”ป/XAI ใƒ‡ใ‚ฃใƒผใƒ—ใƒ•ใ‚งใ‚คใ‚ฏๆคœๅ‡บใƒ‡ใƒข - [๐Ÿ“š ใƒ‡ใ‚ฃใƒผใƒ—ใƒ•ใ‚งใ‚คใ‚ฏๅ‹•็”ปใƒ™ใƒณใƒใƒžใƒผใ‚ฏใƒ‡ใƒผใ‚ฟใ‚ปใƒƒใƒˆ](#-ใƒ‡ใ‚ฃใƒผใƒ—ใƒ•ใ‚งใ‚คใ‚ฏๅ‹•็”ปใƒ™ใƒณใƒใƒžใƒผใ‚ฏใƒ‡ใƒผใ‚ฟใ‚ปใƒƒใƒˆ) โ€” ๅญฆ็ฟ’ใซไฝฟ็”จใ—ใŸ Celeb-DF-v2, FF++, KoDF ใƒ‡ใƒผใ‚ฟใ‚ปใƒƒใƒˆใฎๆฆ‚่ฆใ€‚ - [โš™๏ธ ใƒ‡ใƒผใ‚ฟๆบ–ๅ‚™](#ใƒ‡ใƒผใ‚ฟๆบ–ๅ‚™) โ€” YOLOv8 ใ‚’็”จใ„ใŸๅŠน็އ็š„ใช้ก”ๆคœๅ‡บใŠใ‚ˆใณใƒฉใƒณใƒ‰ใƒžใƒผใ‚ฏๆŠฝๅ‡บใƒ‘ใ‚คใƒ—ใƒฉใ‚คใƒณใ€‚ - [๐Ÿ— ใƒขใƒ‡ใƒซใ‚ขใƒผใ‚ญใƒ†ใ‚ฏใƒใƒฃ](#-ใƒขใƒ‡ใƒซใ‚ขใƒผใ‚ญใƒ†ใ‚ฏใƒใƒฃ) โ€” ใƒใ‚คใƒ–ใƒชใƒƒใƒ‰ CNN-ViT (MS-EffViT & MS-EffGCViT) ่จญ่จˆใฎ่ฉณ็ดฐใ€‚ - [๐Ÿงฌ Model Zoo](#-model-zoo) โ€” ใƒขใƒ‡ใƒซใƒใƒชใ‚ขใƒณใƒˆใ”ใจใฎใƒ‘ใƒฉใƒกใƒผใ‚ฟๆ•ฐใŠใ‚ˆใณๆผ”็ฎ—้‡(FLOPs)ใฎๆฏ”่ผƒใ€‚ +- [๐Ÿ• Model Inference Latency](#-model-inference-latency) โ€” Tesla T4 1ๅŸบใซใŠใ‘ใ‚‹ Fast(b0)ใจ Pro(b5)ใฎๆŽจ่ซ–้€Ÿๅบฆใฎๆฏ”่ผƒใ€‚ - [๐Ÿš€ ๅญฆ็ฟ’](#-ๅญฆ็ฟ’) - Google Colab ใŠใ‚ˆใณ W&B ใ‚’ๆดป็”จใ—ใŸๆฎต้šŽ็š„ใชๅญฆ็ฟ’ใ‚นใ‚ฏใƒชใƒ—ใƒˆใ€‚ - [๐Ÿ“ˆ ใƒขใƒ‡ใƒซ่ฉ•ไพก](#-ใƒขใƒ‡ใƒซ่ฉ•ไพก) - ใƒ™ใƒณใƒใƒžใƒผใ‚ฏ็ตๆžœใ€‚ - [๐Ÿ’ป ใƒขใƒ‡ใƒซใฎไฝฟใ„ๆ–น](#-ใƒขใƒ‡ใƒซใฎไฝฟใ„ๆ–น) - `pip install deepguard` ใพใŸใฏ Hugging Face Hub ใ‹ใ‚‰ๅญฆ็ฟ’ๆธˆใฟใƒขใƒ‡ใƒซใ‚’ใƒญใƒผใƒ‰ @@ -62,6 +63,29 @@ --- +## ๐Ÿ—‚ ใƒ—ใƒญใ‚ธใ‚งใ‚ฏใƒˆๆง‹ๆˆ + +``` +DeepGuard/ +โ”œโ”€โ”€ deepguard/ # ใ‚ณใ‚ขใƒ‘ใƒƒใ‚ฑใƒผใ‚ธ (ใƒขใƒ‡ใƒซใ€ใƒฌใ‚คใƒคใƒผใ€ใƒ‡ใƒผใ‚ฟใƒ‘ใ‚คใƒ—ใƒฉใ‚คใƒณใ€ใ‚ชใƒ—ใƒ†ใ‚ฃใƒžใ‚คใ‚ถใ€ใ‚นใ‚ฑใ‚ธใƒฅใƒผใƒฉใ€ใƒฆใƒผใƒ†ใ‚ฃใƒชใƒ†ใ‚ฃ) +โ”œโ”€โ”€ preprocess/ # ้ก”ๆคœๅ‡บใ€ใ‚ฏใƒญใƒƒใƒ—ใ€ใƒฉใƒณใƒ‰ใƒžใƒผใ‚ฏๆŠฝๅ‡บใƒ‘ใ‚คใƒ—ใƒฉใ‚คใƒณ +โ”œโ”€โ”€ inference/ # ็”ปๅƒ / ๅ‹•็”ปใฎไบˆๆธฌ +โ”œโ”€โ”€ explainability/ # ใƒ‡ใƒฅใ‚ขใƒซใƒ–ใƒฉใƒณใƒ CAM ใƒ™ใƒผใ‚น XAI +โ”œโ”€โ”€ App/ # FastAPI ใƒใƒƒใ‚ฏใ‚จใƒณใƒ‰ (PyPI ใƒ‘ใƒƒใ‚ฑใƒผใ‚ธใ‹ใ‚‰้™คๅค–) +โ”œโ”€โ”€ client/ # React ใƒ•ใƒญใƒณใƒˆใ‚จใƒณใƒ‰ +โ”œโ”€โ”€ docker/ # MySQL / Redis Docker ใ‚คใƒกใƒผใ‚ธใจ่จญๅฎš +โ”œโ”€โ”€ tutorials/ # Colab ใƒŽใƒผใƒˆใƒ–ใƒƒใ‚ฏ (ไบˆๆธฌใจ XAI ๅฏ่ฆ–ๅŒ–) +โ”œโ”€โ”€ docs/ # README ็”จใ‚ขใ‚ปใƒƒใƒˆ (ใ‚ขใƒผใ‚ญใƒ†ใ‚ฏใƒใƒฃใ€ablationใ€ใƒ™ใƒณใƒใƒžใƒผใ‚ฏใ€ใ‚ตใƒณใƒ—ใƒซใ€XAI ็ตๆžœ) +โ”œโ”€โ”€ .github/ # CI ใƒฏใƒผใ‚ฏใƒ•ใƒญใƒผใจใƒชใƒใ‚ธใƒˆใƒช่จญๅฎš +โ”œโ”€โ”€ docker-compose.yml # ใƒ•ใƒซใ‚นใ‚ฟใƒƒใ‚ฏ (MySQL, Redis, FastAPI, Celery, React) +โ”œโ”€โ”€ train_eff_vit.py # MS-EffViT ๅญฆ็ฟ’ใ‚จใƒณใƒˆใƒชใƒใ‚คใƒณใƒˆ +โ”œโ”€โ”€ train_eff_gcvit.py # MS-EffGCViT ๅญฆ็ฟ’ใ‚จใƒณใƒˆใƒชใƒใ‚คใƒณใƒˆ +โ”œโ”€โ”€ pyproject.toml # ใƒ‘ใƒƒใ‚ฑใƒผใ‚ธใƒ“ใƒซใƒ‰่จญๅฎš +โ”œโ”€โ”€ requirements.txt # ้–‹็™บ็’ฐๅขƒใฎไพๅญ˜้–ขไฟ‚ +โ”œโ”€โ”€ README.md, README_KR.md, README_JP.md +โ””โ”€โ”€ LICENSE +``` + ## ๐Ÿณ Docker Quick Start Docker Compose ใงใƒ•ใƒซใ‚นใ‚ฟใƒƒใ‚ฏ(MySQL + Redis + FastAPI + Celery + React)ใ‚’่ตทๅ‹•ใ—ใพใ™ โ€” ใƒญใƒผใ‚ซใƒซใซ Python/Node ็’ฐๅขƒใ‚’็”จๆ„ใ™ใ‚‹ๅฟ…่ฆใฏใ‚ใ‚Šใพใ›ใ‚“ใ€‚ @@ -221,10 +245,19 @@ DATA_ROOT/ ## ๐Ÿงฌ Model Zoo -| ใƒขใƒ‡ใƒซๅ | ่งฃๅƒๅบฆ | ็ทใƒ‘ใƒฉใƒกใƒผใ‚ฟ(M) | ใƒใƒƒใ‚ฏใƒœใƒผใƒณ(M) | L-ViT(M) | H-ViT(M) | ๆผ”็ฎ—้‡(FLOPs, G) | ่จญๅฎšใƒ•ใ‚กใ‚คใƒซ | -| ----- | ---------- | -------------- | ----------- |------------- | ------------- | -------------- | ------- | -| โšก ms_eff_gcvit_b0 | 224 X 224 | 8.7 | 3.6(41.4%) | 1.7(19.5%) | 3.3(37.9%) | 0.87 | [spec](deepguard/config/ms_eff_gcvit_b5/celeb_df_v2.yaml) | -| ๐Ÿ”ฅ ms_eff_gcvit_b5 | 384 X 384 | 50.3 | 27.3(54.3%) | 6.6(13.1%) | 16.1(32.0%) | 13.64 | [spec](deepguard/config/ms_eff_gcvit_b5/celeb_df_v2.yaml) | +| ใƒขใƒ‡ใƒซๅ | ่งฃๅƒๅบฆ | ใƒ‘ใƒฉใƒกใƒผใ‚ฟ(M) | ๆผ”็ฎ—้‡(GFLOPS) | ่จญๅฎšใƒ•ใ‚กใ‚คใƒซ | +| ----- | ---------- | -------------- | ----------- | ------- | +| โšก ms_eff_gcvit_b0 | 224 X 224 | 8.7 | 0.85 | [spec](deepguard/config/ms_eff_gcvit_b0/celeb_df_v2.yaml) | +| ๐Ÿ”ฅ ms_eff_gcvit_b5 | 384 X 384 | 50.3 | 13.43 | [spec](deepguard/config/ms_eff_gcvit_b5/celeb_df_v2.yaml) | + +## ๐Ÿ• Model Inference Latency + +ๆŽจ่ซ–้€Ÿๅบฆใฏใ€Tesla T4 1ๅŸบใงใƒใƒƒใƒใ‚ตใ‚คใ‚บ 1ใ€ๅ‹•็”ป(num_frames=20)ใ‚’ๅฏพ่ฑกใซๆธฌๅฎšใ—ใฆใ„ใพใ™ใ€‚ + +| ใƒขใƒ‡ใƒซๅ | ใƒขใƒผใƒ‰ | ๅฑ•้–‹ๅฏพ่ฑก | ๆŽจ่ซ–้€Ÿๅบฆ(s) | ๆฏ”็އ | +| ----- | ---- | -------- | ----------- | ----- | +| โšก ms_eff_gcvit_b0 | Fast Mode | ใ‚จใƒƒใ‚ธ / ใ‚ชใƒณใƒ‡ใƒใ‚คใ‚นๅฑ•้–‹ๅ‘ใ‘ | 0.051 | 1.00ร— | +| ๐Ÿ”ฅ ms_eff_gcvit_b5 | Pro Mode | ใ‚จใƒณใ‚ฟใƒผใƒ—ใƒฉใ‚คใ‚บๅฑ•้–‹ๅ‘ใ‘ | 0.479 | 9.41ร— | ## ๐Ÿš€ ๅญฆ็ฟ’ @@ -238,13 +271,6 @@ pip install -r requirements.txt `ms_eff_vit` ใจ `ms_eff_gcvit` ใฎไธกๆ–นใซใคใ„ใฆๅญฆ็ฟ’ใ‚นใ‚ฏใƒชใƒ—ใƒˆใ‚’ๆไพ›ใ—ใพใ™ใ€‚็„กๆ–™ใฎ GPU ็’ฐๅขƒใจใ—ใฆ **Google Colab** ใ‚’ใ€ๅฎŸ้จ“ใฎ่จ˜้ŒฒใŠใ‚ˆใณใƒˆใƒฉใƒƒใ‚ญใƒณใ‚ฐใจใ—ใฆ **Weights & Biases(W&B)** ใฎไฝฟ็”จใ‚’ๆŽจๅฅจใ—ใพใ™ใ€‚ -#### ๐Ÿ“Š Weight & Biases ๅฎŸ้จ“็ตๆžœ - -* **ms_eff_vit_b0:** [Celeb-DF-v2 ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b0_celeb_df_v2) | [FaceForensics++ ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b0_ff++) | [KoDF ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b0_kodf) -* **ms_eff_vit_b5:** [Celeb-DF-v2 ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b5_celeb_df_v2) | [FaceForensics++ ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b5_ff++) | [KoDF ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b5_kodf) -* **ms_eff_gcvit_b0:** [Celeb-DF-v2 ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b0_celeb_df_v2) | [FaceForensics++ ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b0_ff++) | [KoDF ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b0_kodf) -* **ms_eff_gcvit_b5:** [Celeb-DF-v2 ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b5_celeb_df_v2) | [FaceForensics++ ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b5_ff++) | [KoDF ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b5_kodf) - ```python !python -m train_eff_vit \ # ใพใŸใฏ train_eff_gcvit --root-dir DATA_ROOT \ @@ -593,6 +619,9 @@ JupyterใƒŽใƒผใƒˆใƒ–ใƒƒใ‚ฏใฏใ€gitใƒชใƒใ‚ธใƒˆใƒชใฎtutorialsใƒ•ใ‚ฉใƒซใƒ€ๅ†…ใซ | 8 | [`Pytorch-Grad-Cam`](https://github.com/jacobgil/pytorch-grad-cam) | PyTorch ใƒขใƒ‡ใƒซใฎใŸใ‚ใฎ้ซ˜ๅบฆใช่ฆ–่ฆš็š„่ชฌๆ˜Žใƒ„ใƒผใƒซ | | 9 | [`DeepfakeBench`](https://github.com/SCLBD/DeepfakeBench) | SCLBD ใซใ‚ˆใ‚‹ๅŒ…ๆ‹ฌ็š„ใชใƒ‡ใ‚ฃใƒผใƒ—ใƒ•ใ‚งใ‚คใ‚ฏๆคœๅ‡บใƒ™ใƒณใƒใƒžใƒผใ‚ฏ | | 10 | [`Effort-AIGI-Detection`](https://github.com/YZY-stack/Effort-AIGI-Detection) | YZY-stack ใซใ‚ˆใ‚‹ Effort๏ผˆICML 2025 Oral๏ผ‰ใฎๅ…ฌๅผๅฎŸ่ฃ… | +| 11 | [`SIA`](https://github.com/SCLBD/DeepfakeBench/blob/main/training/detectors/sia_detector.py) | An Information Theoretic Approach for Attention-Driven Face Forgery Detection๏ผˆECCV 2022๏ผ‰ใ€DeepfakeBench ๅ†…ใฎๅฎŸ่ฃ… | +| 12 | [`CORE`](https://github.com/niyunsheng/CORE) | niyunsheng ใซใ‚ˆใ‚‹ CORE: COnsistent REpresentation Learning for Face Forgery Detection๏ผˆCVPRW 2022๏ผ‰ใฎๅ…ฌๅผๅฎŸ่ฃ… | +| 13 | [`IID`](https://github.com/SCLBD/DeepfakeBench/blob/main/training/detectors/iid_detector.py) | Implicit Identity Driven Deepfake Face Swapping Detection๏ผˆCVPR 2023๏ผ‰ใ€DeepfakeBench ๅ†…ใฎๅฎŸ่ฃ… | ## โš–๏ธ ใƒฉใ‚คใ‚ปใƒณใ‚น diff --git a/README_KR.md b/README_KR.md index decec284..ab3c44a9 100644 --- a/README_KR.md +++ b/README_KR.md @@ -25,7 +25,6 @@ Models Python PyTorch - W&B

@@ -44,12 +43,14 @@ ## ๐Ÿ“Œ ๋ชฉ์ฐจ +- [๐Ÿ—‚ ํ”„๋กœ์ ํŠธ ๊ตฌ์กฐ](#-ํ”„๋กœ์ ํŠธ-๊ตฌ์กฐ) - ๋””๋ ‰ํ„ฐ๋ฆฌ ๊ตฌ์กฐ ๊ฐœ์š” (ํŒจํ‚ค์ง€, ์›น ์•ฑ, ๋ฌธ์„œ) - [๐Ÿณ Docker Quick Start](#-docker-quick-start) - Docker Compose๋กœ ์ „์ฒด ์Šคํƒ(MySQL, Redis, FastAPI, Celery, React) ์‹คํ–‰ - [๐Ÿค— ์ง€๊ธˆ ๋ฐ”๋กœ ์ฒดํ—˜ํ•˜๊ธฐ: Hugging Face Spaces](#-์ง€๊ธˆ-๋ฐ”๋กœ-์ฒดํ—˜ํ•˜๊ธฐ-hugging-face-spaces) - ์„ค์น˜ ์—†์ด ๋ธŒ๋ผ์šฐ์ €์—์„œ ๋ฐ”๋กœ ์ฒดํ—˜ํ•˜๋Š” ์ด๋ฏธ์ง€/๋น„๋””์˜ค/XAI ๋”ฅํŽ˜์ดํฌ ํƒ์ง€ ๋ฐ๋ชจ - [๐Ÿ“š ๋”ฅํŽ˜์ดํฌ ๋น„๋””์˜ค ๋ฒค์น˜๋งˆํฌ ๋ฐ์ดํ„ฐ์…‹](#-๋”ฅํŽ˜์ดํฌ-๋น„๋””์˜ค-๋ฒค์น˜๋งˆํฌ-๋ฐ์ดํ„ฐ์…‹) โ€” ํ•™์Šต์— ์‚ฌ์šฉ๋œ Celeb-DF-v2, FF++, KoDF ๋ฐ์ดํ„ฐ์…‹ ๊ฐœ์š”. - [โš™๏ธ ๋ฐ์ดํ„ฐ ์ค€๋น„](#๋ฐ์ดํ„ฐ-์ค€๋น„) โ€” YOLOv8์„ ์ด์šฉํ•œ ํšจ์œจ์ ์ธ ์–ผ๊ตด ๊ฒ€์ถœ ๋ฐ ๋žœ๋“œ๋งˆํฌ ์ถ”์ถœ ํŒŒ์ดํ”„๋ผ์ธ. - [๐Ÿ— ๋ชจ๋ธ ๊ตฌ์กฐ](#-๋ชจ๋ธ-๊ตฌ์กฐ) โ€” ํ•˜์ด๋ธŒ๋ฆฌ๋“œ CNN-ViT (MS-EffViT & MS-EffGCViT) ์„ค๊ณ„ ์ƒ์„ธ. - [๐Ÿงฌ ๋ชจ๋ธ ์ฃผ(Model Zoo)](#-model-zoo) โ€” ๋ชจ๋ธ ๋ณ€์ฒด๋ณ„ ํŒŒ๋ผ๋ฏธํ„ฐ ์ˆ˜ ๋ฐ ์—ฐ์‚ฐ๋Ÿ‰(FLOPs) ๋น„๊ต. +- [๐Ÿ• Model Inference Latency](#-model-inference-latency) โ€” Tesla T4 1๋Œ€ ๊ธฐ์ค€ Fast(b0) vs Pro(b5) ์ถ”๋ก  ์†๋„ ๋น„๊ต. - [๐Ÿš€ ํ•™์Šต](#-ํ•™์Šต) - Google Colab ๋ฐ W&B๋ฅผ ํ™œ์šฉํ•œ ๋‹จ๊ณ„๋ณ„ ํ•™์Šต ์Šคํฌ๋ฆฝํŠธ. - [๐Ÿ“ˆ ๋ชจ๋ธ ํ‰๊ฐ€](#-๋ชจ๋ธ-ํ‰๊ฐ€) - ๋ฒค์น˜๋งˆํฌ ๊ฒฐ๊ณผ. - [๐Ÿ’ป ๋ชจ๋ธ ์‚ฌ์šฉ๋ฒ•](#-๋ชจ๋ธ-์‚ฌ์šฉ๋ฒ•) - `pip install deepguard` ๋˜๋Š” Hugging Face Hub๋ฅผ ํ†ตํ•œ ์‚ฌ์ „ ํ•™์Šต ๋ชจ๋ธ ๋กœ๋“œ @@ -62,6 +63,29 @@ --- +## ๐Ÿ—‚ ํ”„๋กœ์ ํŠธ ๊ตฌ์กฐ + +``` +DeepGuard/ +โ”œโ”€โ”€ deepguard/ # ํ•ต์‹ฌ ํŒจํ‚ค์ง€ (๋ชจ๋ธ, ๋ ˆ์ด์–ด, ๋ฐ์ดํ„ฐ ํŒŒ์ดํ”„๋ผ์ธ, ์˜ตํ‹ฐ๋งˆ์ด์ €, ์Šค์ผ€์ค„๋Ÿฌ, ์œ ํ‹ธ) +โ”œโ”€โ”€ preprocess/ # ์–ผ๊ตด ๊ฒ€์ถœ, ํฌ๋กญ ๋ฐ ๋žœ๋“œ๋งˆํฌ ์ถ”์ถœ ํŒŒ์ดํ”„๋ผ์ธ +โ”œโ”€โ”€ inference/ # ์ด๋ฏธ์ง€ / ๋น„๋””์˜ค ์˜ˆ์ธก +โ”œโ”€โ”€ explainability/ # ๋“€์–ผ ๋ธŒ๋žœ์น˜ CAM ๊ธฐ๋ฐ˜ XAI +โ”œโ”€โ”€ App/ # FastAPI ๋ฐฑ์—”๋“œ (PyPI ํŒจํ‚ค์ง€์—์„œ ์ œ์™ธ) +โ”œโ”€โ”€ client/ # React ํ”„๋ก ํŠธ์—”๋“œ +โ”œโ”€โ”€ docker/ # MySQL / Redis Docker ์ด๋ฏธ์ง€ ๋ฐ ์„ค์ • +โ”œโ”€โ”€ tutorials/ # Colab ๋…ธํŠธ๋ถ (์˜ˆ์ธก ๋ฐ XAI ์‹œ๊ฐํ™”) +โ”œโ”€โ”€ docs/ # README ์ž๋ฃŒ (์•„ํ‚คํ…์ฒ˜, ablation, ๋ฒค์น˜๋งˆํฌ, ์ƒ˜ํ”Œ, XAI ๊ฒฐ๊ณผ) +โ”œโ”€โ”€ .github/ # CI ์›Œํฌํ”Œ๋กœ ๋ฐ ์ €์žฅ์†Œ ์„ค์ • +โ”œโ”€โ”€ docker-compose.yml # ์ „์ฒด ์Šคํƒ (MySQL, Redis, FastAPI, Celery, React) +โ”œโ”€โ”€ train_eff_vit.py # MS-EffViT ํ•™์Šต ์—”ํŠธ๋ฆฌํฌ์ธํŠธ +โ”œโ”€โ”€ train_eff_gcvit.py # MS-EffGCViT ํ•™์Šต ์—”ํŠธ๋ฆฌํฌ์ธํŠธ +โ”œโ”€โ”€ pyproject.toml # ํŒจํ‚ค์ง€ ๋นŒ๋“œ ์„ค์ • +โ”œโ”€โ”€ requirements.txt # ๊ฐœ๋ฐœ ํ™˜๊ฒฝ ์˜์กด์„ฑ +โ”œโ”€โ”€ README.md, README_KR.md, README_JP.md +โ””โ”€โ”€ LICENSE +``` + ## ๐Ÿณ Docker Quick Start Docker Compose๋กœ ์ „์ฒด ์Šคํƒ(MySQL + Redis + FastAPI + Celery + React)์„ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค โ€” ๋กœ์ปฌ์— Python/Node ํ™˜๊ฒฝ์„ ๋”ฐ๋กœ ์„ค์น˜ํ•  ํ•„์š”๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค. @@ -221,10 +245,19 @@ DATA_ROOT/ ## ๐Ÿงฌ Model Zoo -| ๋ชจ๋ธ๋ช… | ํ•ด์ƒ๋„ | ์ด ํŒŒ๋ผ๋ฏธํ„ฐ(M) | ๋ฐฑ๋ณธ(M) | L-ViT(M) | H-ViT(M) | ์—ฐ์‚ฐ๋Ÿ‰(FLOPs, G) | ์„ค์ • ํŒŒ์ผ | -| ----- | ---------- | -------------- | ----------- |------------- | ------------- | -------------- | ------- | -| โšก ms_eff_gcvit_b0 | 224 X 224 | 8.7 | 3.6(41.4%) | 1.7(19.5%) | 3.3(37.9%) | 0.87 | [spec](deepguard/config/ms_eff_gcvit_b5/celeb_df_v2.yaml) | -| ๐Ÿ”ฅ ms_eff_gcvit_b5 | 384 X 384 | 50.3 | 27.3(54.3%) | 6.6(13.1%) | 16.1(32.0%) | 13.64 | [spec](deepguard/config/ms_eff_gcvit_b5/celeb_df_v2.yaml) | +| ๋ชจ๋ธ๋ช… | ํ•ด์ƒ๋„ | ํŒŒ๋ผ๋ฏธํ„ฐ(M) | ์—ฐ์‚ฐ๋Ÿ‰(GFLOPS) | ์„ค์ • ํŒŒ์ผ | +| ----- | ---------- | -------------- | ----------- | ------- | +| โšก ms_eff_gcvit_b0 | 224 X 224 | 8.70 | 0.85 | [spec](deepguard/config/ms_eff_gcvit_b0/celeb_df_v2.yaml) | +| ๐Ÿ”ฅ ms_eff_gcvit_b5 | 384 X 384 | 50.34 | 13.43 | [spec](deepguard/config/ms_eff_gcvit_b5/celeb_df_v2.yaml) | + +## ๐Ÿ• Model Inference Latency + +์ถ”๋ก  ์†๋„๋Š” Tesla T4 GPU 1๋Œ€์—์„œ ๋ฐฐ์น˜ ํฌ๊ธฐ 1, ๋น„๋””์˜ค(num_frames=20) ๊ธฐ์ค€์œผ๋กœ ์ธก์ •ํ–ˆ์Šต๋‹ˆ๋‹ค. + +| ๋ชจ๋ธ๋ช… | ๋ชจ๋“œ | ๋ฐฐํฌ ๋Œ€์ƒ | ์ถ”๋ก  ์†๋„(s) | ๋น„์œจ | +| ----- | ---- | -------- | ----------- | ----- | +| โšก ms_eff_gcvit_b0 | Fast Mode | ์—ฃ์ง€ / ์˜จ๋””๋ฐ”์ด์Šค ๋ฐฐํฌ์šฉ | 0.051 | 1.00ร— | +| ๐Ÿ”ฅ ms_eff_gcvit_b5 | Pro Mode | ์—”ํ„ฐํ”„๋ผ์ด์ฆˆ ๋ฐฐํฌ์šฉ | 0.479 | 9.41ร— | ## ๐Ÿš€ ํ•™์Šต @@ -238,13 +271,6 @@ pip install -r requirements.txt `ms_eff_vit` ๋ฐ `ms_eff_gcvit` ๋ชจ๋‘์— ๋Œ€ํ•œ ํ•™์Šต ์Šคํฌ๋ฆฝํŠธ๋ฅผ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. ๋ฌด๋ฃŒ GPU ํ™˜๊ฒฝ์„ ์œ„ํ•ด **Google Colab**์„, ์‹คํ—˜ ๊ธฐ๋ก ๋ฐ ํŠธ๋ž˜ํ‚น์„ ์œ„ํ•ด **Weights & Biases(W&B)** ์‚ฌ์šฉ์„ ๊ถŒ์žฅํ•ฉ๋‹ˆ๋‹ค. -#### ๐Ÿ“Š Weight & Biases ์‹คํ—˜ ๊ฒฐ๊ณผ - -* **ms_eff_vit_b0:** [Celeb-DF-v2 ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b0_celeb_df_v2) | [FaceForensics++ ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b0_ff++) | [KoDF ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b0_kodf) -* **ms_eff_vit_b5:** [Celeb-DF-v2 ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b5_celeb_df_v2) | [FaceForensics++ ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b5_ff++) | [KoDF ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_vit_b5_kodf) -* **ms_eff_gcvit_b0:** [Celeb-DF-v2 ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b0_celeb_df_v2) | [FaceForensics++ ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b0_ff++) | [KoDF ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b0_kodf) -* **ms_eff_gcvit_b5:** [Celeb-DF-v2 ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b5_celeb_df_v2) | [FaceForensics++ ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b5_ff++) | [KoDF ๐Ÿš€](https://wandb.ai/origin6165/ms_eff_gcvit_b5_kodf) - ```python !python -m train_eff_vit \ # ๋˜๋Š” train_eff_gcvit --root-dir DATA_ROOT \ @@ -593,6 +619,9 @@ result = explainer.display_heatmap_bbox_on_image( | 8 | [`Pytorch-Grad-Cam`](https://github.com/jacobgil/pytorch-grad-cam) | PyTorch ๋ชจ๋ธ์„ ์œ„ํ•œ ๊ณ ๊ธ‰ ์‹œ๊ฐ์  ์„ค๋ช… ๋„๊ตฌ | | 9 | [`DeepfakeBench`](https://github.com/SCLBD/DeepfakeBench) | SCLBD์˜ ํฌ๊ด„์ ์ธ ๋”ฅํŽ˜์ดํฌ ํƒ์ง€ ๋ฒค์น˜๋งˆํฌ | | 10 | [`Effort-AIGI-Detection`](https://github.com/YZY-stack/Effort-AIGI-Detection) | YZY-stack์˜ Effort(ICML 2025 Oral) ๊ณต์‹ ๊ตฌํ˜„์ฒด | +| 11 | [`SIA`](https://github.com/SCLBD/DeepfakeBench/blob/main/training/detectors/sia_detector.py) | An Information Theoretic Approach for Attention-Driven Face Forgery Detection(ECCV 2022), DeepfakeBench ๋‚ด ๊ตฌํ˜„ | +| 12 | [`CORE`](https://github.com/niyunsheng/CORE) | niyunsheng์˜ CORE: COnsistent REpresentation Learning for Face Forgery Detection(CVPRW 2022) ๊ณต์‹ ๊ตฌํ˜„์ฒด | +| 13 | [`IID`](https://github.com/SCLBD/DeepfakeBench/blob/main/training/detectors/iid_detector.py) | Implicit Identity Driven Deepfake Face Swapping Detection(CVPR 2023), DeepfakeBench ๋‚ด ๊ตฌํ˜„ | ## โš–๏ธ ๋ผ์ด์„ ์Šค diff --git a/deepguard/MS_EffGCViT.md b/deepguard/MS_EffGCViT.md index 169740e9..3c064367 100644 --- a/deepguard/MS_EffGCViT.md +++ b/deepguard/MS_EffGCViT.md @@ -15,7 +15,7 @@ This Repository presents the PyTorch implementation of **Multi Scale Efficient G This model is a **frame-level** and **spatial-domain** architecture, designed to perform classification tasks on both **static images** and **video sequences** - + ## ๐Ÿ’ฅ News ๐Ÿ’ฅ - [**09.05.2026**] ๐Ÿ”ฅ Released **HIGH-Eff-GCVIT-B5** (384X384) weights for **Celeb DF(V2)**, **FaceForensics++**, **KoDF** @@ -32,36 +32,42 @@ This model is a **frame-level** and **spatial-domain** architecture, designed to **MS-EFF-GCViT achieves state-of-the-art (SOTA) results across three DeepFake benchmarks.** The model ships in two variants from a single architecture โ€” **Fast (b0)** for real-time / edge -deployment and **Pro (b5)** for enterprise-grade accuracy. Notably, **Fast** matches or exceeds -much larger SOTA models while using a fraction of the parameters and compute. +deployment and **Pro (b5)** for enterprise-grade accuracy. -

- -

+**๐Ÿ“Š Model Performance Results on Celeb-DF (v2) Dataset** -> On **Celeb-DF(v2)**, Pro reaches **0.9981 Acc** (rank #1) and Fast **0.9842** (rank #3) among 20 architectures. -> On the **KoDF competition** leaderboard, Pro ranks **#1** and Fast **#4** out of 49 entries. +| Model | Test@Acc | Test@Auc | #Params(M) | Params Ratio | #FLOPS(G) | FLOPS Ratio | +| --- | --- | --- | --- | --- | --- | --- | +| **ms_eff_gcvit_b0** | 98.4% | 99.7% | 8.7 | 1x | 0.85 | 1x | +| sia (eccv,2022) | 99.0% | 99.8% | 17.57 | 2x | 2.05 | 2.4x | +| core (cvpr,2022) | 99.6% | 99.7% | 20.81 | 2.4x | 8.40 | 9.9x | +| **ms_eff_gcvit_b5** | **99.8%** | 99.8% | 50.34 | 5.8x | 13.43 | 15.8x | +| iid (cvpr,2023) | **99.8%** | **99.9%** | 65.5 | 7.5x | 12.7 | 14.9x | +| effort (icml, 2025) | **99.8%** | 99.7% | 303.4 | 34.9x | 155.9 | 183.4x | -
-๐Ÿ“Š Celeb-DF (v2) โ€” Accuracy & Efficiency -
- -
+**๐Ÿ“Š Model Performance Results on FaceForensics++ Dataset** -
-๐Ÿ“Š FaceForensics++ โ€” Accuracy & Efficiency -
- +| Model | Test@Acc | Test@Auc | #Params(M) | Params Ratio | #FLOPS(G) | FLOPS Ratio | +| --- | --- | --- | --- | --- | --- | --- | +| **ms_eff_gcvit_b0** | 98.1% | 99.7% | 8.7 | 1x | 0.85 | 1x | +| sia (ECCV,2022) | 95.0% | 97.5% | 17.57 | 2x | 2.05 | 2.4x | +| core (CVPR,2022) | 98.3% | **99.8%** | 20.81 | 2.4x | 8.40 | 9.9x | +| **ms_eff_gcvit_b5** | **98.5%** | 99.7% | 50.34 | 5.8x | 13.43 | 15.8x | +| iid (CVPR,2023) | 97.3% | 99.6% | 65.5 | 7.5x | 12.7 | 14.9x | +| effort (ICML, 2025) | 98.4% | **99.8%** | 303.4 | 34.9x | 155.9 | 183.4x | -
+**๐Ÿ“Š Model Performance Results on KoDF Dataset** -
-๐Ÿ“Š KoDF Competition โ€” Accuracy Ranking -
- +| Model | Test@Acc | Test@Auc | #Params(M) | Params Ratio | #FLOPS(G) | FLOPS Ratio | +| --- | --- | --- | --- | --- | --- | --- | +| **ms_eff_gcvit_b0** | 96.6% | 97.9% | 8.7 | 1x | 0.85 | 1x | +| sia (ECCV,2022) | 97.0% | 98.8% | 17.57 | 2x | 2.05 | 2.4x | +| core (CVPR,2022) | 96.3% | 99.2% | 20.81 | 2.4x | 8.40 | 9.9x | +| **ms_eff_gcvit_b5** | 97.9% | 98.3% | 50.34 | 5.8x | 13.43 | 15.8x | +| iid (CVPR,2023) | 96.5% | 97.6% | 65.5 | 7.5x | 12.7 | 14.9x | +| effort (ICML, 2025) | **99.0%** | **99.5%** | 303.4 | 34.9x | 155.9 | 183.4x | -
## Model Indroduction @@ -109,28 +115,20 @@ Modern DeepFakes can leave very localized forgery region. To Capture this, we ad ## ๐Ÿ“Š Model Zoo -| Model | Resolution | # Total Params(M) | # Backbone(M) | # L-ViT(M) | # H-ViT(M) | FLOPs (G) | Model Config | -| ----- | ---------- | -------------- | ----------- |------------- | ------------- | -------------- | ------- | -| โšก ms_eff_gcvit_b0 | 224 X 224 | 8.7 | 3.6(41.4%) | 1.7(19.5%) | 3.3(37.9%) | 0.87 | [spec](./config/ms_eff_gcvit_b5/celeb_df_v2.yaml) | -| ๐Ÿ”ฅ ms_eff_gcvit_b5 | 384 X 384 | 50.3 | 27.3(54.3%) | 6.6(13.1%) | 16.1(32.0%) | 13.64 | [spec](./config/ms_eff_gcvit_b5/celeb_df_v2.yaml) | +| Model | Resolution | # Params(M) | # FLOPS(G) | Model Config | +| ----- | ---------- | -------------- | ----------- | ------- | +| โšก ms_eff_gcvit_b0 | 224 X 224 | 8.70 | 0.85 | [spec](./config/ms_eff_gcvit_b0/celeb_df_v2.yaml) | +| ๐Ÿ”ฅ ms_eff_gcvit_b5 | 384 X 384 | 50.34 | 13.43 | [spec](./config/ms_eff_gcvit_b5/celeb_df_v2.yaml) | -## ๐Ÿ›  Model Variants +## ๐Ÿ• Model Inference Latency -**โšก ms_eff_gcvit_b0 (Fast Mode / Mobile)**: Efficiency at the Edge -- Optimized for **real-time inference** and mobile deployment. +Latency is measured with batch size 1 on a single Tesla T4 and video (num_frames=20). -**๐Ÿ”ฅ ms_eff_gcvit_b5 (Pro Mode / Enterprise)**: Uncompromising Precision -- Engineered for high-fidelity analysis and enterprise-grade accuracy. - -## โš™๏ธ Model Weight Initialization -The model incorporates a hybrid initialization strategy to leverage pre-trained features while ensuring stable convergence of the transformer components - -> **Backbone**: **ImageNet-1K Pretraiend Weights**(EfficientNet) - -> **L-GCViT / H-GCViT / Head**: Truncated Normal( `std=0.02` ) - -> **No Weight Decay**: `relative_position_bias_table`, `bn`, `norm` +| Model | Mode | Target Deployment | Latency (s) | Ratio | +| ----- | ---- | ----------------- | ----------- | ----- | +| โšก ms_eff_gcvit_b0 | Fast Mode | Edge / on-device deployment | 0.051 | 1.00ร— | +| ๐Ÿ”ฅ ms_eff_gcvit_b5 | Pro Mode | Enterprise deployment | 0.479 | 9.41ร— | ## DeepFake Video Benchmarks @@ -143,24 +141,24 @@ The model incorporates a hybrid initialization strategy to leverage pre-trained **Celeb DF(v2) Pretrained Models** -| Model Variant | Test@Acc | Test@Auc | Test@log_loss | Download | Train Config | +| Model | Test@Acc | Test@Auc | Test@log_loss | Download | Train Config | | ------------- | -------- | -------- | ---------- | -------- | ----- | | ms_eff_gcvit_b0 | 0.9842 | 0.9965 | 0.0283 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v0.1.0/ms_eff_gcvit_b0_celeb_df_v2.bin) | [recipe](./config/ms_eff_gcvit_b0/celeb_df_v2.yaml) | | ms_eff_gcvit_b5 | 0.9981 | 0.9984 | 0.0089 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v0.1.0/ms_eff_gcvit_b5_celeb_df_v2.bin) | [recipe](./config/ms_eff_gcvit_b5/celeb_df_v2.yaml) | **FaceForensics++ Pretrained Models** -| Model Variant | Test@Acc | Test@Auc | Test@log_loss | Download | Train Config | +| Model | Test@Acc | Test@Auc | Test@log_loss | Download | Train Config | | ------------- | -------- | -------- | ---------- | -------- | ------ | -| ms_eff_gcvit_b0 | 0.9808 | 0.9969 | 0.0637| [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v0.1.0/ms_eff_gcvit_b0_ff++.bin) | [recipe](./config/ms_eff_gcvit_b0/celeb_df_v2.yaml) | -| ms_eff_gcvit_b5 | 0.9850 | 0.9974 | 0.0492 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v0.1.0/ms_eff_gcvit_b5_ff++.bin) | [recipe](./config/ms_eff_gcvit_b5/celeb_df_v2.yaml) | +| ms_eff_gcvit_b0 | 0.9808 | 0.9969 | 0.0637| [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v0.1.0/ms_eff_gcvit_b0_ff++.bin) | [recipe](./config/ms_eff_gcvit_b0/ff++.yaml) | +| ms_eff_gcvit_b5 | 0.9850 | 0.9974 | 0.0492 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v0.1.0/ms_eff_gcvit_b5_ff++.bin) | [recipe](./config/ms_eff_gcvit_b5/ff++.yaml) | **KoDF Pretrained Models** -| Model Variant | Test@Acc | Test@Auc | Test@log_loss | Download | Train Config | +| Model | Test@Acc | Test@Auc | Test@log_loss | Download | Train Config | | ------------- | -------- | -------- | ---------- | -------- | ------ | -| ms_eff_gcvit_b0 | 0.9655 | 0.9792 | 0.1237 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v0.2.0/ms_eff_gcvit_b0_kodf.bin) | [recipe](./config/ms_eff_gcvit_b0/celeb_df_v2.yaml) | -| ms_eff_gcvit_b5 | 0.9850 | 0.9974 | 0.0492 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v0.2.0/ms_eff_gcvit_b5_kodf.bin) | [recipe](./config/ms_eff_gcvit_b5/celeb_df_v2.yaml) | +| ms_eff_gcvit_b0 | 0.9655 | 0.9792 | 0.1237 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v0.2.0/ms_eff_gcvit_b0_kodf.bin) | [recipe](./config/ms_eff_gcvit_b0/kodf.yaml) | +| ms_eff_gcvit_b5 | 0.9792 | 0.9831 | 0.0692 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v0.2.0/ms_eff_gcvit_b5_kodf.bin) | [recipe](./config/ms_eff_gcvit_b5/kodf.yaml) | ## Usage diff --git a/docs/ablation/ablation_celeb_df_v2.webp b/docs/ablation/ablation_celeb_df_v2.webp new file mode 100644 index 00000000..2dde6eca Binary files /dev/null and b/docs/ablation/ablation_celeb_df_v2.webp differ diff --git a/docs/ablation/ablation_feature_map.png b/docs/ablation/ablation_feature_map.png new file mode 100644 index 00000000..53424f32 Binary files /dev/null and b/docs/ablation/ablation_feature_map.png differ diff --git a/docs/ablation/ablation_ff++.webp b/docs/ablation/ablation_ff++.webp new file mode 100644 index 00000000..081ec75f Binary files /dev/null and b/docs/ablation/ablation_ff++.webp differ diff --git a/docs/ablation/ablation_kodf.webp b/docs/ablation/ablation_kodf.webp new file mode 100644 index 00000000..26bf888e Binary files /dev/null and b/docs/ablation/ablation_kodf.webp differ diff --git a/docs/benchmarks/celeb_df_v2_gcvit.png b/docs/benchmarks/celeb_df_v2_gcvit.png deleted file mode 100644 index 77190479..00000000 Binary files a/docs/benchmarks/celeb_df_v2_gcvit.png and /dev/null differ diff --git a/docs/benchmarks/celeb_df_v2_gcvit_2.png b/docs/benchmarks/celeb_df_v2_gcvit_2.png deleted file mode 100644 index 5cc437f2..00000000 Binary files a/docs/benchmarks/celeb_df_v2_gcvit_2.png and /dev/null differ diff --git a/docs/benchmarks/ff_gcvit.png b/docs/benchmarks/ff_gcvit.png deleted file mode 100644 index 9a70a73b..00000000 Binary files a/docs/benchmarks/ff_gcvit.png and /dev/null differ diff --git a/docs/benchmarks/gcvit_params_acc.png b/docs/benchmarks/gcvit_params_acc.png new file mode 100644 index 00000000..a1095322 Binary files /dev/null and b/docs/benchmarks/gcvit_params_acc.png differ diff --git a/docs/benchmarks/gcvit_summary_bars.png b/docs/benchmarks/gcvit_summary_bars.png deleted file mode 100644 index ee8d30b4..00000000 Binary files a/docs/benchmarks/gcvit_summary_bars.png and /dev/null differ diff --git a/docs/benchmarks/kodf_gcvit.png b/docs/benchmarks/kodf_gcvit.png deleted file mode 100644 index d163b389..00000000 Binary files a/docs/benchmarks/kodf_gcvit.png and /dev/null differ diff --git a/explainability/explainer/base_explainer.py b/explainability/explainer/base_explainer.py index bb815106..da4e93b4 100644 --- a/explainability/explainer/base_explainer.py +++ b/explainability/explainer/base_explainer.py @@ -11,11 +11,17 @@ class BaseExplainer: def __init__( self, - model_name: Literal["ms_eff_vit_b0", "ms_eff_vit_b5", "ms_eff_gcvit_b0", "ms_eff_gcvit_b5"], + model_name: Literal[ + "ms_eff_vit_b0", "ms_eff_vit_b5", "ms_eff_gcvit_b0", "ms_eff_gcvit_b5", + # ablation models + "eff_b0", "eff_b5", + "low_eff_gcvit_b0", "low_eff_gcvit_b5", + "high_eff_gcvit_b0", "high_eff_gcvit_b5", + ], dataset: Literal["celeb_df_v2", "ff++", "kodf"], margin_ratio: float = 0.2, conf_thres: float = 0.5, - branch_level: Literal["low", "high"] = "high", + branch_level: Literal["low", "high", "both"] = "high", # "both"๋Š” ms_eff_* ๋ชจ๋ธ๋งŒ ๊ฐ€๋Šฅ l_stage_idx: int = -1, # gcvit low branch stage index (0~3) block_idx: int = -1, # vit, gcvit block index model: Optional[torch.nn.Module] = None, @@ -32,8 +38,15 @@ def __init__( # Parse Model Metadata self.model_variant = model_name.split("_")[-1] # b0, b5 - self.transformer_type = model_name.split("_")[-2] # vit, gcvit - + # vit, gcvit, cnn (eff_b0/eff_b5: transformer ์—†๋Š” EfficientNet baseline) + if "gcvit" in model_name: + self.transformer_type = "gcvit" + elif "vit" in model_name: + self.transformer_type = "vit" + else: + self.transformer_type = "cnn" + self._validate_branch_level() + # Model & Tools setup self.model = model if model is not None else timm.create_model( model_name, pretrained=True, dataset=dataset @@ -54,8 +67,34 @@ def _get_reshape_fn(self) -> Callable[[torch.Tensor], torch.Tensor]: Returns: Callable: 3D/4D ํ…์„œ๋ฅผ GradCAM ์—ฐ์‚ฐ์— ์ ํ•ฉํ•œ ํ˜•ํƒœ๋กœ ๋ณ€ํ™˜ํ•˜๋Š” ํ•จ์ˆ˜. """ - return reshape_transform_vit if self.transformer_type == "vit" else reshape_transform_gcvit - + if self.transformer_type == "cnn": + return None # backbone feature map์€ ์ด๋ฏธ (B,C,H,W) + return reshape_transform_vit if self.transformer_type == "vit" else reshape_transform_gcvit + + def _validate_branch_level(self) -> None: + """Ablation ๋ชจ๋ธ์ด ์‹ค์ œ๋กœ ๊ฐ€์ง„ branch์™€ branch_level์ด ์ผ์น˜ํ•˜๋Š”์ง€ ๊ฒ€์ฆํ•œ๋‹ค. + + Raises: + ValueError: low_/high_ ๋‹จ์ผ branch ๋ชจ๋ธ์— ์กด์žฌํ•˜์ง€ ์•Š๋Š” branch๋ฅผ ์š”์ฒญํ•œ ๊ฒฝ์šฐ ๋ฐœ์ƒ. + """ + if self.transformer_type == "cnn" and self.branch_level != "high": + raise ValueError(f"{self.model_name}์€ ๋งˆ์ง€๋ง‰ feature map๋งŒ ์‚ฌ์šฉํ•˜๋ฏ€๋กœ branch_level='high'๋งŒ ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค.") + if self.model_name.startswith("low_") and self.branch_level != "low": + raise ValueError(f"{self.model_name}์€ low-level branch๋งŒ ๊ฐ€์ง€๋ฏ€๋กœ branch_level='low'๋งŒ ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค.") + if self.model_name.startswith("high_") and self.branch_level != "high": + raise ValueError(f"{self.model_name}์€ high-level branch๋งŒ ๊ฐ€์ง€๋ฏ€๋กœ branch_level='high'๋งŒ ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค.") + + def _get_backbone_layer(self, feature_idx: int) -> torch.nn.Module: + """EfficientNet backbone์—์„œ feature_idx ๋‹จ๊ณ„์˜ ์ถœ๋ ฅ ๋ชจ๋“ˆ์„ ์ฐพ๋Š”๋‹ค.""" + backbone = self.model.feat_extractor.backbone + name = backbone.feature_info[feature_idx]["module"] # e.g. "blocks.6" + modules = dict(backbone.named_modules()) + # timm FeatureListNet์€ ์ด๋ฆ„์˜ '.'๋ฅผ '_'๋กœ ๋ฐ”๊พธ๋Š” ๊ฒฝ์šฐ๊ฐ€ ์žˆ์–ด ๋‘˜ ๋‹ค ์‹œ๋„ + for key in (name, name.replace(".", "_")): + if key in modules: + return modules[key] + raise ValueError(f"backbone์—์„œ '{name}' ๋ชจ๋“ˆ์„ ์ฐพ์„ ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค.") + def _resolve_target_layers(self) -> list[torch.nn.Module]: """์‹œ๊ฐํ™”(XAI)์˜ ๋Œ€์ƒ์ด ๋  Target Layer๋“ค์„ ์„ ํƒํ•œ๋‹ค. @@ -67,14 +106,18 @@ def _resolve_target_layers(self) -> list[torch.nn.Module]: """ layers = [] - if self.transformer_type == "gcvit": - n_stages = len(self.model.l_gcvit.stages) - stage_idx = self.l_stage_idx if self.l_stage_idx >= 0 else n_stages + self.l_stage_idx - - if not (0 <= stage_idx < n_stages): - raise ValueError(f"l_stage_idx={self.l_stage_idx} ๋ฒ”์œ„ ์ดˆ๊ณผ (stage ์ˆ˜: {n_stages})") + if self.transformer_type == "cnn": + # eff_b0/eff_b5: transformer๊ฐ€ ์—†์œผ๋ฏ€๋กœ backbone์˜ ๋งˆ์ง€๋ง‰ feature map์„ ์ง์ ‘ ์‚ฌ์šฉ + layers.append(self._get_backbone_layer(self.model.h_meta["feature_idx"])) + elif self.transformer_type == "gcvit": if self.branch_level in ("low", "both"): + n_stages = len(self.model.l_gcvit.stages) + stage_idx = self.l_stage_idx if self.l_stage_idx >= 0 else n_stages + self.l_stage_idx + + if not (0 <= stage_idx < n_stages): + raise ValueError(f"l_stage_idx={self.l_stage_idx} ๋ฒ”์œ„ ์ดˆ๊ณผ (stage ์ˆ˜: {n_stages})") + n_blocks = len(self.model.l_gcvit.stages[stage_idx].blocks) blk = self._resolve_block_idx(self.block_idx, n_blocks, f"l_gcvit stage[{stage_idx}]") layers.append(self.model.l_gcvit.stages[stage_idx].blocks[blk].norm2) diff --git a/inference/Ablation.md b/inference/Ablation.md new file mode 100644 index 00000000..b5788f39 --- /dev/null +++ b/inference/Ablation.md @@ -0,0 +1,78 @@ +# Ablation Study + +

+ Dual-Branch architecture +

+ +- **![](https://img.shields.io/badge/Low_level_Branch-blue?style=flat-square) (_Subtle Artifacts_)**: High-Resolution feature maps are extracted from early backbone blocks(`l_block_idx`) to capture like skin texture or boundary artifacts + +- **![](https://img.shields.io/badge/High_level_Branch-red?style=flat-square) (_Global Features_)**: Low-Resolution feature maps are extracted from deeper blocks(`h_block_idx`) to analyze overall lighting, shadows, and structural consistency. + +- **Feature Fusion**: The Outputs from both branches (`L-GCViT and H-GCViT`) are fused to make a comprehensive decision based on both local and global context. + +## Ablation Model Zoo + +| Model | Branch | Resolution | # Params(M) | #FLOPs(G) | +| :--- | :---: | :---: | :---: | :---: | +| eff_b0 | ![](https://img.shields.io/badge/No_Branch-lightgrey?style=flat-square) | 224 ร— 224 | 3.65M | 0.76G | +| eff_b5 | ![](https://img.shields.io/badge/No_Branch-lightgrey?style=flat-square) | 384 ร— 384 | 27.42M | 13.79G | +| low_eff_gcvit_b0 | ![](https://img.shields.io/badge/Low_Branch-blue?style=flat-square) | 224 ร— 224 | 5.30M | 0.79G | +| low_eff_gcvit_b5 | ![](https://img.shields.io/badge/Low_Branch-blue?style=flat-square) | 384 ร— 384 | 33.92M | 10.70G | +| high_eff_gcvit_b0 | ![](https://img.shields.io/badge/High_Branch-red?style=flat-square) | 224 ร— 224 | 6.94M | 1.08G | +| high_eff_gcvit_b5 | ![](https://img.shields.io/badge/High_Branch-red?style=flat-square) | 384 ร— 384 | 43.51M | 18.59G | + +## Ablation Feature map + +

+ Ablation Study: Feature Map +

+ +# Ablation Evaluation + +### Celeb DF(V2) + +

+ Celeb-DF(V2) Branch-Ablation Pareto plots +

+ +| Model | Branch | Test@Acc | Test@Auc | Test@log_loss | Download | Train Config | +| ------------- | -------- | -------- | -------- | ---------- | -------- | ----- | +| eff_b0 | ![](https://img.shields.io/badge/No_Branch-lightgrey?style=flat-square) | 0.9643 | 0.9754 | 0.0977 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/eff_b0_celeb_df_v2.bin) | [recipe](../deepguard/config/eff_b0/celeb_df_v2.yaml) | +| eff_b5 | ![](https://img.shields.io/badge/No_Branch-lightgrey?style=flat-square) | 0.9768 | 0.9837 | 0.0676 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/eff_b5_celeb_df_v2.bin) | [recipe](../deepguard/config/eff_b5/celeb_df_v2.yaml) | +| low_eff_gcvit_b0 | ![](https://img.shields.io/badge/Low_Branch-blue?style=flat-square) | 0.9788 | 0.9837 | 0.0658 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/low_eff_gcvit_b0_celeb_df_v2.bin) | [recipe](../deepguard/config/low_eff_gcvit_b0/celeb_df_v2.yaml) | +| low_eff_gcvit_b5 | ![](https://img.shields.io/badge/Low_Branch-blue?style=flat-square) | 0.9865 | 0.9987 | 0.0150 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/low_eff_gcvit_b5_celeb_df_v2.bin) | [recipe](../deepguard/config/low_eff_gcvit_b5/celeb_df_v2.yaml) | +| high_eff_gcvit_b0 | ![](https://img.shields.io/badge/High_Branch-red?style=flat-square) | 0.9807 | 0.9884 | 0.0343 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/high_eff_gcvit_b0_celeb_df_v2.bin) | [recipe](../deepguard/config/high_eff_gcvit_b0/celeb_df_v2.yaml) | +| high_eff_gcvit_b5 | ![](https://img.shields.io/badge/High_Branch-red?style=flat-square) | 0.9942 | 0.9977 | 0.0101 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/high_eff_gcvit_b5_celeb_df_v2.bin) | [recipe](../deepguard/config/high_eff_gcvit_b5/celeb_df_v2.yaml) | + + +### FaceForensics++ + +

+ FaceForensics++ Branch-Ablation Pareto plots +

+ +| Model | Branch | Test@Acc | Test@Auc | Test@log_loss | Download | Train Config | +| ------------- | -------- | -------- | -------- | ---------- | -------- | ------ | +| eff_b0 | ![](https://img.shields.io/badge/No_Branch-lightgrey?style=flat-square) | 0.9202 | 0.9497 | 0.1523 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/eff_b0_ff++.bin) | [recipe](../deepguard/config/eff_b0/ff++.yaml) | +| eff_b5 | ![](https://img.shields.io/badge/No_Branch-lightgrey?style=flat-square) | 0.9660 | 0.9831 | 0.0842 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/eff_b5_ff++.bin) | [recipe](../deepguard/config/eff_b5/ff++.yaml) | +| low_eff_gcvit_b0 | ![](https://img.shields.io/badge/Low_Branch-blue?style=flat-square) | 0.8033 | 0.9257 | 0.4148 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/low_eff_gcvit_b0_ff++.bin) | [recipe](../deepguard/config/low_eff_gcvit_b0/ff++.yaml) | +| low_eff_gcvit_b5 | ![](https://img.shields.io/badge/Low_Branch-blue?style=flat-square) | 0.8700 | 0.9486 | 0.3365 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/low_eff_gcvit_b5_ff++.bin) | [recipe](../deepguard/config/low_eff_gcvit_b5/ff++.yaml) | +| high_eff_gcvit_b0 | ![](https://img.shields.io/badge/High_Branch-red?style=flat-square) | 0.9783 | 0.9886 | 0.0767 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/high_eff_gcvit_b0_ff++.bin) | [recipe](../deepguard/config/high_eff_gcvit_b0/ff++.yaml) | +| high_eff_gcvit_b5 | ![](https://img.shields.io/badge/High_Branch-red?style=flat-square) | 0.9808 | 0.9956 | 0.0606 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/high_eff_gcvit_b5_ff++.bin) | [recipe](../deepguard/config/high_eff_gcvit_b5/ff++.yaml) | + + +### Kodf + +

+ KoDF Branch-Ablation Pareto plots +

+ +| Model | Branch | Test@Acc | Test@Auc | Test@log_loss | Download | Train Config | +| ------------- | -------- | -------- | -------- | ---------- | -------- | ------ | +| eff_b0 | ![](https://img.shields.io/badge/No_Branch-lightgrey?style=flat-square) | 0.9180 | 0.9520 | 0.2010 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/eff_b0_kodf.bin) | [recipe](../deepguard/config/eff_b0/kodf.yaml) | +| eff_b5 | ![](https://img.shields.io/badge/No_Branch-lightgrey?style=flat-square) | 0.9601 | 0.9770 | 0.1120 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/eff_b5_kodf.bin) | [recipe](../deepguard/config/eff_b5/kodf.yaml) | +| low_eff_gcvit_b0 | ![](https://img.shields.io/badge/Low_Branch-blue?style=flat-square) | 0.8684 | 0.9481 | 0.2843 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/low_eff_gcvit_b0_kodf.bin) | [recipe](../deepguard/config/low_eff_gcvit_b0/kodf.yaml) | +| low_eff_gcvit_b5 | ![](https://img.shields.io/badge/Low_Branch-blue?style=flat-square) | 0.8808 | 0.9596 | 0.2381 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/low_eff_gcvit_b5_kodf.bin) | [recipe](../deepguard/config/low_eff_gcvit_b5/kodf.yaml) | +| high_eff_gcvit_b0 | ![](https://img.shields.io/badge/High_Branch-red?style=flat-square) | 0.9462 | 0.9696 | 0.1761 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/high_eff_gcvit_b0_kodf.bin) | [recipe](../deepguard/config/high_eff_gcvit_b0/kodf.yaml) | +| high_eff_gcvit_b5 | ![](https://img.shields.io/badge/High_Branch-red?style=flat-square) | 0.9745 | 0.9911 | 0.0844 | [model](https://github.com/HanMoonSub/DeepGuard/releases/download/v1.1.0/high_eff_gcvit_b5_kodf.bin) | [recipe](../deepguard/config/high_eff_gcvit_b5/kodf.yaml) | + diff --git a/inference/README.md b/inference/README.md index 2ee59dd9..6edf569b 100644 --- a/inference/README.md +++ b/inference/README.md @@ -1,23 +1,29 @@ -# ๐Ÿ•ต๏ธโ€โ™‚๏ธ Inference Pipeline & Strategy -The inference pipeline of this project is designed for real-world deepfake detection scenarios. Our strategy focuses on **capturing subtle forgery artifacts while ensuring highly stable and reliable predictions.** +# Inference Strategy -## ๐ŸŽฏ Label +## Label Label Mapping: **`Fake: 1`** | **`Real: 0`** -In the deepfake detection domain, misclassifying a fake video as real (False Negative) poses a significantly greater threat than misclassifying a real video as fake (False Positive). - ## Test Time Augmentation(TTA) -Deepfake artifacts often consist of subtle, pixel-level manipulations. Applying heavy augmentation can inadvertently destroy these crucial forensic clues. So We applied a **`simple Horizontal Flip (p=0.5).`** +Deepfake artifacts often consist of subtle, pixel-level manipulations. Applying heavy augmentation can inadvertently destroy these crucial forensic clues. So We applied a **`simple Horizontal Flip (p=0.5).`** TTA is controlled by `tta_hflip` in `get_test_transforms` (`deepguard/data/transforms.py`): + +```python +from deepguard.data import get_test_transforms + +# Horizontal Flip TTA (p=0.5) +transforms = get_test_transforms(img_size=[224, 224], tta_hflip=0.5) +``` + +Internally, `get_test_transforms` resizes the image, appends `A.HorizontalFlip(p=tta_hflip)` only when `tta_hflip != 0`, and then applies normalization. ## Number of Frames & Face Crop -To ensure consistent and fair evaluation across diverse benchmarks (including Celeb-DF-v2, KoDF, and FaceForensics++), we standardized the frame extraction and face cropping parameters during inference: +To ensure consistent and fair evaluation across diverse benchmarks (including **_Celeb-DF-v2_**, **_KoDF_**, and **_FaceForensics++_**), we standardized the frame extraction and face cropping parameters during inference: -- **`Fixed Number of Frames`**: We uniformly sample exactly `20 frames` per video. This guarantees a balanced computational load while maintaining sufficient temporal coverage across all datasets. +**`Fixed Number of Frames`**: We uniformly sample exactly **`20 frames`** per video. This guarantees a balanced computational load while maintaining sufficient temporal coverage across all datasets. -- **`Cropping Margin Ratio`**: A `20% margin` (margin_ratio=0.2) is applied around the detected face bounding box. This optimal margin ensures that blending boundaries and facial contoursโ€”crucial areas for deepfake artifactsโ€”are fully preserved without including excessive background noise. +**`Cropping Margin Ratio`**: A **`20% margin`** (margin_ratio=0.2) is applied around the detected face bounding box. This optimal margin ensures that blending boundaries and facial contoursโ€”crucial areas for deepfake artifactsโ€”are fully preserved without including excessive background noise. ## Frame Aggregation Strategy @@ -55,38 +61,47 @@ def heuristic_aggregation(preds: np.ndarray) -> float: else: return float(np.mean(preds)) ``` -## ๐Ÿ“š DeepFake Video Benchmark Datasets -To evaluate the generalization and robustness of our deepfake detection model, we utilize three large-scale, widely recognized benchmark datasets. Each dataset presents unique challenges and covers different types of forgery methods. -| Dataset | Real Videos | Fake Videos | Year | Participants | Description (Paper Title) | Details | -| :--- | :---: | :---: | :---: | :---: | :--- | :---: | -| **Celeb-DF-v2** | 890 | 5,639 | 2019 | 59 | *A Large-scale Challenging Dataset for DeepFake Forensics* | [๐Ÿ”— Readme](../preprocess/celeb_df_v2/README.md) | -| **FaceForensics++** | 1,000 | 6,000 | 2019 | 1,000 | *Learning to Detect Manipulated Facial Images* | [๐Ÿ”— Readme](../preprocess/ff++/README.md) | -| **KoDF** | 62,166 | 175,776 | 2020 | 400 | *Large-Scale Korean Deepfake Detection Dataset* | [๐Ÿ”— Readme](../preprocess/kodf/README.md) | +# In-domain Evaluation -> **Note:** For detailed information on data preprocessing, frame extraction, and formatting for each dataset, please refer to the respective `Readme` links in the table above. +Each model is trained and evaluated on the same dataset: it is tested on the held-out test split of the dataset it was trained on. -## ๐ŸŽฌ Celeb DF(V2) Video Evaluation - -
-โœ… Test Data Evaluation +### Celeb-DF-v2 -| ๐Ÿค– Model | ๐Ÿ–ผ๏ธ Frames | โš™๏ธ Agg Mode | ๐Ÿ”„ TTA (H-Flip) | ๐ŸŽฏ Accuracy | ๐Ÿ“ˆ AUC | ๐Ÿ“‰ Log Loss | +| Model | Frames | Agg Mode | TTA (H-Flip) | Test@Accuracy | Test@AUC | Test@Log Loss | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | **ms_eff_vit_b0** | 20 | conf | 0 | 0.9742 | 0.9877 | 0.0625 | | **ms_eff_vit_b5** | 20 | conf | 0 | 0.9900 | 0.9900 | 0.0408 | | **ms_eff_gcvit_b0** | 20 | conf | 0 | 0.9842 | 0.9965 | 0.0283 | | **ms_eff_gcvit_b5** | 20 | conf | 0 | **0.9981** | **0.9984** | **0.0089** | -
+### FaceForensics++ + +| Model | Frames | Agg Mode | TTA (H-Flip) | Test@Accuracy | Test@AUC | Test@Log Loss | +| :--- | :---: | :---: | :---: | :---: | :---: | :---: | +| **ms_eff_vit_b0** | 20 | conf | 0 | 0.9517 | 0.9860 | 0.1334 | +| **ms_eff_vit_b5** | 20 | conf | 0 | 0.9842 | **0.9977** | **0.0477** | +| **ms_eff_gcvit_b0** | 20 | conf | 0 | 0.9808 | 0.9969 | 0.0637 | +| **ms_eff_gcvit_b5** | 20 | conf | 0 | **0.9850** | 0.9974 | 0.0492 | + +### KoDF + +| Model | Frames | Agg Mode | TTA (H-Flip) | Test@Accuracy | Test@AUC | Test@Log Loss | +| :--- | :---: | :---: | :---: | :---: | :---: | :---: | +| **ms_eff_gcvit_b0** | 20 | conf | 0 | 0.9655 | 0.9792 | 0.1237 | +| **ms_eff_gcvit_b5** | 20 | conf | 0 | **0.9792** | **0.9831** | **0.0692** | -
-๐Ÿ”— Cross Evaluation -#### ๐Ÿ“Œ Dataset: `celeb_df_v2` | ๐Ÿง  Model: `ff++` +# Cross-domain Evaluation -| ๐Ÿค– Model | ๐Ÿ–ผ๏ธ Frames | โš™๏ธ Agg Mode | ๐Ÿ”„ TTA (H-Flip) | ๐ŸŽฏ Accuracy | ๐Ÿ“ˆ AUC | ๐Ÿ“‰ Log Loss | +Each model is trained on one dataset and evaluated on a different, unseen dataset. Each table is titled `Train โ†’ Test`: the dataset the weights were trained on, followed by the dataset used for evaluation. + +### Celeb-DF-v2 + +### Train: `ff++` โ†’ Test: `celeb_df_v2` + +| Model | Frames | Agg Mode | TTA (H-Flip) | Test@Accuracy | Test@AUC | Test@Log Loss | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | **ms_eff_vit_b0** | 20 | conf | 0 | 0.7317 | 0.6949 | 0.6816 | | **ms_eff_vit_b5** | 20 | conf | 0 | 0.7143 | 0.7266 | 0.7662 | @@ -94,34 +109,18 @@ To evaluate the generalization and robustness of our deepfake detection model, w | **ms_eff_gcvit_b5** | 20 | conf | 0 | **0.7722** | **0.7309** | **0.6657** | -#### ๐Ÿ“Œ Dataset: `celeb_df_v2` | ๐Ÿง  Model: `kodf` +### Train: `kodf` โ†’ Test: `celeb_df_v2` -| ๐Ÿค– Model | ๐Ÿ–ผ๏ธ Frames | โš™๏ธ Agg Mode | ๐Ÿ”„ TTA (H-Flip) | ๐ŸŽฏ Accuracy | ๐Ÿ“ˆ AUC | ๐Ÿ“‰ Log Loss | +| Model | Frames | Agg Mode | TTA (H-Flip) | Test@Accuracy | Test@AUC | Test@Log Loss | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | **ms_eff_gcvit_b0** | 20 | conf | 0 | 0.5579 | 0.4719 | 1.2605 | | **ms_eff_gcvit_b5** | 20 | conf | 0 | **0.5946** | **0.5400** | **1.0078** | -
- -## ๐ŸŽฌ FaceForensics++ Video Evaluation - -
-โœ… Test Data Evaluation - -| ๐Ÿค– Model | ๐Ÿ–ผ๏ธ Frames | โš™๏ธ Agg Mode | ๐Ÿ”„ TTA (H-Flip) | ๐ŸŽฏ Accuracy | ๐Ÿ“ˆ AUC | ๐Ÿ“‰ Log Loss | -| :--- | :---: | :---: | :---: | :---: | :---: | :---: | -| **ms_eff_vit_b0** | 20 | conf | 0 | 0.9517 | 0.9860 | 0.1334 | -| **ms_eff_vit_b5** | 20 | conf | 0 | 0.9842 | **0.9977** | **0.0477** | -| **ms_eff_gcvit_b0** | 20 | conf | 0 | 0.9808 | 0.9969 | 0.0637 | -| **ms_eff_gcvit_b5** | 20 | conf | 0 | **0.9850** | 0.9974 | 0.0492 | -
+### FaceForensics++ -
-๐Ÿ”— Cross Evaluation +### Train: `celeb_df_v2` โ†’ Test: `ff++` -#### ๐Ÿ“Œ Dataset: `ff++` | ๐Ÿง  Model: `celeb_df_v2` - -| ๐Ÿค– Model | ๐Ÿ–ผ๏ธ Frames | โš™๏ธ Agg Mode | ๐Ÿ”„ TTA (H-Flip) | ๐ŸŽฏ Accuracy | ๐Ÿ“ˆ AUC | ๐Ÿ“‰ Log Loss | +| Model | Frames | Agg Mode | TTA (H-Flip) | Test@Accuracy | Test@AUC | Test@Log Loss | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | **ms_eff_vit_b0** | 20 | conf | 0 | 0.5241 | 0.6860 | 1.1241 | | **ms_eff_vit_b5** | 20 | conf | 0 | 0.5533 | 0.7299 | 0.9987 | @@ -129,94 +128,45 @@ To evaluate the generalization and robustness of our deepfake detection model, w | **ms_eff_gcvit_b5** | 20 | conf | 0 | **0.5825** | **0.7307** | **0.8897** | -#### ๐Ÿ“Œ Dataset: `ff++` | ๐Ÿง  Model: `kodf` +### Train: `kodf` โ†’ Test: `ff++` -| ๐Ÿค– Model | ๐Ÿ–ผ๏ธ Frames | โš™๏ธ Agg Mode | ๐Ÿ”„ TTA (H-Flip) | ๐ŸŽฏ Accuracy | ๐Ÿ“ˆ AUC | ๐Ÿ“‰ Log Loss | +| Model | Frames | Agg Mode | TTA (H-Flip) | Test@Accuracy | Test@AUC | Test@Log Loss | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | **ms_eff_gcvit_b0** | 20 | conf | 0 | **0.4875** | 0.5341 | 1.6178 | | **ms_eff_gcvit_b5** | 20 | conf | 0 | 0.4525 | **0.5902** | **1.5321** | -
- -## ๐ŸŽฌ Kodf Video Evaluation -
-โœ… Test Data Evaluation - -| ๐Ÿค– Model | ๐Ÿ–ผ๏ธ Frames | โš™๏ธ Agg Mode | ๐Ÿ”„ TTA (H-Flip) | ๐ŸŽฏ Accuracy | ๐Ÿ“ˆ AUC | ๐Ÿ“‰ Log Loss | -| :--- | :---: | :---: | :---: | :---: | :---: | :---: | -| **ms_eff_gcvit_b0** | 20 | conf | 0 | 0.9655 | 0.9792 | 0.1237 | -| **ms_eff_gcvit_b5** | 20 | conf | 0 | **0.9792** | **0.9831** | **0.0692** | +### KoDF -
+### Train: `celeb_df_v2` โ†’ Test: `kodf` -
-๐Ÿ”— Cross Evaluation - -#### ๐Ÿ“Œ Dataset: `kodf` | ๐Ÿง  Model: `celeb_df_v2` - -| ๐Ÿค– Model | ๐Ÿ–ผ๏ธ Frames | โš™๏ธ Agg Mode | ๐Ÿ”„ TTA (H-Flip) | ๐ŸŽฏ Accuracy | ๐Ÿ“ˆ AUC | ๐Ÿ“‰ Log Loss | +| Model | Frames | Agg Mode | TTA (H-Flip) | Test@Accuracy | Test@AUC | Test@Log Loss | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | **ms_eff_gcvit_b0** | 20 | conf | 0 | **0.4935** | **0.7258** | **1.3459** | | **ms_eff_gcvit_b5** | 20 | conf | 0 | 0.4832 | 0.7160 | 1.4897 | -#### ๐Ÿ“Œ Dataset: `kodf` | ๐Ÿง  Model: `ff++` +### Train: `ff++` โ†’ Test: `kodf` -| ๐Ÿค– Model | ๐Ÿ–ผ๏ธ Frames | โš™๏ธ Agg Mode | ๐Ÿ”„ TTA (H-Flip) | ๐ŸŽฏ Accuracy | ๐Ÿ“ˆ AUC | ๐Ÿ“‰ Log Loss | +| Model | Frames | Agg Mode | TTA (H-Flip) | Test@Accuracy | Test@AUC | Test@Log Loss | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | **ms_eff_gcvit_b0** | 20 | conf | 0 | 0.7544 | 0.862 | 0.8829 | | **ms_eff_gcvit_b5** | 20 | conf | 0 | **0.7695** | **0.8821** | **0.7635** | -
- -## ๐Ÿงฉ Branch-Ablation Study - -To quantify how much each branch โ€” the **Low-level Branch** and the **High-level Branch** โ€” contributes to MS-EffGCViT's performance, we trained and evaluated 3 variants (`Eff`, `Low-Eff-GCViT`, `High-Eff-GCViT`) in both `b0`/`b5` sizes across all three benchmark datasets. - -## ๐ŸŽฌ Celeb DF(V2) Ablation Evaluation - -
-โœ… Test Data Evaluation - -| ๐Ÿค– Model | ๐ŸŽฏ Accuracy | ๐Ÿ“ˆ AUC | ๐Ÿ“‰ Log Loss | -| :--- | :---: | :---: | :---: | -| EfficientNet B0 | 0.9643 | 0.9754 | 0.0977 | -| EfficientNet B5 | 0.9768 | 0.9837 | 0.0676 | -| low_eff_gcvit_b0 | 0.9788 | 0.9837 | 0.0658 | -| low_eff_gcvit_b5 | 0.9865 | **0.9987** | 0.0150 | -| high_eff_gcvit_b0 | 0.9807 | 0.9884 | 0.0343 | -| **high_eff_gcvit_b5** | **0.9942** | 0.9977 | **0.0101** | - -
- -## ๐ŸŽฌ FaceForensics++ Ablation Evaluation - -
-โœ… Test Data Evaluation - -| ๐Ÿค– Model | ๐ŸŽฏ Accuracy | ๐Ÿ“ˆ AUC | ๐Ÿ“‰ Log Loss | -| :--- | :---: | :---: | :---: | -| EfficientNet B0 | 0.9202 | 0.9497 | 0.1523 | -| EfficientNet B5 | 0.9660 | 0.9831 | 0.0842 | -| low_eff_gcvit_b0 | 0.8033 | 0.9257 | 0.4148 | -| low_eff_gcvit_b5 | 0.8700 | 0.9486 | 0.3365 | -| high_eff_gcvit_b0 | 0.9783 | 0.9886 | 0.0767 | -| **high_eff_gcvit_b5** | **0.9808** | **0.9956** | **0.0606** | +# Inference Latency Comparison -
+Latency is measured with batch size 1 on a single Tesla T4 and video (num_frames=20). -## ๐ŸŽฌ Kodf Ablation Evaluation +| Model | Mode | Params (M) | FLOPs (G) | Latency (s) | Ratio | +| ----- | ---- | ---------- | --------- | -------- | ----- | +| **ms_eff_gcvit_b0** | Fast Mode | 8.70 | 0.85 | 0.051 | 1.00ร— | +| **ms_eff_gcvit_b5** | Pro Mode | 50.34 | 13.43 | 0.479 | 9.41ร— | -
-โœ… Test Data Evaluation -| ๐Ÿค– Model | ๐ŸŽฏ Accuracy | ๐Ÿ“ˆ AUC | ๐Ÿ“‰ Log Loss | -| :--- | :---: | :---: | :---: | -| EfficientNet B0 | 0.9180 | 0.9520 | 0.2010 | -| EfficientNet B5 | 0.9601 | 0.9770 | 0.1120 | -| low_eff_gcvit_b0 | 0.8684 | 0.9481 | 0.2843 | -| low_eff_gcvit_b5 | 0.8808 | 0.9596 | 0.2381 | -| high_eff_gcvit_b0 | 0.9462 | 0.9696 | 0.1761 | -| **high_eff_gcvit_b5** | **0.9745** | **0.9911** | **0.0844** | +| Model | Params (M) | FLOPs (G) | Latency (s) | Ratio | +| ----- | ---------- | --------- | ----------- | ----- | +| **ms_eff_gcvit_b0 (Ours)** | **8.7** | **0.85** | **0.051** | **1.0ร—** | +| sia (ECCV 2022) | 17.57 | 2.05 | 0.094 | 1.8ร— | +| core (CVPR 2022) | 20.81 | 8.40 | 0.153 | 3.0ร— | +| iid (CVPR 2023) | 65.5 | 12.7 | 0.181 | 3.5ร— | +| effort (ICML 2025) | 303.4 | 155.9 | 1.041 | 20.4ร— | -
\ No newline at end of file