Author: Sai Likhith Kanuparthi - Senior AI Infrastructure & Systems Engineer
- GitHub: sailikhithk
- LinkedIn: sailikhithk
- Portfolio: sailikhith.me
A production-reliability layer for detecting AI and Synthetic Intelligence (SI) generated images, aimed at journalists, fact-checkers, and national-security analysts who cannot afford to be fooled by deepfakes but also cannot afford false accusations.
Most open-source detectors publish a single score and call it a day. SAI reports a calibrated probability, an uncertainty estimate, and a verdict that can refuse to commit when the evidence is weak. The eval harness measures cross-generator generalization - the failure mode that breaks detectors the day a new generator ships.
Synthetic image generation has crossed the photorealistic threshold. Stable Diffusion XL, Midjourney v6, DALL-E 3, and FLUX produce images that pass casual human inspection. For newsrooms, intelligence analysts, and election-integrity teams, a wrong call has asymmetric cost: flagging a real photo as fake destroys credibility, while missing a fake enables disinformation.
The public landscape of detectors falls into two camps:
- Academic research code - strong on novel signals, weak on calibration, uncertainty, and deployment. Trained on one generator, evaluated on the same generator.
- Closed platforms - opaque, no way to audit confidence, no way to measure generalization to new generators.
SAI fills the gap between these: a multi-signal detector with the production-reliability primitives that real deployments need.
Three model-agnostic signals feed into a calibration layer that produces a calibrated probability, an uncertainty estimate, and a verdict that can refuse to commit when evidence is weak:
flowchart TD
IMG["Input Image"] --> FREQ["FrequencySignal\nDCT spectrum analysis\nHigh-freq energy ratio\nCross-channel correlation"]
IMG --> RECON["ReconstructionSignal\nWavelet denoiser residual\nResidual energy\nSpectral concentration"]
IMG --> NOISE["NoiseResidualSignal\nPRNU fingerprint\nChannel variance asymmetry\nSpatial consistency"]
FREQ --> ENS["Ensemble\nWeighted combination"]
RECON --> ENS
NOISE --> ENS
ENS --> CAL["Temperature Calibration\nsigmoid(logit(raw) / T)"]
CAL --> EP["Epistemic uncertainty\nSignal disagreement"]
CAL --> AL["Aleatoric uncertainty\nWeight entropy"]
EP --> TOTAL["total_uncertainty"]
AL --> TOTAL
TOTAL --> GATE{"uncertainty >=\nrefuse_threshold?"}
GATE -->|Yes| INC["Verdict: inconclusive\n(refuse to commit)"]
GATE -->|No| SCORE{"calibrated_score\n>= 0.5?"}
SCORE -->|Yes| AI["Verdict: ai"]
SCORE -->|No| REAL["Verdict: real"]
style FREQ fill:#0f3460,stroke:#e94560,color:#fff
style RECON fill:#0f3460,stroke:#e94560,color:#fff
style NOISE fill:#0f3460,stroke:#e94560,color:#fff
style CAL fill:#16213e,stroke:#e94560,color:#fff
style GATE fill:#e94560,stroke:#fff,color:#fff
style INC fill:#1a1a2e,stroke:#e94560,color:#fff
style AI fill:#1a1a2e,stroke:#e94560,color:#fff
style REAL fill:#1a1a2e,stroke:#e94560,color:#fff
Cross-generator evaluation exposes the failure mode that breaks most open-source detectors - a new generator the detector has never seen:
flowchart LR
subgraph TRAIN["Calibration training set"]
GA["Generator A\n(e.g. BigGAN)"]
GB["Generator B\n(e.g. Real camera)"]
end
subgraph TEST["Held-out test"]
GC["Generator C\n(e.g. Midjourney v6)"]
end
PIPE["DetectorPipeline\n(frozen signals)"] --> CALFIT["Fit temperature T\non A + B"]
GA --> CALFIT
GB --> CALFIT
CALFIT --> EVAL["Evaluate on C"]
GC --> EVAL
EVAL --> REPORT["Report\nAUROC, ECE\nRefusal-aware accuracy\nPer-generator breakdown"]
style TRAIN fill:#0f3460,stroke:#e94560,color:#fff
style TEST fill:#1a1a2e,stroke:#e94560,color:#fff
style REPORT fill:#16213e,stroke:#e94560,color:#fff
| Primitive | What it does | Why it matters |
|---|---|---|
| Multi-signal ensemble | Frequency, reconstruction, noise, semantic, metadata, cross-image | No single signal generalizes; ensemble hedges |
| Temperature calibration | Maps raw scores to true probabilities | A 0.9 score should mean 90% chance of AI, not 60% |
| Epistemic uncertainty | Signal disagreement | Catches inputs the detector has never seen |
| Aleatoric uncertainty | Signal confidence (distance from 0.5) | Catches inputs where no signal is confident |
| Refusal verdict | Returns "inconclusive" when uncertainty is high | Prevents forced wrong calls on hard inputs |
| Cross-generator eval | Train calibration on generators A,B; test on C | Measures the new-generator failure mode |
| JPEG compression awareness | Detects and compensates for JPEG block artifacts | Prevents false "real" on platform-re-encoded AI images |
| Metadata fingerprinting | EXIF/IPTC camera data, FBMD, software tags | Deterministic AI detection from file headers |
| Cross-image consistency | Batch-level color/edge/histogram variance | Detects AI-generated series (carousels, batch gens) |
SAI looks at six different artifact families. Each signal has strengths and blind spots; the ensemble is what makes the verdict reliable:
Diffusion and GAN generators leave periodic artifacts in the DCT spectrum that real-camera captures do not. This signal measures:
- High-frequency energy ratio: AI images often have anomalous spectral tails.
- Cross-channel spectral correlation: generators synthesize channels jointly, producing a cross-channel signature distinct from the per-channel Bayer-pattern demosaicing of a real camera.
Synthetic images live on the generator's manifold and reconstruct cleanly under a generic natural-image denoiser. Real photographs contain sensor noise and demosaicing artifacts that survive as residual. This signal measures:
- Residual energy: low residual energy suggests a smooth synthetic manifold.
- Residual spectral concentration: AI residuals concentrate energy in low frequencies (a generator failure mode); real residuals spread across high frequencies (sensor noise).
Uses a wavelet denoiser (no torch dependency) as the reconstructor.
Real cameras leave a Photo Response Non-Uniformity (PRNU) noise fingerprint from sensor demosaicing and lens distortion. Synthetic images lack this fingerprint. This signal measures:
- Channel variance asymmetry: real cameras have channel-dependent noise (Bayer pattern); AI generators typically do not.
- Spatial consistency of noise: real PRNU has stable spatial structure; AI noise is more uniform.
- JPEG blockiness detection: detects 8x8 DCT block artifacts from platform re-encoding (Instagram, Twitter). When strong JPEG compression is detected, the signal weight is reduced up to 70% because JPEG artifacts mimic PRNU spatial structure, causing false "real" verdicts on AI images that were uploaded to social media.
Pixel-level statistical signals miss semantic-level artifacts that are obvious to humans. This signal measures content-level AI signatures:
- Color palette size: AI images often have a limited, synthetic color palette (few unique colors relative to image size).
- Dominant color concentration: AI images with solid/gradient backgrounds have a high fraction of pixels matching one dominant color.
- Background uniformity: samples corner regions and measures variance. AI images with synthetic backgrounds have very low corner variance.
- Channel correlation: AI generators synthesize RGB channels jointly, producing higher channel-to-channel correlation than real camera images.
- Edge density: AI diagrams have uniform edge density outside the natural range of photographic content.
- Content-awareness: when high edge density is detected (text/diagram content), the signal reduces its weight and pulls toward neutral, because text-heavy images create natural-looking diversity that masks AI signatures.
AI/SI-generated images and platform-re-encoded images carry metadata fingerprints and provenance records that pixel-level signals cannot detect. This signal examines file headers:
- C2PA cryptographic provenance: inspects binary headers for C2PA / JUMBF manifests.
Detects digital source type assertions such as
trainedAlgorithmicMedia,compositeSynthetic, andc2pa.synthetic(deterministic synthetic signal: score 1.0, weight 1.0). Conversely, detects hardware camera certificates (c2pa.capture) for cryptographically signed physical captures (score 0.05, weight 1.0). - AI / SI software tags: detects generator signatures across both traditional AI and frontier Synthetic Intelligence (SI) tooling (Midjourney v6, Stable Diffusion 3, Flux.1, DALL-E 3, Imagen 3, Grok-2, Kling, Luma, Ideogram, Recraft).
- FBMD fingerprint: Facebook/Meta Binary Metadata in IPTC SpecialInstructions. Present when an image has been uploaded to Instagram/Facebook. Combined with no camera EXIF, this is a strong indicator of platform stripping.
- EXIF camera absence: real photographs carry EXIF metadata (camera make, model, GPS, exposure settings). AI-generated images typically have NO camera EXIF.
- Progressive JPEG: Instagram and many AI pipelines produce progressive JPEGs. Real camera JPEGs are typically baseline.
- JFIF-only metadata: images with only JFIF metadata (no EXIF, no camera data) are typically generated/synthetic.
This signal requires the image file path (not just the pixel array) because metadata is stored in file headers, not pixels.
AI-generated image series (Instagram carousels, batch generations from the same prompt) have extremely uniform color and texture statistics across images. Real photo sets have high variance due to different lighting, angles, subjects, and camera conditions. This signal operates on a BATCH of images:
- Color statistics variance: mean RGB across the batch. AI batches have stddev < 5; real photo sets have stddev > 15.
- Edge density variance: edge density should vary across real photos. AI-generated series have near-constant edge density.
- Histogram correlation: pairwise histogram correlation across images. AI batches have high correlation (>0.8); real photos have low (<0.5).
This signal is only available when detecting multiple images together via
detect_batch(). For single-image detection, it returns a neutral score
with low weight.
Post-hoc calibration that maps raw ensemble scores to well-calibrated
probabilities. Fit a single temperature T on a held-out labeled set:
calibrated_score = sigmoid(logit(raw_score) / T)
This is Platt scaling applied to the ensemble output. It does not change ranking (AUROC is invariant), only the probability interpretation.
Two forms of uncertainty are reported:
- Epistemic (model uncertainty): weighted variance of signal scores. High when signals disagree - typically on inputs from a generator the detector has never seen.
- Aleatoric (data uncertainty): average distance of signal scores from 0.5 (how confident each signal is). High when signals are near 0.5 (uncertain), low when signals are near 0 or 1 (confident).
When total_uncertainty >= refuse_threshold, the verdict is
inconclusive rather than ai or real. This is the refusal mechanism
that prevents forced wrong calls.
The harness is the production-reliability core. It measures:
- AUROC: standard discrimination metric.
- Accuracy at threshold 0.5.
- Refusal-aware accuracy: accuracy over non-refused inputs only. A high refusal-aware accuracy with a non-trivial refusal rate is the goal: the detector is right when it commits, and honest when it is unsure.
- Expected Calibration Error (ECE): reliability of probability scores.
- Per-generator AUROC: discrimination broken down by generator.
- Cross-generator generalization: fit calibration on generators A, B; evaluate on held-out generator C. This exposes the new-generator failure mode that breaks most open-source detectors.
pip install -e .Requires Python 3.9+ with numpy, scipy, scikit-learn, scikit-image,
PyWavelets, Pillow, pydantic, typer, rich, and matplotlib. Optional
torch backend for neural signals (pip install -e ".[torch]"). Dev
dependencies: pip install -e ".[dev]".
# Detect a single image (uses C2PA provenance and metadata when file path is provided)
sai detect path/to/image.png
# Detect with Synthetic Intelligence (SI) terminology
sai detect path/to/image.png --terminology si
# Detect with forensic/synthetic terminology
sai detect path/to/image.png --terminology synthetic
# Detect with JSON output (for pipelines)
sai detect path/to/image.png --json
# Detect a batch of images (uses cross-image consistency signal)
sai detect-batch path/to/image_directory/ --terminology si
# Evaluate on a directory of real vs AI images
sai eval-dir real/ ai/ --generator sd-xl
# Fit temperature calibration on a labeled set
sai calibrate real/ ai/ --out calibration.jsonfrom sai.pipeline import DetectorPipeline
from sai.io import load_image
pipeline = DetectorPipeline()
# Single image detection (with metadata signal)
result = pipeline.detect(load_image("image.png"), file_path="image.png")
print(result.verdict) # "ai", "real", or "inconclusive"
print(result.calibrated_score) # well-calibrated probability
print(result.total_uncertainty) # 0..1, high = refuse
# Batch detection (with cross-image consistency signal)
images = [load_image(f"img{i}.png") for i in range(10)]
results = pipeline.detect_batch(images, file_paths=[f"img{i}.png" for i in range(10)])
for r in results:
print(r.verdict, r.calibrated_score)Cross-generator evaluation:
from sai.eval import cross_generator_eval
from sai.pipeline import DetectorPipeline
samples = [...] # list of {image, label, generator}
pipeline = DetectorPipeline()
report = cross_generator_eval(
pipeline, samples,
train_generators=["sd-xl", "real-camera"],
held_out_generator="midjourney-v6",
)
print(report.auroc, report.ece, report.refusal_rate)SAI/
sai/
signals/ # Detection signals (frequency, reconstruction, noise,
# semantic, metadata, cross-image)
calibration.py # Temperature scaling + uncertainty quantification
pipeline.py # Ensemble pipeline (single + batch detection)
eval.py # Evaluation harness + cross-generator generalization
cli.py # sai detect / sai detect-batch / sai eval / sai calibrate
io.py # Image loading
tests/ # Synthetic fixtures + signal sanity + calibration checks
docs/ # Architecture and method notes
pyproject.toml
LICENSE
- The signals are model-agnostic and do not require training data. They trade raw discrimination power for generalization: they will not beat a fine-tuned detector on the generator it was trained on, but they will not collapse on a new generator either.
- The calibration fit needs a labeled set. For production use, fit on a representative sample of your deployment distribution.
- The refusal threshold is a policy knob. For newsroom use, set it high (refuse often). For triage use, set it low (commit often, audit later).
- This is not a forensic provenance tool. It tells you whether an image is AI-generated, not which generator produced it or whether it was edited.
Evaluated on 600 images from the GenImage benchmark (200 real MS-COCO + 200 BigGAN + 200 ADM), CPU-only, no GPU.
| Experiment | AUROC | Accuracy | ECE |
|---|---|---|---|
| BigGAN vs Real | 0.9403 | 0.6650 | 0.1796 |
| ADM vs Real | 0.6634 | - | - |
| All generators (uncalibrated) | 0.8019 | 0.6650 | 0.1796 |
| All generators (calibrated) | 0.8019 | 0.6650 | 0.1046 |
Cross-generator generalization (train calibration on one generator, test on another):
| Train on | Test on | AUROC | ECE |
|---|---|---|---|
| BigGAN | ADM | 0.6634 | 0.1145 |
| ADM | BigGAN | 0.9403 | 0.2968 |
Key findings:
- GAN artifacts (BigGAN) are detectable at AUROC 0.94; diffusion artifacts (ADM) are harder at AUROC 0.66. No single signal generalizes to all generators.
- Calibration does not transfer across generator families (ECE degrades from 0.18 to 0.30 when fitted on ADM and applied to BigGAN).
- The refusal mechanism achieves 96.15% accuracy when it commits (at 87% refusal), but the default threshold (0.40) is too aggressive and needs tuning for practical deployment.
Full results and analysis: docs/results.md.
Synthetic image detection is a textbook national-security capability: election disinformation, fabricated evidence in conflict zones, impersonation of public figures, and forged documentation all rely on the inability of analysts to reliably distinguish real from synthetic. SAI's contribution is not a better classifier - it is the reliability layer that makes a classifier safe to deploy in high-cost decisions:
- Calibrated probabilities mean an analyst can reason about risk rather than reading a binary verdict.
- Uncertainty quantification means the detector can refuse instead of being forced into a wrong call.
- Cross-generator evaluation means the deployment team can measure how the detector will perform on next month's generator, not just today's.
MIT - see LICENSE.
Sai Likhith Kanuparthi is a Senior AI Infrastructure & Systems Engineer at Airbnb, building production GenAI platforms with 30+ foundation model integrations. 7+ years shipping production AI systems at Airbnb, Southwest Airlines, Shell, Eli Lilly, and Oracle. MS Computer Science from NYU. Published at SPE ADIPEC 2022. Open-source contributor to LangChain, LiteLLM, Ragas, and LiveKit.
- GitHub: github.com/sailikhithk
- LinkedIn: linkedin.com/in/sailikhithk
- Portfolio: sailikhith.me
- Other open-source projects:
- llm-production-engineering - Field notes on LLM serving in production
- mamba-from-scratch - S4, Mamba-1, Mamba-2 in pure PyTorch
Sai Likhith Kanuparthi Synthetic AI Image Detector SAI deepfake detection
AI-generated image detection image forensics PRNU DCT frequency analysis
wavelet denoiser temperature calibration uncertainty quantification
epistemic uncertainty aleatoric uncertainty cross-generator generalization
GenImage BigGAN ADM diffusion model detection GAN detection
national security election integrity disinformation detection
calibrated probability refusal verdict Python scikit-image NumPy SciPy