Facial-Recognition based IoT Secure Vault System engineered with Dual-ESP32 Microcontrollers, Deep Learning Facial Recognition (OpenCV 5.0 YuNet + SFace), Hardware RTC Deep Sleep Power Optimization, and Two-Factor Out-of-Band Telegram OTP Anti-Spoofing Verification.
- System Architecture & Motivation
- Hardware Bill of Materials (BOM)
- Pinout & Circuit Schematics
- Finite State Machine (FSM) & System Workflow
- AI Face Recognition Engine (OpenCV 5.0 DNN)
- Repository & Project Structure
- REST API Specification
- Step-by-Step Installation & Flashing Guide
- Interactive Face Enrollment & Stream Controls
- Troubleshooting & Hardware FAQ
Standard biometric locker systems are susceptible to 2D presentation attacks (such as holding up a high-resolution printed photo or smartphone screen displaying the vault owner's face).
This system implements Multi-Factor Out-of-Band Authentication:
- Factor 1 (Biometrics): Real-time neural network face detection (YuNet) and 128-dimensional embedding verification (SFace) with cosine similarity scoring.
- Factor 2 (Possession / Out-of-Band OTP): When a registered face is recognized, a cryptographic 4-digit One-Time Password (OTP) is generated with a strict 2-minute validity window. This OTP is instantly transmitted directly to the owner's personal Telegram App alongside a timestamped snapshot of the person standing at the vault.
- Factor 3 (Matrix Keypad Verification): The physical vault unlocks only when the user enters the exact matching OTP on the 4x4 matrix keypad.
- Intruder Defense: If an unauthorized or unrecognized person approaches, the system immediately sounds a loud Piezo Siren Alarm, logs the intruder's photo, and dispatches a high-priority warning alert to the owner's Telegram.
┌───────────────────────────────┐
│ Python Master AI Server │
│ - OpenCV 5.0 YuNet + SFace │
│ - Flask REST API (Port 5000)│
│ - Zeroconf mDNS Broadcaster │
└───────▲───────────────┬───────┘
│ │
POST /api/trigger_scan│ │ GET /capture
(JSON Response) │ │ (MJPEG Stream)
│ ▼
┌───────────────────────────────┐ │ ┌───────────────────────────────┐
│ Main ESP32 Vault Controller │───────┘ │ ESP32-CAM Node │
│ - RTC GPIO 33 Deep Sleep │ │ - OV2640 Camera Sensor │
│ - 16x2 I2C LCD Display │ │ - MJPEG Video Server │
│ - 4x4 Matrix Keypad Input │ │ - High-Power Flash LED │
│ - SG90/MG90S Servo Lock │ │ - mDNS (esp32cam.local) │
│ - Piezo Acoustic Buzzer │ └───────────────────────────────┘
└───────────────┬───────────────┘ │
│ │ Async Dispatch
▼ ▼
[Physical Vault Lock] ┌─────────────────┐
(0° Lock / 90° Unlock) │ Telegram Cloud │
│ Owner Device │
└─────────────────┘
| Item | Component | Model / Spec | Purpose |
|---|---|---|---|
| 1 | Main Controller MCU | ESP32 Dev Module (NodeMCU-32S / ESP-WROOM-32) | Core system orchestrator, peripheral control, power management |
| 2 | Camera Node MCU | ESP32-CAM AI-Thinker Module (OV2640 Sensor) | High-speed MJPEG video streaming & single-frame captures |
| 3 | Proximity Sensor | FC-51 / LM393 IR Obstacle Sensor Module | Low-power proximity detection & hardware RTC interrupt trigger |
| 4 | Display Module | 16x2 Character LCD with PCF8574 I2C Backpack | Visual user guidance, countdowns, and system diagnostics |
| 5 | Input Keypad | 4x4 Membrane / Matrix Keypad Module | Secure OTP entry, submission (#), and manual lock (*) |
| 6 | Actuator | TowerPro SG90 / MG90S Micro Servo Motor | Mechanical deadbolt locking mechanism (0° Locked, 90° Unlocked) |
| 7 | Acoustic Transducer | 5V Piezo Electric Buzzer | Keypress clicks, success chimes, and intruder alarm siren |
| 8 | USB-UART Programmer | FTDI FT232RL USB-to-TTL Adapter | Flashing firmware to ESP32-CAM (with GPIO 0 to GND jumper) |
| 9 | Power Supply | 5V 2A DC Regulated Power Supply / DC to DCBuck Converter | Stable rail power to prevent brownout dips during Wi-Fi transmissions |
ESP32 DEV MODULE
┌──────────────────┐
│ 3V3 GND ├─── [Common Ground Bus]
│ EN GPIO23
│ GPIO36 GPIO22 ├─── [I2C LCD SCL]
│ GPIO39 GPIO1
│ GPIO34 GPIO3
│ GPIO35 GPIO21 ├─── [I2C LCD SDA]
[Keypad Col 3] ───┤ GPIO32 GND
[IR Sensor OUT] ───┤ GPIO33 GPIO19 ├─── [Buzzer Positive (+)]
[Keypad Col 2] ───┤ GPIO25 GPIO18 ├─── [Servo PWM / Signal]
[Keypad Col 1] ───┤ GPIO26 GPIO5
[Keypad Row 4] ───┤ GPIO27 GPIO17
[Keypad Row 3] ───┤ GPIO14 GPIO16 ├─── [Keypad Row 2] (Strapping Safe)
[Keypad Row 1] ───┤ GPIO13 GPIO4 ├─── [Keypad Col 4]
│ GND GND
│ VIN (5V) 3V3
└──────────────────┘
| Peripheral | Peripheral Pin | Main ESP32 Pin | Logic Voltage | Engineering Notes |
|---|---|---|---|---|
| IR Proximity Sensor | VCC |
VIN / 5V |
5V / 3.3V | Adjust onboard potentiometer for ~25–40 cm detection distance |
GND |
GND |
0V (GND) | Common ground rail | |
OUT |
GPIO 33 |
3.3V (RTC) | RTC EXT0 Wakeup Pin. Armed with RTC internal pull-up in sleep | |
| I2C LCD 16x2 | VCC |
VIN / 5V |
5V | LCD backlight requires 5V for crisp contrast (Addr: 0x27) |
GND |
GND |
0V (GND) | Common ground rail | |
SDA |
GPIO 21 |
3.3V / 5V | I2C Serial Data line | |
SCL |
GPIO 22 |
3.3V / 5V | I2C Serial Clock line | |
| Servo Motor | VCC (Red) |
VIN / 5V |
5V | Power from 5V rail (peak currents can exceed 500mA) |
GND (Black/Brown) |
GND |
0V (GND) | Common ground rail | |
PWM (Orange/Yellow) |
GPIO 18 |
3.3V PWM | 0° = Vault Locked, 90° = Vault Unlocked |
|
| Piezo Buzzer | Positive (+) |
GPIO 19 |
3.3V PWM | Driven via ESP32 tone() frequency modulation |
Negative (-) |
GND |
0V (GND) | Common ground rail | |
| 4x4 Keypad | Row 1 (R1) |
GPIO 13 |
Digital IO | |
Row 2 (R2) |
GPIO 16 |
Digital IO | GPIO 16 used instead of GPIO 12 to prevent boot-loop strapping faults | |
Row 3 (R3) |
GPIO 14 |
Digital IO | ||
Row 4 (R4) |
GPIO 27 |
Digital IO | ||
Col 1 (C1) |
GPIO 26 |
Digital IO | ||
Col 2 (C2) |
GPIO 25 |
Digital IO | ||
Col 3 (C3) |
GPIO 32 |
Digital IO | ||
Col 4 (C4) |
GPIO 4 |
Digital IO |
ESP32-CAM AI-THINKER
┌────────────────┐
[Stable 5V 2A] ─┤ 5V GND ├── [Common Ground]
│ 3V3 GPIO12
│ IO16 GPIO13
[FTDI GND] ─────┤ IO0 GPIO15
│ GND GPIO14
│ VCC GPIO2
[FTDI TX] ──────┤ U0R (RX) GPIO4 ── [Onboard High-Power Flash LED]
[FTDI RX] ──────┤ U0T (TX) GND
└────────────────┘
- Brownout Protection: Hardware brownout detector is disabled in software (
WRITE_PERI_REG(RTC_CNTL_BROWN_OUT_REG, 0)) to maintain stability during Wi-Fi transmission bursts. - Camera Sensor: OV2640 configured with SCCB bus (
SIOD: GPIO 26,SIOC: GPIO 27).
stateDiagram-v2
[*] --> STATE_SLEEP
STATE_SLEEP --> STATE_SCANNING : IR Proximity Trigger (RTC EXT0 on GPIO 33)
state STATE_SCANNING {
[*] --> RequestServerScan
RequestServerScan --> EvaluateResponse
EvaluateResponse --> Recognized : JSON status == "RECOGNIZED"
EvaluateResponse --> Unknown : JSON status == "UNKNOWN"
EvaluateResponse --> NoFaceOrError : status == "NO_FACE" / Error
}
Recognized --> STATE_OTP_INPUT : LCD "Enter OTP" & Beep Success
Unknown --> STATE_ALARM : Sound Siren Alarm & Alert Telegram
NoFaceOrError --> STATE_SLEEP : LCD "No Face Detected" ➡️ Deep Sleep
state STATE_OTP_INPUT {
[*] --> Await4DigitKeypad
Await4DigitKeypad --> VerifyWithServer : Press '#' key
VerifyWithServer --> OtpValid : Server response authenticated == True
VerifyWithServer --> OtpInvalid : Server response authenticated == False
Await4DigitKeypad --> OtpTimeout : 45-second timeout exceeded
}
OtpValid --> STATE_UNLOCKED : Rotate Servo 90°
OtpInvalid --> STATE_SLEEP : LCD "Invalid OTP" ➡️ Deep Sleep
OtpTimeout --> STATE_SLEEP : LCD "OTP Timeout" ➡️ Deep Sleep
state STATE_UNLOCKED {
[*] --> AutoLockCountdown : Start 120-second timer
AutoLockCountdown --> LockTriggered : 120s expired OR user pressed '*'
LockTriggered --> LockServo : Rotate Servo 0°
}
LockServo --> STATE_SLEEP : Re-arm RTC EXT0 & Enter Deep Sleep
STATE_ALARM --> STATE_SLEEP : 3s Alarm Finished ➡️ Deep Sleep
The server runs on OpenCV 5.0 Deep Neural Networks, replacing legacy Haar Cascades with state-of-the-art CNN architectures:
[Raw Video Frame (640x480)]
│
▼
┌─────────────────────────┐
│ cv2.FaceDetectorYN │ ──▶ Ultra-fast CNN Face Detection (YuNet)
│ (YuNet 2023mar ONNX) │ Extracts bounding box [x, y, w, h] + 5 facial landmarks
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ cv2.FaceRecognizerSF │ ──▶ Landmark alignment & crop (recognizer.alignCrop)
│ (SFace 2021dec ONNX) │ ──▶ Deep 128-dimensional embedding extraction (recognizer.feature)
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ Cosine Distance Matcher │ ──▶ Compares query vector against enrolled embeddings
│ Threshold: ≥ 0.363 │ score = recognizer.match(feat1, feat2, FR_COSINE)
└────────────┬────────────┘
│
┌───────┴───────┐
▼ ▼
[Score ≥ 0.363] [Score < 0.363]
RECOGNIZED UNKNOWN INTRUDER
- Models Stored in
server/models/:face_detection_yunet.onnx(~230 KB): Compact, high-speed mobile-grade CNN detector.face_recognition_sface.onnx(~37 MB): High-accuracy sphere-face neural network embedding extractor.
- Automatic Cache Download: If models are missing upon launch, the server automatically downloads them from the official OpenCV HuggingFace CDN.
cam/
├── platformio.ini # Dual-environment PlatformIO config (esp32cam & esp32main)
├── README.md # Complete system documentation, pinouts & setup guide
├── include/ # PlatformIO shared header directory
├── lib/ # PlatformIO local library directory
├── src/
│ ├── cam_main.cpp # Firmware for ESP32-CAM Node (MJPEG /stream, /capture, /flash)
│ └── controller_main.cpp # Firmware for Main ESP32 Controller (IR, Keypad, LCD, Servo, Buzzer)
└── server/
├── config.py # Server configuration (IPs, Ports, Telegram Token & Chat ID)
├── face_recognition_server.py # OpenCV 5.0 DNN Recognition + Flask REST API + Zeroconf mDNS
├── requirements.txt # Python dependencies (opencv-python, Flask, requests, zeroconf)
├── models/ # ONNX Neural Network Model weights (YuNet + SFace)
│ ├── face_detection_yunet.onnx
│ └── face_recognition_sface.onnx
├── faces/ # Enrolled authorized face photos (e.g. John_Doe.jpg)
└── snapshots/ # Timestamped capture logs from access triggers
Triggered by the Main ESP32 when the IR proximity sensor detects a person.
- Response (Recognized):
{ "status": "RECOGNIZED", "person": "Roman Shrestha", "message": "Face recognized! OTP sent to Telegram." } - Response (Unknown Intruder):
{ "status": "UNKNOWN", "person": "Unknown", "message": "Unknown face detected! Warning alert sent to Telegram." } - Response (No Face in Frame):
{ "status": "NO_FACE", "message": "No face detected in camera frame." }
Triggered when the user submits a 4-digit code on the 4x4 keypad (#).
- Request Payload:
{ "otp": "4829" } - Response (Success):
{ "authenticated": true, "person": "Roman Shrestha", "message": "Access Granted" }
Reports state transitions (e.g., UNLOCKED or LOCKED) to log timestamps to Telegram.
GET /stream: Real-time multipart MJPEG video stream (multipart/x-mixed-replace).GET /capture: Captures and returns a single JPEG still frame.GET /flash?state=1|0: Remotely turns ON (1) or OFF (0) the high-power Flash LED.GET /status: Health ping returning{"status":"ok", "device":"ESP32-CAM"}.
- VS Code with the PlatformIO IDE extension installed.
- Python 3.9+ with
pip.
- Connect your FTDI Adapter to the ESP32-CAM:
- FTDI
VCC (5V)➡️ ESP32-CAM5V - FTDI
GND➡️ ESP32-CAMGND - FTDI
TX➡️ ESP32-CAMU0R - FTDI
RX➡️ ESP32-CAMU0T - Connect a jumper wire between
GPIO 0andGND(puts board into flashing mode).
- FTDI
- Press the RST button on the back of the ESP32-CAM.
- In your VS Code terminal, run:
pio run -e esp32cam --target upload
- Once it says
[SUCCESS], remove theGPIO 0toGNDjumper wire and press RST.
- Plug the Main ESP32 Dev Board into your PC via standard USB cable.
- Ensure the Serial Monitor is closed (to avoid COM port access conflicts).
- In your terminal, run:
pio run -e esp32main --target upload
- Open Telegram on your phone or PC and message
@BotFather. - Send
/newbot, choose a name (e.g.,FaceVault), and obtain yourBOT_TOKEN. - Message
@userinfobotor@raw_data_botto retrieve your numericalCHAT_ID. - Open
server/config.pyand enter your credentials:TELEGRAM_BOT_TOKEN = "YOUR_BOT_TOKEN_HERE" TELEGRAM_CHAT_ID = "YOUR_NUMERICAL_CHAT_ID_HERE"
⚠️ Critical Activation Step: Open Telegram, search for your newly created bot username, and tapSTART(or send/start). Telegram requires users to initiate the conversation before a bot can message them.
- Open a terminal in the
server/directory:cd server pip install -r requirements.txt - Start the master recognition server:
python face_recognition_server.py
- The server will:
- Automatically broadcast
esp32server.localover mDNS viazeroconf. - Resolve
esp32cam.localto connect to the video stream. - Start the live OpenCV GUI stream window.
- Listen for REST API triggers on port
5000.
- Automatically broadcast
When the OpenCV live stream window is focused on your desktop:
| Hotkey | Action | Description |
|---|---|---|
s |
Enroll New Face | Captures the current live frame, prompts you in the terminal for the person's name (e.g., John_Doe), saves the image to server/faces/, and re-indexes deep embeddings on the spot. |
f |
Toggle Flash LED | Sends a remote HTTP command to turn the ESP32-CAM's high-power flash LED ON/OFF over Wi-Fi. |
q |
Quit Server | Safely stops the video stream and closes the window. |
- Cause: The PlatformIO Serial Monitor or another terminal is actively holding the COM port open.
- Fix: Click the Trash Can icon (🗑️ Kill Terminal) on the bottom right in VS Code to terminate the active Serial Monitor, then run the upload command again.
- Check Sensor Logic Level: Standard IR sensors drop to
0V(LOW) when triggered. If using a PIR sensor that outputs3.3V(HIGH), update Line 29 insrc/controller_main.cppto#define IR_ACTIVE_LEVEL HIGH. - Calibrate Sensitivity: Rotate the small blue potentiometer on the IR sensor module until the detection LED turns ON only when a hand is within 20–40 cm.
- Windows Firewall: If your Wi-Fi is set to "Public Network", Windows Defender Firewall blocks incoming connections to port 5000. In Windows Settings ➡️ Network & Internet ➡️ Wi-Fi, change your network profile to Private Network.
- Auto-Discovery: The Python server automatically broadcasts
esp32server.localusingzeroconf, enabling the ESP32 to adapt dynamically across router IP reassignments.
- Contrast Adjustment: Rotate the small potentiometer on the back of the I2C backpack with a small flathead screwdriver until the characters become crisp and visible.
- Fix: Open Telegram, search for your bot's username (e.g.,
@FaceVault_v1_Bot), and clickSTART. Telegram prevents bots from messaging users who haven't initiated contact.
- OpenCV Zoo: YuNet and SFace models provided by the OpenCV Zoo Project under Apache 2.0.
- PlatformIO & Espressif: Built with the Arduino-ESP32 core.
- Author: Developed as an IoT Anti-Spoofing Biometric Security Architecture.