Skip to content
casdadPublic

About

Multimodal visual learning demo with YOLOv8, Flask, Socket.IO, and optional Zhipu AI.

Topics

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

AI Study

AI Study(智视学习)是一个基于 Flask、YOLOv8 和智谱 AI 的多模态视觉学习演示平台。它可以从摄像头或上传的视频中实时识别常见物体,并在用户自行配置 API Key 后提供中文物体讲解和对话功能。

本仓库面向本地学习和演示。它不是生产级监控、安防或自动决策系统。

Features

  • YOLOv8 摄像头与视频文件实时检测
  • 浏览器中的 MJPEG 检测画面和物体列表
  • 可选的智谱 AI 中文物体讲解与聊天
  • Flask-SocketIO 实时状态更新
  • 上传文件扩展名、体积和访问路径限制
  • 未配置 API Key 时仍可独立运行视觉检测

Architecture

flowchart LR
    A[Camera or uploaded video] --> B[OpenCV frame capture]
    B --> C[YOLOv8 detection]
    C --> D[Annotated MJPEG stream]
    C --> E[Detected object names]
    E --> F{ZHIPU_API_KEY configured?}
    F -->|Yes| G[Zhipu AI explanation and chat]
    F -->|No| H[Local detection only]
    D --> I[Browser UI]
    G --> I
    H --> I
Loading

Project Structure

AI-study/
├── app.py                  # Flask/Socket.IO server and inference pipeline
├── templates/index.html    # Browser interface
├── static/                 # Front-end assets and ignored runtime uploads
├── scripts/smoke_inference.py
├── tests/                  # Security and route smoke tests
├── yolov8n.pt              # Small baseline YOLOv8 checkpoint
├── requirements.txt
├── requirements-dev.txt
├── .env.example
└── SECURITY.md

Requirements

  • Python 3.10 or newer
  • A webcam or a supported video file for interactive detection
  • CPU is supported; a CUDA-compatible PyTorch installation is optional
  • Internet access is required only for智谱 AI calls and front-end CDN assets

Main dependencies: Flask, Flask-SocketIO, OpenCV, Ultralytics, NumPy, Pillow, PyTorch and zhipuai.

Installation

git clone https://github.com/casdad/AI-study.git
cd AI-study
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Linux/macOS activation:

source .venv/bin/activate

For GPU acceleration, install the PyTorch build matching your CUDA environment before installing the remaining requirements. See the official PyTorch installation guide.

Configuration

Visual detection works without an API key. To enable智谱 AI features, set the key in the process environment; never write it into source code:

$env:ZHIPU_API_KEY = "your-key"
$env:FLASK_SECRET_KEY = "a-long-random-value"

Supported settings are documented in .env.example. The application does not automatically load .env; set variables in the shell or through your process manager.

Usage

Start the local application:

python app.py

Open http://127.0.0.1:7000, choose the default webcam or upload an MP4/AVI/MOV/MKV/WEBM video, and select 开始识别.

The safe default binds only to 127.0.0.1. Do not expose this development server to the public Internet. If remote access is required, add authentication, HTTPS, a production WSGI deployment and a reverse proxy first.

Model

The bundled yolov8n.pt checkpoint is used through the Ultralytics API. Input frames are decoded by OpenCV; the model returns bounding boxes, COCO class labels and model confidence values. Annotated frames are rendered by Ultralytics and streamed to the UI.

Bundled checkpoint SHA-256: f59b3d833e2ff32e194b5bb8e08d211dc7c5bdf144b90d2c8412c47ccfc83b36.

Override the checkpoint path when needed:

$env:YOLO_MODEL_PATH = "C:\path\to\your-model.pt"
python app.py

Only load checkpoints from trusted sources. PyTorch checkpoints may contain executable serialized objects.

Testing

Install development dependencies and run:

python -m compileall app.py scripts tests
python -m pytest -q
python scripts/smoke_inference.py --model yolov8n.pt --device cpu

The smoke inference uses a generated blank frame only to verify model loading, preprocessing, inference and postprocessing. It is not a detection-quality benchmark and produces no fabricated demo screenshot.

Verified on 2026-09-03 with Python 3.10 and CPU inference:

  • Dependency consistency: passed (pip check)
  • Python compilation: passed
  • Automated tests: 8 passed
  • Local server smoke test: HTTP 200
  • Blank-frame model smoke test: model load 39.733 ms, pipeline inference 4183.624 ms, 0 detections

These timings describe one local smoke run only; they are not FPS or accuracy claims.

Inputs and Outputs

Inputs:

  • Webcam index 0
  • Uploaded MP4, AVI, MOV, MKV or WEBM video, up to 250 MB by default
  • Optional browser chat messages

Outputs:

  • MJPEG stream with YOLO bounding boxes
  • Detected COCO object class names
  • Optional智谱 AI descriptions and chat responses

Runtime uploads are stored under static/uploads/, ignored by Git, and should be deleted when no longer needed.

Security and Privacy

  • API keys are accepted only through environment variables.
  • Uploaded file identifiers cannot be used to request arbitrary local paths.
  • AI and user-generated text is HTML-escaped before rendering.
  • The server defaults to localhost with debug mode disabled.
  • Video frames, detected labels and chat content may be sensitive. Review the智谱 AI privacy and retention terms before enabling API-backed features.

See SECURITY.md for reporting guidance.

Limitations

  • The global in-memory state is intended for one local interactive session, not concurrent multi-user deployment.
  • There is no user authentication, database or persistent conversation storage.
  • Recognition is limited by the bundled COCO-trained model and does not imply domain-specific accuracy.
  • The repository does not include a redistributable example video, so users must provide their own non-sensitive media. -智谱 AI features require a valid account, network access and may incur provider charges.

License

No project-wide license has been selected. Public visibility does not by itself grant reuse rights. Third-party libraries and the YOLO checkpoint remain subject to their respective licenses and terms.

About

Multimodal visual learning demo with YOLOv8, Flask, Socket.IO, and optional Zhipu AI.

Topics

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages