ShotSense

What it is

• A computer vision app that watches basketball shot videos and automatically scores makes vs. misses, then breaks down shooting form (elbow angle, knee angle, release angle) frame by frame.
• Combines a Faster R-CNN detector for tracking the ball and hoop with OpenPose for body pose estimation, plus a curve-fit trajectory model for estimating release angle from the ball's path.
• Started as coursework and grew into a research project. The trajectory-tracing and curve-fitting work behind it was published at IEEE MIT URTC 2024.

Use cases

• Give a player or coach objective, repeatable feedback on shooting form instead of relying on eyeballing a video.
• Batch-score practice footage to track makes/misses and form metrics (release angle, elbow/knee angle) over time.
• Pull detection results programmatically through the JSON API to feed into another training or analytics tool.

Tech stack

• Backend: Python, Flask, Gunicorn
• Computer vision / ML: TensorFlow 1.x (Faster R-CNN object detection), OpenPose (pose estimation), OpenCV (frame processing), NumPy/SciPy (trajectory curve fitting), scikit-learn (data analysis), Matplotlib
• Frontend: Flask templates (Jinja2), HTML/CSS
• Deployment: Heroku (Procfile, Aptfile for buildpack system deps), Travis CI

Project structure

app.py                          Flask app entrypoint and routes (upload, detection, API)
basketball_path_curve_fit.py    Parabolic curve fitting for ball trajectory / release angle
research/                       Offline research pipeline behind the URTC 2024 paper
  cv2_frameshow.py              Detection, tracking, Bounce-Around + SWISAC fitting, make/miss
  logger_util.py                Per-video debug logger
  results/                      Summary CSVs and sample per-video outputs
src/
  app_helper.py                 Video/image processing pipeline glue
  config.py                     Shared shooting-result state (attempts, makes, angles)
  utils.py                      Detection and drawing helpers
inference_graph/
  frozen_inference_graph.pb     Trained Faster R-CNN detection model (frozen graph)
OpenPose/
  openpose/                     Compiled OpenPose Python bindings
  models/                       OpenPose pose model weights and getModels scripts
static/
  css/, img/                    App styling and README/UI images
  detections/                   Sample detection and trajectory-fit output images
  uploads/                      Sample videos/images used by the app
templates/                      Jinja2 templates for the web UI (index, results, analysis, detection)

Setup

Clone the repo:

git clone https://github.com/vasishtk1/ShotSense.git
cd ShotSense

Install dependencies:

pip install -r requirements.txt

A GPU with CUDA is required to run the video analysis pipeline, since OpenPose needs a CUDA device.

Run the app:

python app.py

Detection API

Submit a POST request to /detection_json with image as the key and your image as the value to get raw JSON detection results back.

Requirements

• Python 3.6
• CUDA-capable GPU (for OpenPose / video analysis)
• See requirements.txt for pinned package versions

requirements.txt intentionally pins older packages (Flask 2.0.0, Werkzeug 0.16.0, Pillow 6.0.0, TensorFlow 1.15.2) to match the OpenPose/TensorFlow 1.x GPU setup this project depends on. Some of those versions have known CVEs. Upgrade them if you deploy this somewhere other than a local, controlled environment, but note that bumping TensorFlow will likely require reworking the OpenPose and detection graph integration.

Research pipeline

The web app above is the interactive demo. The research work behind the paper lives in research/, as a separate offline pipeline that batch-processes shot videos:

• Detects the ball and rim with the same Faster R-CNN graph, tracks each across frames, and caches the detections so the analysis can be rerun without redoing perception.
• Fits the ball's free-flight trajectory with two methods: Bounce-Around, which grows a window outward from the apex of the arc while the fit stays above an R² threshold, and SWISAC, a RANSAC variant that samples contiguous windows in time rather than random subsets, since detection outliers here are temporally clustered.
• Estimates the release angle from the derivative of the fitted parabola, and predicts on- or off-course by extrapolating that parabola into the rim — which is what makes a prediction possible even though no detector can see the ball once it is inside the net.

See research/README.md for how to run it, what each output plot shows, and the known limitations.

Background and credits

This project builds on the original AI-basketball-analysis project by chonyy, extended here with basketball trajectory tracing and curve fitting for release-angle estimation. Shooting pose analysis is powered by OpenPose, which is licensed for noncommercial research use only, see LICENSE, which mirrors CMU's OpenPose License.

The trajectory-tracing and curve-fitting work is described in:

Kartik, Vasisht, and Ross Greer. "Optimizing Basketball Shot Trajectory using Image Segmentation Techniques for Training Feedback." 2024 IEEE MIT URTC. IEEE, 2024.