Clarity

Clarity is a mobile app that helps visually impaired users read text around them. It has two modes depending on how far away the text is.

Near mode is for close-up text such as labels and small print. It reads the detected text aloud, and users can tap an individual line to hear only that line.

Far mode is for distant text such as signs. As this text is often too small for OCR to recognize from the full image, a fine-tuned YOLOv11 model first locates the text regions, which are then cropped and passed to EasyOCR.

_{Mode toggle, Near mode reading a label, and Far mode reading distant text}

How Far Mode Works

Far mode follows this pipeline:

1. YOLOv11 detects regions that may contain text.
2. Nearby boxes on the same line are merged.
3. Each merged region is cropped from the image.
4. EasyOCR reads the text inside each crop.
5. The results are sorted into reading order and returned to the app.

Model Training

EasyOCR’s built-in detector worked well for nearby text, however, it struggled with very distant text

To improve this, I fine-tuned YOLOv11 on approximately 1,200 images from the ICDAR scene-text dataset which was then trained using a Kaggle GPU to locate far text regions in real world scenarios

YOLO handles locating the distant text, while EasyOCR handles reading it.

Tech Stack

• React Native with Expo
• Python and FastAPI
• PyTorch
• Ultralytics YOLOv11
• EasyOCR
• OpenCV

The phone captures images, sends them to the FastAPI backend, and plays the returned text aloud. The backend runs the detection and OCR models before returning the results as JSON.

Running the App

Start the backend:

cd backend

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/uvicorn app.main:app --host 0.0.0.0 --port 8000

Then start the mobile app:

cd mobile
npm install
npx expo start

Set BACKEND_URL in mobile/App.js to your computer’s local IP address. Then scan the QR code using Expo Go to open the app on your phone.