Skip to content

Project.md

Latest commit

 

History

History
254 lines (212 loc) · 12.7 KB

Project.md

File metadata and controls

254 lines (212 loc) · 12.7 KB
Raw

Image Captioning Meme Generator — Project Spec

Overview

Revive and complete an image captioning project. The original prototype used two Hugging Face transformer models — an object detector and an image captioner — to analyze images. The goal now is to build a working, end-to-end web application — a FastAPI backend serving the ML pipeline and a React frontend — that takes an image, generates a caption, and turns that caption into a meme.

What already worked (from the original prototype)

  • Object detection with facebook/detr-resnet-50 — returns labels, scores, and bounding boxes (e.g. detected cat, couch, remote, sports ball, person with >0.99 confidence on test images).
  • Image captioning with Salesforce/blip-image-captioning-large — generated natural-language captions like "woman sitting on the beach with her dog and a cell phone."

Both models ran via the Hugging Face transformers pipeline and produced good results independently.

Problems that killed the original project (must be solved)

  1. Combining the two models was awkward — the original plan tried to merge detection + captioning and got stuck. Decision: don't force a merge. Use BLIP captioning as the primary path; object detection is optional enrichment.
  2. GPT/LLM API integration was too complex — they couldn't get the caption rewriting working. This step is core, not optional — turning a plain caption into a meme/joke is the whole point of the project. The fix for the original blocker is a clean, well-abstracted integration behind one function, not skipping the feature. (A no-key dev fallback may exist purely for local testing, but the shipped product requires the LLM step.)
  3. Writing text onto the image was never figured out — meme rendering wasn't implemented.
  4. No user interface — it stayed a script. Solution: a FastAPI backend exposing the pipeline as endpoints, with a React frontend for upload, preview, and download.
  5. A web scraper broke — the original used old.reddit + BeautifulSoup for image sourcing and it broke (markup changes, rate limits). Replace scraping entirely with the official Unsplash API — structured JSON, free key, no breakage, legally clean for reuse. See "Image sourcing" below.

Target architecture

┌─────────────────────────────┐         ┌──────────────────────────────────────┐
│   React frontend (Vite)     │         │          FastAPI backend             │
│                             │         │                                      │
│  - image upload / drag-drop │  POST   │  POST /api/caption                   │
│  - "fetch from Unsplash"    │ ──────► │    multipart image in                │
│  - preview & loading state  │  image  │    ML pipeline runs (see below)      │
│  - download button          │ ◄────── │    meme image out (PNG bytes/base64) │
│                             │         │                                      │
│                             │  GET    │  GET /api/unsplash?query=...         │
│                             │ ──────► │    returns candidate image(s)        │
└─────────────────────────────┘         └──────────────────────────────────────┘
                                                       │
                              ML pipeline (inside the backend, per request):
                                                       │
                                Image ──► BLIP captioning ──► raw caption
                                  │         (Salesforce/blip-image-captioning-large)
                                  │
                                  ├──► (optional) DETR detection ──► detected objects
                                  │         (facebook/detr-resnet-50)
                                  ▼
                                Claude API (Haiku) ──► meme-style caption   [CORE]
                                  ▼
                                Pillow renderer ──► caption drawn on image
                                  ▼
                                returned to frontend

Models are loaded once at backend startup, not per request.

Tech stack

Backend (Python 3.10+)

  • fastapi + uvicorn for the API server
  • transformers + torch for the models
  • Pillow for drawing text on images (this solves the "write caption on image" problem they never cracked)
  • anthropic SDK for the meme rewrite (Claude Haiku), key from ANTHROPIC_API_KEY
  • python-dotenv to load .env (works everywhere, not just in an IDE terminal)
  • httpx (or requests) for the Unsplash fetch
  • python-multipart for handling file uploads in FastAPI

Frontend

  • React via Vite (npm create vite@latest -- --template react)
  • Plain fetch/axios to call the backend — no need for a heavy state library
  • Minimal, clean UI: drag-and-drop upload, an Unsplash search box, a preview area with a loading state, and a download button

Why this over Gradio: a real backend/frontend split is a stronger portfolio piece, matches the FastAPI + React stack already in use elsewhere, and keeps the ML pipeline cleanly separated from the UI so either side can change independently.

Build steps

Suggested layout:

image-captioning/
  backend/
    main.py            # FastAPI app, endpoints, CORS, startup model load
    caption.py         # BLIP captioning
    detect.py          # DETR detection (optional)
    memeify.py         # Claude rewrite + Pillow rendering
    unsplash_source.py # Unsplash fetch
    fonts/             # bundled Impact-style TTF
    samples/           # curated sample images
    requirements.txt
    .env.example
  frontend/
    (Vite React app)
  README.md

Backend

  1. Scaffold — backend/ with requirements.txt, .env.example, FastAPI app in main.py with CORS enabled for the Vite dev origin (http://localhost:5173).
  2. Captioning module — caption.py: load BLIP once at startup (FastAPI lifespan/startup event), function generate_caption(image) -> str.
  3. Detection module (optional) — detect.py: load DETR once, function detect_objects(image) -> list[label], filtered by a score threshold (~0.9).
  4. Meme-text module — memeify.py:
    • rewrite_caption(caption, objects=None) -> str — core feature. Calls the Anthropic API (anthropic SDK, Claude Haiku) to rewrite the plain caption as a short, funny meme caption. Keep the prompt simple but make the joke the point ("rewrite this image caption as a short, punchy, funny meme caption"). Dev-only fallback returns the caption unchanged if no key is set, but the product depends on this step.
    • render_meme(image, text) -> image — uses Pillow to draw bold uppercase text with a black outline, top or bottom, auto-wrapping and auto-scaling font to image width. Bundle a font file (e.g. a free Impact-style TTF) so it works cross-platform.
  5. Unsplash module (optional) — unsplash_source.py: function fetch_images(query, count) -> list[image] using the official Unsplash API (/photos/random or /search/photos). Key from UNSPLASH_ACCESS_KEY.
  6. Endpoints in main.py:
    • POST /api/caption — accepts a multipart image upload, runs the full pipeline, returns the rendered meme (PNG bytes, or base64 + the generated caption text as JSON).
    • GET /api/unsplash?query=... — returns candidate image(s) from Unsplash.
    • GET /api/health — simple readiness check (useful since model load is slow).

Frontend 7. Scaffold — frontend/ via Vite React. Set the backend base URL (e.g. a Vite env var VITE_API_URL=http://localhost:8000). 8. UI — drag-and-drop / file-picker upload, optional Unsplash search box, a "Generate meme" button, a preview area with a clear loading state (model inference on CPU is slow — show a spinner), and a download button for the result. 9. Wire it together and test on the sample images (cats on couch, dog on beach) plus a live Unsplash fetch.

Key implementation requirements

  • Load models once at backend startup — use FastAPI's lifespan/startup event; model loading is slow, never reload per request. Expose /api/health so the frontend can tell when the backend is actually ready.
  • CPU-friendly — the original ran on CPU (Intel Xeon). Don't assume a GPU; use device_map / .to() only if CUDA is available.
  • Enable CORS on the backend for the Vite dev origin (http://localhost:5173) so the React app can call the API in development.
  • LLM caption rewriting is core — the meme/joke generation is the product, via the Anthropic Claude API. A no-key fallback exists only for local dev; the shipped app requires ANTHROPIC_API_KEY. Load it with python-dotenv.
  • Self-contained font — don't rely on system fonts being installed.
  • Image sourcing via official Unsplash API, not scraping — key from UNSPLASH_ACCESS_KEY, handle the rate limit (50 req/hr on the demo tier), respect Unsplash attribution requirements if displayed publicly.
  • Pin major dependency versions in requirements.txt.

Environment variables

Create a backend/.env file (real keys, gitignored — never commit) based on this backend/.env.example template (placeholders, safe to commit):

# backend/.env.example

# --- Required ---
# Anthropic API key for the caption-to-meme rewrite (Claude Haiku).
# Get one at https://console.anthropic.com/settings/keys
ANTHROPIC_API_KEY=sk-ant-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# --- Optional ---
# Unsplash Access Key for the "fetch from Unsplash" image source.
# Register an app at https://unsplash.com/developers (free tier: 50 req/hr)
UNSPLASH_ACCESS_KEY=your_unsplash_access_key_here

# --- Optional tuning (sensible defaults if omitted) ---
# Anthropic model used for the rewrite
ANTHROPIC_MODEL=claude-haiku-4-5-20251001
# Allowed CORS origin for the frontend dev server
FRONTEND_ORIGIN=http://localhost:5173

The frontend needs only the backend URL, set in frontend/.env:

# frontend/.env
VITE_API_URL=http://localhost:8000

Rules:

  • Commit .env.example (placeholders only); add .env to .gitignore.
  • Load backend vars with python-dotenv (load_dotenv() at startup) so they work from any terminal, not just an IDE-injected one.
  • The app must run with only ANTHROPIC_API_KEY set; Unsplash features degrade gracefully (or are hidden) when its key is absent.

Image sourcing

Two ways to feed images into the pipeline:

  1. File upload (primary, always available) — user uploads an image in the UI. Ship a small curated sample folder too, so the demo works with zero setup.

  2. Unsplash fetch (optional, replaces the original broken scraper) — pull fresh images by search term to caption and meme-ify.

    • Use the official Unsplash API: /photos/random?query=<term> or /search/photos?query=<term>. Free developer key from unsplash.com/developers, read from UNSPLASH_ACCESS_KEY.
    • Download the image bytes from the returned urls.regular field.
    • Respect the demo-tier rate limit (50 requests/hour) and Unsplash's attribution guidelines if images are shown publicly.

    This is the same "pull a fresh image on demand" goal as the original Reddit scraper, but on a stable, official, legally-clean API instead of HTML scraping that breaks without warning.

Deliverables

  • A FastAPI backend runnable with uvicorn main:app --reload (from backend/).
  • A React (Vite) frontend runnable with npm run dev.
  • backend/requirements.txt and a frontend/package.json.
  • .env.example documenting ANTHROPIC_API_KEY (required) and UNSPLASH_ACCESS_KEY (optional).
  • README.md with setup and run instructions for both the backend and frontend, how to configure the keys, and the local URLs (backend on :8000, frontend on :5173).
  • Clean module separation in the backend (caption / detect / memeify / unsplash / main).

Stretch goals (only if core works)

  • Let the user edit the generated caption in the UI before re-rendering the meme.
  • Top-text / bottom-text split.
  • Multiple meme template styles, selectable in the frontend.
  • Show detected objects as tags alongside the caption.
  • Stream/poll backend readiness so the UI disables "Generate" until models load.
  • Dockerize backend + frontend for one-command startup.