🤖 InterfaceAI

🚀 Project Overview

InterfaceAI is a Chrome extension that turns natural language into action.
Ask it to “set up an AWS instance,” “book a flight,” or “open my calculus lecture,” and it will understand your intent, navigate webpages, and execute tasks directly on your screen.

By combining intent recognition, vision-based action execution, and contextual awareness, InterfaceAI goes beyond existing tools with autonomous webpage exploration and personalized digital profiles—building the next generation of AI agents that make technology work for you.

View the Kanban / Task Board and more Info / Links

---

🔑 Key Features (Planned for MVP)

• 💬 Natural language task execution (e.g., “Set up AWS instance”)
• 👀 Vision-based screen understanding and action execution
• 🌐 Autonomous webpage exploration
• 🧠 Contextual awareness for multi-step tasks
• 🪪 Personalized digital user profiles
• 🔒 Secure and privacy-first agent design

---

🛠️ Technologies

• Frontend: React + Chrome Extension APIs
• Backend / Infra: Flask, SQL
• ML / AI: Google Gemini API, OCR / Vision Models
• Hosting / Deployment: GCP (Cloud Run to easily deploy our docker containers, BigTable, BigQuery)

---

Contributor 👨‍💻

Photo	Contributor	Role & Tools	Notable Contributions
	Sachin Gupta Project Lead	LangGraph · Mem0 · GCP	Ideation
	Adrian Najmi Open Source Dev	Python · JavaScript · TypeScript · Chrome Extension APIs	Action Execution
	Hassan Ibrahim Backend	Python · Flask	Database Design
	Luna Aljammal Backend + DB	Python · Flask · Redis · GCP	Database Integration
	Andrew Wu Frontend	Python · JavaScript · GCP	UI, Frontend Development, Auth, Database
	Mohit Gedela Vision AI	Python · Java · JavaScript	Vision AI Development
	Alex Melnbardis AI	Python · TypeScript · LangGraph · Chrome Extension APIs · GCP	Vision AI
	David Olejniczak CI/CD + Cloud	CGP · Docker · Github Actions	CI/CD
	Anhad Chawla AI Specialist + Solution Architect	Python · LangGraph · GCP · BigTable	Agentic Workflows

---

System Diagrams

Architecture

• Each service is isolated to allow for independant scaling

User Interface

• Frictionless, Clean, Modern (Liquid Glass inspired design)

Langgraph Loop simulating Human Browser Behavior

• Agent can reason based on goal and use/call its tools (goto, click, fill_input, back, etc) to interact with the website
• Session state to track when its looping and force it to do a different approach/technique

Database

• Google Sign-in / Auth
• User level memory to store preferences and information (prefers google over yahoo search, phone number, email, last name, etc)
• General Agent memory (learnings from previous successful/failed sessions and efficiency tips like "instead of going to youtube and typing in search bar" just do a direct "goto(https://www.youtube.com/results?search_query=xyz)"

Vision AI

• Multi-modal model for detecting text and icons (like "gear" icon = settings page) in a screenshot
• Can also explain what is going on in the screen and what the main blocker is
• Used whenever the model gets stuck or for recognizing icons or for elements not part of the DOM/HTML

Action Execution

• Done via a chrome extension content script so it can be done right in your current tab
• Better compared to alternatives like playwright which open in a Chromium app (not Chrome) and within a new window/page
• Provides the functionality to navigate pages, click elements, type into fields, select form options, upload files, scroll, and inspect on-screen content (HTML / DOM)
• When the agent in the backend calls a tool like "click", via the websocket it is routed here and thus the approriate function is called to do the action on the screen

---

For More Information...

Quick Start Guide - Get started in minutes

DevOps Cheatsheet - Docker and local development

CI/CD Documentation - Tests, linters, and formatters

View the Project Roadmap

---

🧪 Local Development (Docker)

Services

• frontend: Chrome Extension (frontend)
• backend: Flask API at http://localhost:5000 (backend/app/main.py)
• vision-ai: Minimal Flask service (vision-ai/service.py)

Run

docker compose up --build

Health checks:

• Backend: GET http://localhost:5000/health
• Relay: POST http://localhost:5000/api/relay with { "message": "hello" }

To stop:

docker compose down

---

🧩 Chrome Extension Frontend

This repo includes a Chrome extension UI in frontend/:

• frontend/manifest.json
• frontend/popup.html
• frontend/popup.js

The popup sends a POST to http://localhost:5000/api/relay and displays the response. CORS and host_permissions are configured to allow this in development.

Load the Extension

1. Open Chrome and go to chrome://extensions.
2. Toggle on "Developer mode" (top-right).
3. Click "Load unpacked" and select the frontend/ directory.
4. Click the extension icon (puzzle piece) → pin "InterfaceAI MVP".
5. Open the popup, type a message, and click "Send to backend".

If you see CORS issues, ensure Docker services are running and the backend is reachable at http://localhost:5000.

---

📁 Repository Layout

interface-ai/
├── docker-compose.yml
├── backend/
│   ├── Dockerfile
│   ├── requirements.txt
│   └── app/
│       └── main.py
├── frontend/
│   ├── manifest.json
│   ├── popup.html
│   └── popup.js
├── vision-ai/
│   ├── Dockerfile
│   └── service.py
└── scripts/

---

🧰 Notes

• Backend CORS allows http://localhost:* and chrome-extension://* for simple dev flow.
• Update frontend/manifest.json host_permissions if backend URL changes.