A robotics and AI project exploring how autonomous agents can perceive, remember, reason and act inside physical environments using ROS2, AI models, semantic vector memory and agentic planning workflows.
| 📦ROS2 Package/📂Folder | Description |
|---|---|
📦 aws-robomaker-small-warehouse-world (submodule) |
Simulation Environment • AWS RoboMaker small warehouse is used as the Gazebo world in which the robot is launched and tested. |
📦 Depth-Anything-3-ROS2 (submodule) |
Depth Estimation Layer • Performs monocular depth estimation from RGB camera input without requiring dedicated depth sensors. • Raw depth output is consumed by vision_spatial_mapping.• Normalized depth maps are consumed by Rviz for better visualization. |
📦 my_robot_description |
Robot Description & Kinematics • Unified robot description format is an XML format file used to describe robot’s physical/geometry & visual features(links & joints). • Used for generating kinematic model (TF frames) • Used for describing sensor (camera, LiDAR) & actuator (diff-drive, joint state publisher) placement. |
📦 my_robot_bringup |
Configuration & Launch files • It contains the configuration for Rviz, Nav2, gazebo_ros_bridge. • RViz is used for 3D visualization of robot model & TF verification. • Gazebo bridge package provides a network bridge, which enables the exchange of messages between ROS & Gazebo Transport. • Launch files for ROS2 nodes (RViz, gz-sim, DA3, etc.) |
📦 my_robot_interfaces |
Custom Interfaces • Custom ROS2 message and service interface definitions. |
📦 fastvlm_caption_rqt_overlay |
Real-Time Caption Visualization • RQT plugin overlaying FastVLM-generated captions on live camera images for real-time visualization. |
📦 vision_spatial_mapping |
Spatial Mapping Pipeline • Receives camera frames, DA3 depth output and TF transforms. • Extracts depth from the central pixel region to estimate 3D coordinates in the map frame. • The camera field-of-view is intentionally narrowed to reduce spatial estimation error introduced by central-region depth approximation. • Uploads the image to Supabase File Storage and publishes the image reference and spatial coordinates to the RabbitMQ message broker. |
📂 image_caption_publisher |
Live captioning • Sends a HTTP/2 request with a configurable timeout, containing image bytes to the backend server. • The captioning loop runs at 2 Hz. Only one caption request is allowed at a time, preventing multiple pending requests from accumulating. Each new request uses the latest available image frame. As a tradeoff, some intermediate scenes may be skipped if caption generation takes longer than the request interval. |
📂robot_agent |
Agentic Planning Layer • Implements the core planning, reasoning and tool orchestration loop using LangGraph. The AI agent is powered by a locally running Qwen3.5:9B model. • Uses dedicated prompts and subgraphs with isolated state to enable context engineering, preventing the main agent's context from being bloated with intermediate search results |
| Service | Purpose |
|---|---|
clip_faiss_server |
Visual-Spatial Semantic Memory Backend • v1.0.0 → Uvicorn ASGI Server + FastAPI Backend Framework for automatic generation of interactive OpenAPI/Swagger UI documentation + CLIP for multimodal embedding generation + FAISS for vector storage and semantic retrieval (IndexFlatIP, cosine similarity) through REST APIs + Pydantic for providing schema validation.• v2.0.0 → Supports asynchronous event-driven image ingestion via RabbitMQ using MQTT plugin + server-side FastVLM captioning only for semantically distinct scenes + claim-check pattern using Supabase File Storage to support large image payloads (>1MB) |
physical_robot_agent_mcp_server |
Extended Nav2 MCP Server Extends the Nav2 MCP server by integrating a FAISS-based semantic search tool for agent memory retrieval, transforming navigation into a memory-augmented autonomous agent system. |
Setup and installation instructions are currently being documented.