RIMURU - OS reincarnated, AI - RAPHAEL (Great Sage)

AI-Native Linux Distribution

Product Requirements Document (PRD) v1.0

Project Status: Open Source - Seeking Contributors
Repository: github.com/dbisina/rimuru
License: GPL-3.0
Current Phase: Pre-Alpha Development

Vision

Build a Linux distribution where AI is the primary interface, not an assistant. Users control their entire system through natural language, voice, and intelligent automation, while the AI operates as a privileged "super-user" with deep system integration.

The North Star

By 2026, a user should be able to:

Boot the system and say: "Set up my development environment for Rust"
The AI autonomously: installs toolchains, configures editors, clones relevant repos, and opens documentation
The user never touches the terminal or clicks a menu

Two-Track Development Strategy

Track 1: Usable System (6-12 Months)

Goal: Ship a working AI-augmented desktop that users can daily-drive

Deliverables:

Natural language system control
Voice input integration
AI interaction panel (overlay UI)
GUI/CLI/Browser app automation
70-80% success rate on common tasks

Target Users: Power users, developers, early adopters

Track 2: Full Vision (2-4 Years)

Goal: Transform into true AI-native OS with deep kernel integration

Deliverables:

eBPF monitoring layer
Kernel-level resource optimization
Self-healing infrastructure
95%+ autonomous operation
Production-ready for general users

Target Users: Everyone

System Architecture

High-Level Overview

┌──────────────────────────────────────────────────────────┐
│                    USER INTERACTION                       │
│  ┌────────────┬──────────────┬─────────────────────────┐ │
│  │   Voice    │   Search     │   AI Panel (Super+AI)   │ │
│  │  (Whisper) │   (Rofi/Ulauncher) │  (Overlay UI)    │ │
│  └────────────┴──────────────┴─────────────────────────┘ │
└──────────────────────────────────────────────────────────┘
                          ↓
┌──────────────────────────────────────────────────────────┐
│              AI ORCHESTRATION LAYER                       │
│  ┌─────────────────┬──────────────────┬────────────────┐ │
│  │  Intent Parser  │  Goal Planner    │  Model Router  │ │
│  │  (NLU)          │  (Multi-step)    │  (Local only)  │ │
│  └─────────────────┴──────────────────┴────────────────┘ │
│  ┌─────────────────┬──────────────────┬────────────────┐ │
│  │  Memory System  │  Learning Engine │  Safety Guard  │ │
│  │  (Context/Prefs)│  (Pattern adapt) │  (Undo/Verify) │ │
│  └─────────────────┴──────────────────┴────────────────┘ │
└──────────────────────────────────────────────────────────┘
                          ↓
┌──────────────────────────────────────────────────────────┐
│            UNIVERSAL CONTROL PLANE                        │
│  ┌──────────┬──────────┬───────────┬──────────────────┐  │
│  │ Terminal │ GUI Apps │ Browser   │ Creative Suite   │  │
│  │ Control  │ Control  │ Control   │ (API-first)      │  │
│  └──────────┴──────────┴───────────┴──────────────────┘  │
└──────────────────────────────────────────────────────────┘
                          ↓
┌──────────────────────────────────────────────────────────┐
│         PERCEPTION ENGINE (OSWorld-Enhanced)              │
│  ┌──────────────────┬─────────────────┬────────────────┐ │
│  │ Vision Pipeline  │ Accessibility   │ Process Mon.   │ │
│  │ (Screenshot+OCR) │ (AT-SPI2 Tree)  │ (Logs/Status)  │ │
│  └──────────────────┴─────────────────┴────────────────┘ │
│  ┌──────────────────────────────────────────────────────┐│
│  │ OSWorld Integration (Pattern Library + SoM Overlay)  ││
│  └──────────────────────────────────────────────────────┘│
└──────────────────────────────────────────────────────────┘
                          ↓
┌──────────────────────────────────────────────────────────┐
│              LINUX BASE SYSTEM                            │
│  Wayland/Sway | systemd | D-Bus | Standard Linux Stack   │
└──────────────────────────────────────────────────────────┘

User Experience: AI-Native Interaction

Primary Interaction Methods

1. Voice Control (Always Listening - Optional)

Implementation:

Local voice recognition via Whisper.cpp
Wake word: "Raphael" or "Hey Raphael" (configurable)
Privacy-first: All processing on-device

Usage:

User: "Raphael, open my Python project from yesterday"
AI: [Opens VSCode, loads recent project, starts dev server]
AI: "Project loaded. Dev server running on port 3000."

User: "Find all TODO comments in the codebase"
AI: [Searches, creates list, opens in editor]
AI: "Found 23 TODOs. Displayed in search panel."

Settings:

Toggle always-listening mode
Push-to-talk mode (default: Super+V)
Voice feedback on/off
Custom wake words

2. AI Search Panel (Universal Launcher)

Activation: Press Super key (replaces traditional app launcher)

Interface:

┌─────────────────────────────────────────────────────────┐
│  What would you like to do?                             │
│  ┌───────────────────────────────────────────────────┐  │
│  │ Set up Python environment for ML                  │  │
│  └───────────────────────────────────────────────────┘  │
│                                                          │
│  Suggestions:                                            │
│  • Continue editing main.py                              │
│  • Check system updates                                  │
│  • Open yesterday's browser session                      │
│                                                          │
│  Recent Tasks:                                           │
│  ✓ Installed Docker (2 hours ago)                        │
│  ✓ Compiled Rust project (yesterday)                     │
└─────────────────────────────────────────────────────────┘

Capabilities:

Natural language input (not just app names)
Contextual suggestions based on time, recent activity
Shows AI's understanding before execution
Quick confirmation: Enter = Execute, Ctrl+E = Edit plan

3. AI Interaction Panel (Persistent Assistant)

Activation: Super+A or three-finger swipe

Interface: Slide-in panel from right side of screen

┌─────────────────────────────────┐
│  Raphael's Workshop             │
├─────────────────────────────────┤
│                                 │
│  Current Goal:                  │
│  → Installing CUDA drivers      │
│     [==============    ] 70%    │
│                                 │
│  Recent Actions:                │
│  ✓ Downloaded driver package    │
│  ✓ Stopped display manager      │
│  → Running installer...         │
│                                 │
│  [Pause] [Stop] [Show Details]  │
│                                 │
├─────────────────────────────────┤
│  Ask anything...                │
│  ┌───────────────────────────┐  │
│  │                           │  │
│  └───────────────────────────┘  │
└─────────────────────────────────┘

Features:

Always visible (can minimize to icon)
Real-time activity feed
Interrupt/pause any operation
Ask questions without stopping work
Direct file/app access from panel

🤖 AI Engine Specifications

Model Stack (100% Local)

Primary Models (Ollama-based):

Fast Reactive (Always Running):
  model: llama3.2:3b-instruct-q4_K_M
  purpose: Quick decisions, UI understanding, immediate responses
  memory: ~2.5GB VRAM
  latency: <100ms
  
Balanced Reasoning (On-demand):
  model: llama3.1:8b-instruct-q4_K_M
  purpose: Multi-step planning, complex tasks
  memory: ~5GB VRAM
  latency: 1-3s
  
Deep Planning (Complex tasks):
  model: qwen2.5:14b-instruct-q4_K_M
  purpose: Novel problems, long-term planning
  memory: ~9GB VRAM
  latency: 3-8s

Vision Understanding:
  model: llava:13b-q4_K_M
  purpose: Screenshot analysis, UI element detection
  memory: ~8GB VRAM
  latency: 2-5s per image

Code Specialist:
  model: deepseek-coder:6.7b-instruct-q4_K_M
  purpose: Code generation, terminal commands, scripts
  memory: ~4GB VRAM
  latency: 1-2s

Voice Recognition:
  model: whisper.cpp (base.en)
  purpose: Voice-to-text conversion
  memory: ~500MB RAM
  latency: <500ms (streaming)

Model Router Logic

def route_task(task):
    """
    Intelligently select model based on task requirements
    """
    
    # Vision required?
    if task.has_screenshot or task.requires_ui_understanding:
        return 'llava:13b'
    
    # Code/terminal task?
    if task.type in ['code', 'terminal', 'script']:
        return 'deepseek-coder:6.7b'
    
    # Complexity-based routing
    if task.estimated_steps <= 2 and task.confidence > 0.8:
        return 'llama3.2:3b'  # Fast path
    
    if task.estimated_steps <= 5:
        return 'llama3.1:8b'  # Balanced
    
    # Complex/novel tasks
    return 'qwen2.5:14b'  # Deep reasoning

Hardware Requirements

Minimum (Usable):

CPU: 6 cores (AMD Ryzen 5 / Intel i5 or better)
RAM: 16GB
GPU: 6GB VRAM (GTX 1660 / RX 5600 or better)
Storage: 100GB free space

Recommended (Smooth):

CPU: 8+ cores
RAM: 32GB
GPU: 8GB+ VRAM (RTX 3060 / RX 6700 or better)
Storage: 256GB SSD

Optimal (Fast):

CPU: 12+ cores
RAM: 64GB
GPU: 12GB+ VRAM (RTX 4070 / RX 7800 XT or better)
Storage: 512GB NVMe

Application Control Methods

Priority System (Top to Bottom)

1. Native APIs (Best - 95% Reliability)

When available, always use direct APIs:

# Blender
import bpy
bpy.ops.mesh.primitive_cube_add(location=(0, 0, 0))

# GIMP
from gimpfu import *
pdb.gimp_image_scale(image, 1920, 1080)

# Inkscape
subprocess.run(['inkscape', '--export-png=output.png', 'input.svg'])

# VSCode
# Via Extension API (custom extension)
vscode.commands.executeCommand('workbench.action.files.save')

2. Accessibility Tree (Second Best - 85% Reliability)

Use AT-SPI2 for semantic understanding:

from pyatspi import Registry, utils

def click_button_by_name(name):
    """
    Find and click button using accessibility
    """
    desktop = Registry.getDesktop(0)
    
    for app in desktop:
        for window in app:
            button = find_element_recursive(
                window, 
                role='button',
                name=name
            )
            if button:
                button.click()
                return True
    return False

# Usage
click_button_by_name("Save")

3. OSWorld Pattern Library (Adaptive - 75% Reliability)

Pre-recorded interaction patterns with visual grounding:

Pattern Structure:

{
  "pattern_id": "firefox_bookmark_page",
  "app": "firefox",
  "description": "Bookmark current page",
  "trigger_conditions": {
    "app_name": "firefox",
    "window_title_contains": ["Mozilla Firefox"]
  },
  "steps": [
    {
      "type": "keyboard",
      "action": "Ctrl+D",
      "verify": "wait_for_dialog",
      "timeout_ms": 1000
    },
    {
      "type": "vision_verify",
      "expect": "bookmark_dialog_visible",
      "confidence_threshold": 0.8
    },
    {
      "type": "accessibility",
      "target": {"role": "text", "name": "Name"},
      "action": "set_text",
      "value": "{bookmark_name}"
    },
    {
      "type": "keyboard",
      "action": "Return"
    }
  ],
  "fallback_steps": [
    {
      "type": "vision_click",
      "target": "star_icon",
      "description": "Click bookmark star in URL bar"
    }
  ],
  "success_indicators": [
    "bookmark_star_filled",
    "bookmark_added_notification"
  ],
  "success_rate": 0.92,
  "last_verified": "2025-01-20"
}

Pattern Adaptation:

class AdaptivePatternPlayer:
    def execute_pattern(self, pattern, context):
        """
        Execute pattern with real-time adaptation
        """
        for step in pattern['steps']:
            # Take screenshot before action
            before = capture_screen()
            
            # Execute step
            success = self.execute_step(step, context)
            
            if not success:
                # Try fallback
                for fallback in pattern.get('fallback_steps', []):
                    if self.execute_step(fallback, context):
                        success = True
                        break
            
            if not success:
                # Use vision to understand what went wrong
                after = capture_screen()
                analysis = self.vision_model.analyze_failure(
                    before, after, step
                )
                
                # AI generates new approach
                new_step = self.ai_generate_alternative(step, analysis)
                if self.execute_step(new_step, context):
                    # Learn this new approach
                    pattern['steps'][i] = new_step
                    self.save_pattern_update(pattern)
                    success = True
            
            if not success:
                return False
        
        return True

4. Vision + Input Simulation (Fallback - 60% Reliability)

When nothing else works:

class VisionGuidedControl:
    def click_element(self, description):
        """
        Find and click element using vision
        """
        # Capture screen
        screenshot = capture_screen()
        
        # Use Set-of-Marks overlay
        marked_image, elements = self.som_overlay(screenshot)
        
        # Ask vision model
        prompt = f"""
        This is a screenshot with numbered UI elements.
        Find the element matching: "{description}"
        
        Respond with just the number of the correct element.
        """
        
        response = self.vision_model.query(marked_image, prompt)
        element_num = int(response.strip())
        
        # Click the element
        element = elements[element_num]
        pyautogui.click(element.center_x, element.center_y)
        
        # Verify action
        time.sleep(0.5)
        after = capture_screen()
        return self.verify_state_change(screenshot, after)

OSWorld Integration (The Secret Sauce)

What We Take from OSWorld

OSWorld is not the foundation, but a performance booster for reliability.

1. Set-of-Marks (SoM) Overlay

Purpose: Make vision models more accurate at UI interaction

class SetOfMarksProcessor:
    def __init__(self):
        self.ui_detector = load_model('yolov8-ui-elements')
    
    def create_overlay(self, screenshot):
        """
        Detect interactive elements and number them
        """
        # Detect buttons, inputs, links, etc.
        elements = self.ui_detector.detect(screenshot)
        
        # Filter to interactive only
        interactive = [e for e in elements if e.is_interactive()]
        
        # Sort by visual hierarchy (top-left to bottom-right)
        interactive.sort(key=lambda e: (e.y, e.x))
        
        # Create overlay
        overlay = screenshot.copy()
        for idx, elem in enumerate(interactive):
            # Draw bounding box
            cv2.rectangle(overlay,
                         (elem.x1, elem.y1),
                         (elem.x2, elem.y2),
                         (255, 0, 0), 2)
            
            # Add number marker
            cv2.circle(overlay,
                      (elem.x1, elem.y1),
                      radius=15,
                      color=(255, 0, 0),
                      thickness=-1)
            
            cv2.putText(overlay, str(idx),
                       (elem.x1-5, elem.y1+5),
                       cv2.FONT_HERSHEY_BOLD,
                       0.5, (255, 255, 255), 2)
        
        return overlay, interactive
    
    def interact(self, element_number, action='click'):
        """
        Execute action on numbered element
        """
        elem = self.current_elements[element_number]
        
        if action == 'click':
            click(elem.center_x, elem.center_y)
        elif action == 'type':
            click(elem.center_x, elem.center_y)
            # Text provided separately

AI Interaction:

AI sees: Screenshot with numbered elements [0-25]
AI response: "Click element 7 to save the file"
System: Executes click on element #7

Advantage: Vision model doesn't need to describe locations ("top right blue button"), just picks a number.

2. Trajectory Recording & Replay

Human demonstrates task → AI learns to replicate

class TrajectoryRecorder:
    def start_recording(self, task_name):
        """
        Record human demonstration
        """
        self.trajectory = {
            'task': task_name,
            'timestamp': time.time(),
            'steps': [],
            'environment': self.capture_environment()
        }
        
        # Hook input events
        self.listen_to_inputs()
    
    def on_input_event(self, event):
        """
        Capture each interaction with full context
        """
        step = {
            'timestamp': time.time(),
            'type': event.type,  # click, key, scroll
            
            # Raw action
            'raw_coords': (event.x, event.y) if hasattr(event, 'x') else None,
            'raw_key': event.key if hasattr(event, 'key') else None,
            
            # Semantic context
            'screenshot_before': capture_screen(),
            'accessibility_tree': get_a11y_tree(),
            'element_clicked': find_element_at(event.x, event.y),
            
            # Visual grounding
            'visual_description': self.vision_model.describe_region(
                screenshot, event.x, event.y, radius=100
            ),
            
            # App context
            'active_window': get_active_window(),
            'process_name': get_focused_process()
        }
        
        self.trajectory['steps'].append(step)
    
    def save_trajectory(self):
        """
        Save as replayable pattern
        """
        # Convert to pattern format
        pattern = self.trajectory_to_pattern(self.trajectory)
        
        # Save to library
        save_pattern(pattern)

Replay with Adaptation:

class AdaptiveReplayer:
    def replay_trajectory(self, pattern, context):
        """
        Replay with intelligent adaptation
        """
        for step in pattern['steps']:
            # Don't use raw coordinates - use semantic matching
            if step['type'] == 'click':
                # Find element by semantic description
                element = self.find_element_semantic(
                    description=step['visual_description'],
                    a11y_hint=step['element_clicked']
                )
                
                if element:
                    # Click found element (coordinates may differ)
                    click(element.center_x, element.center_y)
                else:
                    # Fallback to vision similarity
                    location = self.find_visual_match(
                        current_screen(),
                        step['screenshot_before'],
                        step['raw_coords']
                    )
                    click(*location)

3. Multi-Modal Grounding

Combine vision + accessibility for robust understanding

class MultiModalGrounding:
    def find_element(self, description):
        """
        Use all available information to locate element
        """
        # Method 1: Accessibility tree (fastest, most reliable)
        a11y_result = self.search_a11y_tree(description)
        if a11y_result and a11y_result.confidence > 0.9:
            return a11y_result
        
        # Method 2: OCR text matching
        ocr_text = self.extract_text_from_screen()
        text_match = self.fuzzy_match(description, ocr_text)
        if text_match and text_match.confidence > 0.8:
            return text_match
        
        # Method 3: Vision model
        screenshot = capture_screen()
        vision_result = self.vision_model.locate(screenshot, description)
        if vision_result and vision_result.confidence > 0.7:
            return vision_result
        
        # Method 4: Combined reasoning
        combined = self.ai_reason_about_location(
            description=description,
            a11y_tree=get_a11y_tree(),
            screenshot=screenshot,
            ocr_text=ocr_text
        )
        
        return combined

🛠️ Technical Stack

Core Technologies

Operating System:
  base_distro: Arch Linux (rolling release for latest packages)
  alternative_bases:
    - NixOS (for reproducibility)
    - Fedora Silverblue (for atomic updates)
  display_server: Wayland
  compositor: Sway (i3-like tiling) or Hyprland (eye candy)
  init_system: systemd
  shell: zsh with starship prompt

AI/ML Infrastructure:
  model_server: Ollama (local LLM serving)
  inference_engine: llama.cpp (optimized inference)
  vector_db: ChromaDB (for memory/RAG)
  embeddings: sentence-transformers
  vision: LLaVA + Florence-2
  voice: whisper.cpp (local STT)
  tts: piper-tts (local text-to-speech)

System Integration:
  language_primary: Rust (core system, performance-critical)
  language_secondary: Python (AI/ML, rapid prototyping)
  language_tertiary: Go (networking, services)
  
  ipc: D-Bus (system-wide messaging)
  rpc: gRPC (internal services)
  websocket: For UI communication

Perception Stack:
  screen_capture:
    wayland: wlr-screencopy protocol
    x11_fallback: scrot / imagemagick
  
  accessibility: pyatspi2 (AT-SPI2 Python bindings)
  
  vision_ml:
    - opencv-python (computer vision)
    - ultralytics (YOLOv8 for UI detection)
    - easyocr (text extraction)
    - pytesseract (backup OCR)
  
  window_management:
    - i3ipc-python (Sway/i3 control)
    - pywlroots (Wayland compositor interaction)

Action Execution:
  input_simulation:
    wayland: ydotool (maintained fork)
    x11_fallback: xdotool
    virtual_input: python-uinput
  
  terminal_automation:
    - pexpect (interactive CLI)
    - ptyprocess (PTY allocation)
    - sh (subprocess wrapper)
  
  browser_automation:
    - playwright (modern, multi-browser)
    - selenium (fallback)

UI/UX:
  ai_panel: GTK4 with Layer Shell
  search_launcher: Rofi (Wayland fork) or Ulauncher
  voice_indicator: Custom overlay widget
  notifications: mako (Wayland) or dunst (X11)
  theming: Catppuccin or Nord (dark themes)

Data & Storage:
  relational: SQLite (embedded database)
  analytics: DuckDB (for query analysis)
  vector: ChromaDB (embeddings)
  cache: Redis (optional, for performance)

Monitoring & Logging:
  system_metrics: psutil (Python)
  file_events: pyinotify (inotify wrapper)
  process_tracking: procfs parsing
  logging: structured logging (loguru)

Development:
  package_manager: paru or yay (AUR helper)
  build_system: meson (Rust+C mixed)
  testing: pytest (Python), cargo test (Rust)
  ci_cd: GitHub Actions
  documentation: mdBook or Sphinx

📁 Repository Structure

rimuru/
├── README.md
├── LICENSE (GPL-3.0)
├── CONTRIBUTING.md
├── docs/
│   ├── architecture/
│   │   ├── system-overview.md
│   │   ├── ai-engine.md
│   │   ├── perception.md
│   │   └── control-methods.md
│   ├── user-guide/
│   │   ├── installation.md
│   │   ├── quick-start.md
│   │   ├── voice-setup.md
│   │   └── customization.md
│   ├── developer/
│   │   ├── setup.md
│   │   ├── architecture-deep-dive.md
│   │   ├── adding-app-support.md
│   │   └── pattern-creation.md
│   └── api/
│       ├── rest-api.md
│       └── dbus-interface.md
│
├── core/
│   ├── Cargo.toml (Rust workspace)
│   ├── orchestrator/ (Main AI coordination)
│   │   ├── src/
│   │   │   ├── main.rs
│   │   │   ├── intent_parser.rs
│   │   │   ├── goal_planner.rs
│   │   │   ├── model_router.rs
│   │   │   └── safety_guard.rs
│   │   └── Cargo.toml
│   │
│   ├── perception/ (Multi-modal sensing)
│   │   ├── src/
│   │   │   ├── vision.rs
│   │   │   ├── accessibility.rs
│   │   │   ├── process_monitor.rs
│   │   │   └── screen_capture.rs
│   │   └── Cargo.toml
│   │
│   └── executor/ (Action execution)
│       ├── src/
│       │   ├── input_sim.rs
│       │   ├── terminal.rs
│       │   ├── gui_control.rs
│       │   └── browser.rs
│       └── Cargo.toml
│
├── ai/
│   ├── models/ (Model management)
│   │   ├── __init__.py
│   │   ├── router.py
│   │   ├── ollama_client.py
│   │   └── vision_model.py
│   │
│   ├── planning/ (Goal decomposition)
│   │   ├── __init__.py
│   │   ├── goal_planner.py
│   │   ├── step_generator.py
│   │   └── fallback_handler.py
│   │
│   ├── memory/ (Context & learning)
│   │   ├── __init__.py
│   │   ├── vector_store.py
│   │   ├── conversation_history.py
│   │   └── user_preferences.py
│   │
│   └── osworld/ (OSWorld integration)
│       ├── __init__.py
│       ├── som_overlay.py
│       ├── trajectory_recorder.py
│       ├── pattern_library.py
│       └── adaptive_replay.py
│
├── control/
│   ├── terminal/ (CLI automation)
│   │   ├── __init__.py
│   │   ├── command_executor.py
│   │   ├── pty_controller.py
│   │   └── shell_parser.py
│   │
│   ├── gui/ (GUI app control)
│   │   ├── __init__.py
│   │   ├── accessibility.py
│   │   ├── input_simulation.py
│   │   ├── window_manager.py
│   │   └── app_apis/
│   │       ├── blender.py
│   │       ├── gimp.py
│   │       ├── inkscape.py
│   │       └── vscode.py
│   │
│   └── browser/ (Web automation)
│       ├── __init__.py
│       ├── playwright_controller.py
│       ├── cdp_client.py
│       └── web_utils.py
│
├── ui/
│   ├── ai-panel/ (GTK4 overlay UI)
│   │   ├── src/
│   │   │   ├── main.rs
│   │   │   ├── panel.rs
│   │   │   ├── activity_feed.rs
│   │   │   └── chat_interface.rs
│   │   └── Cargo.toml
│   │
│   ├── launcher/ (Search/command bar)
│   │   ├── src/
│   │   │   ├── main.rs
│   │   │   ├── search.rs
│   │   │   ├── suggestions.rs
│   │   │   └── executor.rs
│   │   └── Cargo.toml
│   │
│   └── voice-indicator/
│       └── src/main.rs
│
├── patterns/ (OSWorld pattern library)
│   ├── core/ (Essential patterns)
│   │   ├── file_operations.json
│   │   ├── browser_navigation.json
│   │   ├── terminal_tasks.json
│   │   └── system_settings.json
│   │
│   ├── apps/ (App-specific patterns)
│   │   ├── firefox/
│   │   ├── vscode/
│   │   ├── blender/
│   │   ├── gimp/
│   │   └── ...
│   │
│   └── custom/ (User-recorded patterns)
│       └── .gitignore (don't commit user patterns)
│
├── services/ (System daemons)
│   ├── raphael-orchestrator.service
│   ├── raphael-voice.service
│   └── raphael-perception.service
│
├── config/
│   ├── default.toml (Default configuration)
│   ├── models.toml (Model definitions)
│   └── keybindings.toml (Keyboard shortcuts)
│
├── scripts/
│   ├── install.sh (Full system installation)
│   ├── setup-models.sh (Download AI models)
│   ├── build.sh (Build all components)
│   └── dev-setup.sh (Development environment)
│
├── tests/
│   ├── integration/ (End-to-end tests)
│   ├── unit/ (Component tests)
│   └── fixtures/ (Test data)
│
└── assets/
    ├── icons/ (UI icons)
    ├── sounds/ (Audio feedback)
    └── themes/ (Visual themes)

Development Roadmap

Phase 1: Foundation (Months 1-3)

Goal: Core infrastructure working, basic automation functional

Milestones:

Month 1: Perception & Basic Control

Screen capture pipeline (Wayland + X11)
Accessibility tree parsing (AT-SPI2)
Basic input simulation (keyboard/mouse)
Terminal command execution (pexpect)
Process monitoring

Deliverable: Can capture screen, parse UI, execute terminal commands

Month 2: AI Integration

Ollama integration and model management
Model router implementation
Intent parsing (natural language → goals)
Basic goal planning (single-step tasks)
Memory system (SQLite + ChromaDB)

Deliverable: Can understand "install package X" and execute it

Month 3: Basic UI

Deliverable: User can interact via search panel and voice

Phase 2: OSWorld Integration (Months 4-6)

Goal: Reliable GUI automation using OSWorld techniques

Milestones:

Month 4: Set-of-Marks System

Deliverable: Can click GUI elements via vision + numbering

Month 5: Pattern Library

Trajectory recording system
Pattern storage format
Adaptive pattern replay
Pattern verification
Create 50+ core patterns (Firefox, VSCode, Files, Terminal)

Deliverable: 80% success rate on common tasks via patterns

Month 6: Multi-Modal Grounding

Vision + A11y fusion
Fallback chain (API → A11y → Pattern → Vision)
Confidence scoring
Automatic pattern learning from user corrections

Deliverable: Robust control across different UI types

Phase 3: Application Support (Months 7-9)

Goal: Support 30+ common applications

Milestones:

Month 7: Browser Control

Deliverable: Can control web browsers and web apps

Month 8: Creative Suite

Deliverable: Can perform creative tasks autonomously

Month 9: Developer Tools

Deliverable: Can set up dev environments autonomously

Phase 4: Intelligence & Learning (Months 10-12)

Goal: System learns and improves over time

Milestones:

Month 10: Memory & Context

Long-term user preference storage
Contextual awareness (time, location, activity)
Pattern success rate tracking
Automatic pattern updates
Conversation history with RAG

Deliverable: System remembers user preferences and improves

Month 11: Proactive Behavior

Deliverable: System anticipates needs and suggests actions

Month 12: Safety & Reliability

Deliverable: Safe autonomous operation with rollback

Phase 5: Polish & Release (Months 13-18)

Goal: Production-ready alpha release

Milestones:

Months 13-15: Performance & Optimization

Model quantization optimization
Reduce latency (<500ms for common tasks)
Battery optimization for laptops
Memory usage optimization
GPU scheduling improvements

Months 16-17: Documentation & Testing

Month 18: Alpha Release

Success Metrics (Phase 1 - 12 Months)

Quantitative Goals

Reliability:
  target: 80% success rate
  measurement: "Success rate on 100 common tasks"
  tasks_tested:
    - Install package
    - Open application
    - Search the web
    - Create file/folder
    - Edit configuration
    - Git operations
    - Browser automation
    - File operations
  
Performance:
  latency_simple: "<500ms (e.g., 'open Firefox')"
  latency_complex: "<5s (e.g., 'set up Python env')"
  battery_impact: "<15% overhead on laptops"
  memory_usage: "<4GB base (excluding models)"
  
Coverage:
  applications_supported: "50+ apps"
  patterns_library: "200+ patterns"
  use_cases: "1000+ tested scenarios"
  
User Satisfaction:
  alpha_users: "100+ active testers"
  retention: ">60% weekly active"
  nps_score: ">40"

Qualitative Goals

"Wow" Factor: First-time users should have at least one "this is magic" moment
Trust: Users feel confident letting AI handle sensitive tasks
Transparency: Always clear what AI is doing and why
Interruptible: Can stop/pause any action instantly
Learning: System improves with use, doesn't repeat mistakes

Key Features (Differentiation)

What Makes Rimuru Unique

1. True AI-Native Interface

Not: AI chatbot in a sidebar Is: AI as the primary system interface

Traditional Linux:
  User → Application → Task

Rimuru:
  User → AI → Application(s) → Task
  
User never directly touches apps unless they want to

2. Transparent Autonomous Operation

Not: Black box automation Is: Visible, understandable AI actions

Visual Indicators:

Ghost cursor showing AI's intended clicks
Overlay labels explaining each action
Live activity feed in AI panel
Confidence scores for decisions
Ability to pause and inspect at any time

3. 100% Local & Private

Not: Cloud API dependent Is: Everything runs on your hardware

Privacy Guarantees:

No data sent to cloud
All models run locally
Voice processing on-device
No telemetry (optional anonymous usage stats)
Full control over data

4. OSWorld-Enhanced Reliability

Not: Brittle screen scraping Is: Multi-modal understanding with learned patterns

Reliability Stack:

Native APIs (when available)
Accessibility tree (semantic)
Learned patterns (demonstrated)
Vision + AI (last resort)

Each layer improves success rate

5. Learning & Adaptation

Not: Static automation scripts Is: System that improves over time

Learning Mechanisms:

User corrections → pattern updates
Success/failure tracking → confidence adjustment
New app interactions → automatic pattern extraction
Usage patterns → proactive suggestions

Safety & Security

Safety Mechanisms

1. Confidence-Based Execution

class SafetyGuard:
    def should_execute_autonomously(self, action):
        """
        Determine if action can run without user approval
        """
        confidence = self.calculate_confidence(action)
        risk_level = self.assess_risk(action)
        
        # High confidence + low risk = auto execute
        if confidence > 0.95 and risk_level == 'low':
            return True, 'auto'
        
        # Medium confidence = show plan, countdown
        if confidence > 0.80 and risk_level in ['low', 'medium']:
            return True, 'countdown_5s'
        
        # Everything else = ask first
        return False, 'require_approval'
    
    def assess_risk(self, action):
        """
        Categorize action risk
        """
        risk_indicators = {
            'high': [
                'rm -rf', 'sudo', 'format', 'delete',
                'send email', 'post online', 'payment'
            ],
            'medium': [
                'install', 'uninstall', 'modify config',
                'git push', 'network request'
            ],
            'low': [
                'open', 'read', 'search', 'navigate',
                'copy', 'view'
            ]
        }
        
        # Check action against patterns
        for risk, patterns in risk_indicators.items():
            if any(p in action.lower() for p in patterns):
                return risk
        
        return 'medium'  # Default to caution

2. Comprehensive Undo System

class UndoSystem:
    def before_action(self, action):
        """
        Create checkpoint before any system change
        """
        checkpoint = {
            'timestamp': time.time(),
            'action': action,
            'snapshots': {}
        }
        
        if action.affects_filesystem:
            # Snapshot affected files/directories
            checkpoint['snapshots']['files'] = self.snapshot_files(
                action.target_paths
            )
        
        if action.affects_packages:
            # Record package state
            checkpoint['snapshots']['packages'] = self.get_package_list()
        
        if action.affects_config:
            # Backup config files
            checkpoint['snapshots']['configs'] = self.backup_configs(
                action.config_files
            )
        
        if action.affects_database:
            # Export database state
            checkpoint['snapshots']['db'] = self.export_db_state()
        
        self.checkpoint_stack.append(checkpoint)
        return checkpoint.id
    
    def undo_last_action(self):
        """
        Rollback most recent change
        """
        if not self.checkpoint_stack:
            return False, "Nothing to undo"
        
        checkpoint = self.checkpoint_stack.pop()
        
        # Restore in reverse order of changes
        for snapshot_type, snapshot_data in checkpoint['snapshots'].items():
            self.restore_snapshot(snapshot_type, snapshot_data)
        
        return True, f"Undid: {checkpoint['action'].description}"
    
    def undo_multiple(self, count):
        """
        Undo last N actions
        """
        for _ in range(count):
            success, msg = self.undo_last_action()
            if not success:
                break

Undo UI:

╔════════════════════════════════════════╗
║  Recent Actions (Can Undo)             ║
╠════════════════════════════════════════╣
║  ✓ Installed Docker              [↶]   ║
║  ✓ Modified .bashrc              [↶]   ║
║  ✓ Created project folder        [↶]   ║
║  ✓ Opened VSCode                 [↶]   ║
╠════════════════════════════════════════╣
║  [Undo Last] [Undo Multiple] [Clear]   ║
╚════════════════════════════════════════╝

3. Sandboxed Testing Mode

class SandboxMode:
    def enable_sandbox(self):
        """
        All actions execute in safe simulation
        """
        self.mode = 'sandbox'
        
        # File operations → temp directory
        self.sandbox_root = tempfile.mkdtemp()
        
        # Package operations → dry-run only
        self.package_manager.set_dry_run(True)
        
        # Network → logged but not executed
        self.network.set_intercept_mode(True)
        
        # System changes → logged only
        self.system.set_simulation_mode(True)
    
    def execute_in_sandbox(self, action):
        """
        Run action safely, log results
        """
        result = self.simulate_action(action)
        
        # Show user what WOULD happen
        self.show_preview(result)
        
        # User can approve to execute for real
        if user_approves():
            self.disable_sandbox()
            actual_result = self.execute_action(action)
            self.enable_sandbox()
            return actual_result
        
        return result

Sandbox UI Indicator:

┌─────────────────────────────────────┐
│ 🛡️  SANDBOX MODE ACTIVE            │
│ All actions are simulated           │
│ No real changes will be made        │
│ [Disable Sandbox] [Keep Protected]  │
└─────────────────────────────────────┘

4. Permission Boundaries

# config/permissions.toml

[filesystem]
allowed_read = [
    "~/*",
    "/usr/share/*",
    "/etc/*"
]
allowed_write = [
    "~/Documents/*",
    "~/Downloads/*",
    "~/Projects/*"
]
forbidden_write = [
    "~/.ssh/*",
    "~/.gnupg/*",
    "/etc/passwd",
    "/etc/shadow"
]

[network]
allowed_domains = [
    "*"  # All domains by default
]
forbidden_domains = [
    # Add domains AI should never access
]
require_approval = [
    "POST",    # Any POST request needs approval
    "DELETE",  # Any DELETE request needs approval
]

[system]
allowed_commands = [
    "apt", "pacman", "dnf",
    "git", "cargo", "npm",
    "systemctl --user",
]
forbidden_commands = [
    "dd",
    "mkfs",
    "fdisk",
    "parted",
    "rm -rf /",
]
require_sudo_approval = true

[applications]
can_modify = [
    "firefox",
    "code",
    "blender",
    # ... apps AI can control
]
cannot_modify = [
    "gnome-keyring",
    "keepassxc",
    # ... sensitive apps
]

Security Mechanisms

1. Model Integrity Verification

class ModelSecurity:
    def verify_model(self, model_path):
        """
        Ensure model hasn't been tampered with
        """
        # Check SHA256 hash against known good
        current_hash = self.calculate_hash(model_path)
        expected_hash = self.get_official_hash(model_name)
        
        if current_hash != expected_hash:
            raise SecurityError(
                f"Model {model_name} failed integrity check!"
            )
        
        # Verify model signature (if available)
        if self.has_signature(model_path):
            self.verify_signature(model_path)

2. Input Sanitization

class InputValidator:
    def sanitize_user_input(self, user_input):
        """
        Prevent injection attacks
        """
        # Remove dangerous shell characters
        dangerous_chars = [';', '|', '&', ', '`']
        for char in dangerous_chars:
            if char in user_input:
                # Log potential attack
                self.log_security_event(
                    'suspicious_input',
                    user_input
                )
                # Remove or escape
                user_input = user_input.replace(char, '')
        
        return user_input
    
    def validate_file_path(self, path):
        """
        Ensure path is within allowed boundaries
        """
        real_path = os.path.realpath(path)
        
        # Check against forbidden paths
        for forbidden in self.forbidden_paths:
            if real_path.startswith(forbidden):
                raise SecurityError(
                    f"Access to {real_path} is forbidden"
                )
        
        return real_path

3. Audit Logging

class AuditLog:
    def log_action(self, action, result):
        """
        Complete audit trail of all AI actions
        """
        log_entry = {
            'timestamp': datetime.now().isoformat(),
            'user': os.getenv('USER'),
            'action_type': action.type,
            'action_description': action.description,
            'command_executed': action.command,
            'files_accessed': action.files_accessed,
            'files_modified': action.files_modified,
            'network_requests': action.network_requests,
            'result': result.status,
            'error': result.error if result.failed else None,
            'checkpoint_id': result.checkpoint_id
        }
        
        # Write to append-only log
        self.write_to_audit_log(log_entry)
        
        # Also store in searchable database
        self.db.insert_audit_entry(log_entry)

Audit Log Viewer:

$ rimuru audit --last 10

2025-01-25 14:32:15  [SUCCESS]  Install package 'docker'
  Command: sudo pacman -S docker
  Files modified: /var/lib/pacman/local/docker-*
  Checkpoint: chk_001a2b3c

2025-01-25 14:35:22  [SUCCESS]  Create directory 'projects/rust-app'
  Command: mkdir -p ~/projects/rust-app
  Files created: /home/user/projects/rust-app
  Checkpoint: chk_001a2b3d

2025-01-25 14:36:18  [FAILED]   Install package 'nonexistent-pkg'
  Command: sudo pacman -S nonexistent-pkg
  Error: target not found: nonexistent-pkg
  Checkpoint: None (no changes made)

Contributing

How to Contribute

We welcome contributions from:

Developers: Core features, bug fixes, optimizations
ML Engineers: Model fine-tuning, pattern recognition
UX Designers: Interface improvements, accessibility
Technical Writers: Documentation, tutorials
Testers: Bug reports, edge case discovery
Pattern Creators: Record interaction patterns for apps

Contribution Areas

High Priority (Need Help Now)

Pattern Library Creation
- Record interaction patterns for common apps
- We need 200+ patterns for Phase 2
- Tools: Pattern recorder (we'll build together)
UI Element Detection Model
- Train YOLOv8 on UI elements (buttons, inputs, etc.)
- Need dataset of annotated screenshots
- We'll provide annotation tools
App API Integration
- Write Python wrappers for app APIs
- Blender, GIMP, Inkscape, Krita, etc.
- Document each API's capabilities
Testing & QA
- Test on different hardware (AMD/NVIDIA/Intel)
- Test different DEs (KDE, GNOME, Sway)
- Report edge cases and failures

Medium Priority

Documentation
- User guides for common tasks
- Video tutorials
- Architecture deep-dives
- API documentation
Performance Optimization
- Reduce inference latency
- Optimize memory usage
- Battery life improvements
Accessibility
- Screen reader compatibility
- Keyboard-only operation
- High-contrast themes

Future (After Phase 1)

Multi-language Support
- Translate UI
- Multi-lingual voice input
- International keyboard layouts
Platform Ports
- macOS version (via Homebrew)
- Windows WSL2 support
- Mobile (Android/Linux phones)

Development Setup

# Clone repository
git clone https://github.com/dbisina/rimuru.git
cd rimuru

# Run development setup script
./scripts/dev-setup.sh

# This will:
# 1. Install system dependencies
# 2. Set up Python virtual environment
# 3. Install Rust toolchain
# 4. Download AI models
# 5. Build core components
# 6. Run initial tests

# Start development services
./scripts/dev-run.sh

# Run tests
cargo test      # Rust tests
pytest tests/   # Python tests

Coding Standards

Rust Code:

Use rustfmt for formatting
Run clippy for linting
Document public APIs
Write unit tests for new features

Python Code:

Follow PEP 8 style guide
Use type hints
Docstrings for all functions
80% test coverage minimum

Commit Messages:

<type>(<scope>): <subject>

<body>

<footer>

Types: feat, fix, docs, style, refactor, test, chore
Scope: component name (orchestrator, perception, ui, etc.)

Example:
feat(perception): Add Set-of-Marks overlay system

- Implement UI element detection using YOLOv8
- Create numbered overlay rendering
- Add vision model integration for element selection

Closes #42

Community & Support

Communication Channels

GitHub:

Issues: Bug reports, feature requests
Discussions: Questions, ideas, show-and-tell
Pull Requests: Code contributions

Matrix/Discord (Coming Soon):

Real-time chat
Help & support
Development coordination

Forum (Coming Soon):

Long-form discussions
Tutorials & guides
Community patterns sharing

Getting Help

For Users:

Check documentation: docs/user-guide/
Search existing issues
Ask in Discussions
Join Matrix/Discord

For Developers:

Read architecture docs: docs/architecture/
Check developer guide: docs/developer/
Ask in dev channel
Open draft PR for feedback

License

GPL-3.0 License

Rimuru is free and open-source software. You can:

✅ Use it for any purpose
✅ Study and modify the source code
✅ Distribute copies
✅ Distribute modified versions

Requirements:

Disclose source code
Include original license
State changes made
Use same license for derivative works

See LICENSE for full details.

🎓 Research & Citations

Built On The Shoulders Of Giants

This project integrates research and techniques from:

OSWorld (Zhou et al., 2024)

Multi-modal UI understanding
Set-of-Marks visual prompting
Trajectory learning for GUI control

GPT-4V (OpenAI, 2023)

Vision-language model architecture
Screenshot understanding

Agent frameworks:

AutoGPT - Autonomous agent architecture
LangChain - Tool composition patterns

UI Automation:

Selenium/Playwright - Browser control patterns
AT-SPI2 - Accessibility standards

If you use Rimuru in research, please cite:

@software{rimuru2025,
  title={Rimuru: AI-Native Linux Distribution},
  author={Rimuru Contributors},
  year={2025},
  url={https://github.com/dbisina/rimuru}
}

Long-Term Vision (Beyond Phase 1)

Track 2: Years 2-4

Once the base system is solid, we'll pursue the full vision:

Year 2: Deep Integration

eBPF monitoring layer
Kernel-level resource optimization
Native app protocol (apps designed for AI control)
Self-healing infrastructure

Year 3: Intelligence Advancement

Fine-tuned models on user interactions
Federated learning (privacy-preserving)
Multi-agent coordination
Proactive system maintenance

Year 4: Ecosystem Maturity

App developer SDK (build AI-controllable apps)
Hardware partnerships (optimized for AI workloads)
Enterprise features
Stable 1.0 release

❓ FAQ

Q: Does this replace my desktop environment? A: No, it works with your existing DE (GNOME, KDE, Sway, etc.)

Q: What if the AI makes a mistake? A: Every action can be undone. High-risk actions require approval.

Q: Does it need internet? A: No, everything runs locally. Internet only needed for optional updates.

Q: What hardware do I need? A: Minimum: 16GB RAM, 6GB VRAM. Recommended: 32GB RAM, 8GB VRAM.

Q: Can I use my own models? A: Yes! Ollama supports custom models. You can fine-tune or import.

Q: Is my data safe? A: Yes. Everything stays on your device. No telemetry, no cloud.

Q: What apps does it support? A: Currently 50+ apps. Adding more each release. You can add your own.

Q: Can I turn off the AI? A: Yes. Works as a normal Linux system with AI features disabled.

Q: How is this different from GitHub Copilot? A: Copilot is for code. This controls your entire OS and applications.

Q: Will there be a Windows/Mac version? A: Maybe later. Linux first because of open-source ecosystem.

🚀 Get Started

For Users (When Released)

# Download installer
curl -fsSL https://rimuru.org/install.sh | sh

# Or build from source
git clone https://github.com/dbisina/rimuru.git
cd rimuru
./scripts/install.sh

For Contributors (Now)

# Clone and set up development environment
git clone https://github.com/dbisina/rimuru.git
cd rimuru
./scripts/dev-setup.sh

# Join the community
# - Star the repo
# - Read CONTRIBUTING.md
# - Pick an issue labeled "good first issue"
# - Join Matrix/Discord for discussions

📊 Project Status

Current Phase: Pre-Alpha Development
Version: 0.0.1-dev
Contributors: Seeking initial team
First Release Target: Q4 2025

Progress:

🙏 Acknowledgments

Inspiration:

Xerox PARC - Vision of human-computer interaction
Unix philosophy - Do one thing well (AI operating the computer)
Science fiction - Jarvis, HAL, TARS (minus the homicidal tendencies)
Rimuru's Great Sage from Reincarnated as a slime

Special Thanks:

OSWorld team for groundbreaking GUI control research
Ollama team for local LLM infrastructure
LLaMA team for open-source models
Linux community for the ultimate hackable OS

Let's build the future of human-computer interaction together.

Star ⭐ the repo | Join the discussion | Contribute code

Last Updated: January 2026
Document Version: 1.0
Status: Living Document - Will evolve with project

Name		Name	Last commit message	Last commit date
ai		ai
assets		assets
config		config
control		control
core		core
docs		docs
patterns		patterns
scripts		scripts
services		services
tests		tests
ui		ui
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation

RIMURU - OS reincarnated, AI - RAPHAEL (Great Sage)

AI-Native Linux Distribution

Product Requirements Document (PRD) v1.0

Vision

The North Star

Two-Track Development Strategy

Track 1: Usable System (6-12 Months)

Track 2: Full Vision (2-4 Years)

System Architecture

High-Level Overview

User Experience: AI-Native Interaction

Primary Interaction Methods

1. Voice Control (Always Listening - Optional)

2. AI Search Panel (Universal Launcher)

3. AI Interaction Panel (Persistent Assistant)

🤖 AI Engine Specifications

Model Stack (100% Local)

Model Router Logic

Hardware Requirements

Application Control Methods

Priority System (Top to Bottom)

1. Native APIs (Best - 95% Reliability)

2. Accessibility Tree (Second Best - 85% Reliability)

3. OSWorld Pattern Library (Adaptive - 75% Reliability)

4. Vision + Input Simulation (Fallback - 60% Reliability)

OSWorld Integration (The Secret Sauce)

What We Take from OSWorld

1. Set-of-Marks (SoM) Overlay

2. Trajectory Recording & Replay

3. Multi-Modal Grounding

🛠️ Technical Stack

Core Technologies

📁 Repository Structure

Development Roadmap

Phase 1: Foundation (Months 1-3)

Phase 2: OSWorld Integration (Months 4-6)

Phase 3: Application Support (Months 7-9)

Phase 4: Intelligence & Learning (Months 10-12)

Phase 5: Polish & Release (Months 13-18)

Success Metrics (Phase 1 - 12 Months)

Quantitative Goals

Qualitative Goals

Key Features (Differentiation)

What Makes Rimuru Unique

1. True AI-Native Interface

2. Transparent Autonomous Operation

3. 100% Local & Private

4. OSWorld-Enhanced Reliability

5. Learning & Adaptation

Safety & Security

Safety Mechanisms

1. Confidence-Based Execution

2. Comprehensive Undo System

3. Sandboxed Testing Mode

4. Permission Boundaries

Security Mechanisms

1. Model Integrity Verification

2. Input Sanitization

3. Audit Logging

Contributing

How to Contribute

Contribution Areas

High Priority (Need Help Now)

Medium Priority

Future (After Phase 1)

Development Setup

Coding Standards

Community & Support

Communication Channels

Getting Help

License

🎓 Research & Citations

Built On The Shoulders Of Giants

Long-Term Vision (Beyond Phase 1)

Track 2: Years 2-4

❓ FAQ