Project Refactoring Proposal: lance-graph

Current Status: Beta (v0.3.x)
Target: Production-Grade GraphRAG Engine (v1.0)
Core Objective: Transition from a "Cypher-to-SQL Transpiler" to a "Native Neuro-Symbolic Query Engine" capable of handling both semantic vectors and graph topology in a single execution pass.

1. Architectural Diagnosis

The current architecture of lance-graph tightly couples the Cypher Parser (nom), the Logical Planner, and the DataFusion Execution Engine into a single crate. This creates several friction points:

1. Bloated Dependencies: Consumers who just want to parse Cypher or validate schemas must pull in the heavy datafusion and tokio dependencies.
2. Rigid Execution: The engine translates Cypher directly to DataFusion LogicalPlans. This makes it difficult to inject custom "Graph-Native" operators (like recursive graph traversal or vector search) that don't map cleanly to SQL joins.
3. Python Friction: The Python bindings are likely second-class citizens, making integration with AI frameworks (LangChain/LlamaIndex) difficult.

2. Structural Refactoring (The Workspace Pattern)

We recommend migrating the repository to a Cargo Workspace structure. This aligns with lance and lance-context, enforcing clean separation of concerns.

Proposed Directory Layout

lance-graph/
├── Cargo.toml            # Workspace definition
├── README.md             # Unified documentation
├── docs/                 # Architecture Decision Records (ADRs)
├── crates/
│   ├── lance-graph/           # The "Facade" Crate (Re-exports everything)
│   │   └── src/lib.rs
│   │
│   ├── lance-graph-core/      # Pure Rust types (No heavy dependencies)
│   │   ├── src/ast/           # Cypher AST definitions
│   │   ├── src/schema.rs      # GraphConfig, NodeMapping, RelationshipMapping
│   │   └── src/error.rs
│   │
│   ├── lance-graph-planner/   # The Compiler (Cypher -> LogicalPlan)
│   │   └── src/translator.rs  # Rewrites AST into DataFusion LogicalPlan
│   │
│   └── lance-graph-physical/  # Custom Execution Kernels
│       ├── src/traversal.rs   # Optimized Variable-Length Path operator
│       └── src/vector.rs      # Custom Vector Search UDFs
│
├── python/               # First-class Python Package
│   ├── Cargo.toml        # PyO3 bindings
│   ├── pyproject.toml    # Maturin build config
│   └── lance_graph/      # Python wrapper code
│
└── examples/             # End-to-end GraphRAG examples

3. Technical Roadmap & Next Steps

Phase 1: Decoupling & Stability (The "Core" Fixes)

Goal: Fix existing bugs and stabilize the foundation.

• Extract lance-graph-core: Move GraphConfig and AST definitions into a lightweight crate. This allows other tools (like lance-context) to import graph schemas without importing the query engine.
• Fix Column Naming (Issue Inconsistent column names returned from the execute method #30): The current transpiler relies on implicit DataFusion naming, which breaks when DataFusion optimizes plans.
  • Action: Implement a strict "Final Projection" pass that forces output column names to match the user's Cypher aliases explicitly.
• Implement UNWIND (Issue Support UNWIND in the lance-graph #75): This is critical for RAG.
  • Action: Map Cypher UNWIND to DataFusion's unnest operator.

Phase 2: The GraphRAG Capabilities

Goal: Enable hybrid search (Vector + Graph) within a single query.

• Native Vector Search Operator: Currently, users must do vector search outside lance-graph and pass IDs in.
  • Action: Implement a Cypher custom function vector_search(node.embedding, $query_vector).
  • Optimization: Push this down to the Lance Scanner so it uses the IVF-PQ index instead of scanning all nodes.
• Variable Length Paths: Optimize MATCH (a)-[*1..3]->(b).
  • Current State: Likely compiles to a fixed chain of JOINs (slow).
  • Target: Implement a custom DataFusion UserDefinedLogicalNode for recursive traversal that utilizes Lance's random-access capabilities.

Phase 3: Mutability (The "Delta" Layer)

Goal: Allow agents to write to the graph (add nodes/edges).

Since Lance is immutable, we cannot support direct CREATE or SET easily.

• Design "Graph Delta":
  • Create a temporary DeltaTable (Arrow IPC file) for new nodes/edges created during a session.
  • Update the Planner to inject a UNION ALL that reads from both the Base Lance Table and the Delta Table.
  • Implement compact() to merge Deltas into Lance files periodically.

Phase 4: Python Ecosystem Integration

Goal: Make it the default GraphRAG engine for Python agents.

• Async PyO3 Bindings: Ensure query() releases the Python GIL so agents can run parallel queries.
• LangChain Adapter: Build a LanceGraphRetriever for LangChain that accepts natural language, converts to Cypher (via LLM), and executes.
• NetworkX Export: Add a to_networkx() method to result sets for easy visualization in Python.