VXD - Vector Database with Hybrid Search

🚀 Open Source License

This is an open source project and everyone is allowed to use it in whatever way they want. Absolutely no restrictions whatsoever.

Feel free to:

• Use it in commercial projects
• Modify and distribute
• Create derivative works
• Use it for research and education
• Integrate it into your applications

📧 Support

If you need support or have questions, please send an email to: vxd@launchxlabs.ai

✨ Key Features

VXD implements cutting-edge vector database technology with the following advanced features:

🔍 Hybrid Search

• Dense Vector Search: High-performance HNSW (Hierarchical Navigable Small World) algorithm for approximate nearest neighbor search
• Sparse Vector Search: BM25-style sparse vector retrieval for keyword-based search
• Hybrid Retrieval: Combines dense and sparse search results for optimal relevance

🧠 Hybrid Graph Memory Management

• Ultra Low RAM Usage: Innovative memory management techniques that minimize RAM consumption
• Level-based Storage: Base layer vectors stored in LMDB for persistence, upper layers kept in memory
• Thread-local Vector Caching: Safe and efficient vector data access patterns
• Dynamic Memory Allocation: Efficient memory usage that scales with your data

🏗️ Advanced Technical Features

• FP16 Support: Half-precision floating-point for reduced memory footprint
• Multi-threading: Concurrent search and insertion operations
• ACID Transactions: Reliable data consistency with LMDB backend
• Configurable Parameters: Tunable M, ef_construction, and other HNSW parameters
• Delete Operations: Soft deletes with proper cleanup
• Index Persistence: Save and load indices to/from disk
• Hybrid Vector Storage: Seamless integration of dense and sparse vectors

🛠️ Compilation

Prerequisites

• C++17 compatible compiler (GCC, Clang, MSVC)
• CMake 3.10 or higher
• OpenSSL development libraries
• Eigen3 library (for sparse vector operations)

Build Instructions

mkdir build && cd build
cmake ..
make -j4

For debug builds:

mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Debug -DVX_DEBUG=ON ..
make -j4

🚀 Running VXD

Basic Usage

./vxd

Configuration

Data Directory

Set your data directory path:

export VXD_DATA_DIR=<your_data_directory>
./vxd

Environment Variables

• VXD_DATA_DIR: Directory where VXD will store index files and data
• VXD_PORT: HTTP server port (default: 8080)
• VXD_HOST: HTTP server host (default: 0.0.0.0)

Example Usage

# Set data directory
export VXD_DATA_DIR=/path/to/your/data

# Run VXD server
./vxd

The server will start and listen for HTTP requests on the configured port (default: 8080).

📡 API Endpoints

VXD provides a RESTful HTTP API for vector operations:

• POST /index/{index_id}/vectors - Add vectors to an index
• POST /index/{index_id}/search - Search for similar vectors
• DELETE /index/{index_id}/vectors/{vector_id} - Delete a vector
• GET /index/{index_id}/stats - Get index statistics
• POST /index/{index_id}/hybrid - Add hybrid vectors (dense + sparse)
• POST /index/{index_id}/hybrid/search - Hybrid search

🏗️ Architecture

Core Components

• HNSW Algorithm: Modified for hybrid operations and memory efficiency
• LMDB Storage: Persistent storage for base layer vectors
• Vector Storage: Efficient vector data management
• ID Mapper: String to numeric ID mapping with LMDB persistence
• Filter System: Advanced filtering capabilities with Roaring bitmaps
• WAL (Write-Ahead Log): Transaction logging for recovery

Memory Management

• Base layer (level 0) vectors stored in LMDB
• Upper layer vectors cached in memory
• Thread-local storage for safe concurrent access
• Automatic memory cleanup and garbage collection

🔧 Advanced Configuration

VXD supports various configuration parameters in settings.hpp:

• DEFAULT_M: Number of connections per node in HNSW graph
• DEFAULT_EF_CONSTRUCT: Size of dynamic candidate list during construction
• MAX_EXTRA_NEIGHBORS: Additional neighbors beyond M for quality
• EARLY_EXIT_BUFFER_*: Early termination thresholds for search optimization

🤝 Contributing

Since this is fully open source with no restrictions, contributions are welcome! Feel free to:

• Submit pull requests
• Report issues
• Suggest improvements
• Add new features

📞 Contact

For support, questions, or collaboration opportunities: Email: vxd@launchxlabs.ai

---

VXD - Powering the next generation of vector search applications with hybrid capabilities and ultra-efficient memory management.