A modern, high-performance C++17 library designed to provide a unified, embeddable interface for interacting with various Large Language Model (LLM) providers. It simplifies the process of integrating LLM capabilities into C++ applications by offering a single, consistent API. ArbiterAI is a foundational component for C++ developers building AI-powered applications that require robust, efficient, and flexible LLM integration without being tied to a single provider.
Primary Goal: Create a high-performance, easy-to-use, robust, and extensible C++ library that can be seamlessly integrated into other applications, abstracting away LLM provider complexities.
Secondary Goal: Develop a standalone server application that uses the ArbiterAI library to serve an OpenAI-compatible API, model management endpoints, and a live stats dashboard for local model inference.
C++ developers, data scientists, and engineers building applications that require LLM integration — particularly those focused on performance, control, and multi-provider flexibility.
- Unified C++ Interface — Simplifies LLM integration with a consistent API across diverse providers
- High Performance — Leverages modern C++17 features and efficient third-party libraries
- Extensibility — Clear provider pattern makes it straightforward to add new LLM services or local models
- Robust Error Handling — Clear and predictable error reporting for debugging and application stability
A single, intuitive API for both standard (synchronous) and streaming (asynchronous) responses, offering a consistent interaction model regardless of the underlying LLM provider.
The ChatClient provides a stateful, session-oriented interface that manages conversation history, tool definitions, caching, and usage statistics per session.
Implemented through a clean provider pattern using the BaseProvider interface, enabling easy integration of new LLM services with minimal changes to core logic.
| Provider | Type | Description |
|---|---|---|
| OpenAI | Cloud | Full support for OpenAI's ChatCompletion API |
| Anthropic | Cloud | Integration with Anthropic's Messages API |
| DeepSeek | Cloud | Support for DeepSeek's ChatCompletion API |
| OpenRouter | Cloud | Unified interface for models via the OpenRouter aggregation service |
| Llama.cpp | Local | Integration with llama.cpp for local model inference (currently disabled in build) |
| Mock | Testing | Deterministic testing provider with echo tag support |
The ModelManager dynamically loads model configurations from multiple sources:
- JSON configuration files with schema validation (
schemas/model_config.schema.json) - Remote configuration updates via Git through
ConfigDownloader - Environment variables for API keys (e.g.,
OPENAI_API_KEY,ANTHROPIC_API_KEY) - Layered configuration — remote, local, and override paths with clear precedence
A well-defined ErrorCode enum for predictable, granular error reporting across all operations.
Support for LLM tool/function calling with JSON schema-based parameter definitions via ToolDefinition and ToolCall structures.
Optional CacheManager provides TTL-based caching at both session and global scope to reduce API costs and latency.
The CostManager tracks spending with global and per-session limits, persisting cost state across application restarts.
Asynchronous streamingCompletion function handles Server-Sent Events (SSE) for real-time token delivery in interactive applications.
| Library | Purpose |
|---|---|
| cpr | HTTP requests |
| nlohmann/json | JSON parsing and generation |
| nlohmann/json-schema-validator | JSON schema validation |
| spdlog | Logging |
| libgit2 | Git operations for config downloads |
| picosha2 | SHA256 hashing for file verification |
| cpp-httplib | HTTP server (proxy example) |
| Google Test | Testing framework |
- Developer Guide — Architecture, API reference, and component details
- Testing Guide — Mock provider and testing strategies
- Development Process — Workflow and task tracking
- Examples — Example applications and usage patterns