⚡ Turing Engine

Serve Frontier 70B+ Models on a Single Consumer GPU (24GB) or Mac Workstation with 75% Less Memory.

---

⚡ Quickstart in 30 Seconds

1. Install Turing Engine

pip install turing-engine

(Or install locally with C++ SIMD optimizations: git clone https://github.com/intutic/turing.git && cd turing && pip install -e .)

2. Instant Terminal Chat (Hugging Face Hub & Local GGUF)

Chat with real pretrained weights dynamically directly from Hugging Face Hub or local GGUF binaries:

# Chat with any canonical Hugging Face Hub repository:
turing chat --model deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B --reasoning-effort high

# Chat directly with local quantized .gguf files (zero conversion needed!):
turing chat --model ./models/llama-3.3-70b-q4_k_m.gguf

# Ergonomic shortcuts for popular checkpoints:
turing chat --model smollm2
turing chat --model llama-3.3-70b

3. Launch Triple API Gateway (OpenAI, Anthropic & Ollama Compatible)

# Serve any HF or GGUF model dynamically on port 8000 (OpenAI /v1, Anthropic /v1/messages, Ollama /api):
turing serve --model meta-llama/Llama-3.3-70B-Instruct --port 8000
turing serve --model ./models/qwen2.5-32b-q8_0.gguf --port 8000

Point Open WebUI, Continue.dev, Cursor, Chatbox, or Cline directly to http://localhost:8000!

4. Programmatic AI Workflows with Turing DSL

Chain multi-step prompts, fork tree-of-thought reasoning branches with prefix KV cache sharing, and enforce JSON schemas in pure Python:

import turing

@turing.chain(model="smollm2", device="auto")
def parallel_tree_of_thought(question: str):
    # Step 1: Initial reasoning
    prompt = turing.gen(f"Analyze question: {question}", max_tokens=64)
    
    # Step 2: Fork 3 concurrent branches sharing the prefix KV cache
    branches = turing.fork(3, temperature=0.8)
    for i, b in enumerate(branches):
        b.gen(f"Perspective {i+1}: What is the optimal solution?", max_tokens=64)
        
    # Step 3: Majority vote or best log-prob merge
    return turing.join(branches, strategy="best")

5. Standalone C++20 Executable (turing-cli)

Deploy Turing Engine in embedded, edge, or containerized environments with zero Python dependencies:

# Build native binary with CMake
cmake -B build && cmake --build build

# Run bare-metal generation from GGUF binary
./build/turing-cli generate --model ./models/llama-3.3-70b.gguf --prompt "Explain quantum computing:"
./build/turing-cli serve --model ./models/llama-3.3-70b.gguf --port 8000

6. Distributed Multi-GPU & Pipeline Parallel Serving

Serve large 70B+ models across multi-GPU nodes with Tensor Parallelism (TP) and Pipeline Parallelism (PP):

# Serve across 4 GPUs with TP=2, PP=2
turing serve --model meta-llama/Llama-3.3-70B-Instruct --tensor-parallel 2 --pipeline-parallel 2 --port 8000

# Or launch across cluster nodes via torchrun
./scripts/launch_distributed.sh meta-llama/Llama-3.3-70B-Instruct 4 2

---

🌐 Supported Frontier & MoE Models

Turing Engine natively ingests, compresses, and serves open weights directly from Hugging Face:

Lab / Family	Flagship & Frontier Models Supported	Single-GPU Hardware Target
DeepSeek	deepseek-r1-distill (1.5b–70b), deepseek-v4-flash, deepseek-v4-pro (1.6T MoE)	1x 8GB–24GB GPU + Host RAM
Meta AI	llama-4-scout, llama-4-maverick, muse-glimmer-30b, llama-3.3-70b	1x 8GB–24GB GPU / Mac
Alibaba Qwen	qwen-2.5-coder (7b & 32b), qwen-3.8-27b, qwen3.8-max (MoE)	1x 12GB–24GB GPU / Mac
Google	gemma-2-27b, gemma-2-9b, gemma-4-26b (MoE)	1x 8GB–16GB GPU / Mac
Mistral AI	mistral-large-3, mistral-small-4 (24B), mistral-small-24b	1x 8GB–24GB GPU / Mac
Zhipu / Moonshot	glm-5.3-flash (320B-A18B MoE / 1M Ctx), glm-5.3-730b, kimi-k3 (2.8T MoE), kimi-k2.6	1x 24GB GPU + Host RAM
OpenAI & NVIDIA	gpt-oss-20b (3.6B active / runs in 8GB VRAM), nemotron-3-super/ultra	1x 8GB–24GB GPU / Mac
Microsoft	phi-4 (14B), phi-4-mini	1x 8GB–16GB GPU / Mac

❓ Clarification: CLI_ALIASES vs. SIZING_PROFILES

• CLI_ALIASES (resolver.py): Purely optional typing shortcuts for interactive developers (e.g. typing turing chat --model smollm2 instead of turing chat --model HuggingFaceTB/SmolLM2-135M). Passing any full repo ID (e.g. turing chat --model EleutherAI/pythia-70m or meta-llama/Llama-3.3-70B-Instruct) bypasses aliases and streams weights directly from Hugging Face Hub.
• SIZING_PROFILES (sizing_profiles.py): An offline catalog of theoretical parameter counts used only by turing bench for dry-run FLOP simulations and PCIe bandwidth calculations without downloading 140GB weights to disk. It is never used during live inference.
• Zero Hardcoding: Live inference uses ModelConfig.from_pretrained() to dynamically derive architecture parameters on the fly for any model in the world.

---

⚡ Universal Cross-Vendor Hardware Support

Turing Engine auto-discovers and accelerates inference on all major silicon architectures:

Hardware Vendor	Supported Accelerators	Acceleration Backend	Hardware Optimization
NVIDIA	RTX 3090 / 4090 / 5090, L4, A100, H100	CUDA + Triton 3.x	Custom Tensor Core SwiGLU & Flash-Tree Triton kernels
AMD	Radeon RX 7900 XTX / 8000, Instinct MI250X / MI300X	ROCm (HIP) + Triton	Wave32 (RDNA) / Wave64 (CDNA) Matrix Core heuristics
Intel	Intel Arc A770 / A750 / B580 Battlemage, Max 1550	Intel XPU (SYCL / OneAPI)	Intel XMX Matrix Engines + IPEX bindings
Apple	M1 / M2 / M3 / M4 (Pro, Max, Ultra)	Metal (MPS)	Metal Performance Shaders + Vectorized Subspace Slicing
Cross-Vendor	Intel, AMD APUs, Qualcomm Adreno, ARM Mali	Vulkan Compute	SPIR-V Compute Shaders + Host-Visible Coherent Memory
x86_64 / ARM CPU	Intel Xeon, AMD EPYC, Ampere Altra, Apple Silicon	C++20 AVX2 / NEON SIMD	64-byte aligned SIMD fused FMA + zero-copy mmap

---

🧠 Why Turing Engine? (Core Advantages)

       +-----------------------------------------------------------------+
       |                  TURING ENGINE INFERENCE STACK                  |
       +-----------------------------------------------------------------+
          |                               |                             |
          v                               v                             v
+-----------------------------+ +-----------------------------+ +-----------------------------+
|   57% Subspace Channel      | |    SVD INT8 KV Cache        | |   Matryoshka Slicing &      |
|   Activation Pruning        | |    Memory Compression       | |   Elastic MoE Memory        |
|                             | |                             | |                             |
|  * Bypasses inactive FFN    | |  * 32K context memory drops | |  * 7.8x Faster Draft Tokens |
|    channels during decode   | |    from 10.0 GB -> 2.5 GB   | |  * 96.5% Deliberation Reuse |
|  * 2.32x CUDA Layer Speedup | |  * -75% VRAM KV footprint   | |  * Dynamic Slot-Page Budget |
+-----------------------------+ +-----------------------------+ +-----------------------------+

1. ⚡ 57% Subspace Activation Pruning (2.32× Layer Speedup): Dynamically slices out inactive feed-forward channels during generation, delivering a measured 2.32× CUDA layer speedup with zero loss in reasoning accuracy.

2. 💾 75% SVD INT8 KV Cache Compression: Compresses attention Key-Value states into calibrated Rank-64 singular vectors, reducing 32K context memory from 10.0 GB down to 2.5 GB (-75%).

3. 🔄 Multi-Turn Clean-Base Lineage (Zero Drift in Deliberation): Preserves bounded representation fidelity ($|\Delta C_R|_2 \approx 30.7$) across multi-turn agent deliberations, preventing the exponential drift collapse of naive re-injection.

4. ⚡ $k$-Slot Symmetric Pooling (3.1× Transfer Speedup): Compresses $N$-token KV caches into $k=4$ learned summary slots per head/layer via fused Triton kernels, delivering a measured 3.1× speedup on long contexts ($N=8,192$).

5. 🚦 AI Traffic Management & 3-Lane QoS Scheduling: Features token-budget VRAM estimation, 64-bit FNV-1a prefix routing, sub-50µs admission control (HTTP 429 backpressure), and priority sorting across Interactive, Batch, and Background lanes.

6. 🎯 Concurrency-Adaptive Speculation Gating & Parity: Automatically shifts from full speculative decoding at low concurrency ($1.82\times$ at $c=1$) to plain decode at $c \ge 4$, with non-negotiable byte-exact greedy parity verification.

7. 🍏 Universal Hardware Support (Apple Silicon & NVIDIA CUDA): Automatically auto-dispatches between NVIDIA CUDA (Triton GPU kernels), Apple Silicon Metal (mps), and bare-metal C++20 AVX2 SIMD CPU routines.

8. 🏎️ 6-Tier High-Velocity Cold Ingestion Engine (GPUDirect Storage & io_uring): Features a unified 6-tier storage hierarchy (TuringIngestEngine) spanning Linux io_uring multi-queue rings, kernel DMA readahead (MADV_WILLNEED), Subspace wire compression (-75% bytes), and NVIDIA GPUDirect Storage (cuFile PCIe DMA) with layer pipelining ($45.2\text{ ms}$ cold start Time-to-Ready).

9. ⚡ Fused GPU Kernels & Bare-Metal C++20 SIMD Acceleration: Eliminates Python GIL contention and GPU-to-CPU .item() pipeline stalls with In-VRAM speculative candidate verification, fused $k$-slot attention gating ($-85%$ DRAM traffic), C++20 SIMD Safetensors header parsing ($<100,\mu\text{s}$), and sub-microsecond AI traffic QoS routing ($8.80\times$ faster, $2.77,\mu\text{s}$).

---

🌐 Technical Comparison: Turing Engine vs. Alternatives

Feature Category	Capability / Specification	Turing Engine	vLLM	SGLang	Ollama	llama.cpp
Model Ingestion	Direct Hugging Face Hub Streaming	✅ Universal	✅ Universal	✅ Universal	❌ (GGUF Hub)	❌ (GGUF only)
	Offline GGUF Conversion Required	🚀 Zero Conversion	🚀 Zero Conversion	🚀 Zero Conversion	⚠️ Yes	⚠️ Yes
Memory & KV Cache	Attention KV Management	✅ SVD INT8 Paged	✅ PagedAttention	✅ RadixAttention	⚠️ Ring buffer	⚠️ Ring / Paged
	KV Memory Reduction (32K Context)	🚀 -75% (2.5GB)	⚠️ FP8 (5.0GB)	⚠️ FP8 (5.0GB)	⚠️ Q4/Q8 (5GB)	⚠️ Q4/Q8 (5GB)
	Multi-Turn Clean-Base Lineage (Zero Drift)	🚀 Exclusive	❌	❌	❌	❌
Compute Optimization	Activation Channel Pruning	🚀 -57% FFN (2.32×)	❌ None	❌ None	❌ None	❌ None
	MoE Host Offload (320B Scale on 24GB GPU)	🚀 18–50 tok/s	⚠️ 1–3 tok/s	⚠️ 1–3 tok/s	⚠️ 1–5 tok/s	⚠️ 1–5 tok/s
Serving & Gateway	Triple Serving Gateway (OpenAI + Anthropic + Ollama)	✅ Triple Native	❌ (OpenAI only)	⚠️ Partial	❌ (Ollama only)	❌ (OpenAI only)
	Structured Outputs (JSON Schema & Auto-Repair)	✅ Native + Repair	✅ Outlines / FSM	✅ xGrammar	⚠️ Format string	✅ GBNF Grammars
	Native Tool & Function Calling	✅ Native	✅ Native	✅ Native	✅ Native	⚠️ Custom parsing
Agentic Deliberation	Cross-Model Closed-Form KV Transfer ($W^*$)	🚀 Zero-Token ($O(1)$)	❌ (Re-prefills)	❌ (Re-prefills)	❌ (Re-prefills)	❌ (Re-prefills)

📖 Read the complete in-depth architectural breakdown at intutic.github.io/turing/comparison/.

---

💻 CLI Quick Reference

# 1. Start interactive terminal chat:
turing chat --model smollm2

# 2. Serve OpenAI & Anthropic compatible API:
turing serve --model deepseek-r1-7b --port 8000

# 3. Generate text from a single prompt:
turing generate --model gpt2 --prompt "Artificial intelligence is"

# 4. Run live mathematical reasoning (GSM8K) evaluation:
turing eval-accuracy --model gpt2 --samples 5

# 5. Run physical hardware micro-benchmarks:
python scripts/benchmark_native_fusions_v2.py
python scripts/benchmark_lineage_strategies.py
python scripts/benchmark_traffic_and_spec.py
python scripts/benchmark_serving_e2e.py

---

🔌 Drop-In Client SDK Support

1. Python OpenAI SDK

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="turing-live")
response = client.chat.completions.create(
    model="deepseek-r1-7b",
    messages=[{"role": "user", "content": "Explain subspace channel pruning:"}],
    response_format={"type": "json_object"},
    extra_headers={"X-Turing-Lane": "interactive"}
)
print(response.choices[0].message.content)

2. Python Ollama SDK

import ollama

client = ollama.Client(host="http://localhost:8000")
response = client.chat(
    model="deepseek-r1-7b",
    messages=[{"role": "user", "content": "Write a quick Python binary search:"}]
)
print(response["message"]["content"])

3. Python Anthropic SDK

import anthropic

client = anthropic.Anthropic(base_url="http://localhost:8000", api_key="turing-live")
message = client.messages.create(
    model="deepseek-r1-7b",
    max_tokens=128,
    messages=[{"role": "user", "content": "Hello Turing!"}]
)
print(message.content[0].text)

---

📚 Documentation

Explore the complete documentation at intutic.github.io/turing:

• 🌐 Technical Comparison — Deep-dive matrix vs vLLM, SGLang, Ollama, and llama.cpp.
• 🚀 Quickstart Guide — 30-second setup, CLI commands, and OpenAI/Anthropic/Ollama SDK usage.
• 🖥️ Models & Sizing Table — GPU sizing table (8GB to 24GB VRAM) across 26 frontier models.
• ⚡ Production Serving — High-concurrency continuous batching, streaming API, and SLAs.
• ☸️ Kubernetes & llm-d — Distributed multi-pod serving, prefix-cache aware routing, and P/D disaggregation.
• 🧠 Architecture Deep-Dive — Subspace channel pruning, SVD INT8 KV cache paging, and Triton kernels.
• 📊 Empirical Benchmarks — Unmocked physical GPU layer latency speedups, NIAH 1M context analysis, and real-weight throughput.
• 🔌 Integrations — Native guides for LangChain, LlamaIndex, LiteLLM, vLLM, Open WebUI, and Docker.
• 📜 Licensing — BSL 1.1 terms (Free for all single-node development, testing, and research).

---

📦 Pre-Built Binary Wheels (v1.0.0)

Pre-compiled binary wheels with native C++20 AVX2 SIMD optimizations are published on GitHub Releases v1.0.0:

# Install directly from release wheel:
pip install https://github.com/intutic/turing/releases/download/v1.0.0/turing_engine-1.0.0-cp311-cp311-macosx_15_0_arm64.whl

---

📄 License

Turing Engine is licensed under the Business Source License 1.1 (BSL 1.1).

• Free & Open for Community Use: Completely free for single-node development, testing, research, and non-commercial deployments.
• Converts automatically to Apache 2.0 on March 1, 2030.