ActiveHarness

⚠️ Work in progress. The API is under active development and may change between versions without notice.

📛 Naming note. As of v0.3.0, ActiveHarness::Agent was renamed to ActiveHarness::Request — it's a single resilient LLM call (model chain, fallback, retry, hooks), not an autonomous entity, and Request describes that more accurately. The Agent name is reserved for a future, separate abstraction — something that uses a model to decide and take actions — which doesn't exist in this gem yet.

Running a single LLM call is easy. Running a reliable, observable, cost-controlled AI system is not.

ActiveHarness is a Ruby framework for building production-grade LLM pipelines — with deep observability, consensus-based decisions, automatic fallbacks, and real-time cost and timing control. Made for Rails, works in plain Ruby too.

ActiveHarness gem gives you the scaffolding to build multi-step pipelines where every request is under full control: its inputs are directed, its outputs are observed, its errors are retried, and its cost is tracked. You define the logic; ActiveHarness handles the infrastructure.

What is a "Harness"?

A harness in software is scaffolding that keeps a component under control — directing its inputs, observing its outputs, and enforcing rules around it. ActiveHarness does exactly that for LLM requests.

Build AI-Based Pipelines!

Build multi-step, trackable, cost-effective, and reliable AI flows with a clean, Rails-native DSL.

Build Nested AI Pipelines!

Group related steps into reusable sub-pipelines, and compose complex workflows from smaller ones. Each pipeline is just another step, with its own stop conditions, context forwarding, and execution time tracking.

Compose Hybrid Pipelines!

Orchestrate deterministic and AI steps together.

Control the Cost of Your AI Calls!

With ActiveHarness you can track time, tokens, and dollars for every request call, pipeline step, and tribunal.

Cost in Application	Provider's Cost

Use Consensus-Based Decisions!

Use Tribunals to run multiple requests in parallel and make Verdicts based on their agreement — improving reliability and reducing biases and hallucinations.

Provide Event Tracing & Observability!

Use power of event hooks to log and trace every step of your AI flows, from individual request calls to multi-step pipelines and parallel tribunals.

Event Tracing Architecture	Grafana Dashboard

Backend Agnostic — Built on OpenTelemetry, ready for any collector (Jaeger, Datadog, Honeycomb, or custom).

Use Memory to make your requests stateful!

Store conversation history in JSON, SQLite and PostgreSQL. Inject memory into prompts to make requests that remember past interactions.

Visualize Models' Cost and Type

Add Streaming (SSE)!

Rails App	Console

Evaluate & Moderate Content with Jev!

Not every request needs a chat model. Use provider: :vercel to reach Jev, TypeSafe AI's "System One" evaluation model, via Vercel AI Gateway — ask typed score/choice/noul questions in one call and get back typed, probabilistic answers instead of free text. A natural fit for classification, routing, and content moderation.

See docs/JEV.md for the full request/response schema and use-case examples (ticket triage, refund detection, tool-call risk gating, content moderation).

Key Capabilities

Capability	What it means
Multi-step Pipelines	Chain requests sequentially, with per-step stop conditions and context forwarding
Tribunal Consensus	Run multiple requests in parallel and accept the result only if they agree (unanimous, majority, or custom)
Automatic Fallbacks	If a model fails, the next one in the chain takes over — zero extra code
Retry Policy	Exponential backoff per model, globally configurable or per-request
Full Observability	Lifecycle hooks on every request event: before_call, after_call, retry, failure — log, stream, or act
Real-time Streaming	SSE-ready token streaming from any request into your Rails response
Execution Time Tracking	Per-request and per-pipeline timing built in
Token  Cost Tracking	Know exactly what each call cost in tokens and dollars
Rails-native DSL	Clean file structure, Railtie integration, generator support
Event Tracing	OpenTelemetry integration for distributed tracing of requests, tribunals, and pipelines

File Structure

File structure for Ruby and Ruby on Rails applications:

Place all of your AI-related code in app/ai to keep it organized and separate from your core application logic. You can further organize it into subdirectories for prompts, requests, tribunals, pipelines, and memory.

app/
├── models/
├── controllers/
├── views/
└── ai/
    ├── prompts/      # system prompt classes
    ├── requests/       # request classes
    ├── tribunals/    # parallel verdict panels
    ├── pipelines/    # multi-step pipelines
    └── memory/       # custom memory classes

Prompt Documentation

• How Prompts Work
• Minimal Prompt
• Multi-line Prompt
• Dynamic Content with @context
• Tuning with @params
• Using @input in the Prompt
• JSON Output Prompt
• Using @memory for Conversation History
• Respecting @context_window
• Generator

Request Documentation

• How to Create Your First Request in 5 Minutes
• How to Provide Fallbacks
• Model Options
• Modifying the Model Chain at Runtime
• How to Track Retries and Failures
• How to Use with RubyLLM
• JSON Output and Parsing
• Lifecycle Events
• Custom Providers
• Streaming in the Console
• Streaming in a Rails App
• Image Generation
• Audio Transcription
• Jev / TypeSafe AI Evaluation (Vercel AI Gateway)

Pipeline Documentation

• How to Create Your First Pipeline in 5 Minutes
• Step Types
• Stop Conditions
• Tribunal Step
• Context: Accessing Previous Step Results
• Lifecycle Events
• Event Streams
• Full Example

Nested Pipelines Documentation

• How It Works
• Minimal Example
• Stopping the Outer Pipeline from the Inside
• transform — Why It Is Required
• Accessing Inner Results from the Outer Context
• Event Streams
• Multiple Levels of Nesting
• Reference

Tribunal Documentation

• How to Create Your First Tribunal in 5 Minutes
• Tribunal from Different Requests
• Verdict Strategies
• Custom Verdict Logic
• Tolerating Partial Failures
• Same Request, Different Models
• Runtime Model Prepend per Request
• Direct Usage
• Lifecycle Events

Memory Documentation

• How Memory Works
• JsonFile Memory
• Custom Memory Class
• Managing Memory via Request Callbacks
• Injection Patterns
• Filtering History with to_messages
• Memory API Reference
• Memory with namespace:
• Sharing Memory Across Pipeline Steps
• PostgreSQL Backend
• SQLite Backend

Installation and Configuration

• Installation
• Rails Setup
• Plain Ruby Setup
• Configuration
• Providers Reference
• Global Settings
• Generators

Tracing and Observability

• How It Works
• Simple Logging with Hooks
• OpenTelemetry Setup
• RequestTracing Concern
• TribunalTracing Concern
• PipelineTracing Concern
• Span Hierarchy
• Connecting to a Backend

License

MIT © the-teacher