Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,10 @@ required-features = ["derive", "openai", "tools"]
name = "mock_testing_example"
required-features = ["derive", "mock"]

[[example]]
name = "fixture_record_replay"
required-features = ["derive", "mock"]

[[example]]
name = "retry_attempt_ledger"
required-features = ["derive", "openai"]
Expand Down
43 changes: 42 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,7 @@ complete copy-paste recipes.
| Read a chart with Kimi K3 | [`kimi_k3_multimodal_example.rs`](examples/kimi_k3_multimodal_example.rs) | Downloads a labeled revenue chart, sends it through Moonshot's OpenAI-compatible endpoint, and returns typed values plus calculated insights. |
| Put extraction behind an axum handler | [`axum_handler_example.rs`](examples/axum_handler_example.rs) | Injects any `LLMClient` into typed JSON request and response handling, tested in-process. |
| Test without network | [`mock_testing_example.rs`](examples/mock_testing_example.rs) | Scripts realistic responses through the real deserialization, validation, and re-ask path. |
| Record and replay a sanitized fixture | [`fixture_record_replay.rs`](examples/fixture_record_replay.rs) | Persists a versioned interaction with usage and attempts, then strictly replays it offline. |
| Use a local model with Ollama | [`ollama_local_example.rs`](examples/ollama_local_example.rs) | Connects to the keyless local endpoint through the same structured-output API. |
| Choose a provider at runtime | [`runtime_provider_example.rs`](examples/runtime_provider_example.rs) | Parses `provider/model` into one `AnyClient`, including aggregator model IDs with slashes. |
| Reuse an existing schemars model | [`schemars_bridge_example.rs`](examples/schemars_bridge_example.rs) | Materializes `JsonSchema + Serde` types through the transparent `Schemars<T>` adapter. |
Expand Down Expand Up @@ -958,6 +959,46 @@ feature pulls in only the lightweight path-aware decoder and works without the H
client; streaming and tool-loop mocking light up when the `streaming` / `tools`
features are also enabled. See `examples/mock_testing_example.rs`.

### Record, sanitize, and replay fixtures

`FixtureRecorder<C>` is an `LLMClient` wrapper for turning representative
non-streaming calls into versioned JSON fixtures. Sanitization is mandatory and
runs before a request or response is retained. Credential-shaped JSON fields are
redacted structurally, inline media bytes are never stored, and the callback lets
you remove domain-specific identifiers:

```rust
use rstructor::{
Fixture, FixtureRecorder, FixtureSanitizer, Instructor, LLMClient, OpenAIClient,
};
use serde::{Deserialize, Serialize};

#[derive(Debug, PartialEq, Instructor, Serialize, Deserialize)]
struct Position { account_id: String, symbol: String, quantity: i64 }

fn sanitizer() -> FixtureSanitizer {
FixtureSanitizer::new(|text| text.replace("HF-ALPHA-001", "[ACCOUNT]"))
}

let recorder = FixtureRecorder::new(OpenAIClient::from_env()?, sanitizer());
let position: Position = recorder.extract("HF-ALPHA-001 owns 1,000 AAPL shares").await?;
recorder.save("tests/fixtures/position.fixture.json")?;

// CI: no API key or network.
let fixture = Fixture::load("tests/fixtures/position.fixture.json")?;
let replay = fixture.replay_with_sanitizer(sanitizer());
let replayed: Position = replay.extract("HF-ALPHA-001 owns 1,000 AAPL shares").await?;
assert_eq!(replayed, position);
replay.assert_finished()?;
```

Replay is ordered and strict. Operation, sanitized prompt, target schema, and
media metadata must match before an interaction is consumed, so prompt or type
drift fails locally instead of silently returning a stale fixture. The fixture
also preserves token usage, attempt reports, typed provider errors, HTTP status,
and request IDs. See `examples/fixture_record_replay.rs` for a runnable,
key-free round trip based on a real 10-K metric.

## Feature Flags

```toml
Expand All @@ -970,7 +1011,7 @@ rstructor = { version = "0.5.1", features = ["openai", "anthropic", "grok", "gem
- `logging` — Tracing integration
- `streaming` — Streaming via `generate_stream` / `materialize_iter` / `materialize_stream` (opt-in)
- `tools` — Tool/function calling via `Toolbox` + `client.with_tools(..).run(..)` (opt-in)
- `mock` — `MockClient` for offline unit testing (opt-in; see [Testing](#testing-offline))
- `mock` — `MockClient` plus record/sanitize/replay fixtures for offline testing (opt-in; see [Testing](#testing-offline))

All features are on by default. For a **schema-only build** — generate JSON Schema from your types with no networking, `tokio`, or `reqwest` — disable the providers:

Expand Down
32 changes: 32 additions & 0 deletions docs/COOKBOOK.md
Original file line number Diff line number Diff line change
Expand Up @@ -309,6 +309,38 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
Scripted payloads still pass through the production Serde and `Instructor`
validation path; only the provider transport is replaced.

## Record and replay a sanitized fixture

The [fixture example](../examples/fixture_record_replay.rs) records a complete
non-streaming interaction and replays it without a key or network. The required
sanitizer runs before values enter the in-memory fixture; inline media bytes are
never persisted.

```rust
use rstructor::{Fixture, FixtureRecorder, FixtureSanitizer, LLMClient, OpenAIClient};

fn sanitizer() -> FixtureSanitizer {
FixtureSanitizer::new(|text| text.replace("PRIVATE-ACCOUNT", "[ACCOUNT]"))
}

# async fn example() -> Result<(), Box<dyn std::error::Error>> {
let recorder = FixtureRecorder::new(OpenAIClient::from_env()?, sanitizer());
let fill: Fill = recorder.extract("PRIVATE-ACCOUNT bought 5,000 AAPL at 238").await?;
recorder.save("tests/fixtures/fill.fixture.json")?;

let fixture = Fixture::load("tests/fixtures/fill.fixture.json")?;
let replay = fixture.replay_with_sanitizer(sanitizer());
let replayed: Fill = replay.extract("PRIVATE-ACCOUNT bought 5,000 AAPL at 238").await?;
assert_eq!(replayed, fill);
replay.assert_finished()?;
# Ok(())
# }
```

Replay matches the sanitized operation, prompt, schema, and media metadata in
order. A mismatch leaves the interaction unconsumed and reports only the field
that differed, so assertion failures do not echo fixture contents.

## Use a local model through Ollama

The [Ollama example](../examples/ollama_local_example.rs) is safe to run in CI:
Expand Down
54 changes: 54 additions & 0 deletions examples/fixture_record_replay.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
//! Record, sanitize, save, and strictly replay an offline fixture.
//!
//! Run with:
//! `cargo run --example fixture_record_replay --features mock`

use rstructor::{
Fixture, FixtureRecorder, FixtureSanitizer, Instructor, LLMClient, MockClient, TokenUsage,
};
use serde::{Deserialize, Serialize};

const APPLE_2023_PROMPT: &str = "From Apple Inc.'s 2023 Form 10-K: Total net sales were $383,285 million for the fiscal year ended September 30, 2023. Extract the issuer, fiscal year, and net sales in USD millions.";

#[derive(Debug, PartialEq, Instructor, Serialize, Deserialize)]
struct FilingMetric {
issuer: String,
fiscal_year: u16,
net_sales_usd_millions: u64,
}

fn fixture_sanitizer() -> FixtureSanitizer {
// Replace account IDs, customer names, or other private strings here.
// Credential-shaped JSON fields and inline media bytes are redacted separately.
FixtureSanitizer::new(str::to_owned)
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// A live recording run can wrap OpenAIClient, AnthropicClient, GeminiClient,
// or GrokClient in exactly the same way. MockClient keeps this example key-free.
let source = MockClient::new().with_response_and_usage(
r#"{"issuer":"Apple Inc.","fiscal_year":2023,"net_sales_usd_millions":383285}"#,
TokenUsage::new("recorded-provider-model", 67, 24),
);
let recorder = FixtureRecorder::new(source, fixture_sanitizer());
let recorded = recorder
.extract_with_report::<FilingMetric>(APPLE_2023_PROMPT)
.await?;
assert_eq!(recorded.data.net_sales_usd_millions, 383_285);

std::fs::create_dir_all("tmp")?;
recorder.save("tmp/apple-2023-10k.fixture.json")?;

let fixture = Fixture::load("tmp/apple-2023-10k.fixture.json")?;
let replay = fixture.replay_with_sanitizer(fixture_sanitizer());
let replayed = replay
.extract_with_report::<FilingMetric>(APPLE_2023_PROMPT)
.await?;
assert_eq!(replayed.data, recorded.data);
assert_eq!(replayed.report, recorded.report);
replay.assert_finished()?;

println!("recorded and replayed {} interaction", fixture.len());
Ok(())
}
Loading
Loading