imbh-go

A Go binding for IMBH — an embeddable, Rust observability database (logs, traces, metrics on Apache DataFusion) — with zero-copy Apache Arrow query results, fused onto Go's scheduler via sable.

You open an IMBH database from Go, feed it OpenTelemetry (OTLP) data, and query it with SQL or typed observability queries. Query results cross the language boundary as Arrow record batches without a copy: Go holds IMBH's Arc-refcounted Arrow buffers by pointer and releases them when done.

db, _ := imbhgo.OpenInMemory()
defer db.Close()
ctx := context.Background()

db.IngestOTLPLogs(otlpBytes)                 // what any OTLP/HTTP exporter sends

logs, _ := db.QueryLogsTyped(ctx, imbhgo.LogQuery{Service: "checkout", Match: "error"})
for _, e := range logs {
    fmt.Println(e.Time, e.Service, e.Body)
}

Status: early but working. The full path — open → ingest → SQL, typed queries, and the LGTM query languages (PromQL / LogQL / TraceQL), all zero-copy → errors → cancellation → backpressure — is implemented and tested (83 tests under -race, including on-disk durability, concurrency-under-load, and the admin / lifecycle / cursor-paging surface), and the memory-ownership protocol is leak-verified two independent ways (an in-process counter and a Valgrind gate). Dependencies are now external — imbh from crates.io and sable git-pinned — so it builds without local checkouts, and CI (.github/workflows/ci.yml) runs the whole gate on linux/amd64 and linux/arm64 for every push and pull request. See Feature coverage for what is and isn't exposed, and Limitations.

---

How it works

flowchart LR
    Go["Go program<br/>(imbhgo package)"]

    subgraph lib["libimbhgo.a — one static library"]
        direction LR
        Sable["sable<br/>Rust ⇄ Go runtime fusion"]
        Imbh["imbh::Db<br/>DataFusion + Tantivy"]
    end

    Go -->|"OTLP ingest (bytes)"| Sable
    Go -->|"SQL / typed queries"| Sable
    Sable --> Imbh
    Imbh -.->|"zero-copy Arrow batches<br/>(C Data Interface)"| Go

Loading

• Everything links into one combined static library (libimbhgo.a) containing IMBH, sable's runtime, and the binding's handlers.
• Queries stream one Arrow batch per await: the goroutine parks between batches while IMBH's engine runs, so no OS thread is blocked and the query executes lazily (bounded memory).
• Each result batch crosses via the Arrow C Data Interface (FFI_ArrowArray), imported by arrow-go/cdata — no serialization, no GoBytes copy.

The full design is in ARCHITECTURE.md.

Requirements

• Go 1.26.4 (pinned — sable reaches Go's internal ABI via //go:linkname). go.mod enforces this toolchain, so Go downloads it automatically.
• cgo (a C toolchain) to link the static library into your binary.
• The github.com/apache/arrow-go/v18 module (for reading result batches).
• Rust (stable, 1.96+) is needed only if you build the library from source. Consumers using a prebuilt archive (below) do not need Rust.

Prebuilt libimbhgo.a archives are published per platform (see Using a prebuilt library).

Using a prebuilt library (no Rust build)

You can depend on the binding without ever building the Rust side. cmd/imbhgo-fetch downloads the prebuilt libimbhgo.a for your platform from the matching GitHub release, verifies its checksum, and prints the CGO_LDFLAGS to link against it:

go get github.com/moriyoshi/imbh-go@v0.8.0
eval "$(go run github.com/moriyoshi/imbh-go/cmd/imbhgo-fetch@v0.8.0 -print-env)"
go build -tags sable_extern_lib ./...

• The sable_extern_lib build tag is required — it makes sable's Go package link the combined archive rather than its own libsable.a.

• -print-env emits POSIX shell syntax on every platform, including Windows — the shell that consumes the line is the one you invoked the tool from, and on Windows CI that is normally git-bash / MSYS2. From a native Windows shell, ask for its dialect explicitly:

  for /f "delims=" %i in ('go run github.com/moriyoshi/imbh-go/cmd/imbhgo-fetch@v0.8.0 -print-env -shell cmd') do @%i

  go run github.com/moriyoshi/imbh-go/cmd/imbhgo-fetch@v0.8.0 -print-env -shell powershell | Invoke-Expression

• The fetch tool caches the archive under your user cache dir (override with -dest) and re-detects glibc vs musl on Linux automatically (override with -libc).

• Prebuilt cells: Linux amd64/arm64 (glibc and musl), macOS amd64/arm64, and Windows amd64 (best-effort). Each archive is pinned to the Go toolchain in go.mod.

Building from source

To build the static library yourself (co-development, or a platform with no prebuilt), build the Rust side first; the Go package links it.

make            # cargo build --release in rust/  → rust/target/release/libimbhgo.a
make test       # go test -tags sable_extern_lib -race ./...

Then compile your program with the sable_extern_lib build tag:

go build -tags sable_extern_lib ./...

(The per-platform cgo linker flags live in link_linux.go / link_darwin.go / link_windows.go; they default to rust/target/release for a local build, and defer to CGO_LDFLAGS for a prebuilt.)

Cutting a release (maintainers)

make release VERSION=v0.1.2   # rewrite the version everywhere, then build + vet + race-test

That rewrites internal/release.Version — which the release workflow requires the tag to match — along with the version references in this README and in cmd/imbhgo-fetch, then runs the local gate. It stops there and prints the remaining steps: commit, merge to main, then, from an up-to-date main:

make tag                 # creates the signed tag named by internal/release.Version
git push origin v0.1.2   # this publishes

Pushing the tag is what triggers .github/workflows/release.yml, which cross-builds every prebuilt cell, attaches the archives plus SHA256SUMS to the release, and smoke-tests the result as a consumer would.

A pushed tag is permanent. proxy.golang.org caches the tag → commit mapping immutably as soon as anyone resolves it, whether or not the release workflow succeeded, so a bad tag cannot be moved — only abandoned for the next number. Two guards keep the tag from ever disagreeing with the tree: make check-version (run by CI on every push and PR) rejects a half-applied bump or a version that has gone backwards past a published tag, and make tag takes the tag name from the constant rather than from an argument, so there is nothing to mistype.

Quick start

A complete runnable tour lives in examples/quickstart — it ingests OTLP logs and metrics and queries them three ways:

make            # build the static library
make example    # go run -tags sable_extern_lib ./examples/quickstart

The package initializes the runtime automatically on first use — just import and go.

package main

import (
	"context"
	"fmt"

	imbhgo "github.com/moriyoshi/imbh-go"
)

func main() {
	db, err := imbhgo.OpenInMemory() // or imbhgo.Open("/path/to/data")
	if err != nil {
		panic(err)
	}
	defer db.Close()
	ctx := context.Background()

	// Ingest OTLP/HTTP export-request protobuf bytes (from any OTel SDK exporter).
	if _, err := db.IngestOTLPLogs(otlpLogBytes); err != nil {
		panic(err)
	}

	// Query with SQL — results stream as zero-copy Arrow batches.
	rows, err := db.Query(ctx, "SELECT service, count(*) AS n FROM logs GROUP BY service")
	if err != nil {
		panic(err)
	}
	defer rows.Close()
	for {
		rec, ok, err := rows.Next()
		if err != nil {
			panic(err)
		}
		if !ok {
			break
		}
		fmt.Println(rec) // an arrow.RecordBatch
		rec.Release()    // return the buffers when done
	}
}

Querying

SQL (lazy, zero-copy)

Query streams the result batch-by-batch through IMBH's lazy scan. Tables: logs, spans, metrics_gauge, metrics_sum, metrics_histogram, and more. IMBH's UDFs are available in SQL — including histogram_quantile, matches (full-text), and json_get_str:

rows, _ := db.Query(ctx,
    "SELECT histogram_quantile(0.95, explicit_bounds, bucket_counts) AS p95 " +
        "FROM metrics_histogram WHERE metric = 'http.server.duration'")

Query returns *Rows; iterate with Next() (arrow.RecordBatch, ok bool, err error) until ok is false, then check Err(). Every query takes a context.Context as its first argument — cancel a running query by cancelling that ctx, and a parked Next is interrupted and reports context.Canceled from Err().

Typed queries

Endpoint-shaped queries as native Go structs. Each has a raw form (returns *Rows of Arrow batches) and a decoded form (returns Go structs).

// Logs → []LogEntry
entries, _ := db.QueryLogsTyped(ctx, imbhgo.LogQuery{
    Service: "checkout",
    Match:   "timeout",       // full-text on the body
    Limit:   100,
})

// Metric range → a Matrix (one series per group-by label set)
m, _ := db.QueryMetricsTyped(ctx, imbhgo.MetricQuery{
    Metric:  "cpu.utilization",
    Step:    int64(time.Second),
    Start:   start.UnixNano(),
    End:     end.UnixNano(),
    GroupBy: []string{"host"},
})
for _, s := range m.Series {
    fmt.Println(s.Labels, len(s.Points))
}

// Span RED metrics → []SpanMetricPoint (calls / errors / p50 / p95 / p99)
red, _ := db.QuerySpanMetricsTyped(ctx, imbhgo.SpanMetricsQuery{
    Service: "checkout",
    Step:    int64(time.Minute),
    Start:   start.UnixNano(),
    End:     end.UnixNano(),
})

Typed queries collect eagerly (they materialize the result), so prefer them for bounded results (a limited page, a fixed-step range) and use Query(ctx, sql) for large, unbounded scans.

Every GroupBy / attribute predicate (LogVolumeBy, MetricQuery.GroupBy, SpanMetricsQuery.GroupBy, LogQuery.AttrEq and friends) also accepts "service.name" — or its column spelling "service" — even though service is a resource attribute lifted into a built-in column rather than a record attribute. Requires imbh 0.3.0: earlier versions resolved the key against the record attributes, silently collapsing a per-service breakdown into one empty-labelled series with the counts merged.

Tailing logs on the arrival clock

Every record carries two instants: Time (when the event happened) and ObservedTime (when ingest received it). A follow loop must watermark on the second. Event time is not monotone in arrival — a record can be emitted before one already delivered and still land after it, whether because ingest batches or because the line's own timestamp was trusted — so a loop that advances a Start bound by the newest event time it has seen silently drops those records. Ordering by arrival cannot be overtaken that way:

var watermark int64 // last arrival instant delivered
for {
    entries, err := db.QueryLogsTyped(ctx, imbhgo.LogQuery{
        Service:       "checkout",
        ObservedAfter: watermark,                 // strict: observed_time > watermark
        OrderBy:       imbhgo.LogOrderObservedTime,
        Limit:         500,
    })
    if err != nil {
        return err
    }
    for _, e := range entries {
        handle(e)
        if e.ObservedTime > watermark {
            watermark = e.ObservedTime
        }
    }
    time.Sleep(pollInterval)
}

ObservedTime is nullable — an OTLP producer need not send one, and LogEntry.ObservedTime is then 0. Such records sort last in either direction and never match ObservedAfter (SQL NULL > t is unknown), so a record that cannot be placed against a watermark is left out of the loop rather than replayed on every poll. OrderBy is orthogonal to Backward, which picks the direction along the chosen axis; the default is LogOrderTime, and an unrecognized value is rejected rather than silently falling back. Requires imbh 0.6.0.

LGTM query languages (PromQL / LogQL / TraceQL)

IMBH implements the Grafana-stack query languages as explicitly-versioned compatibility profiles, and the binding exposes them over the same zero-copy Arrow path. Constructs outside a profile are rejected with a clear diagnostic rather than silently approximated.

// PromQL → labeled series. Metric names resolve against the stored catalog
// (OTel dots map to Prometheus underscores: cpu.util → cpu_util).
series, _ := db.QueryPromQLSeries(ctx, "rate(http_requests_total[5m])", start, end, step)
for _, s := range series {
    fmt.Println(s.Labels, len(s.Points))
}

// LogQL has two result shapes, as in Loki:
lines,  _ := db.QueryLogQLLines(ctx, `{service="checkout"} |= "error"`, start, end, 100) // streams
counts, _ := db.QueryLogQLSeries(ctx, `count_over_time({service="checkout"}[5m])`, start, end, step) // matrix

// TraceQL → matching traces + the span ids its spanset selected.
// Note attribute scope: service.name is a *resource* attribute.
matches, _ := db.QueryTraceQLMatches(ctx, `{ resource.service.name = "checkout" }`, start, end)

// …and follow a match through to that trace's spans (zero-copy).
spans, _ := db.GetTraceSpans(ctx, matches[0].TraceID)

Each also has a raw *Rows form (QueryPromQL, QueryLogQL, QueryTraceQL, GetTrace). Like every query method, all take a context.Context first argument for cancellation.

Ingesting

IngestOTLPLogs/IngestOTLPTraces/IngestOTLPMetrics take the protobuf bytes an OTLP/HTTP exporter sends (an ExportLogsServiceRequest, etc.) and return a Receipt:

r, _ := db.IngestOTLPLogs(otlpBytes)
fmt.Println(r.Accepted, r.Rejected, r.Durable)

Data is queryable immediately, before any Flush(). Flush() seals the in-memory buffer into an on-disk segment (to bound memory).

Close() runs IMBH's clean shutdown — drain the async-ingest worker, join the background maintenance thread, a final seal, commit_pending — and returns the error that shutdown reported. The handle is released either way, so Close() is never retryable and a second call is a no-op returning nil. An error means the on-disk state is not the tidy post-shutdown one; it does not mean rows were lost. Whatever the final seal could not write stays in the WAL and replays on the next Open — the same state a crash would have left. defer db.Close() stays valid where you do not need to check.

Backpressure

By default ingest is unbounded and never refused. To cap concurrency, set a global in-flight limit and use the Try* variants, which return ErrBackpressure instead of piling on:

imbhgo.SetMaxInFlight(64)
if _, err := db.TryIngestOTLPLogs(otlpBytes); errors.Is(err, imbhgo.ErrBackpressure) {
    // shed load or retry with backoff
}
stats := imbhgo.RuntimeStats() // InFlight, Rejected, MaxInFlight, …

The cap is process-global and also bounds concurrent open result streams (an open Rows holds a slot until Close).

Zero-copy: lifetime rules (important)

Result batches wrap IMBH-owned Arrow buffers that are freed when you Release() the batch:

• Always Close() a Rows and Release() each RecordBatch you take from Next.
• Values read from a batch alias its buffers. In particular, arrow-go's String.Value(i) and binary accessors return data that points into the Arrow buffer without copying. Anything you keep past that batch's Release() must be copied out (e.g. strings.Clone for strings). The typed decoders (QueryLogsTyped, etc.) do this for you.

Feature coverage

What this binding exposes today, against IMBH's own surface. "Reachable via SQL" means the capability works but through db.Query(ctx, sql) rather than a dedicated method.

Ingest & lifecycle

IMBH capability	Go binding	Notes
OTLP ingest (logs / traces / metrics)	✅ IngestOTLPLogs / …Traces / …Metrics	returns a Receipt
Backpressure-aware ingest	✅ TryIngestOTLP*, SetMaxInFlight, RuntimeStats	ErrBackpressure at the cap
Open in-memory / on-disk	✅ OpenInMemory, Open(path)
flush (seal buffer → segment)	✅ Flush
close (drain, join maintenance, final seal, commit_pending)	✅ Close() error	the handle is released even when the shutdown errors; un-sealed rows stay in the WAL for the next open to replay
Read-only open (open_read_only)	✅ OpenReadOnly	many-reader / single-writer; writes on the handle are rejected
Builder options (memory budget, WAL mode, retention, compression, maintenance, promote)	✅ OpenWith(DbOptions)	host-runtime-Handle variants (async ingest, runtime-driven maintenance) deferred. Since imbh 0.8.0 the promote list and the retention policy are durable database state, so omitting PromoteKeys / RetentionDays inherits what the database carries — ResetPromote / ResetRetention are the explicit "clear it" spellings, and a read-only open always adopts the durable promote set
Flush policy (DbBuilder::flush, imbh 0.2.0)	✅ DbOptions.Flush	imbh's spec string ("interval=5s,wal=64MiB", or "manual"); needs MaintenanceBackgroundNs set, since that is what runs the scheduler
Duplicate-timestamp policy (DbBuilder::duplicates, imbh 0.5.0)	✅ DbOptions.Duplicates	imbh's spec string: "error_on_read" (default, fails the PromQL read), "last_wins" (collapse at read), "reject[,recent=N]" (drop at ingest → Receipt.Rejected, DbStats.IngestRejected). Since imbh 0.6.0 "last_wins" also collapses the typed metric reads (QueryMetricsTyped / QueryMetricInstant), which previously aggregated both points
Ops: stats, compact, maintain, snapshot, segments, segment_files, durable_through, export (Arrow IPC)	✅ Stats / Compact / Maintain / Snapshot / Segments / SegmentFiles / DurableThrough / Export (+ExportRecords)	writer-only ops error on a read-only handle. SegmentFiles answers correctly on a reader since imbh 0.8.0 — it used to report an empty list, indistinguishable from "no segments"
Bounded compaction (compact_bounded, imbh 0.8.0)	✅ CompactBounded(maxPartitions)	a pass costs the corpus, not the request; the bound makes it incremental. Partitions past it are untouched rather than half-done, so there is nothing to resume — call it again. CompactionReport.SegmentsConverged counts the 1→1 rewrites a promote change leaves behind
Pending housekeeping (commit_pending, imbh 0.8.0)	✅ CommitPending → PendingReport	the embedded half of out-of-process segment rewriting: validate an external preparer's records, one manifest delta. Maintain now does it on every pass too (MaintenanceReport.Pending*)
Durable promote set (promote / set_promote, imbh 0.8.0)	✅ Promoted / SetPromoted(keys) → []string	the database's set, not the handle's. SetPromoted seals first, persists, and returns what is in effect; segments sealed before a key was promoted answer through the attributes JSON until Compact converges them
Durable retention policy (retention, imbh 0.8.0)	✅ Retention → RetentionPolicy	both bounds optional; an external housekeeper applies the host's rules instead of inventing its own
Attribute statistics (attribute_stats, imbh 0.8.0)	✅ AttributeStats(AttrStatsOptions) → AttrStatsReport	cardinality, sigma (per-segment selectivity), the cardinality curve over a window ladder, and both verdicts — promote this key and index it, up to this width — computed upstream. Takes no lock and writes nothing, so it runs on a reader beside a live writer; PromotionCandidates feeds SetPromoted

Query surfaces

IMBH capability	Go binding	Notes
SQL over logs / spans / metrics_*	✅ Query(ctx, sql)	lazy, zero-copy, cancellable
SQL UDFs (histogram_quantile, matches, json_get_str, hex)	✅ via SQL	incl. computed quantiles
Typed log query	✅ QueryLogs, QueryLogsTyped → []LogEntry	curated field subset (below)
Metric range (resampled)	✅ QueryMetrics, QueryMetricsTyped → Matrix	GroupBy splits series
Raw metric samples	✅ QueryMetricPoints, …Typed → []MetricPoint	unaggregated; histogram → raw Rows
Span RED metrics	✅ QuerySpanMetrics, …Typed → []SpanMetricPoint	calls / errors / p50 / p95 / p99
One trace's spans	✅ GetTrace, GetTraceSpans → []Span	composes with TraceQL matches
Assembled Trace tree	✅ GetTraceForest, AssembleTrace → []*TraceNode	parent→child forest; orphans surface as roots
Log pagination (LogPage cursor)	✅ QueryLogPage → LogPage{Entries,Next,Stats}	opaque resume Cursor + per-page QueryStats
logs().volume / volume_by	✅ LogVolume / LogVolumeBy → []VolumeBucket	time-bucketed counts, optional group_by
logs().count	✅ CountLogs(ctx, LogQuery) → uint64	full count(*) over the filter; ignores Limit/Backward
Arrival axis (observed_after / LogOrder, imbh 0.6.0)	✅ LogQuery.ObservedAfter / LogQuery.OrderBy	filter and order on ingest time instead of event time; LogEntry.ObservedTime carries it back
traces().search (TraceSummary)	✅ SearchTraces → []TraceSummary	full attr-predicate set on TraceQuery
metrics().instant (Vector)	✅ QueryMetricInstant → []InstantSample	last sample per series
metrics().catalog / series / exemplars	✅ MetricCatalog / MetricSeries / MetricExemplars
metrics().dimensions (imbh 0.7.0)	✅ MetricDimensions → []MetricDimension	label axes + their distinct values; kind-agnostic, so it answers for a histogram, which PromQL cannot select bare; reports the promoted service axis that MetricSeries leaves out
attrs() discovery (names, values)	✅ AttrNames / AttrValues

LGTM query languages

Capability	Go binding	Notes
PromQL	✅ QueryPromQL, QueryPromQLSeries → []Series	metric names auto-resolved from the catalog (dots→underscores)
LogQL — range aggregation (matrix)	✅ QueryLogQL, QueryLogQLSeries → []Series	count_over_time, rate, …
LogQL — bare selector (streams)	✅ QueryLogQLLines → []LogEntry	{service="x"} |= "err"
TraceQL	✅ QueryTraceQL, QueryTraceQLMatches → []TraceMatch	returns trace + selected span ids
Out-of-profile constructs	❌ by design	rejected with a stable diagnostic rather than approximated

Typed-query field coverage

LogQuery now covers IMBH's builder broadly: Service, Match, AttrEq, Start, End, Limit, Backward, plus TraceID/SpanID correlation, SeverityAtLeast, the richer attribute predicates (AttrExists, AttrMatches, AttrIn/AttrNotIn, AttrGt/Ge/Lt/Le, AttrRegex), and the arrival axis (ObservedAfter, OrderBy); cursor paging is via QueryLogPage. TraceQuery (trace search) exposes service / name / status / kind / duration bounds / time range / limit and the same attribute-predicate set. SpanMetricsQuery and MetricQuery cover the common fields.

Transport & correctness

Property	Status
Zero-copy Arrow results (C Data Interface)	✅ all query surfaces
Lazy streaming (bounded memory)	✅ SQL only — typed/LGTM queries collect eagerly upstream
Context cancellation	✅ ctx first arg on every query
Error surfacing	✅ Rows.Err(), Go errors on byte ops
Leak-verified ownership	✅ live-batch counter + Valgrind gate

Limitations & roadmap

Honest current state:

• API stability. Pre-1.0, so breaking changes land in minor releases. In v0.8.0, two behavior changes carry no compile error with them: DbOptions.PromoteKeys and DbOptions.RetentionDays left unset now inherit the database's stored setting instead of resetting it (imbh 0.8.0 made both durable state — use ResetPromote / ResetRetention to clear), and a filter on an attribute key promoted after some rows were sealed now matches those rows through the attributes JSON, where it used to silently match none of them. Also in v0.8.0: SegmentFiles on a read-only handle returns the real paths rather than an empty list. v0.7.0 changed (*DB).Close() to return error instead of nothing — a failed final seal used to be silent; existing defer db.Close() call sites still compile.
• Packaging. Consumed as external deps — imbh from crates.io (0.8.0) and sable git-pinned to a main commit — so no local checkouts are required to build. CI runs the standard gate on every push and pull request; cold build is minutes; the static archive is large.
• Durability. Covered by TestDurabilityReopen (ingest → flush → close → reopen the same path → query without re-ingest, exercising WAL replay / segment reload). Read-only opens are tested too.
• Concurrency. Stress-tested by TestConcurrentQueries (48 goroutines × 60 iters, mixed SQL + typed queries against one shared Db, all under -race).
• Portability. linux/{amd64,arm64} and windows/amd64 are gated in CI: full build + go test -race, with the Windows leg building for x86_64-pc-windows-gnu and linking through the cgo directives natively. Durable (on-disk) DBs need imbh 0.1.1 or newer on Windows — 0.1.0 failed every on-disk open there (imbh#3). The Apple cells are compile-checked in CI (cargo check on a macOS runner) and built on release tags, but the Go suite has never run there.
• Query surface. SQL, typed queries, LGTM (PromQL / LogQL / TraceQL), cursor-paged logs (QueryLogPage
  • QueryStats), trace search, log volume, metric catalog / series / exemplars / instant, and attribute discovery are all exposed, alongside the read-only / builder-options / ops admin surface — see Feature coverage for the exact matrix. Remaining: typed and LGTM queries collect eagerly upstream (only SQL streams lazily), logs().count is reachable via SQL rather than a dedicated method, and the two host-runtime-Handle builder variants (async ingest, runtime-driven maintenance) are deferred.

Testing

make test            # go test -tags sable_extern_lib -race ./...
make leak-valgrind   # Valgrind buffer-leak gate (requires valgrind)

License

Licensed under the Apache License, Version 2.0. This binding also links IMBH (Apache-2.0) and sable (MIT).