Proposal: add optional DuckLake helpers without changing the core provider

[luisquintanilla]

Summary

DuckDB.NET already executes ordinary DuckLake SQL through DuckDB.NET.Data: callers can LOAD ducklake, ATTACH 'ducklake:...', USE the catalog, and then use normal commands, parameters, transactions, readers, schema APIs, Arrow, and appenders. This proposal is for a small optional convenience package rather than a DuckLake implementation inside the ADO.NET provider.

Proposed public API

Package: DuckDB.NET.DuckLake

Target frameworks: net8.0 and net10.0

Dependency: DuckDB.NET.Data only. The package contains no DuckLake extension and no native DuckDB binaries.

Attach options

public sealed record DuckLakeAttachOptions
{
    public required string Catalog { get; init; }
    public required string Alias { get; init; }
    public string? DataPath { get; init; }
    public bool ReadOnly { get; init; }
    public ulong? SnapshotVersion { get; init; }
    public DateTimeOffset? SnapshotTime { get; init; }
    public bool AutomaticMigration { get; init; }
    public bool UseAfterAttach { get; init; } = true;
}

Catalog is the complete DuckLake URI, for example ducklake:metadata.ducklake; accepting the complete URI keeps local, SQLite, PostgreSQL, and other DuckLake catalog forms possible without adding catalog-specific fields to the provider.

Connection helpers

public static class DuckLakeExtensions
{
    public static void LoadDuckLakeExtension(this DuckDBConnection connection);

    public static void AttachDuckLake(
        this DuckDBConnection connection,
        DuckLakeAttachOptions options);

    public static DuckDBConnection OpenDuckLake(
        string duckDbConnectionString,
        DuckLakeAttachOptions options);
}

Example

using DuckDB.NET.Data;
using DuckDB.NET.DuckLake;

var options = new DuckLakeAttachOptions
{
    Catalog = "ducklake:metadata.ducklake",
    Alias = "lake",
    DataPath = "data_files"
};

using var connection = DuckLakeExtensions.OpenDuckLake(
    "Data Source=:memory:;enable_external_access=true",
    options);

using var command = connection.CreateCommand();
command.CommandText = "SELECT * FROM events ORDER BY id";
using var reader = command.ExecuteReader();

For an already open connection, the equivalent is:

connection.AttachDuckLake(options);

UseAfterAttach defaults to true, so the helper also executes USE "lake". Set it to false when callers want to address the attached catalog explicitly, such as SELECT * FROM lake.events.

SQL generated by the helper

For the example options, the helper emits the equivalent of:

LOAD ducklake;
ATTACH 'ducklake:metadata.ducklake' AS "lake" (DATA_PATH 'data_files');
USE "lake";

Identifiers and string literals are quoted by the helper because DuckDB cannot parameterize the catalog URI, alias, or DATA_PATH in ATTACH. The helper rejects null characters, requires the catalog URI to start with ducklake:, rejects blank aliases, and rejects specifying both SnapshotVersion and SnapshotTime. Snapshot timestamps are rendered as invariant UTC values.

Behavioral and security contract

• The helper is load-only. It executes LOAD ducklake; it never executes INSTALL, enables external access, enables autoinstall, or creates secrets.
• The caller/deployment must provision a DuckLake extension matching the DuckDB native library version and platform. A controlled extension directory or custom repository can be configured through the existing DuckDB.NET connection-string options.
• AttachDuckLake and LoadDuckLakeExtension require an open DuckDBConnection.
• OpenDuckLake opens and returns a normal DuckDBConnection; if loading or attaching fails, it disposes the connection before rethrowing.
• ATTACH and USE are connection-session state. Reopening a connection requires loading/attaching the lake again.
• The helper does not change DuckDBConnectionStringBuilder, reinterpret Data Source, or make DuckLake state global.
• ReadOnly, snapshot selection, and automatic migration are passed through to DuckLake’s ATTACH options. SnapshotVersion and SnapshotTime are mutually exclusive.

Proposed scope

Add the optional managed-only package with the API above, plus:

• validated SQL generation with explicit string-literal and identifier quoting;
• integration coverage for attach/use, parameters, transactions, snapshots, read-only reopen, schema, Arrow, and appender behavior;
• documentation showing both raw SQL and the helper API.

The core DuckDB.NET.Data connection-string builder and native bindings would remain unchanged. DuckLake catalog backends, storage, credentials, and extension provisioning remain deployment concerns.

Compatibility evidence

The current DuckDB.NET native baseline is DuckDB v1.5.6. A compatibility harness and real-extension tests verified INSTALL/LOAD, repeated LOAD, DATA_PATH, SNAPSHOT_VERSION, SNAPSHOT_TIME, READ_ONLY, AUTOMATIC_MIGRATION, USE, ordinary SQL, transactions, schema inspection, Arrow reads, reconnect/read-only behavior, and appender persistence. Appender behavior should be treated as DuckDB/DuckLake-version coupled rather than as a permanent provider guarantee; SQL or a native temporary table followed by INSERT INTO ... SELECT ... remains the portable fallback.

Questions for maintainers

1. Is an optional DuckDB.NET.DuckLake helper package useful and appropriate in this repository, or should support remain documentation-only / live in a separate package?
2. Is the load-only extension policy the right security and deployment boundary?
3. Is this API shape and naming appropriate, especially the complete catalog URI, UseAfterAttach, and OpenDuckLake factory?
4. Should the helper API be released as stable, or initially marked preview while DuckLake attach syntax and extension packaging evolve?
5. Is it acceptable for CI integration tests to provision the matching extension with INSTALL ducklake in an explicitly enabled test path, or should CI use a separately provisioned extension directory/artifact?
6. Should appender behavior be documented only as verified compatibility, or should the project make a stronger support promise for a specific DuckDB/DuckLake version range?

Non-goals

• Reimplementing DuckLake catalog or storage semantics in C#.
• Bundling the DuckLake extension or platform-native libraries in DuckDB.NET packages.
• Adding DuckLake-specific fields to DuckDBConnectionStringBuilder or changing the meaning of Data Source.
• Hiding extension installation, external-access changes, credentials, or secrets behind a convenience method.

If this direction is accepted, I can follow up with a focused PR targeting develop, including the helper package, tests, CI, and documentation.

[luisquintanilla]

Clarification on the packaging question: “separate package” means a separate GitHub repository, not merely another project in this repository.

My current recommendation is: keep the proposed thin adapter in this repository as a separate project and publish it as a separate NuGet package. It is tightly coupled to DuckDB.NET.Data.DuckDBConnection, needs to be tested against the provider’s exact native-version matrix, and is small enough that a second repository would add release/CI/documentation overhead without creating real independence. The core provider would still remain unchanged.

A separate repository would become the better home if this grows into an independent .NET DuckLake SDK, supports multiple .NET DuckDB providers, develops catalog/credential/storage/migration abstractions beyond SQL attachment, needs an independent release cadence, or is taken on by the DuckLake project/community itself. The PR decision should explicitly resolve same-repository separate project versus separate repository before implementation is submitted.

[luisquintanilla]

@Giorgi thanks for turning it into a discussion. Happy to contribute the code but would be good to hear from other folks if this is something they'd be interested in.

[Giorgi]

@luisquintanilla Sounds good. To be honest, I haven't used DuckLake myself, so hearing from people who do would help. Let's leave this open for feedback, and if there's interest we can figure out the best home for it.

Out of curiosity, are you using it at Microsoft, or is this from a personal project?