test(benchmarks): measure what a quotation query costs on the client by xperiandri · Pull Request #56 · fsprojects/FSharp.Azure.Cosmos

xperiandri · 2026-10-11T12:01:06Z

Proposed Changes

Depends on #42 (which contains #48). The branch is built on the branch of #42, so until that is merged this pull request also shows its seven commits; its own are the last two. It targets main because the build workflow runs only for pull requests into main.

ADR 0001 (#34) plans to translate a quotation once, to cache the translation and to bind only the captured values on every later call. That design rests on costs nobody had measured. Phase 0 of the ADR asks for the measurements before the translator is written, and this pull request adds them: a benchmark project and its first numbers.

benchmarks/Cosmos.Quotations.Benchmarks is a BenchmarkDotNet console application. The solution builds it; nothing runs it, it is run by hand.

Class	Question
`QuotationBenchmarks`	What does a call cost on the client once its translation is cached: evaluating the quotation, walking it for a cache key and the captured values, finding the cached query text, building the `QueryDefinition`?
`EvaluationBenchmarks`	What does it cost to turn an expression that does not read the document, such as `filter.MinAge`, into a parameter value: by reflection, or by compiling it?
`SqlModelBenchmarks`	What does `FSharp.Azure.Cosmos.Sql` (#42) add to a translation that is not cached: building, validating and printing the syntax tree?
`RoundTripBenchmarks`	What does one request cost: a point read and a query against the emulator?

ShapeKey.fs is a prototype of the pass that computes the cache key, written to be measured and nothing else. A quotation cannot be a dictionary key as it is: every evaluation of <@ … @> builds a new tree, and two such trees are not equal. The prototype walks the tree once and writes down its structure, with the values taken out, as a sequence of numbers that can be compared and hashed. It covers the nodes of the sample query and refuses every other one.

What came out (the README of the project has the tables and how to run them):

	Mean
Evaluating the quotation literal, a predicate of four conditions	75 µs
The same for one condition	24 µs
Everything the library does after that on a cached call: the walk, the lookup, the `QueryDefinition`	about 1 µs
A captured value by reflection / compiled	0.2 to 0.4 µs / 98 to 705 µs
Building, validating and printing the syntax tree	about 1 µs
A point read / a query on the local emulator	1.4 ms / 2.1 ms

The quotation literal is the cost of a cached call, and it is paid inside FSharp.Core. The compiler stores a literal as pickled bytes, and Expr.Deserialize40 unpickles the whole tree on every evaluation, resolving every type and member through reflection again. A cache of translations cannot avoid that, because the library receives the tree only after it is built. Reported upstream as Unpickle a quotation literal once, not on every evaluation dotnet/fsharp#20764, which asks FSharp.Core to unpickle a literal once.
A query held in a value pays none of it, which is what the compiled queries of the ADR are for. Against a request, an inline quotation adds between 1 and 5 percent.
Values are evaluated by reflection, never compiled: compiling costs 400 to 1700 times as much.
The SQL model is not a cost.

Types of changes

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Benchmarks only; the packages are unchanged.

Checklist

Build and tests pass locally
I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added necessary documentation (if appropriate)

Further comments

How far to trust the numbers: the machine did other work during the runs. Three runs within two hours differed by a factor of up to three, so the times are orders of magnitude; the allocations were the same every time. The conclusions above hold in every run.
The prototype has no tests of its own. Before anything is measured, the benchmark class checks that the same predicate over other values has the same key, that another predicate has another key, and that the number of values equals the number of parameters of the query.
New packages, both MIT and only for this project: BenchmarkDotNet 0.15.8 and Newtonsoft.Json 13.0.3. The SDK loads Newtonsoft.Json in the constructor of the client and declares no dependency on it (recorded in the ADR); a test project gets it from the test host, a console application has to reference it, and the first run of the round trip benchmarks failed without it. 13.0.3 is the version the test projects resolve already, so the central pin changes nothing for them.
The round trip benchmarks use the client of the test infrastructure (Emulator.createClient) and name their database with the prefix of the test databases, so the sweep of a later test run deletes it if a benchmark run is aborted.
build.fs: Clean also cleans the output of the projects under benchmarks/.
Verification: dotnet build FSharp.Azure.Cosmos.slnx in Debug and Release with the same 19 warnings as main, none in the new project; Fantomas clean; all sixteen benchmarks ran, the two against the Windows emulator included.
No changelog entry: nothing ships.

🤖 Generated with Claude Code

…mutableArray` patterns Starts src/Cosmos.Sql (FSharp.Azure.Cosmos.Sql, net10.0, FSharp.Core only), the project of the Cosmos DB SQL model of ADR 0001 (sections 4, 6 and 15), with the two collection helpers that the syntax tree is built from, so that they can be reviewed on their own before the tree, the printer and the validator follow. EquatableArray<'T> is a struct over ImmutableArray<'T> with sequence equality and a combined hash code. ImmutableArray's own Equals and EqualityComparer<T>.Default compare the reference of the wrapped array, which a struct tuple, a default-comparer HashSet or a C# caller would see, and a default array never equals an empty one. The nodes of the syntax tree will hold their children in this wrapper, so that their equality is structural under every comparer. The EquatableArray module creates and maps the arrays; its unsafeOfArray wraps an array without copying it, as ImmutableCollectionsMarshal.AsImmutableArray does, for a caller that gives the array away. Implicit conversions to and from ImmutableArray wrap and unwrap without copying; F# applies them without a warning at method arguments, and with warning FS3391 elsewhere. The struct partial active patterns Arr0, Arr1, Arr2, Arr3 and ArrN match an ImmutableArray by its length without allocating: each returns a struct value option or a bool, and its payload is a struct tuple. tests/Cosmos.Sql.Tests is the new emulator-free MSTest project, with 15 tests under the category Collections: a file for the struct, one for the module and one for the patterns. Both projects join the solution and the solution filter, and the agent instructions list them. The library is not packed yet (IsPackable is false): a push to main publishes every packable project to GitHub Packages, and the collection types alone are no package. The commit that gives the package its metadata removes the setting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Adds the syntax tree of the Cosmos DB NoSQL query language to FSharp.Azure.Cosmos.Sql: one union case or record per non-terminal of the SDK grammar sql.g4, named after the SDK's SqlObjects classes (ADR 0001 sections 6 and 15). There is no case for raw text and no shift operator; integer literals are int64 and numbers keep the SDK's integer/double split. Child sequences are EquatableArray<'T>, so the equality of the tree is structural under every comparer. SyntaxTests checks it on whole trees: separately built trees with equal children are equal, hash alike and find each other as dictionary keys, which the plan cache needs. Keywords holds the reserved words of the grammar and its identifier rule. THIRD-PARTY-NOTICES.md attributes the material taken from the Azure Cosmos DB .NET SDK (MIT) and joins the solution items. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The benchmarks of #56 give the cost model of section 11 its first numbers. A quotation literal costs 24 to 75 µs per evaluation inside FSharp.Core, which unpickles the tree every time; the shape key, the plan lookup and the QueryDefinition together cost about 1 µs; a value costs a fraction of a microsecond by reflection and hundreds of times more compiled; a request to the emulator takes 1.4 to 2.1 ms. The record states what follows for the design: the plan cache does not remove the cost of an inline quotation, compiled queries do, and the precompilation tool of Phase 5 stays undecided until the translation itself can be measured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

🟡 Changes recommended

Inline substitution can evaluate one named parameter multiple times and emit inconsistent literals.

1 open finding

Cache encoded parameter values across substitutions · New

What changed in this PR

Adds Phase 0 measurements for ADR 0001, supported by the stacked Cosmos SQL model changes.

Changes:

Adds the SQL AST, printer, validator, function catalog, and unit tests.
Adds quotation, evaluation, SQL-model, and emulator round-trip benchmarks.
Documents results and updates solution, dependencies, cleanup, packaging, and notices.

File	Description
`.github/copilot-instructions.md`	Documents new projects.
`CHANGELOG.md`	Announces SQL package.
`Directory.Packages.props`	Pins benchmark dependencies.
`FSharp.Azure.Cosmos.slnf`	Adds new projects.
`FSharp.Azure.Cosmos.slnx`	Adds new projects and notice.
`THIRD-PARTY-NOTICES.md`	Records SDK-derived material.
`build/build.fs`	Cleans benchmark outputs.
`benchmarks/Cosmos.Quotations.Benchmarks/EvaluationBenchmarks.fs`	Measures closed-expression evaluation.
`benchmarks/Cosmos.Quotations.Benchmarks/FSharp.Azure.Cosmos.Quotations.Benchmarks.fsproj`	Defines benchmark application.
`benchmarks/Cosmos.Quotations.Benchmarks/Program.fs`	Runs selected benchmarks.
`benchmarks/Cosmos.Quotations.Benchmarks/QuotationBenchmarks.fs`	Measures cached-query client costs.
`benchmarks/Cosmos.Quotations.Benchmarks/README.md`	Documents execution and results.
`benchmarks/Cosmos.Quotations.Benchmarks/RoundTripBenchmarks.fs`	Measures emulator requests.
`benchmarks/Cosmos.Quotations.Benchmarks/SampleQuery.fs`	Supplies benchmark query forms.
`benchmarks/Cosmos.Quotations.Benchmarks/ShapeKey.fs`	Prototypes quotation cache keys.
`benchmarks/Cosmos.Quotations.Benchmarks/SqlModelBenchmarks.fs`	Measures SQL-model operations.
`src/Cosmos.Sql/AssemblyInfo.fs`	Adds generated assembly metadata.
`src/Cosmos.Sql/Catalog.fs`	Exposes function lookup and calls.
`src/Cosmos.Sql/CatalogData.fs`	Defines built-in function data.
`src/Cosmos.Sql/EquatableArray.fs`	Adds structurally equal arrays.
`src/Cosmos.Sql/FSharp.Azure.Cosmos.Sql.fsproj`	Defines SQL package.
`src/Cosmos.Sql/FunctionSpec.fs`	Models function specifications.
`src/Cosmos.Sql/ImmutableArrayPatterns.fs`	Adds immutable-array patterns.
`src/Cosmos.Sql/Inline.fs`	Implements parameter substitution.
`src/Cosmos.Sql/Keywords.fs`	Defines identifiers and keywords.
`src/Cosmos.Sql/Printer.fs`	Prints SQL syntax trees.
`src/Cosmos.Sql/Syntax.fs`	Defines the SQL AST.
`src/Cosmos.Sql/ValidationError.fs`	Defines validation diagnostics.
`src/Cosmos.Sql/Validator.fs`	Validates SQL trees.
`tests/Cosmos.Sql.Tests/Ast.fs`	Adds test AST helpers.
`tests/Cosmos.Sql.Tests/CatalogTests.fs`	Tests the function catalog.
`tests/Cosmos.Sql.Tests/EquatableArrayModuleTests.fs`	Tests array helpers.
`tests/Cosmos.Sql.Tests/EquatableArrayTests.fs`	Tests structural arrays.
`tests/Cosmos.Sql.Tests/FSharp.Azure.Cosmos.Sql.Tests.fsproj`	Defines SQL test project.
`tests/Cosmos.Sql.Tests/ImmutableArrayPatternsTests.fs`	Tests length patterns.
`tests/Cosmos.Sql.Tests/InlineTests.fs`	Tests inline substitution.
`tests/Cosmos.Sql.Tests/PrinterTests.fs`	Tests SQL printing.
`tests/Cosmos.Sql.Tests/SyntaxTests.fs`	Tests AST equality.
`tests/Cosmos.Sql.Tests/TestCategories.fs`	Defines test categories.
`tests/Cosmos.Sql.Tests/testconfig.json`	Enables parallel tests.
`tests/Cosmos.Sql.Tests/ValidatorTests.fs`	Tests validation rules.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@name

Printer writes the syntax tree as query text following the rules of the SDK's SqlObjectTextSerializer without pretty printing (ADR 0001 section 6): bare aliases; property accesses as alias["name"] with JSON escaping, or alias.name under PropertyStyle.DotWhenSafe when the name is a safe identifier; every binary, unary, conditional, coalesce, IN, BETWEEN and LIKE expression in parentheses; the SDK's string escaping (the solidus stays as is, like in the SDK); integers in full and other numbers with "R", both in the invariant culture; lower-case literal keywords; the udf. prefix; the clause order of the grammar. The text equals the SDK baselines after whitespace normalization; it differs only where the SDK writes "NOT LIKE" and a space after GROUP BY. Printing is total and has two modes: print writes parameters as @name, while printInline and printPredicateInline first run Inline.substituteParameters, which replaces every parameter, in every clause and subquery, with the constant an encoder gives for it. The encoder contract is one function from a parameter name to the literal, if any; Inline.ofJsonElement turns a value written by a serializer into such a literal. printPredicate writes the "FROM c WHERE ..." text that the filter predicate of a patch takes. Errors of the substitution are ValidationError values. The golden tests cover every node kind and cite the SDK baselines they reproduce (SqlObjectVisitorBaselineTests, the parser baselines and the LINQ baselines of SDK 3.62.0), escaping of every control character, and culture independence under fr-FR and a culture with other number symbols. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The built-in functions of the query language are data (ADR 0001 section 6): a FunctionSpec per name with category, arity, return kind, undefined rule, index usage, availability, determinism, the ORDER BY and ORDER BY RANK flags, the Learn page and a status. CatalogData.fs is the data file; Catalog exposes it as Catalog.all, Catalog.byName (a FrozenDictionary with OrdinalIgnoreCase, since function names are case-insensitive), Catalog.tryFind and Catalog.call, which builds a call with the catalog's spelling after checking that the function is known, specified and given an accepted number of arguments. The catalog is seeded with every name of SqlFunctionCallScalarExpression.Names of SDK 3.62.0 except the internal ones that start with an underscore, plus the documented names the SDK does not list: the INTBIT* spellings, NOW, AGO, NUMBERBIN, REGEXREPLACE, REGEXREPLACEALL, ST_AREA and the GETCURRENT*STATIC functions. The functions the first translator slice needs are specified from their Learn pages (type checks, the string and array functions, aggregates, IIF, the integer functions, basic date and time functions, and the scoring functions VectorDistance, FullTextScore and RRF); every other entry is Unsupported with its reason: not specified yet, undocumented, a keyword of the grammar or the SDK's spelling of a documented name. A test reads the names of the SDK in use by reflection and fails when an SDK update adds or removes one; the test project references Microsoft.Azure.Cosmos for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@name

SqlQuery.validate and SqlQuery.validateWith check a query and its subqueries against the rules of the query language that the grammar cannot express and return them as ImmutableArray<ValidationError>, each with a code and an English message that quotes the offending fragment (ADR 0001 section 6). The rules are those the first translator slice needs: - SELECT * only with exactly one collection in the FROM clause; - TOP never in a subquery; - no parameter in a GROUP BY key or as an ORDER BY key, while the arguments of a function allowed in ORDER BY or ORDER BY RANK may be parameters, as the vector and hybrid search queries on Microsoft Learn and the SDK's query plan baseline pass the query vector; parameter names of the form @name; - ORDER BY items only property paths or calls of functions the catalog allows in ORDER BY; ORDER BY RANK items only scoring functions; functions allowed only in ORDER BY RANK nowhere else; - identifiers of the form [A-Za-z_][A-Za-z_0-9]* that are not reserved words; - non-empty IN lists and select lists; - built-in functions known to the catalog and called with an accepted arity; - finite number literals, integer literals within -2^53..2^53 unless ValidationOptions.AllowLossyInt64 is set, non-negative counts. OFFSET and LIMIT together, counts as literals or parameters, string object keys and the separation of plain and rank ORDER BY items hold by construction of the syntax tree. The documentation of the tree, the keywords, the printer and the catalog now refers to the rules the validator checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Gives the new package its NuGet metadata and packs THIRD-PARTY-NOTICES.md next to README.md and LICENSE, because the package contains material derived from the Azure Cosmos DB .NET SDK (ADR 0001 section 5). The package depends on FSharp.Core only; System.Text.Json is in-box on net10.0. IsPackable=false, which kept the library out of the packages while it held the collection types only, is removed. The project is already part of the solution filter that FAKE packs, the test glob tests/**/*.Tests.??proj picks up the new test project, the coverage assembly filter "-*.Tests;-*.Tests.Infrastructure" excludes it while keeping the library, and fsdocs builds the API documentation of every src project. The changelog lists the package under Unreleased. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ions The solution tree of the agent instructions names every file of src/Cosmos.Sql with what it holds, in compile order, as it does for src/Cosmos. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ADR 0001 plans to translate a quotation once, to cache the translation and to bind only the captured values on every later call (section 11). That plan rests on costs nobody had measured: what FSharp.Core charges for a quotation literal, what one walk over the tree for a cache key costs, what evaluating a value by reflection costs against compiling it, and how all of that compares with one request to the service. benchmarks/Cosmos.Quotations.Benchmarks is a BenchmarkDotNet console application with four classes: - QuotationBenchmarks: the steps of an execution whose translation is cached, one by one and together. - EvaluationBenchmarks: a subexpression that does not read the document, evaluated by reflection and by LeafExpressionConverter. - SqlModelBenchmarks: building, validating and printing a syntax tree of FSharp.Azure.Cosmos.Sql, in both print modes. - RoundTripBenchmarks: a point read and a query against the emulator, through the client of the test infrastructure. ShapeKey is a prototype of the cache key pass, written to be measured and nothing else. It walks a quotation once, writes its structure as a sequence of numbers (kinds of nodes, runtime handles of types, metadata tokens of members, the distance of a variable to its binding) and takes the values out. A node under which nothing reads a lambda parameter is evaluated by reflection and becomes one value. It covers the nodes of the sample query and refuses every other one. The benchmark class checks before it measures that the same predicate over other values has the same key and that another predicate has another one. The project is in both solution files, so the build keeps it compiling; nothing runs it. BenchmarkDotNet 0.15.8 (MIT) is a new entry in Directory.Packages.props, and so is Newtonsoft.Json 13.0.3, the version the test projects resolve already: the SDK loads it in the constructor of the client, a test project gets it from the test host, and a console application has to reference it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The README of the benchmark project says what each class measures, how to run it and what came out on 2026-10-11: - Evaluating a quotation literal is the cost of a cached call: 24 µs for one condition and 75 µs for four, because FSharp.Core unpickles the tree on every evaluation. Everything the library does afterwards, the walk for the cache key, the lookup and the binding, is about 1 µs. - A value that does not come from the document costs a fraction of a microsecond by reflection and 400 to 1700 times as much compiled. - Building, validating and printing the syntax tree is about 1 µs. - A point read on the emulator is 1.4 ms, a query 2.1 ms. The machine did other work during the runs, so the times are orders of magnitude; the README says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

xperiandri and others added 2 commits October 11, 2026 13:46

Copilot AI balanced review requested due to automatic review settings October 11, 2026 12:01

Copilot started reviewing on behalf of xperiandri October 11, 2026 12:01 View session

Copilot AI reviewed Oct 11, 2026

View reviewed changes

Comment thread src/Cosmos.Sql/Inline.fs

xperiandri and others added 7 commits October 11, 2026 14:56

docs: list the files of FSharp.Azure.Cosmos.Sql in the agent instruct…

82913d4

…ions The solution tree of the agent instructions names every file of src/Cosmos.Sql with what it holds, in compile order, as it does for src/Cosmos. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

xperiandri force-pushed the test/quotations-benchmarks branch from 7ab615b to a0e1777 Compare October 11, 2026 13:08

Navigation Menu

Repository navigation

test(benchmarks): measure what a quotation query costs on the client - #56

test(benchmarks): measure what a quotation query costs on the client#56
xperiandri wants to merge 9 commits into
mainfrom
test/quotations-benchmarks

xperiandri commented Oct 11, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

xperiandri commented Oct 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Changes

Types of changes

Checklist

Further comments

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

xperiandri commented Oct 11, 2026 •

edited

Loading