Repository navigation
test: read back what the printer writes with the SDK's own parser - #57
Open
xperiandri wants to merge 25 commits into
Open
xperiandri wants to merge 25 commits into
xperiandri wants to merge 25 commits into
Conversation
Start the Hedgehog MSTest adapter, tests/Hedgehog.MSTest (ADR 0001, sections 5, 14 and 22.6), with the nine files that Hedgehog.NUnit and Hedgehog.Xunit of hedgehogqa/fsharp-hedgehog share and that do not depend on a test framework: Prelude, ReflectionHelpers, AutoGenConfig, IPropertyAttribute, RecheckAttribute, GenAttribute, GenAttribute.Prelude, PropertyContext and InternalLogic, copied from src/Hedgehog.NUnit at the Hedgehog 2.0.4 release commit a469772 in upstream's file layout and compile order. The only change is the namespace, Hedgehog.NUnit becomes Hedgehog.MSTest, so this commit can be compared with upstream line by line; the MSTest attribute and the changes it needs follow separately. The code is Apache-2.0: every copied file names its origin, the copyright and the license in a header comment, and the new THIRD-PARTY-NOTICES.md records the source, the commit, the files and the license text. The same code is proposed upstream as src/Hedgehog.MSTest, so the project keeps upstream's conventions rather than the repository's: nullness checking is off (upstream enables it nowhere, and the copied files give 23 nullness warnings under the repository's Nullable=enable) and .fantomasignore excludes the project, so the Fantomas run of the build leaves upstream's formatting alone. It sets IsTestProject=false and IsPackable=false, because tests/Directory.Build.props marks every project under tests/ as a test project, references MSTest.TestFramework rather than the MSTest metapackage, and is excluded from the test infrastructure reference of tests/Directory.Build.props, because the adapter must not depend on this repository. The copied code calls functions of FSharp.Core that return an option, such as Seq.tryHead and Seq.tryFind. The collection helpers that the root Directory.Build.props compiles into every F# project replace exactly these functions with ones that return a voption, so the project removes that file from its compilation for now; the commit that moves the adapter to the repository's conventions takes the removal out. The project joins both FSharp.Azure.Cosmos.slnx and FSharp.Azure.Cosmos.slnf, which FAKE builds, so every build compiles it, and THIRD-PARTY-NOTICES.md joins the solution items. Directory.Packages.props gets Hedgehog 2.0.4 (Apache-2.0; FSharp.Core >= 8.0.403 and TypeShape 9.0.0 transitively, checked in the nuspec), which contains the former Hedgehog.Experimental auto-generation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Turn the ported sources into an MSTest adapter (ADR 0001, section 22.6).
PropertyAttribute derives from TestMethodAttribute and, like it, applies
to methods only and is not inherited. It keeps upstream's six
constructor overloads and named settings (AutoGenConfig,
AutoGenConfigArgs, Tests, Shrinks, Size), forwards the caller
information of every constructor to the base constructor, as
STATestMethodAttribute does, and overrides ExecuteAsync. MSTest has no
counterpart to NUnit's test builder or xUnit's discoverer, so this one
attribute is the framework layer; PropertiesAttribute is copied from
upstream with the same settings.
Each generated case and each shrink step is one call of
ITestMethod.InvokeAsync, never MethodInfo.Invoke, so MSTest creates the
test class, injects the TestContext and runs TestInitialize, TestCleanup
and Dispose for every case, and applies a [<Timeout>] to each of them.
The values of a [<DataRow>] or [<DynamicData>] row, which MSTest passes
in ITestMethod.Arguments, are the leading arguments and also appear in
the report; only the parameters after them are generated. Generated
IDisposable arguments are disposed after their invocation. The run is
folded into one TestResult, because several results from one
ExecuteAsync appear as separate results of one test: a failure carries
Hedgehog's report with the shrunk arguments, the exception of the
counterexample as inner exception (unwrapped from MSTest's internal
TestFailedException), the output of its invocation and a
[<Recheck("...")>] to add next to the method's Property attribute,
which keeps the attribute's settings, such as its AutoGenConfig, so that
the recheck data replays this counterexample. A run that gives up after
Hedgehog's 100 discards fails, as in upstream.
Additions beyond upstream: a uint64 Seed setting on both attributes;
Assert.Inconclusive in an invocation discards the case, and fails it
while rechecking, because a recheck cannot discard its only case; a run
whose TestContext cancellation token is cancelled stops without
shrinking, reading the token before every invocation, because MSTest
gives the TestContext a new token source before each TestCleanup, and
checking it again after the invocation, whatever its outcome: MSTest
reports a test method that returns without observing its cancelled
token as passed, and after the last case nothing else would notice the
token, so the property stops in the same way wherever the cancellation
reached it; the run still disposes the arguments generated for a case
it keeps from running;
an invocation whose outcome is none of Passed, Failed, Inconclusive and
Timeout, such as Error, was not run as a test by MSTest, so the run
stops at it without shrinking and the property reports the outcome and
the exception of that invocation instead of a counterexample.
The token comes from TestContext.Current, experimental in MSTest 4.2.3
(MSTESTEXP), in one helper with a scoped #nowarn "57".
Departures from upstream, each recorded in the header of the changed
file: class settings come from the reflected type of the method, so a
property inherited from an abstract base gets the settings of each
derived test class, and a derived class's [<Properties>] win over its
base's; a GenAttribute is found through a new non-generic GenAttribute
base wherever it sits in the inheritance chain (upstream checks only the
direct base type and silently auto-generates otherwise); generic
methods, which MSTest 4.2.3 discovers, and return types other than unit,
Task and ValueTask, which MSTest 4 cannot discover, are rejected before
any generator is built, so the return-value handling and the
reflection-based invocation of upstream's InternalLogic and
ReflectionHelpers are removed; every invocation is memoised per
shrink-tree node, because Hedgehog 2.0.4 runs an asynchronous case again
whenever it unwraps it and would invoke the counterexample a second time
to read its journal; the recheck, which is synchronous in Hedgehog
2.0.4, runs on the thread pool; the generic arguments of an
AutoGenConfig method are inferred from every place a generic parameter
occurs in its parameter types, nested ones such as 'a[] included, and
every place must give the same type, where upstream takes one argument
type per parameter that is itself a generic parameter, which fails for
'a list or a repeated 'a; NonZeroInt skips a 0 at either end of its
range, where upstream generates from an empty part, and rejects the
range from 0 to 0; Odd and Even are generated from the odd or even
values inside their range and reject a range without one, where
upstream sets or clears the lowest bit of any value of the range and
so leaves an even upper bound of Odd or an odd lower bound of Even by
one; any exception, an invalid AutoGenConfig
type or malformed recheck data included, becomes a result with the
outcome Error instead of escaping ExecuteAsync.
InternalLogic.executeAsync, a function from the method, the data row,
an invoker and a function that reads the cancellation token to a
TestResult, is the entry point of ExecuteAsync; InternalsVisibleTo
exposes it to the adapter's suite, so the suite can run properties that
fail by design without failing itself. The repository turns
GenerateAssemblyInfo off for the FAKE-written AssemblyInfo.fs of the src
projects, so this project turns it back on for the SDK to emit the
InternalsVisibleTo attribute.
Every public type and member is documented, and every reference to a
type or member is a documentation ID that resolves in the documentation
files of the referenced assemblies, including the primary constructors,
whose comments sit before their parameters. THIRD-PARTY-NOTICES.md lists
the two attribute files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The adapter was ported in upstream's style, because the same code was to go into the upstream pull request. That pull request now exists, hedgehogqa/fsharp-hedgehog#487, with its own copy in upstream's style, so the copy in this repository follows the coding guidelines of this repository, its formatting and nullness checking included, as the review of #43 asks: - A setting that is not given is a voption instead of an option: the members of IPropertyAttribute, the fields of PropertyContext, the private constructors and fields of PropertyAttribute and PropertiesAttribute, the generic arguments that AutoGenConfig infers and the generator that a GenAttribute gives. - The project compiles src/Shared/ValueCollections.fs again, like every other F# project, and uses its Seq.tryHead, Seq.tryFind, Seq.tryExactlyOne and Seq.zip, which return a voption or struct tuples. The three files that use them open FSharp.Azure.Cosmos, the namespace of these functions. Prelude loses Option.requireSome and Seq.seqTryExactlyOne. - The generated arguments are an array instead of a list. Hedgehog 2.0.4 has Gen.sequence for sequences and Gen.sequenceList for lists, both over one traversal, and nothing for arrays, so Prelude adds Gen.sequenceArray over Gen.sequence; the values shrink as they did through Gen.sequenceList. Type.getAllAttributes returns an array, and the two parts of a NonZeroInt range are an array. - Collections are joined in array and sequence expressions instead of Array.append and Seq.append: the arguments of an invocation, the values of the journal and the attributes that PropertyContext folds. - The types that the adapter compares other types with are in one module, KnownTypes: Void, Task, ValueTask, IAutoGenConfig and the generic definitions of ValueTask<'T>, Async<'T> and Result<'T, 'E>. typeof is one ldtoken, but typedefof calls GetGenericTypeDefinition on every use (read in the compiled IL), so each type is resolved once, and every comparison goes through the operators of System.Type instead of generic equality. - withTests, withShrinks and withSeed are functions with named parameters instead of values that hold a lambda. - Attributes are found through GetCustomAttributes<'T> and OfType<'T> instead of a type test and a cast per attribute. - The error for an AutoGenConfig type without exactly one config member says "member" where upstream says "property": a method that takes AutoGenConfigArgs is accepted as well (automatic review of #50). - The project is formatted with Fantomas: .fantomasignore no longer lists it, so the Fantomas run of the build covers it. - Nullness checking is on, as everywhere in the repository. A data row and the arguments of an invocation or of a config method may hold null, so they are arrays of objnull. A null array is checked for where it enters: in the setters of AutoGenConfigArgs, which take it as no arguments, and at the entry point that receives the data row from MSTest. Nested exceptions and base types are matched on null instead of tested with isNull, reflection lookups that cannot fail are asserted with nonNull, and a config method that returns no IAutoGenConfig is an error. One call keeps a scoped #nowarn "3261", in a helper of its own: ITestMethod.InvokeAsync declares the elements of its arguments as non-null, although MSTest itself passes it ITestMethod.Arguments, whose elements it declares as nullable. - The pipeline that builds the journal ends in Seq.toArray, so that it stays in one module. The header of every changed file records the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add tests/Hedgehog.MSTest.Tests, an MSTest runner executable (OutputType Exe, EnableMSTestRunner) with method-level parallelism in testconfig.json, that runs 137 tests against the adapter (ADR 0001, section 22.6). Upstream's F# xUnit and NUnit adapter tests at a469772 are rewritten as [<TestClass>] types: generating and shrinking, AutoGenConfig types and their error messages, AutoGenConfigArgs with generic, concrete and mixed arguments, class-level settings and their overrides, attribute subclasses, tuples, shrink limits, recheck data and Size, IDisposable arguments, GenAttribute and the 18 built-in generator attributes. The cases MSTest 4 cannot discover (bool, Result, Async, Task<'T> and Property returns, unresolved generics, module-level properties) and the xUnit-specific ones are left out. Added to them are tests of a generic config parameter used for two arguments, nested in an array type and given two types, of NonZeroInt with 0 at either end of its range and from 0 to 0, and of Odd and Even at the ends of their range, next to Int32.MinValue and Int32.MaxValue and over a range without such a value; three misspelled test names are corrected. The ported files and the shared configurations carry upstream's Apache-2.0 attribution, and THIRD-PARTY-NOTICES.md lists them. The targets that must shrink exactly once fix their seed: with a random one, a run whose first failing value is exactly 2500 has nothing smaller to shrink to, which happened in 8 of 20,000 simulated runs. MSTest-specific tests cover what the adapter adds or does differently, each pinning MSTest behaviour that the adapter relies on: a new instance, TestContext, TestInitialize, TestCleanup and Dispose for every case, with property and constructor injection; DataRow, DynamicData and TestDataRow<struct (string * int)> rows as leading arguments; a property inherited from an abstract base reading the settings of each derived test class, and a derived class's settings and configuration winning over its base's; a GenAttribute derived from a built-in one; the caller information of each of the six PropertyAttribute constructors; Task and ValueTask bodies; Assert.Inconclusive as a discard and as a failure while rechecking; give-ups failing; generic methods and unsupported return types giving an error result; the same Seed giving the same cases; the counterexample invoked once; a token cancelled at a failing case that has smaller values to shrink to, and a timed-out invocation, stopping the run without shrinking, with a per-invocation [<Timeout>] that a whole property exceeds and a TestCleanup that makes MSTest replace the token source; a token cancelled during a passing invocation, an earlier one or the last, stopping the run as well; the arguments of a case that a cancelled token keeps from running still disposed; an invocation that ends with the outcome Error or NotFound, and an exception of the invoker itself, stopping the run without shrinking and giving the property that outcome and that exception instead of a failure. Properties that fail by design never fail the suite. Targets in classes without [<TestClass>], which MSTest does not run, go through Harness and the adapter's entry point InternalLogic.executeAsync with an invoker that creates the class and awaits the method, so the tests inspect the report, the journal and the result. FailingPropertyAttribute runs a property through the real MSTest pipeline instead and passes when its single result has the expected outcome, message, output and inner exception: a failure reports its counterexample, its recheck data and the output of its invocation, [<Recheck>] replays the counterexample of the seed-42 run and passes once the bug is fixed, a data row appears among the parameters, malformed recheck data gives an error result and a give-up fails. Twenty behaviours of the adapter, from the reflected type and the memoised invocations to the single result and the cancellation check before each invocation, were each reverted in the adapter on its own: a test fails for every one. So do the three tests of an invocation that MSTest could not run when such an outcome is treated as a falsified case again. Like the adapter, the suite follows the coding guidelines of this repository, its formatting and nullness checking included; one test pins that a null array of AutoGenConfigArgs, which a caller compiled without nullness checking can give, counts as no arguments. The suite compiles the collection helpers of src/Shared/ValueCollections.fs and references no project of this repository but the adapter, so tests/Directory.Build.props leaves the Cosmos test infrastructure out of it too. The suite joins both FSharp.Azure.Cosmos.slnx and FSharp.Azure.Cosmos.slnf; testsGlob, already narrowed to tests/**/*.Tests.??proj, picks it up and leaves the adapter library out, so DotnetTest runs Hedgehog.MSTest.Tests next to FSharp.Azure.Cosmos.Tests. The coverage report excludes the adapter and Hedgehog, which the suite loads and whose package embeds its PDB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hog MSTest adapter Hedgehog 2.0.4 cannot auto-generate a DateOnly or a TimeOnly: Gen.auto throws for both (run in dotnet fsi against the package), and upstream's adapters have a generator attribute for DateTime and DateTimeOffset only. A property with a parameter of one of these types therefore needed a generator of its own; the review of #43 asks for the two attributes. DateOnlyAttribute generates a date between two dates, both included, and TimeOnlyAttribute a time of day between two times, both included, at the precision of TimeOnly (100 nanoseconds). Both ranges are constant, like those of DateTimeAttribute: they do not grow with the size of a case, and a value shrinks towards the lower end. Without arguments DateOnly generates within the 3650 days from 2000-01-01, the dates of DateTimeAttribute's default range, and TimeOnly any time of day. An attribute argument cannot be a DateOnly or a TimeOnly, so each attribute also takes its range as integers, [<DateOnly(2024, 2, 28, 2024, 3, 1)>] and [<TimeOnly(9, 0, 17, 30)>]; the constructor that takes the two values themselves serves attributes that derive from these. Six tests cover the default and the given ranges through [<Property>] methods and check on 200 sampled values that a range of three values gives each of them, both ends included, and nothing else. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… them A new DotnetListTests target, between DotnetBuild and DotnetTest in both build chains, runs `dotnet test --list-tests` for every test application (ADR 0001, section 22.1). MSTest 4 fails the discovery of a whole assembly when one test method has a signature it cannot run, such as a [<Property>] method that returns bool, while the F# build gives no warning; checked by adding such a method to the adapter's suite: the build succeeded without a warning, and the listing failed with UTA007 and exit code -532462766. Listing reports that before the run, which needs the emulator, starts. DotnetTest and DotnetListTests share dotnetTestApplications, which runs `dotnet test --no-build` in the configuration of the executing targets for every project that testsGlob matches, so the loop over the test applications is written once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The solution tree of the agent instructions lists tests/Hedgehog.MSTest and its suite, and the libraries list Hedgehog. The testing rules say how to write a property test: a method of a [<TestClass>] type marked [<Property>], with generated parameters, never Property.check inside a [<TestMethod>]; that it is an instance method that returns unit, Task or ValueTask and fails by throwing, because MSTest 4 rejects static methods and other return types and one such method fails the discovery of the whole assembly, which --list-tests shows before a run; that MSTest runs [<TestInitialize>] and [<TestCleanup>] for every generated case and every shrink step; and that a failure reports a [<Recheck>] that replays it; and that a DateOnly or TimeOnly parameter needs its generator attribute, because Hedgehog 2.0.4 cannot auto-generate these types. They also record that the adapter and its suite are the exception to the shared test infrastructure reference, the suite referencing the adapter alone, and that their code follows the rules of the file, formatting and nullness checking included, while the copy in the upstream pull request keeps upstream's style, with Apache-2.0 headers and THIRD-PARTY-NOTICES.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mutableArray` patterns Starts src/Cosmos.Sql (FSharp.Azure.Cosmos.Sql, net10.0, FSharp.Core only), the project of the Cosmos DB SQL model of ADR 0001 (sections 4, 6 and 15), with the two collection helpers that the syntax tree is built from, so that they can be reviewed on their own before the tree, the printer and the validator follow. EquatableArray<'T> is a struct over ImmutableArray<'T> with sequence equality and a combined hash code. ImmutableArray's own Equals and EqualityComparer<T>.Default compare the reference of the wrapped array, which a struct tuple, a default-comparer HashSet or a C# caller would see, and a default array never equals an empty one. The nodes of the syntax tree will hold their children in this wrapper, so that their equality is structural under every comparer. The EquatableArray module creates and maps the arrays; its unsafeOfArray wraps an array without copying it, as ImmutableCollectionsMarshal.AsImmutableArray does, for a caller that gives the array away. Implicit conversions to and from ImmutableArray wrap and unwrap without copying; F# applies them without a warning at method arguments, and with warning FS3391 elsewhere. The struct partial active patterns Arr0, Arr1, Arr2, Arr3 and ArrN match an ImmutableArray by its length without allocating: each returns a struct value option or a bool, and its payload is a struct tuple. tests/Cosmos.Sql.Tests is the new emulator-free MSTest project, with 15 tests under the category Collections: a file for the struct, one for the module and one for the patterns. Both projects join the solution and the solution filter, and the agent instructions list them. The library is not packed yet (IsPackable is false): a push to main publishes every packable project to GitHub Packages, and the collection types alone are no package. The commit that gives the package its metadata removes the setting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds the syntax tree of the Cosmos DB NoSQL query language to FSharp.Azure.Cosmos.Sql: one union case or record per non-terminal of the SDK grammar sql.g4, named after the SDK's SqlObjects classes (ADR 0001 sections 6 and 15). There is no case for raw text and no shift operator; integer literals are int64 and numbers keep the SDK's integer/double split. Child sequences are EquatableArray<'T>, so the equality of the tree is structural under every comparer. SyntaxTests checks it on whole trees: separately built trees with equal children are equal, hash alike and find each other as dictionary keys, which the plan cache needs. Keywords holds the reserved words of the grammar and its identifier rule. THIRD-PARTY-NOTICES.md attributes the material taken from the Azure Cosmos DB .NET SDK (MIT) and joins the solution items. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Printer writes the syntax tree as query text following the rules of the SDK's SqlObjectTextSerializer without pretty printing (ADR 0001 section 6): bare aliases; property accesses as alias["name"] with JSON escaping, or alias.name under PropertyStyle.DotWhenSafe when the name is a safe identifier; every binary, unary, conditional, coalesce, IN, BETWEEN and LIKE expression in parentheses; the SDK's string escaping (the solidus stays as is, like in the SDK); integers in full and other numbers with "R", both in the invariant culture; lower-case literal keywords; the udf. prefix; the clause order of the grammar. The text equals the SDK baselines after whitespace normalization; it differs only where the SDK writes "NOT LIKE" and a space after GROUP BY. Printing is total and has two modes: print writes parameters as @name, while printInline and printPredicateInline first run Inline.substituteParameters, which replaces every parameter, in every clause and subquery, with the constant an encoder gives for it. The encoder contract is one function from a parameter name to the literal, if any; Inline.ofJsonElement turns a value written by a serializer into such a literal. printPredicate writes the "FROM c WHERE ..." text that the filter predicate of a patch takes. Errors of the substitution are ValidationError values. The golden tests cover every node kind and cite the SDK baselines they reproduce (SqlObjectVisitorBaselineTests, the parser baselines and the LINQ baselines of SDK 3.62.0), escaping of every control character, and culture independence under fr-FR and a culture with other number symbols. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The built-in functions of the query language are data (ADR 0001 section 6): a FunctionSpec per name with category, arity, return kind, undefined rule, index usage, availability, determinism, the ORDER BY and ORDER BY RANK flags, the Learn page and a status. CatalogData.fs is the data file; Catalog exposes it as Catalog.all, Catalog.byName (a FrozenDictionary with OrdinalIgnoreCase, since function names are case-insensitive), Catalog.tryFind and Catalog.call, which builds a call with the catalog's spelling after checking that the function is known, specified and given an accepted number of arguments. The catalog is seeded with every name of SqlFunctionCallScalarExpression.Names of SDK 3.62.0 except the internal ones that start with an underscore, plus the documented names the SDK does not list: the INTBIT* spellings, NOW, AGO, NUMBERBIN, REGEXREPLACE, REGEXREPLACEALL, ST_AREA and the GETCURRENT*STATIC functions. The functions the first translator slice needs are specified from their Learn pages (type checks, the string and array functions, aggregates, IIF, the integer functions, basic date and time functions, and the scoring functions VectorDistance, FullTextScore and RRF); every other entry is Unsupported with its reason: not specified yet, undocumented, a keyword of the grammar or the SDK's spelling of a documented name. A test reads the names of the SDK in use by reflection and fails when an SDK update adds or removes one; the test project references Microsoft.Azure.Cosmos for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SqlQuery.validate and SqlQuery.validateWith check a query and its subqueries against the rules of the query language that the grammar cannot express and return them as ImmutableArray<ValidationError>, each with a code and an English message that quotes the offending fragment (ADR 0001 section 6). The rules are those the first translator slice needs: - SELECT * only with exactly one collection in the FROM clause; - TOP never in a subquery; - no parameter in a GROUP BY key or as an ORDER BY key, while the arguments of a function allowed in ORDER BY or ORDER BY RANK may be parameters, as the vector and hybrid search queries on Microsoft Learn and the SDK's query plan baseline pass the query vector; parameter names of the form @name; - ORDER BY items only property paths or calls of functions the catalog allows in ORDER BY; ORDER BY RANK items only scoring functions; functions allowed only in ORDER BY RANK nowhere else; - identifiers of the form [A-Za-z_][A-Za-z_0-9]* that are not reserved words; - non-empty IN lists and select lists; - built-in functions known to the catalog and called with an accepted arity; - finite number literals, integer literals within -2^53..2^53 unless ValidationOptions.AllowLossyInt64 is set, non-negative counts. OFFSET and LIMIT together, counts as literals or parameters, string object keys and the separation of plain and rank ORDER BY items hold by construction of the syntax tree. The documentation of the tree, the keywords, the printer and the catalog now refers to the rules the validator checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gives the new package its NuGet metadata and packs THIRD-PARTY-NOTICES.md next to README.md and LICENSE, because the package contains material derived from the Azure Cosmos DB .NET SDK (ADR 0001 section 5). The package depends on FSharp.Core only; System.Text.Json is in-box on net10.0. IsPackable=false, which kept the library out of the packages while it held the collection types only, is removed. The project is already part of the solution filter that FAKE packs, the test glob tests/**/*.Tests.??proj picks up the new test project, the coverage assembly filter "-*.Tests;-*.Tests.Infrastructure" excludes it while keeping the library, and fsdocs builds the API documentation of every src project. The changelog lists the package under Unreleased. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ions The solution tree of the agent instructions names every file of src/Cosmos.Sql with what it holds, in compile order, as it does for src/Cosmos. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The generators of the SQL model and the printer laws need both: the syntax tree, printer and validator of #42 and the property attribute of #43. This merge is only the base of the stacked branch; it goes away when both pull requests are in main and the branch is rebased onto it. THIRD-PARTY-NOTICES.md is added by both; the result keeps the text of #42 and appends the fsharp-hedgehog section of #43. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/Cosmos.Sql.Hedgehog is the library that the generator projects of ADR 0001 (section 22.6) share. This commit adds the helper modules that need no model, in the namespace FSharp.Azure.Cosmos.Tests.Models: - Sample draws values outside of a property. singleWith and multipleWith take a seed, so that the data of a dataset or of a data-driven test can be drawn again: the value at index i has the seed seed + i and the size 1 + i % 100, which keeps the values spread over every size. single and multiple draw at a random seed and a random size. - Gen generates optional values: the sized voption and skippable, where an absent value becomes rarer as the size grows, and optionBalanced, voptionBalanced and skippableBalanced, where it stays as likely as a present one. The module has the name of Hedgehog's own on purpose: a file that opens both namespaces reaches both through Gen. - CollectionGen generates immutable arrays, hash sets, dictionaries and sorted sets, F# sets and maps, each with a Balanced twin that is empty in one draw of two at every size. - TextGen generates names and notes of ASCII letters and digits, and capitalizes by the invariant culture. Skippable comes from FSharp.SystemTextJson 1.4.36, which only this project references. The date and number helpers of the ADR are left for the step that brings the test model: their bounds are that model's. The project is a library, not a test application, and needs neither a test framework nor the Cosmos test infrastructure, so the shared props of the test projects leave that reference out of it, and the coverage report ignores its assembly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sixteen emulator-free tests in the category Generators. Every value is drawn at a fixed seed, so a test sees the same values on every run: - the seeded members of Sample give the same values again, the value at index i has the seed and the size the documentation states, and the random members draw at a size from 1 to 100; - the balanced generators give the absent value, or the empty collection, in about half of 2000 draws at the smallest and the largest size, while the sized ones go from one half at the size 1 to a few draws in a hundred at the size 100; - the collections run from empty to their largest size, and a key that repeats keeps one entry of a dictionary or a map; - capitalize gives "Istanbul" while the current culture is Turkish, and names and notes keep to their lengths and to ASCII letters and digits. The bounds of "about half" are 0.4 and 0.6: nine standard deviations of a share counted over 2000 draws, so they hold for any seed and still fail for a generator that favours one outcome. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The solution tree lists tests/Cosmos.Sql.Hedgehog, and the testing section says where generators live, which Sample members give data that can be drawn again, what the Balanced suffix means, what a generator file opens and how its parameters are named, and that a generator makes valid values by construction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SqlGenerators builds syntax trees for property tests (ADR 0001, section 13, rung 2). sqlQueryGen and scalarExpressionGen build trees that are valid by construction: every rule of SqlQuery.validate is built into the generator, and nothing is generated and then filtered out. SELECT * comes only with one collection, TOP only in the outermost query, an IN list and a select list have an item, GROUP BY keys hold no parameter, an ORDER BY item is a property path or a call of a function allowed there, an ORDER BY RANK item a call of a scoring function, identifiers are no reserved words, integers fit a JSON number and numbers are finite. Function calls come from the catalog with a number of arguments their arity accepts. A valid tree has no ORDER BY and no OFFSET LIMIT in a subquery either: the validator accepts them there, the emulator rejects them. arbitrarySqlQueryGen and arbitraryScalarExpressionGen build any tree, valid or not, for the properties that must hold for every tree. Of 400 such queries 386 were invalid. constantGen builds the constants that JSON can carry, and textGen the text that exercises the escaping of the printer: quotes, backslashes, control characters, letters outside ASCII and a character outside the Basic Multilingual Plane. A tree halves the size it gives to its children and a list inside it has at most three items, so a failing case stays readable: over 400 valid queries the printed text was 152 characters at the median and 1151 at the longest. SqlAutoGenConfig registers the valid generators by parameter type for the property attribute of the Hedgehog MSTest adapter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eleven properties in the category PrinterLaws, each a method marked with the property attribute of the Hedgehog MSTest adapter, so a law that fails is reported with the smallest tree that breaks it: - a generated query has no validation error, which is the test of the generator itself; - a valid query, and any tree at all, is written the same way twice and under the invariant, French, German, Turkish and Persian cultures, in both property styles; - a string literal and a property name in brackets are JSON strings that read back, an integer and a number literal parse back to the same value, and a constant is JSON that Inline.ofJsonElement reads back as the same constant; - the dot style writes a dot exactly for a name that is an identifier and no reserved word; - inlining a value for every parameter leaves none and keeps a valid query valid, and inlining without values reports every parameter of any tree once and under its own name: a tree reports what each of its parameters reports alone. The laws about parameters find them with a traversal of their own (Parameters.ofQuery): in a first version the inline pass was its own oracle, and a pass that skipped the pattern of LIKE passed. Each law now fails for at least one of eleven one-line defects planted in the printer, the inline pass and the generator, which were reverted afterwards. The laws that read the printed text back as a query wait for a parser of the query language. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The solution tree names the generators of the SQL syntax tree, and the testing section says how a property takes its trees (SqlAutoGenConfig on the class or method, an attribute derived from GenAttribute for another generator) and that a law never asks the code it checks for the expected answer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/Cosmos.Sql.Grammar.Generated holds the grammar sql.g4 of the Azure Cosmos DB .NET SDK 3.62.0 and the C# lexer and parser that ANTLR 4.7.2 generated from it, copied unchanged from Microsoft.Azure.Cosmos/src/Query/Core/Parser. Nothing is generated in this repository, so the build needs neither Java nor the ANTLR tool, and the parser is the one the SDK runs. Update-Generated.ps1 copies the files of another SDK tag; run for 3.62.0 it gives these files byte for byte. The project is C# because the sources are. Its only own code is the InternalsVisibleTo attribute that opens the generated types, which are internal, to the F# project that will read their parse trees. The runtime is Antlr4.Runtime.Standard 4.9.3 (BSD-3-Clause), a new entry in Directory.Packages.props. It targets .NET Standard 2.0 without dependencies, where 4.7.2 targets .NET Standard 1.3, and it has to stay below 4.10: these sources do not compile against 4.10.0 or 4.13.1, where the type of SerializedAtn differs. The project is in both solution files, is left out of the shared test infrastructure and of the coverage report, and THIRD-PARTY-NOTICES.md lists the copied files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/Cosmos.Sql.Grammar is the SDK's parser as an oracle for the tests: SdkGrammar.parse reads query text with the generated lexer and parser and gives the SqlQuery it means, or the first syntax error with its line and column. The conversion follows the SDK's own (CstToAstVisitor.cs), with what the syntax tree of this repository asks for: c.name becomes the node of c["name"], a string literal is decoded with every escape of the grammar (the SDK's conversion decodes three), and a number is read with the invariant culture, as an integer when it fits Int64 and as a double otherwise. It is a project of its own, and one that must never reference Microsoft.Azure.Cosmos: the SDK assembly has internal types named sqlParser and sqlLexer too, and in a project that references it F# resolves those names to the SDK's types, which it cannot use. Every test project references the SDK through the test infrastructure, so the conversion cannot live in one. 19 tests in the category Grammar: every clause and every kind of node, the precedence of operators, the escapes of strings, the kinds of numbers, and the errors. Four of them pin what the lexer and the text do that a printer has to know: - a sign directly before a digit belongs to the number, so 1 -2 is two numbers in a row and is rejected, while (- 2) is a unary minus; - a double without a fraction is written like an integer and read back as one; - a chain of joins is read left-deep, whatever tree it was written from; - a step of an input path written with a dot is read as an identifier step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two more laws of the printer, on generated valid queries: - the SDK's parser reads the text in the bracket style as the very tree it was written from; - it reads the dot style as well, and what it reads is written as the same text. Not the same tree there: the dot style writes a string step of an input path like an identifier step. Until now the laws could only read back what JSON and the number formats of .NET read: literals, property names and constants. A printer that wrote a wrong operator, dropped a NOT or wrote DESC as ASC passed all of them. Five such defects were planted, one at a time, and reverted: each fails the first law, and none of the eleven laws before it. The first law found two things a tree can hold and the text cannot, which the generator of valid trees now avoids: - a join on the right side of a join: the text has no brackets around a join, so it reads back left-deep; - a double without a fraction: the printer writes 2.0 as 2, the way the SDK does, and that is read as an integer. A number without a fraction is an integer literal in a valid tree, unless it is written with an exponent or is the negative zero. Both laws passed fifteen runs in a row, 1500 queries each. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The solution tree lists the two grammar projects, and the testing rules say what SdkGrammar.parse is for, why only tests/Cosmos.Sql.Grammar may name the generated parser types and must never reference the SDK, and which three distinctions of a tree are not in its text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The stacked 76-file change includes a new public SQL model, a custom test adapter, and vendored generated parser code requiring final human review.
0 open findings
What changed in this PR
Adds SDK-parser-based round-trip verification for the Cosmos SQL printer, built on the stacked SQL model, generator, and Hedgehog adapter changes.
Changes:
- Vendors the Cosmos SDK grammar/parser and converts parse trees into the project SQL AST.
- Adds parser unit tests and two property-based printer laws.
- Canonicalizes generated valid trees for lossless round trips.
| File | Description |
|---|---|
.github/copilot-instructions.md |
Documents new SQL testing infrastructure. |
CHANGELOG.md |
Records the SQL package. |
Directory.Packages.props |
Adds parser and property-testing dependencies. |
FSharp.Azure.Cosmos.slnf |
Adds new projects to the filter. |
FSharp.Azure.Cosmos.slnx |
Adds new projects to the solution. |
THIRD-PARTY-NOTICES.md |
Records vendored SDK and Hedgehog sources. |
build/build.fs |
Adds test discovery and coverage exclusions. |
src/Cosmos.Sql/AssemblyInfo.fs |
Defines SQL assembly metadata. |
src/Cosmos.Sql/Catalog.fs |
Exposes function lookup and construction. |
src/Cosmos.Sql/CatalogData.fs |
Defines built-in function metadata. |
src/Cosmos.Sql/EquatableArray.fs |
Adds structurally equal immutable arrays. |
src/Cosmos.Sql/FSharp.Azure.Cosmos.Sql.fsproj |
Defines the SQL package project. |
src/Cosmos.Sql/FunctionSpec.fs |
Defines function specifications. |
src/Cosmos.Sql/ImmutableArrayPatterns.fs |
Adds immutable-array active patterns. |
src/Cosmos.Sql/Inline.fs |
Implements parameter substitution. |
src/Cosmos.Sql/Keywords.fs |
Defines keywords and identifier rules. |
src/Cosmos.Sql/Printer.fs |
Implements deterministic SQL printing. |
src/Cosmos.Sql/Syntax.fs |
Defines the SQL syntax tree. |
src/Cosmos.Sql/ValidationError.fs |
Defines validation diagnostics. |
src/Cosmos.Sql/Validator.fs |
Implements SQL AST validation. |
tests/Cosmos.Sql.Grammar.Generated/FSharp.Azure.Cosmos.Sql.Grammar.Generated.csproj |
Builds the vendored parser. |
tests/Cosmos.Sql.Grammar.Generated/InternalsVisibleTo.cs |
Exposes parser internals to the converter. |
tests/Cosmos.Sql.Grammar.Generated/IsqlListener.cs |
Vendors the generated listener interface. |
tests/Cosmos.Sql.Grammar.Generated/IsqlVisitor.cs |
Vendors the generated visitor interface. |
tests/Cosmos.Sql.Grammar.Generated/README.md |
Documents parser provenance and updates. |
tests/Cosmos.Sql.Grammar.Generated/Update-Generated.ps1 |
Refreshes parser sources from an SDK tag. |
tests/Cosmos.Sql.Grammar.Generated/sql.g4 |
Vendors the SDK SQL grammar. |
tests/Cosmos.Sql.Grammar.Generated/sqlBaseListener.cs |
Vendors the generated base listener. |
tests/Cosmos.Sql.Grammar.Generated/sqlBaseVisitor.cs |
Vendors the generated base visitor. |
tests/Cosmos.Sql.Grammar.Generated/sqlLexer.cs |
Vendors the generated lexer. |
tests/Cosmos.Sql.Grammar.Generated/sqlParser.cs |
Vendors the generated parser. |
tests/Cosmos.Sql.Grammar/FSharp.Azure.Cosmos.Sql.Grammar.fsproj |
Defines the grammar-conversion project. |
tests/Cosmos.Sql.Grammar/SdkGrammar.fs |
Converts SDK parse trees into the SQL AST. |
tests/Cosmos.Sql.Hedgehog/CollectionGen.fs |
Adds collection generators. |
tests/Cosmos.Sql.Hedgehog/FSharp.Azure.Cosmos.Sql.Hedgehog.fsproj |
Defines the generator project. |
tests/Cosmos.Sql.Hedgehog/Gen.fs |
Adds optional-value generators. |
tests/Cosmos.Sql.Hedgehog/Sample.fs |
Adds reproducible generator sampling. |
tests/Cosmos.Sql.Hedgehog/Sql.Generators.fs |
Generates canonical valid SQL trees. |
tests/Cosmos.Sql.Hedgehog/SqlAutoGenConfig.fs |
Registers SQL generators by type. |
tests/Cosmos.Sql.Hedgehog/TextGen.fs |
Adds text generators. |
tests/Cosmos.Sql.Tests/Ast.fs |
Adds concise AST test builders. |
tests/Cosmos.Sql.Tests/CatalogTests.fs |
Tests function catalog behavior. |
tests/Cosmos.Sql.Tests/EquatableArrayModuleTests.fs |
Tests array helper functions. |
tests/Cosmos.Sql.Tests/EquatableArrayTests.fs |
Tests structural array behavior. |
tests/Cosmos.Sql.Tests/FSharp.Azure.Cosmos.Sql.Tests.fsproj |
Defines the SQL test application. |
tests/Cosmos.Sql.Tests/GeneratorHelperTests.fs |
Tests generator helpers. |
tests/Cosmos.Sql.Tests/ImmutableArrayPatternsTests.fs |
Tests array active patterns. |
tests/Cosmos.Sql.Tests/InlineTests.fs |
Tests parameter inlining. |
tests/Cosmos.Sql.Tests/PrinterLawsTests.fs |
Adds parser-backed printer laws. |
tests/Cosmos.Sql.Tests/PrinterTests.fs |
Tests deterministic SQL output. |
tests/Cosmos.Sql.Tests/SdkGrammarTests.fs |
Tests parse-tree conversion and edge cases. |
tests/Cosmos.Sql.Tests/SyntaxTests.fs |
Tests syntax-tree equality. |
tests/Cosmos.Sql.Tests/TestCategories.fs |
Defines sorted SQL test categories. |
tests/Cosmos.Sql.Tests/ValidatorTests.fs |
Tests SQL validation rules. |
tests/Cosmos.Sql.Tests/testconfig.json |
Enables method-level test parallelism. |
tests/Directory.Build.props |
Excludes independent helper projects from infrastructure. |
tests/Hedgehog.MSTest.Tests/Common.fs |
Adds shared adapter test fixtures. |
tests/Hedgehog.MSTest.Tests/GenAttributePreludeTests.fs |
Tests built-in generator attributes. |
tests/Hedgehog.MSTest.Tests/Harness.fs |
Adds an adapter execution harness. |
tests/Hedgehog.MSTest.Tests/Hedgehog.MSTest.Tests.fsproj |
Defines the adapter test project. |
tests/Hedgehog.MSTest.Tests/MSTestTests.fs |
Tests MSTest integration behavior. |
tests/Hedgehog.MSTest.Tests/PropertyTests.fs |
Tests property execution and shrinking. |
tests/Hedgehog.MSTest.Tests/testconfig.json |
Configures parallel adapter tests. |
tests/Hedgehog.MSTest/AutoGenConfig.fs |
Resolves automatic generator configurations. |
tests/Hedgehog.MSTest/GenAttribute.Prelude.fs |
Defines standard generator attributes. |
tests/Hedgehog.MSTest/GenAttribute.fs |
Defines custom generator attributes. |
tests/Hedgehog.MSTest/Hedgehog.MSTest.fsproj |
Defines the Hedgehog MSTest adapter. |
tests/Hedgehog.MSTest/IPropertyAttribute.fs |
Defines shared property settings. |
tests/Hedgehog.MSTest/InternalLogic.fs |
Implements property execution and reporting. |
tests/Hedgehog.MSTest/KnownTypes.fs |
Caches reflected framework types. |
tests/Hedgehog.MSTest/Prelude.fs |
Adds adapter utilities. |
tests/Hedgehog.MSTest/PropertiesAttribute.fs |
Defines class-level property settings. |
tests/Hedgehog.MSTest/PropertyAttribute.fs |
Integrates properties with MSTest. |
tests/Hedgehog.MSTest/PropertyContext.fs |
Merges property configuration. |
tests/Hedgehog.MSTest/RecheckAttribute.fs |
Supports counterexample replay. |
tests/Hedgehog.MSTest/ReflectionHelpers.fs |
Adds return-type reflection helpers. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
xperiandri
force-pushed
the
test/cosmos-sql-grammar
branch
from
October 11, 2026 13:07
da3745b to
e1bc325
Compare
xperiandri
added a commit
that referenced
this pull request
Oct 11, 2026
#57 makes the SDK's parser the judge of the printer. The record states how it departs from the plan and why: the SDK's generated sources are copied instead of generated again, so no Java is needed; the conversion is F# and sits in a project that never references the SDK, because the SDK assembly has internal types of the same names as the generated parser; and the runtime package stays below 4.10. It also keeps what the parser taught: a sign directly before a digit belongs to the number, and three distinctions of a tree are not in its text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed Changes
The laws of #55 check the printer against what JSON and the number formats of .NET can read back: literals, property names and constants. Nothing checked the query text as a whole: whether the service's own grammar accepts it, and whether it means the query it was written from. A printer that wrote
+for-, dropped aNOTor wroteDESCasASCpassed every test.This pull request makes the parser of the Azure Cosmos DB .NET SDK the judge (ADR 0001 (#34), section 13, the first two rungs of the test plan): the text of every generated valid query is parsed with it, and the tree it reads has to be the tree the text was written from.
Two projects
tests/Cosmos.Sql.Grammar.Generated(C#) holds the SDK's grammarsql.g4and the lexer and parser that ANTLR generated from it, copied unchanged from the SDK 3.62.0. Nothing is generated here, so the build needs neither Java nor the ANTLR tool, and the parser is the very one the SDK runs.Update-Generated.ps1 -Tag <tag>takes the files of another SDK version; for3.62.0it reproduces them byte for byte.tests/Cosmos.Sql.Grammar(F#) is the oracle the tests call:SdkGrammar.parse : string -> Result<SqlQuery, string>runs the generated parser and converts its parse tree into the syntax tree ofFSharp.Azure.Cosmos.Sql, or gives the first syntax error with its line and column.Why the conversion is a project of its own. The generated types are internal and have no namespace (
sqlParser,sqlLexer). The SDK assembly has internal types of exactly these names, because the SDK compiles the same files. In a project that references the SDK, F# resolvessqlParserto the SDK's type, which it cannot use, and the code does not compile (The namespace or module 'sqlParser' is not defined). Every test project references the SDK through the test infrastructure, so the conversion lives in a project that never does, and the tests call its one public function.The two new laws (
PrinterLawsTests, 100 generated valid queries each):c["name"]c.nameWhat the parser taught, each pinned by a unit test in
SdkGrammarTests:A sign directly before a digit belongs to the number.
1 -2is two numbers in a row and is rejected;(1 - 2)is a subtraction and(- 2)a unary minus. The printer already writes the spaces. Anything that writes query text by hand has to as well.Three distinctions of a tree are not in its text:
a JOIN (b JOIN d)is written like(a JOIN b) JOIN dand read as that;2.0as2, as the SDK does, and2is read as an integer;FROM c["tags"]is writtenFROM c.tags, which is read as an identifier step. That is why the second law compares texts.The generator of valid trees (test: check the Cosmos SQL printer by laws on generated syntax trees #55) now builds joins left-deep and gives a number without a fraction as an integer literal, so a valid tree is one the text keeps.
Types of changes
Tests only; the packages are unchanged.
Checklist
Further comments
Grammar(every clause and kind of node, operator precedence, string escapes, kinds of numbers, errors) and the 2 laws;FSharp.Azure.Cosmos.Sql.Testshas 123 now.NOT INasIN,DESCasASC, and no space after a unary operator. Each fails the first law, the last one both; none of the eleven laws of test: check the Cosmos SQL printer by laws on generated syntax trees #55 fails for any of them.sql.g4, to generate the parser from it and to keep a regeneration script. The SDK keeps its own generated sources in its repository, so they are copied instead: no Java anywhere, and no difference between this parser and the SDK's. The ADR's "fix the emptyLEX_IDENTIFIERalternative" falls away with that: the rule stays as the SDK has it.\n,\t,\uXXXX, …). The SDK's own conversion decodes three (\",\\,\/) and would not read back what the printer writes.Antlr4.Runtime.Standard4.9.3 (BSD-3-Clause). It targets .NET Standard 2.0 without dependencies, where 4.7.2, the version of the generator, targets .NET Standard 1.3. It has to stay below 4.10: the generated sources do not compile against 4.10.0 or 4.13.1 (the type ofSerializedAtnchanged).SqlQuery.validateaccepts a join on the right side of a join, although no text can express it. Whether the validator should report it, or the printer should refuse it, is a question for feat: addFSharp.Azure.Cosmos.Sql, a syntax tree, printer, validator and function catalog for Cosmos DB SQL #42.dotnet build FSharp.Azure.Cosmos.slnxin Debug and Release with the same 19 warnings as the branches it is built on and none in the two new projects (the F# one also with-p:WarnOn=3390);FSharp.Azure.Cosmos.Sql.Tests123 of 123 andHedgehog.MSTest.Tests143 of 143 in both configurations; Fantomas clean. Each of the four commits builds and passes the tests it has (102, 121, 123, 123). The emulator suite was not run locally: nothing it uses changed.🤖 Generated with Claude Code