diff --git a/ai-plans/0061-add-utf-8-floating-point-formatting.md b/ai-plans/0061-add-utf-8-floating-point-formatting.md new file mode 100644 index 0000000..966f26a --- /dev/null +++ b/ai-plans/0061-add-utf-8-floating-point-formatting.md @@ -0,0 +1,67 @@ +# Add UTF-8 Floating-Point Formatting + +## Rationale + +`CanonicalFloatingPointFormatter` currently renders only UTF-16 text, so every UTF-8 consumer has to detour through a `char` representation. The JSON metadata writer shows the cost: `MetadataExtensions.WriteNumberValue` calls `writer.WriteRawValue(value.ToCanonicalString(), skipInputValidation: true)`, which allocates a string for every `Double` and `Single` it writes and then lets `System.Text.Json` transcode it back to ASCII bytes. Other UTF-8 producers would have to format into a temporary `char` span and narrow it themselves. + +Add explicit UTF-8 span overloads that write the existing runtime-independent canonical representation directly into the caller's byte destination. This plan covers the formatting primitive only. The allocation it removes becomes observable once the serializers adopt it, which is deliberate follow-up work. + +## Acceptance Criteria + +- [x] `CanonicalFloatingPointFormatter` exposes `TryFormatUtf8` overloads for `double` and `float` plus public maximum-length constants on both package assets, all with XML documentation, while its existing public API remains source- and behavior-compatible. +- [x] For every finite value, the UTF-8 output is byte-for-byte equal to the ASCII bytes of the existing canonical text and `bytesWritten` equals the `charsWritten` of the UTF-16 overload for the same value. No byte-order mark or terminator is written, and the bytes are produced without routing through a UTF-16 buffer or a framework floating-point formatter. +- [x] UTF-8 formatting rejects NaN and both infinities with the same `ArgumentException` contract as UTF-16 formatting; an insufficient destination returns `false`, writes zero to `bytesWritten`, and leaves the destination unchanged. +- [x] After warm-up of both the Grisu3 and the Dragon4 path, both UTF-8 overloads allocate nothing for `double` and `float`. +- [x] Automated tests cover both output encodings for the named canonical scenarios, the deterministic finite-value and exponent corpora, the forced-Dragon4 path, non-finite values, and insufficient destinations. The formatter test project passes against both the `net10.0` and the `netstandard2.0` library asset. +- [x] `src/Light.PortableResults/Numbers/README.md` records the shared code-unit renderer among the adaptations, and the package release notes mention the new UTF-8 formatting API. +- [x] Both target frameworks build in Release with warnings as errors, package validation succeeds, the Native AOT sample publishes successfully, and test coverage remains above 95%. + +## Technical Details + +### Public API + +The exact public API addition is: + +```csharp +public const int MaximumDoubleLength = 32; +public const int MaximumSingleLength = 24; + +public static bool TryFormatUtf8(double value, Span destination, out int bytesWritten); +public static bool TryFormatUtf8(float value, Span destination, out int bytesWritten); +``` + +Ownership and pooling of the byte storage belong to the caller, so no array-returning `FormatUtf8` API is added. The constants replace the existing private `DoubleTextLength` and `SingleTextLength` fields, which callers currently have to duplicate as magic numbers. They are documented upper bounds with headroom above the true worst cases, and they bound both encodings, because the canonical alphabet is entirely ASCII: digits, sign characters, decimal point, and uppercase `E`. Every output character therefore maps to exactly one UTF-8 byte. The bytes represent the canonical numeric token itself, not JSON-escaped content. + +Required length must be calculated before the first destination write so both encodings retain the all-or-nothing `TryFormat` contract. + +### Shared renderer + +Keep concrete `double` and `float` entry points and digit-generation paths. Generic-math interfaces and static abstract interface members are unavailable to the `netstandard2.0` contract. Instead, share the formatting logic across the output code unit with a private generic core and renderer, specialized on the code-unit type: + +```csharp +private static bool TryRender( + ReadOnlySpan digits, + int scale, + bool isNegative, + int positionalMaximumScale, + Span destination, + out int unitsWritten +) + where TCodeUnit : unmanaged; +``` + +The digit generators already emit ASCII digits into `Span`. The renderer stores those digits and its ASCII punctuation as the selected code-unit type through a small conversion helper that branches on `typeof(TCodeUnit) == typeof(byte)` and reinterprets the value with `Unsafe.As` or `Unsafe.As`. The JIT specializes generic code over value types and folds the `typeof` comparison away, so neither branch survives into the hot loops. `System.Runtime.CompilerServices.Unsafe` is already used by the `netstandard2.0` asset in `Errors.cs`, and `AllowUnsafeBlocks` is enabled, so this adds no dependency. + +The `unmanaged` constraint permits instantiations other than `char` and `byte`, but the renderer is private and only ever instantiated with those two. Do not add a guard clause for the impossible case: an unreachable `throw` is a coverage hole. + +The renderer must not format to `char` and then transcode, call `Encoding`, or duplicate the notation and length-calculation rules in separate UTF-8 and UTF-16 renderers. Because the UTF-16 path keeps writing straight into the caller's span, it gains no intermediate buffer and no extra pass. + +### Test seam + +Retain the per-call forced-Dragon4 test seam for both output encodings without mutable global state. Keep exactly one private `TryFormatCore` per numeric type, so the reflection lookup in `CanonicalFloatingPointFormatterTests` continues to match a single method per numeric type; the tests then bind it per code unit with `MakeGenericMethod` and one delegate type per code unit. Adding per-encoding overloads instead would make that lookup ambiguous. + +Extend the existing corpus assertions to compare the UTF-8 bytes with the ASCII bytes of the same expected canonical strings; the UTF-16 assertions remain the normative compatibility check. The allocation test must warm up both digit generators before using `GC.GetAllocatedBytesForCurrentThread`, using values known to take the Dragon4 fallback in addition to the Grisu3-reachable ones it already covers. + +### Scope + +This issue does not add UTF-8 APIs to `MetadataValue` and does not change JSON or other serializers. Those integrations follow once the primitive has been reviewed, and they are where the removed string allocation becomes measurable. diff --git a/src/Light.PortableResults/Light.PortableResults.csproj b/src/Light.PortableResults/Light.PortableResults.csproj index 8dd2b29..cc21522 100644 --- a/src/Light.PortableResults/Light.PortableResults.csproj +++ b/src/Light.PortableResults/Light.PortableResults.csproj @@ -17,6 +17,8 @@ - Adds a net10.0 asset with DateOnly and TimeOnly metadata APIs while retaining netstandard2.0. - Makes canonical Double and Single text runtime-independent. Output is unchanged on .NET Core 3.0+ hosts and corrected on .NET Framework and legacy Mono hosts. + - Adds allocation-free UTF-8 span formatting for canonical Double and Single values, plus public + maximum-length constants shared by the UTF-16 and UTF-8 formatting APIs. - Emits primitive metadata arrays as ordered, separate HTTP header values. Custom header converters can reuse the new public HttpHeaderValueFormatter. diff --git a/src/Light.PortableResults/Numbers/CanonicalFloatingPointFormatter.cs b/src/Light.PortableResults/Numbers/CanonicalFloatingPointFormatter.cs index f8a3b5d..3e9db0e 100644 --- a/src/Light.PortableResults/Numbers/CanonicalFloatingPointFormatter.cs +++ b/src/Light.PortableResults/Numbers/CanonicalFloatingPointFormatter.cs @@ -1,4 +1,5 @@ using System; +using System.Runtime.CompilerServices; namespace Light.PortableResults.Numbers; @@ -20,16 +21,42 @@ namespace Light.PortableResults.Numbers; /// public static class CanonicalFloatingPointFormatter { + /// + /// A destination size that is always large enough to hold the canonical encoding of any finite + /// double-precision value, in UTF-16 characters or in UTF-8 bytes. + /// + /// + /// This is an upper bound with headroom, not the length of the longest output: no finite value + /// produces this many code units. Size a destination with it and + /// and + /// always succeed. A single value bounds + /// both encodings because the canonical alphabet is entirely ASCII, so every character occupies + /// exactly one UTF-8 byte. + /// + public const int MaximumDoubleLength = 32; + + /// + /// A destination size that is always large enough to hold the canonical encoding of any finite + /// single-precision value, in UTF-16 characters or in UTF-8 bytes. + /// + /// + /// This is an upper bound with headroom, not the length of the longest output: no finite value + /// produces this many code units. Size a destination with it and + /// and + /// always succeed. A single value bounds + /// both encodings because the canonical alphabet is entirely ASCII, so every character occupies + /// exactly one UTF-8 byte. + /// + public const int MaximumSingleLength = 24; + private const int DoubleDigitsLength = 17; - private const int DoubleTextLength = 32; private const int SingleDigitsLength = 9; - private const int SingleTextLength = 24; /// Formats a finite double-precision value. /// Thrown when is not finite. public static string Format(double value) { - Span buffer = stackalloc char[DoubleTextLength]; + Span buffer = stackalloc char[MaximumDoubleLength]; TryFormatCore(value, buffer, out var charsWritten, forceDragon4: false); return buffer.Slice(0, charsWritten).ToString(); } @@ -38,7 +65,7 @@ public static string Format(double value) /// Thrown when is not finite. public static string Format(float value) { - Span buffer = stackalloc char[SingleTextLength]; + Span buffer = stackalloc char[MaximumSingleLength]; TryFormatCore(value, buffer, out var charsWritten, forceDragon4: false); return buffer.Slice(0, charsWritten).ToString(); } @@ -61,12 +88,33 @@ public static bool TryFormat(double value, Span destination, out int chars public static bool TryFormat(float value, Span destination, out int charsWritten) => TryFormatCore(value, destination, out charsWritten, forceDragon4: false); - private static bool TryFormatCore( + /// Attempts to format a finite double-precision value as a UTF-8 numeric token. + /// + /// when the destination is large enough; otherwise . + /// On failure, is zero and the destination is not modified. No byte-order + /// mark or terminator is written. + /// + /// Thrown when is not finite. + public static bool TryFormatUtf8(double value, Span destination, out int bytesWritten) => + TryFormatCore(value, destination, out bytesWritten, forceDragon4: false); + + /// Attempts to format a finite single-precision value as a UTF-8 numeric token. + /// + /// when the destination is large enough; otherwise . + /// On failure, is zero and the destination is not modified. No byte-order + /// mark or terminator is written. + /// + /// Thrown when is not finite. + public static bool TryFormatUtf8(float value, Span destination, out int bytesWritten) => + TryFormatCore(value, destination, out bytesWritten, forceDragon4: false); + + private static bool TryFormatCore( double value, - Span destination, - out int charsWritten, + Span destination, + out int unitsWritten, bool forceDragon4 ) + where TCodeUnit : unmanaged { if (!FloatingPointBits.IsFinite(value)) { @@ -94,16 +142,17 @@ bool forceDragon4 FloatingPointBits.IsNegative(value), positionalMaximumScale: 17, destination, - out charsWritten + out unitsWritten ); } - private static bool TryFormatCore( + private static bool TryFormatCore( float value, - Span destination, - out int charsWritten, + Span destination, + out int unitsWritten, bool forceDragon4 ) + where TCodeUnit : unmanaged { if (!FloatingPointBits.IsFinite(value)) { @@ -131,18 +180,19 @@ bool forceDragon4 FloatingPointBits.IsNegative(value), positionalMaximumScale: 9, destination, - out charsWritten + out unitsWritten ); } - private static bool TryRender( + private static bool TryRender( ReadOnlySpan digits, int scale, bool isNegative, int positionalMaximumScale, - Span destination, - out int charsWritten + Span destination, + out int unitsWritten ) + where TCodeUnit : unmanaged { var useScientificNotation = scale < -3 || scale > positionalMaximumScale; var requiredLength = isNegative ? 1 : 0; @@ -169,37 +219,37 @@ out int charsWritten if (destination.Length < requiredLength) { - charsWritten = 0; + unitsWritten = 0; return false; } var index = 0; if (isNegative) { - destination[index++] = '-'; + destination[index++] = ToCodeUnit((byte) '-'); } if (useScientificNotation) { - destination[index++] = (char) digits[0]; + destination[index++] = ToCodeUnit(digits[0]); if (digits.Length > 1) { - destination[index++] = '.'; + destination[index++] = ToCodeUnit((byte) '.'); CopyDigits(digits[1..], destination, ref index); } - destination[index++] = 'E'; + destination[index++] = ToCodeUnit((byte) 'E'); var exponent = scale - 1; - destination[index++] = exponent < 0 ? '-' : '+'; + destination[index++] = ToCodeUnit(exponent < 0 ? (byte) '-' : (byte) '+'); WriteExponent(exponent < 0 ? -exponent : exponent, destination, ref index); } else if (scale <= 0) { - destination[index++] = '0'; - destination[index++] = '.'; + destination[index++] = ToCodeUnit((byte) '0'); + destination[index++] = ToCodeUnit((byte) '.'); for (var zeroIndex = 0; zeroIndex < -scale; zeroIndex++) { - destination[index++] = '0'; + destination[index++] = ToCodeUnit((byte) '0'); } CopyDigits(digits, destination, ref index); @@ -207,7 +257,7 @@ out int charsWritten else if (scale < digits.Length) { CopyDigits(digits[..scale], destination, ref index); - destination[index++] = '.'; + destination[index++] = ToCodeUnit((byte) '.'); CopyDigits(digits[scale..], destination, ref index); } else @@ -215,38 +265,57 @@ out int charsWritten CopyDigits(digits, destination, ref index); for (var zeroIndex = digits.Length; zeroIndex < scale; zeroIndex++) { - destination[index++] = '0'; + destination[index++] = ToCodeUnit((byte) '0'); } - destination[index++] = '.'; - destination[index++] = '0'; + destination[index++] = ToCodeUnit((byte) '.'); + destination[index++] = ToCodeUnit((byte) '0'); } - charsWritten = index; + unitsWritten = index; return true; } - private static void CopyDigits( + private static void CopyDigits( ReadOnlySpan digits, - Span destination, + Span destination, ref int destinationIndex ) + where TCodeUnit : unmanaged { for (var index = 0; index < digits.Length; index++) { - destination[destinationIndex++] = (char) digits[index]; + destination[destinationIndex++] = ToCodeUnit(digits[index]); } } - private static void WriteExponent(int exponent, Span destination, ref int index) + private static void WriteExponent( + int exponent, + Span destination, + ref int index + ) + where TCodeUnit : unmanaged { if (exponent >= 100) { - destination[index++] = (char) ('0' + exponent / 100); + destination[index++] = ToCodeUnit((byte) ('0' + exponent / 100)); exponent %= 100; } - destination[index++] = (char) ('0' + exponent / 10); - destination[index++] = (char) ('0' + exponent % 10); + destination[index++] = ToCodeUnit((byte) ('0' + exponent / 10)); + destination[index++] = ToCodeUnit((byte) ('0' + exponent % 10)); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static TCodeUnit ToCodeUnit(byte value) + where TCodeUnit : unmanaged + { + if (typeof(TCodeUnit) == typeof(byte)) + { + return Unsafe.As(ref value); + } + + var character = (char) value; + return Unsafe.As(ref character); } } diff --git a/src/Light.PortableResults/Numbers/README.md b/src/Light.PortableResults/Numbers/README.md index ff3547b..00d53c5 100644 --- a/src/Light.PortableResults/Numbers/README.md +++ b/src/Light.PortableResults/Numbers/README.md @@ -39,10 +39,11 @@ Files containing adapted runtime code retain the .NET Foundation MIT header. The buffer is stack-only, requires `AllowUnsafeBlocks`, and leaves headroom above the binary64 shortest-mode maximum. - Replaced runtime-internal memory clearing and copying with bounded fixed-buffer loops. -- Added a direct invariant renderer for this library's notation thresholds, signed uppercase - exponent form, negative zero, and positional whole-number `.0` marker. -- Made `TryFormat` the primary rendering path. `Format` uses a bounded stack buffer and constructs - only the returned string. +- Added a shared code-unit renderer, specialized for UTF-16 characters and UTF-8 bytes, for this + library's notation thresholds, signed uppercase exponent form, negative zero, and positional + whole-number `.0` marker. Both encodings write directly into the caller's destination. +- Made the span-based `TryFormat` and `TryFormatUtf8` methods the primary rendering paths. `Format` + uses a bounded stack buffer and constructs only the returned string. ## Retained code that shortest-unique mode cannot reach diff --git a/tests/Light.PortableResults.Tests/Numbers/CanonicalFloatingPointFormatterTests.cs b/tests/Light.PortableResults.Tests/Numbers/CanonicalFloatingPointFormatterTests.cs index ad02eff..3551928 100644 --- a/tests/Light.PortableResults.Tests/Numbers/CanonicalFloatingPointFormatterTests.cs +++ b/tests/Light.PortableResults.Tests/Numbers/CanonicalFloatingPointFormatterTests.cs @@ -3,6 +3,7 @@ using System.Globalization; using System.Linq; using System.Reflection; +using System.Text; using FluentAssertions; using Light.PortableResults.Metadata; using Light.PortableResults.Numbers; @@ -15,25 +16,22 @@ public sealed class CanonicalFloatingPointFormatterTests private const int CorpusSize = 50_000; private const int CorpusSeed = 0x5EED_0058; - private static readonly ForcedDoubleFormatter ForceDragon4Double = CreateDoubleDragon4Formatter(); - private static readonly ForcedSingleFormatter ForceDragon4Single = CreateSingleDragon4Formatter(); + private const byte UnwrittenByte = 0xCC; - private delegate bool ForcedDoubleFormatter( - double value, - Span destination, - out int charsWritten, - bool forceDragon4 - ); + private static readonly ForcedDoubleCharFormatter ForceDragon4DoubleChars = + CreateDoubleDragon4Formatter(); - private delegate bool ForcedSingleFormatter( - float value, - Span destination, - out int charsWritten, - bool forceDragon4 - ); + private static readonly ForcedDoubleByteFormatter ForceDragon4DoubleBytes = + CreateDoubleDragon4Formatter(); + + private static readonly ForcedSingleCharFormatter ForceDragon4SingleChars = + CreateSingleDragon4Formatter(); + + private static readonly ForcedSingleByteFormatter ForceDragon4SingleBytes = + CreateSingleDragon4Formatter(); public static TheoryData DoubleScenarios => - new() + new () { { 0x3F1A36E2EB1C432DUL, "0.0001" }, { 0x3EE4F8B588E368F1UL, "1E-05" }, @@ -65,7 +63,7 @@ bool forceDragon4 }; public static TheoryData SingleScenarios => - new() + new () { { 0x38D1B717U, "0.0001" }, { 0x3727C5ACU, "1E-05" }, @@ -87,6 +85,28 @@ bool forceDragon4 { 0x00800000U, "1.1754944E-38" } }; + [Fact] + public void MaximumLengthsShouldBoundEveryFiniteValueInBothEncodings() + { + // An all-ones mantissa maximizes the significant digit count, so these sweeps drive both + // digit generators to the longest coefficient each binary exponent can produce. + const ulong doubleMantissa = 0x000FFFFFFFFFFFFFUL; + for (ulong exponent = 0; exponent < 0x7FF; exponent++) + { + var bits = (exponent << 52) | doubleMantissa; + AssertFitsIntoMaximumLength(BitConverter.Int64BitsToDouble((long) bits)); + AssertFitsIntoMaximumLength(BitConverter.Int64BitsToDouble((long) (bits | (1UL << 63)))); + } + + const uint singleMantissa = 0x007FFFFFU; + for (uint exponent = 0; exponent < 0xFF; exponent++) + { + var bits = (exponent << 23) | singleMantissa; + AssertFitsIntoMaximumLength(BitConverter.Int32BitsToSingle((int) bits)); + AssertFitsIntoMaximumLength(BitConverter.Int32BitsToSingle((int) (bits | (1U << 31)))); + } + } + [Theory] [MemberData(nameof(DoubleScenarios))] public void DoubleNamedScenariosShouldUseTheCanonicalEncoding(ulong bits, string expected) @@ -96,6 +116,8 @@ public void DoubleNamedScenariosShouldUseTheCanonicalEncoding(ulong bits, string CanonicalFloatingPointFormatter.Format(value).Should().Be(expected); Format(value).Should().Be(expected); FormatWithDragon4(value).Should().Be(expected); + AssertUtf8Matches(value, expected, forceDragon4: false); + AssertUtf8Matches(value, expected, forceDragon4: true); MetadataValue.FromDouble(value).ToCanonicalString().Should().Be(expected); } @@ -108,6 +130,8 @@ public void SingleNamedScenariosShouldUseTheCanonicalEncoding(uint bits, string CanonicalFloatingPointFormatter.Format(value).Should().Be(expected); Format(value).Should().Be(expected); FormatWithDragon4(value).Should().Be(expected); + AssertUtf8Matches(value, expected, forceDragon4: false); + AssertUtf8Matches(value, expected, forceDragon4: true); MetadataValue.FromSingle(value).ToCanonicalString().Should().Be(expected); } @@ -117,10 +141,22 @@ public void NonFiniteDoubleValuesShouldBeRejectedByEveryOverload() foreach (var value in new[] { double.NaN, double.PositiveInfinity, double.NegativeInfinity }) { var format = () => CanonicalFloatingPointFormatter.Format(value); - var tryFormat = () => CanonicalFloatingPointFormatter.TryFormat(value, new char[32], out _); + var tryFormat = () => + CanonicalFloatingPointFormatter.TryFormat( + value, + new char[CanonicalFloatingPointFormatter.MaximumDoubleLength], + out _ + ); + var tryFormatUtf8 = () => + CanonicalFloatingPointFormatter.TryFormatUtf8( + value, + new byte[CanonicalFloatingPointFormatter.MaximumDoubleLength], + out _ + ); format.Should().Throw().WithParameterName("value"); tryFormat.Should().Throw().WithParameterName("value"); + tryFormatUtf8.Should().Throw().WithParameterName("value"); } } @@ -130,10 +166,22 @@ public void NonFiniteSingleValuesShouldBeRejectedByEveryOverload() foreach (var value in new[] { float.NaN, float.PositiveInfinity, float.NegativeInfinity }) { var format = () => CanonicalFloatingPointFormatter.Format(value); - var tryFormat = () => CanonicalFloatingPointFormatter.TryFormat(value, new char[24], out _); + var tryFormat = () => + CanonicalFloatingPointFormatter.TryFormat( + value, + new char[CanonicalFloatingPointFormatter.MaximumSingleLength], + out _ + ); + var tryFormatUtf8 = () => + CanonicalFloatingPointFormatter.TryFormatUtf8( + value, + new byte[CanonicalFloatingPointFormatter.MaximumSingleLength], + out _ + ); format.Should().Throw().WithParameterName("value"); tryFormat.Should().Throw().WithParameterName("value"); + tryFormatUtf8.Should().Throw().WithParameterName("value"); } } @@ -142,8 +190,12 @@ public void InsufficientDestinationsShouldRemainUnmodified() { Span doubleDestination = stackalloc char[3]; Span singleDestination = stackalloc char[2]; + Span doubleUtf8Destination = stackalloc byte[3]; + Span singleUtf8Destination = stackalloc byte[2]; doubleDestination.Fill('x'); singleDestination.Fill('y'); + doubleUtf8Destination.Fill(0xAA); + singleUtf8Destination.Fill(0xBB); CanonicalFloatingPointFormatter.TryFormat( double.MaxValue, @@ -159,11 +211,29 @@ out var singleCharsWritten ) .Should() .BeFalse(); + CanonicalFloatingPointFormatter.TryFormatUtf8( + double.MaxValue, + doubleUtf8Destination, + out var doubleBytesWritten + ) + .Should() + .BeFalse(); + CanonicalFloatingPointFormatter.TryFormatUtf8( + float.MaxValue, + singleUtf8Destination, + out var singleBytesWritten + ) + .Should() + .BeFalse(); doubleCharsWritten.Should().Be(0); singleCharsWritten.Should().Be(0); + doubleBytesWritten.Should().Be(0); + singleBytesWritten.Should().Be(0); doubleDestination.ToArray().Should().OnlyContain(character => character == 'x'); singleDestination.ToArray().Should().OnlyContain(character => character == 'y'); + doubleUtf8Destination.ToArray().Should().OnlyContain(value => value == 0xAA); + singleUtf8Destination.ToArray().Should().OnlyContain(value => value == 0xBB); } [Fact] @@ -191,6 +261,8 @@ public void RandomFiniteBitPatternsShouldMatchTheRuntimeOracle() FormatWithDragon4(value) .Should() .Be(expected, "Dragon4 must independently match binary64 bits 0x{0:X16}", bits); + AssertUtf8Matches(value, expected, forceDragon4: false); + AssertUtf8Matches(value, expected, forceDragon4: true); doubleCount++; } @@ -211,6 +283,8 @@ public void RandomFiniteBitPatternsShouldMatchTheRuntimeOracle() FormatWithDragon4(value) .Should() .Be(expected, "Dragon4 must independently match binary32 bits 0x{0:X8}", bits); + AssertUtf8Matches(value, expected, forceDragon4: false); + AssertUtf8Matches(value, expected, forceDragon4: true); singleCount++; } } @@ -221,17 +295,17 @@ public void EveryBinaryExponentShouldMatchTheRuntimeOracle() const ulong doubleMantissa = 0x000A5A5A5A5A5A5AUL; for (ulong exponent = 0; exponent < 0x7FF; exponent++) { - var bits = exponent << 52 | doubleMantissa; + var bits = (exponent << 52) | doubleMantissa; AssertMatchesOracle(BitConverter.Int64BitsToDouble((long) bits)); - AssertMatchesOracle(BitConverter.Int64BitsToDouble((long) (bits | 1UL << 63))); + AssertMatchesOracle(BitConverter.Int64BitsToDouble((long) (bits | (1UL << 63)))); } const uint singleMantissa = 0x005A5A5AU; for (uint exponent = 0; exponent < 0xFF; exponent++) { - var bits = exponent << 23 | singleMantissa; + var bits = (exponent << 23) | singleMantissa; AssertMatchesOracle(BitConverter.Int32BitsToSingle((int) bits)); - AssertMatchesOracle(BitConverter.Int32BitsToSingle((int) (bits | 1U << 31))); + AssertMatchesOracle(BitConverter.Int32BitsToSingle((int) (bits | (1U << 31)))); } } @@ -242,23 +316,36 @@ public void FloatingPointSpanFormattingShouldAllocateNothingAfterWarmup() const float singleValue = 123_456_789f; var doubleMetadata = MetadataValue.FromDouble(doubleValue); var singleMetadata = MetadataValue.FromSingle(singleValue); - Span destination = stackalloc char[32]; + Span charDestination = stackalloc char[CanonicalFloatingPointFormatter.MaximumDoubleLength]; + Span byteDestination = stackalloc byte[CanonicalFloatingPointFormatter.MaximumDoubleLength]; for (var index = 0; index < 100; index++) { - CanonicalFloatingPointFormatter.TryFormat(doubleValue, destination, out _); - CanonicalFloatingPointFormatter.TryFormat(singleValue, destination, out _); - doubleMetadata.TryFormatCanonical(destination, out _); - singleMetadata.TryFormatCanonical(destination, out _); + CanonicalFloatingPointFormatter.TryFormat(doubleValue, charDestination, out _); + CanonicalFloatingPointFormatter.TryFormat(singleValue, charDestination, out _); + CanonicalFloatingPointFormatter.TryFormatUtf8(doubleValue, byteDestination, out _); + CanonicalFloatingPointFormatter.TryFormatUtf8(singleValue, byteDestination, out _); + ForceDragon4DoubleChars(doubleValue, charDestination, out _, true); + ForceDragon4SingleChars(singleValue, charDestination, out _, true); + ForceDragon4DoubleBytes(doubleValue, byteDestination, out _, true); + ForceDragon4SingleBytes(singleValue, byteDestination, out _, true); + doubleMetadata.TryFormatCanonical(charDestination, out _); + singleMetadata.TryFormatCanonical(charDestination, out _); } var before = GC.GetAllocatedBytesForCurrentThread(); for (var index = 0; index < 1_000; index++) { - CanonicalFloatingPointFormatter.TryFormat(doubleValue, destination, out _); - CanonicalFloatingPointFormatter.TryFormat(singleValue, destination, out _); - doubleMetadata.TryFormatCanonical(destination, out _); - singleMetadata.TryFormatCanonical(destination, out _); + CanonicalFloatingPointFormatter.TryFormat(doubleValue, charDestination, out _); + CanonicalFloatingPointFormatter.TryFormat(singleValue, charDestination, out _); + CanonicalFloatingPointFormatter.TryFormatUtf8(doubleValue, byteDestination, out _); + CanonicalFloatingPointFormatter.TryFormatUtf8(singleValue, byteDestination, out _); + ForceDragon4DoubleChars(doubleValue, charDestination, out _, true); + ForceDragon4SingleChars(singleValue, charDestination, out _, true); + ForceDragon4DoubleBytes(doubleValue, byteDestination, out _, true); + ForceDragon4SingleBytes(singleValue, byteDestination, out _, true); + doubleMetadata.TryFormatCanonical(charDestination, out _); + singleMetadata.TryFormatCanonical(charDestination, out _); } GC.GetAllocatedBytesForCurrentThread().Should().Be(before); @@ -299,11 +386,53 @@ public void StringFormattingShouldAllocateOnlyTheReturnedStrings() .Be(singleBaseline); } + private static void AssertFitsIntoMaximumLength(double value) + { + // The destinations are sized exactly at the constant, so a successful call is itself the + // proof that the constant bounds this value in this encoding. + Span chars = stackalloc char[CanonicalFloatingPointFormatter.MaximumDoubleLength]; + Span bytes = stackalloc byte[CanonicalFloatingPointFormatter.MaximumDoubleLength]; + + CanonicalFloatingPointFormatter.TryFormat(value, chars, out var charsWritten) + .Should() + .BeTrue(); + CanonicalFloatingPointFormatter.TryFormatUtf8(value, bytes, out var bytesWritten) + .Should() + .BeTrue(); + ForceDragon4DoubleChars(value, chars, out var dragon4CharsWritten, true).Should().BeTrue(); + ForceDragon4DoubleBytes(value, bytes, out var dragon4BytesWritten, true).Should().BeTrue(); + + bytesWritten.Should().Be(charsWritten); + dragon4CharsWritten.Should().Be(charsWritten); + dragon4BytesWritten.Should().Be(charsWritten); + } + + private static void AssertFitsIntoMaximumLength(float value) + { + Span chars = stackalloc char[CanonicalFloatingPointFormatter.MaximumSingleLength]; + Span bytes = stackalloc byte[CanonicalFloatingPointFormatter.MaximumSingleLength]; + + CanonicalFloatingPointFormatter.TryFormat(value, chars, out var charsWritten) + .Should() + .BeTrue(); + CanonicalFloatingPointFormatter.TryFormatUtf8(value, bytes, out var bytesWritten) + .Should() + .BeTrue(); + ForceDragon4SingleChars(value, chars, out var dragon4CharsWritten, true).Should().BeTrue(); + ForceDragon4SingleBytes(value, bytes, out var dragon4BytesWritten, true).Should().BeTrue(); + + bytesWritten.Should().Be(charsWritten); + dragon4CharsWritten.Should().Be(charsWritten); + dragon4BytesWritten.Should().Be(charsWritten); + } + private static void AssertMatchesOracle(double value) { var expected = CanonicalizeRuntimeText(value.ToString("R", CultureInfo.InvariantCulture)); CanonicalFloatingPointFormatter.Format(value).Should().Be(expected); FormatWithDragon4(value).Should().Be(expected); + AssertUtf8Matches(value, expected, forceDragon4: false); + AssertUtf8Matches(value, expected, forceDragon4: true); } private static void AssertMatchesOracle(float value) @@ -311,6 +440,8 @@ private static void AssertMatchesOracle(float value) var expected = CanonicalizeRuntimeText(value.ToString("R", CultureInfo.InvariantCulture)); CanonicalFloatingPointFormatter.Format(value).Should().Be(expected); FormatWithDragon4(value).Should().Be(expected); + AssertUtf8Matches(value, expected, forceDragon4: false); + AssertUtf8Matches(value, expected, forceDragon4: true); } private static string CanonicalizeRuntimeText(string value) => @@ -318,7 +449,7 @@ private static string CanonicalizeRuntimeText(string value) => private static string Format(double value) { - Span destination = stackalloc char[32]; + Span destination = stackalloc char[CanonicalFloatingPointFormatter.MaximumDoubleLength]; CanonicalFloatingPointFormatter.TryFormat(value, destination, out var charsWritten) .Should() .BeTrue(); @@ -327,7 +458,7 @@ private static string Format(double value) private static string Format(float value) { - Span destination = stackalloc char[24]; + Span destination = stackalloc char[CanonicalFloatingPointFormatter.MaximumSingleLength]; CanonicalFloatingPointFormatter.TryFormat(value, destination, out var charsWritten) .Should() .BeTrue(); @@ -336,27 +467,73 @@ private static string Format(float value) private static string FormatWithDragon4(double value) { - Span destination = stackalloc char[32]; - ForceDragon4Double(value, destination, out var charsWritten, true).Should().BeTrue(); + Span destination = stackalloc char[CanonicalFloatingPointFormatter.MaximumDoubleLength]; + ForceDragon4DoubleChars(value, destination, out var charsWritten, true).Should().BeTrue(); return new string(destination[..charsWritten]); } private static string FormatWithDragon4(float value) { - Span destination = stackalloc char[24]; - ForceDragon4Single(value, destination, out var charsWritten, true).Should().BeTrue(); + Span destination = stackalloc char[CanonicalFloatingPointFormatter.MaximumSingleLength]; + ForceDragon4SingleChars(value, destination, out var charsWritten, true).Should().BeTrue(); return new string(destination[..charsWritten]); } - private static ForcedDoubleFormatter CreateDoubleDragon4Formatter() => - (ForcedDoubleFormatter) GetTryFormatCore(typeof(double)).CreateDelegate( - typeof(ForcedDoubleFormatter) - ); + private static void AssertUtf8Matches(double value, string expected, bool forceDragon4) + { + Span chars = stackalloc char[CanonicalFloatingPointFormatter.MaximumDoubleLength]; + Span bytes = stackalloc byte[CanonicalFloatingPointFormatter.MaximumDoubleLength + 1]; + bytes.Fill(UnwrittenByte); + + var charsSucceeded = forceDragon4 ? + ForceDragon4DoubleChars(value, chars, out var charsWritten, true) : + CanonicalFloatingPointFormatter.TryFormat(value, chars, out charsWritten); + var bytesSucceeded = forceDragon4 ? + ForceDragon4DoubleBytes(value, bytes, out var bytesWritten, true) : + CanonicalFloatingPointFormatter.TryFormatUtf8(value, bytes, out bytesWritten); + + charsSucceeded.Should().BeTrue(); + bytesSucceeded.Should().BeTrue(); + bytesWritten.Should().Be(charsWritten); + new string(chars[..charsWritten]).Should().Be(expected); + bytes[..bytesWritten].ToArray().Should().Equal(Encoding.ASCII.GetBytes(expected)); + bytes[bytesWritten].Should().Be(UnwrittenByte); + } - private static ForcedSingleFormatter CreateSingleDragon4Formatter() => - (ForcedSingleFormatter) GetTryFormatCore(typeof(float)).CreateDelegate( - typeof(ForcedSingleFormatter) - ); + private static void AssertUtf8Matches(float value, string expected, bool forceDragon4) + { + Span chars = stackalloc char[CanonicalFloatingPointFormatter.MaximumSingleLength]; + Span bytes = stackalloc byte[CanonicalFloatingPointFormatter.MaximumSingleLength + 1]; + bytes.Fill(UnwrittenByte); + + var charsSucceeded = forceDragon4 ? + ForceDragon4SingleChars(value, chars, out var charsWritten, true) : + CanonicalFloatingPointFormatter.TryFormat(value, chars, out charsWritten); + var bytesSucceeded = forceDragon4 ? + ForceDragon4SingleBytes(value, bytes, out var bytesWritten, true) : + CanonicalFloatingPointFormatter.TryFormatUtf8(value, bytes, out bytesWritten); + + charsSucceeded.Should().BeTrue(); + bytesSucceeded.Should().BeTrue(); + bytesWritten.Should().Be(charsWritten); + new string(chars[..charsWritten]).Should().Be(expected); + bytes[..bytesWritten].ToArray().Should().Equal(Encoding.ASCII.GetBytes(expected)); + bytes[bytesWritten].Should().Be(UnwrittenByte); + } + + private static TFormatter CreateDoubleDragon4Formatter() + where TCodeUnit : unmanaged + where TFormatter : Delegate => + (TFormatter) GetTryFormatCore(typeof(double)) + .MakeGenericMethod(typeof(TCodeUnit)) + .CreateDelegate(typeof(TFormatter)); + + private static TFormatter CreateSingleDragon4Formatter() + where TCodeUnit : unmanaged + where TFormatter : Delegate => + (TFormatter) GetTryFormatCore(typeof(float)) + .MakeGenericMethod(typeof(TCodeUnit)) + .CreateDelegate(typeof(TFormatter)); private static MethodInfo GetTryFormatCore(Type numberType) => typeof(CanonicalFloatingPointFormatter) @@ -377,4 +554,32 @@ private static long MeasureStringAllocations(Func factory) return GC.GetAllocatedBytesForCurrentThread() - before; } + + private delegate bool ForcedDoubleCharFormatter( + double value, + Span destination, + out int charsWritten, + bool forceDragon4 + ); + + private delegate bool ForcedDoubleByteFormatter( + double value, + Span destination, + out int bytesWritten, + bool forceDragon4 + ); + + private delegate bool ForcedSingleCharFormatter( + float value, + Span destination, + out int charsWritten, + bool forceDragon4 + ); + + private delegate bool ForcedSingleByteFormatter( + float value, + Span destination, + out int bytesWritten, + bool forceDragon4 + ); }