Add systematic type-matrix and fuzz tests for expression functions

.github/workflows/fuzz.yml

@@ -0,0 +1,51 @@
+name: Fuzz
+
+# Property-based fuzz tests (tagged :fuzz) run on a schedule rather than as a
+# PR/push gate. They use a fresh random seed each run to DISCOVER inputs that
+# crash a function, so they can legitimately go red — which makes them a poor
+# merge gate but a good nightly signal. When this job fails, ExUnit prints the
+# seed and StreamData prints the shrunk failing input; reproduce locally with
+# `mix test --only fuzz --seed <N>`.
+on:
+  schedule:
+    # 03:00 UTC daily
+    - cron: "0 3 * * *"
+  workflow_dispatch: {}
+
+jobs:
+  fuzz:
+    name: Run fuzz tests
+    runs-on: ubuntu-latest
+    env:
+      # Explore more of the input space than the local default (100); this job
+      # is not latency-bound.
+      FUZZ_MAX_RUNS: "1000"
+    strategy:
+      matrix:
+        include:
+          - elixir: "1.19"
+            otp: "28"
+        cache_version: ["4"]
+
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
+      - name: Set up Elixir
+        uses: erlef/setup-beam@fc68ffb90438ef2936bbb3251622353b3dcb2f93 # v1
+        with:
+          elixir-version: ${{ matrix.elixir }}
+          otp-version: ${{ matrix.otp }}
+      - name: Restore dependencies cache
+        uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5
+        id: mix-cache
+        with:
+          path: deps
+          key: ${{matrix.cache_version}}-${{ runner.os }}-${{ matrix.otp }}-${{ matrix.elixir }}-mix-${{ hashFiles('**/mix.lock') }}
+          restore-keys: ${{matrix.cache_version}}-${{ runner.os }}-${{ matrix.otp }}-${{ matrix.elixir }}-mix-
+      - name: Install Mix Dependencies
+        if: steps.mix-cache.outputs.cache-hit != 'true'
+        run: |
+          mix local.rebar --force
+          mix local.hex --force
+          mix deps.get
+      - name: Run fuzz tests
+        run: mix test --only fuzz

CLAUDE.md

@@ -67,3 +67,10 @@ Expressions preserve types through evaluation (integers, floats, booleans, DateT
 ## Testing
 
 CI tests against Elixir 1.15/OTP 26, 1.18/OTP 27, and 1.19/OTP 28. Format checking runs only on 1.19/OTP 28.
+
+Beyond `@expression_doc` doctests and `expression_test.exs`, the suite includes
+systematic **type-matrix** tests per function category
+(`test/*_functions_type_test.exs`) and **property-based fuzz** tests
+(`test/expression_fuzz_test.exs`, tagged `:fuzz`, excluded from `mix test` —
+run with `mix test --only fuzz`). These document current V1 behavior (pinning
+crashes, not endorsing them). See `TESTING.md` for the full approach.

TESTING.md

@@ -0,0 +1,179 @@
+# Testing Guide
+
+This document describes how the Expression library is tested and how to add
+tests when you change or add a function.
+
+## Test layout
+
+| Location | Purpose |
+|----------|---------|
+| `@expression_doc` annotations in `lib/expression/callbacks/standard.ex` | Happy-path examples that double as doctests |
+| `test/expression_test.exs` | Core engine + hand-written edge cases (incl. the pilot type tests for `upper`, `lower`, `abs`, `round`, `date`) |
+| `test/*_functions_type_test.exs` | Systematic **type-matrix** tests, one file per function category |
+| `test/crash_safe_classification_test.exs` | Deterministic guard that the crash-safe set never raises across the type matrix |
+| `test/expression_fuzz_test.exs` | Property-based **crash-safety** tests (tagged `:fuzz`) |
+| `test/support/*.ex` | Reusable test helpers (compiled via `elixirc_paths` for `:test`) |
+
+The category type-test files are:
+
+- `test/string_functions_type_test.exs`
+- `test/number_functions_type_test.exs`
+- `test/date_functions_type_test.exs`
+- `test/logical_functions_type_test.exs`
+- `test/enum_functions_type_test.exs`
+
+## Running tests
+
+```bash
+# Everything except fuzz tests (the default, fast suite)
+mix test
+
+# A single file
+mix test test/string_functions_type_test.exs
+
+# A single test by line number
+mix test test/string_functions_type_test.exs:42
+
+# Watch mode during development
+mix test.watch
+
+# Property-based fuzz tests ONLY (see "Fuzz tests" below)
+mix test --only fuzz
+```
+
+Fuzz tests are **excluded from the default suite** via
+`ExUnit.start(exclude: [:fuzz])` in `test/test_helper.exs`. You opt into them
+explicitly with `--only fuzz`.
+
+## Guiding philosophy: document current behavior
+
+The type-test and fuzz files **document the V1 engine's actual behavior — they
+do not endorse it.** Many functions raise on unexpected input rather than
+returning an error map. Where that happens, the test pins the exact exception
+so that any future change to the behavior is deliberate, not accidental:
+
+```elixir
+test "nil raises ArithmeticError" do
+  # Known crash behavior, documented not endorsed: Kernel.rem/2 with nil.
+  assert_raise ArithmeticError, fn -> evaluate_with_value("rem(value, 3)", nil) end
+end
+```
+
+When you read `# Known crash behavior, documented not endorsed: ...` or
+`# Surprising: ...` in a test, it is recording a real, observed behavior — not
+the behavior we wish the function had. If you *fix* such a behavior, update the
+corresponding test to assert the new (better) result.
+
+## Test support helpers
+
+### `Expression.Test.TypeTestMatrix`
+
+Imported by every type-test file. Provides a canonical set of sample values for
+each runtime type, plus convenience evaluators:
+
+- `evaluate_with_value(expr, value, extra_context \\ %{})` — evaluate a block
+  expression (no leading `@`) with `value` bound to the `value` key.
+- `complex_value(value, extra \\ %{})` — build a "complex" map carrying a
+  `__value__` key, as produced by flow results.
+- `error_value(message \\ ...)` — build a V1 error map.
+- `all_test_values/0`, `test_values_for/1`, `invalid_values_for/1` — the matrix
+  values themselves.
+
+### `Expression.Test.FuzzHelpers`
+
+Imported by the fuzz file. Provides `StreamData` generators (`any_value/0`,
+`string_value/0`, `list_value/0`, `enumerable_value/0`, …) and
+`assert_no_crash/2`, which fails only if evaluating an expression *raises* or
+`throw`s/`exit`s (returning an error map is fine).
+
+### `Expression.Test.CrashSafe`
+
+The single source of truth for the crash-safe function set. `groups/0`,
+`group/1`, and `all/0` return `{label, expression}` entries consumed by both the
+deterministic classification test and the fuzz suite.
+
+## The context-coercion gotcha
+
+`Expression.Context` coerces context values **before callbacks see them**:
+
+- numeric-looking strings (`"123"`, `"3.14"`) become numbers,
+- ISO-date-looking strings become `Date`/`DateTime`/`Time` structs,
+- `"true"`/`"false"` become booleans.
+
+So `evaluate_with_value("upper(value)", "123")` does **not** pass the string
+`"123"` to `upper` — it passes the integer `123`. To test a function with a
+genuine string argument, embed a **string literal in the expression source**:
+
+```elixir
+# context-coerced: "123" -> 123 before the callback
+evaluate_with_value("fixed(value, 2)", "123")   # => "123.00"
+
+# literal string survives to the callback
+Expression.evaluate_block!(~s|fixed("3.14", 2)|) # => "3.14"
+```
+
+## Fuzz tests
+
+`test/expression_fuzz_test.exs` enforces a single invariant: **the function must
+never raise** on arbitrary input (returning a value or an error map is fine).
+
+The V1 engine does **not** uphold this invariant universally, so the suite
+fuzzes only the subset of functions empirically confirmed crash-safe across the
+whole type matrix. That subset is defined once in `Expression.Test.CrashSafe`
+(`test/support/crash_safe.ex`) and consumed by two suites:
+
+- `test/crash_safe_classification_test.exs` — a **deterministic** guard (runs in
+  the default suite, every CI build) that evaluates each crash-safe function
+  against the full type matrix and fails if any raises. This is what pins the
+  classification; it cannot drift on a lucky seed.
+- `test/expression_fuzz_test.exs` — random exploration on top, for inputs the
+  fixed matrix doesn't contain.
+
+Functions that currently *do* crash are not in `CrashSafe`; each is pinned with
+its exact exception in the relevant `*_functions_type_test.exs` file.
+
+These tests are **not** a merge gate. They use a random seed each run and exist
+to *discover* new crashing inputs, so they can legitimately go red when they
+find one — useful as a signal, but a poor fit for a branch-protection check.
+Instead they run on a schedule via `.github/workflows/fuzz.yml` (nightly, plus a
+manual "Run workflow" button). Run them locally with:
+
+```bash
+mix test --only fuzz
+
+# Search deeper (the scheduled job uses 1000 generations per property):
+FUZZ_MAX_RUNS=1000 mix test --only fuzz
+```
+
+When a run fails, ExUnit prints the seed and StreamData prints the shrunk
+failing input. Reproduce it deterministically with:
+
+```bash
+mix test --only fuzz --seed <N>
+```
+
+If you harden a known-crashing function so it returns an error map instead of
+raising, add it to the appropriate category in `Expression.Test.CrashSafe` (and
+update its type test). Both the deterministic guard and the fuzz suite pick it
+up automatically.
+
+## Adding tests for a new function
+
+When you add a function to `Expression.Callbacks.Standard`:
+
+1. **Add `@expression_doc` examples** in the source for the happy path. These
+   run as doctests automatically.
+2. **Add type-matrix tests** to the matching `test/<category>_functions_type_test.exs`
+   file. Cover, at minimum:
+   - `nil`,
+   - the type(s) the function is meant to accept,
+   - wrong types (numbers, strings, booleans, lists, maps),
+   - a complex value via `complex_value/1` to confirm `__value__` extraction,
+   - any function-specific edge cases (empty string/list, unicode, zero,
+     negatives, leap years, …).
+   **Discover the real behavior by running it** — do not guess. If it crashes,
+   pin the exact exception with `assert_raise` and a `# Known crash behavior`
+   comment.
+3. **If the function is crash-safe**, add it to the relevant `@crash_safe_*`
+   list in `test/expression_fuzz_test.exs` so the no-crash invariant is guarded.
+4. Run `mix format` and make sure `mix test` is green.

mix.exs

@@ -9,6 +9,7 @@ defmodule Expression.MixProject do
       aliases: aliases(),
       version: @version,
       elixir: "~> 1.13",
+      elixirc_paths: elixirc_paths(Mix.env()),
       start_permanent: Mix.env() == :prod,
       deps: deps(),
       package: package(),
@@ -32,6 +33,10 @@ defmodule Expression.MixProject do
     [plt_file: {:no_warn, "priv/plts/expression.plt"}, ignore_warnings: ".dialyzer_ignore.exs"]
   end
 
+  # Test support modules live in test/support and are only compiled for :test.
+  defp elixirc_paths(:test), do: ["lib", "test/support"]
+  defp elixirc_paths(_), do: ["lib"]
+
   # Run "mix help compile.app" to learn about applications.
   def application do
     [
@@ -51,6 +56,7 @@ defmodule Expression.MixProject do
       {:mix_test_watch, "~> 1.0", only: :dev, runtime: false},
       {:nimble_parsec, "~> 1.1"},
       {:number, "~> 1.0"},
+      {:stream_data, "~> 1.0", only: [:test, :dev]},
       {:decimal, "~> 2.0"},
       {:timex, "~> 3.7"}
     ]

mix.lock

@@ -28,6 +28,7 @@
   "parse_trans": {:hex, :parse_trans, "3.4.1", "6e6aa8167cb44cc8f39441d05193be6e6f4e7c2946cb2759f015f8c56b76e5ff", [:rebar3], [], "hexpm", "620a406ce75dada827b82e453c19cf06776be266f5a67cff34e1ef2cbb60e49a"},
   "ssl_verify_fun": {:hex, :ssl_verify_fun, "1.1.7", "354c321cf377240c7b8716899e182ce4890c5938111a1296add3ec74cf1715df", [:make, :mix, :rebar3], [], "hexpm", "fe4c190e8f37401d30167c8c405eda19469f34577987c76dde613e838bbc67f8"},
   "statistex": {:hex, :statistex, "1.1.0", "7fec1eb2f580a0d2c1a05ed27396a084ab064a40cfc84246dbfb0c72a5c761e5", [:mix], [], "hexpm", "f5950ea26ad43246ba2cce54324ac394a4e7408fdcf98b8e230f503a0cba9cf5"},
+  "stream_data": {:hex, :stream_data, "1.3.0", "bde37905530aff386dea1ddd86ecbf00e6642dc074ceffc10b7d4e41dfd6aac9", [:mix], [], "hexpm", "3cc552e286e817dca43c98044c706eec9318083a1480c52ae2688b08e2936e3c"},
   "sweet_xml": {:hex, :sweet_xml, "0.7.5", "803a563113981aaac202a1dbd39771562d0ad31004ddbfc9b5090bdcd5605277", [:mix], [], "hexpm", "193b28a9b12891cae351d81a0cead165ffe67df1b73fe5866d10629f4faefb12"},
   "timex": {:hex, :timex, "3.7.13", "0688ce11950f5b65e154e42b47bf67b15d3bc0e0c3def62199991b8a8079a1e2", [:mix], [{:combine, "~> 0.10", [hex: :combine, repo: "hexpm", optional: false]}, {:gettext, "~> 0.26", [hex: :gettext, repo: "hexpm", optional: false]}, {:tzdata, "~> 1.1", [hex: :tzdata, repo: "hexpm", optional: false]}], "hexpm", "09588e0522669328e973b8b4fd8741246321b3f0d32735b589f78b136e6d4c54"},
   "tzdata": {:hex, :tzdata, "1.1.3", "b1cef7bb6de1de90d4ddc25d33892b32830f907e7fc2fccd1e7e22778ab7dfbc", [:mix], [{:hackney, "~> 1.17", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm", "d4ca85575a064d29d4e94253ee95912edfb165938743dbf002acdf0dcecb0c28"},

test/crash_safe_classification_test.exs

@@ -0,0 +1,32 @@
+defmodule CrashSafeClassificationTest do
+  @moduledoc """
+  Deterministic guard for the crash-safe classification in
+  `Expression.Test.CrashSafe`.
+
+  `ExpressionFuzzTest` explores the same set with random inputs, but it is
+  excluded from the default suite and uses a fresh seed each run. This test pins
+  the classification deterministically: every function claimed crash-safe is
+  evaluated against the entire type matrix on every CI run, and must return a
+  value or error map rather than raising. If someone adds a function to
+  `CrashSafe` that actually crashes on a matrix value, this fails immediately —
+  no random seed required.
+  """
+  use ExUnit.Case, async: true
+
+  import Expression.Test.TypeTestMatrix
+  alias Expression.Test.CrashSafe
+
+  for {label, expr} <- CrashSafe.all() do
+    test "#{label} never raises across the full type matrix" do
+      values = all_test_values()
+
+      results =
+        for value <- values do
+          # Raising here fails the test; a value or error map is acceptable.
+          Expression.evaluate_block!(unquote(expr), %{"value" => value})
+        end
+
+      assert length(results) == length(values)
+    end
+  end
+end