feat(compiler): infer rectangular arrays and tables#64
Open
thiremani wants to merge 25 commits into
Open
Conversation
Refactor Array into a homogeneous one-dimensional type and classify bracket literals from their shape and column schemas. Multi-row homogeneous literals become row-major matrices, while named or heterogeneous-column literals become columnar tables; a bare header row forces an unnamed table without adding a keyword. Add LLVM/runtime lowering, value-copy and cleanup semantics, named column access, printing, ABI mangling, documentation, and focused end-to-end coverage for the new collection types.
Normalize pointer-backed indirect call results before struct and table field dispatch. Preserve table projection ownership and cover selecting a column directly from a returned table.
Copy projected columns before consuming temporary table bases through freeConsumedTemporary. This keeps conditional-frame cleanup synchronized when a projected column feeds another expression.
Reject a bare colon in bracket literals and infer unnamed tables directly from their data rows. Remove the redundant HasHeaderRow flag, standardize examples on four-space indentation, and omit header markers when printing unnamed tables.
Render and document named table headers as :Name rather than : Name. Source whitespace remains permissive and produces no compiler warnings.
Allow empty arrays and header-only tables to remain valid zero-row values until concatenation supplies a concrete element type. Lower, print, and project these values safely while rejecting indexing, element-wise operations, and function calls that require resolved element types. Document the behavior and cover empty table projection and refinement alongside collection shape and unresolved-type errors.
Represent [] as a resolved Array[Empty] so empty arrays can specialize templates and refine through concatenation. Preserve a binding's concrete element type when it is reset to [], while lowering empty values to the canonical null array representation. Document the empty-array rules and cover direct, named, in-place, identity, empty-concat, and type-lock behavior.
Replace the Matrix type with Array{ElemType, Rank}: rank is part of type
identity, dimension lengths are runtime values, and every rank shares one
flat row-major buffer (rank 1 stays a bare runtime-vector pointer; higher
ranks carry {data, dim0..dimN}). Mangling composes as repeated Array_t1_,
so no new ABI entry is needed.
Literals infer rank from shape: stacked scalar rows are rank 2, and
array-valued cells of equal rank and shape stack one rank higher, so
[[1 2] [3 4]] equals the stacked spelling and variables stack as rows.
Mixed scalar/array cells, ragged rows, rank-mismatched operations, and
statically unequal child shapes are compile errors; cells are never
padded with defaults. Indexing removes the outer dimension and returns
an owned copy, so chained indexing reaches inner ranks; OOB reads keep
the per-lane failed-condition contract. Range indexing stays rank-1
outside collectors, while [m[range]] stacks selected rows.
Runtime shape checks (stacking, concatenation, zipping with dynamically
unequal inner dimensions) now abort through array_shape_fail with a
diagnostic naming the mismatched dimensions instead of a silent trap.
Stacked-literal children are released via freeConsumedTemporary so
conditional-frame cleanup stays synchronized.
Document the model in docs/Pluto Array Semantics.md and cover rank-3
construction and chained indexing, stacked collectors, string matrices,
empty resets, and the new shape/rank error paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
thiremani
changed the title
Record grouped rank-N selection and nested range construction as post-PIR work. Distinguish shared statement gates from local value-position && and align the PIR vocabulary with that scope.
Specify that value-position range && binds a domain without collecting it. Document matching row fallbacks and preserve explicit stacking and rectangular shape checks.
Document explicit range binders for multidimensional construction, constant collection with [i && 1], and zero-filled array children for failed value-position conditions. Require PIR to prove skipped child shape or use an explicit compatible fallback.
Preserve inner dimensions when higher-rank OOB cells zero-fill, and distinguish rank-1 empty resets from shaped Empty values. Reject range-driven table cells before code generation and align the empty-table and empty-operand documentation with runtime behavior.
Render the table header marker as a two-space hanging prefix so the first header and first row value begin in the same column. Document this as the preferred source layout while keeping table whitespace non-semantic and all existing spellings accepted.
Keep the README focused on the user-facing overview, move detailed table rules into Array Semantics, and make Range Semantics authoritative for nested domain construction. Trim repeated selection and binder rules from the Memory Model in favor of links to their canonical sections.
Use one flat-buffer allocation and cell-fill path for rank-1 and rectangular scalar literals while leaving dimension calculation with each caller. Restore the deferred unreachable-fallback diagnostic and the no-padding shape guarantee in Range Semantics.
Render one-row AST literals inline while retaining multiline source rendering for tables and multi-row literals. Route all rank-1 runtime string conversions through the existing rank-aware formatter so scalar and higher-rank arrays share cell formatting and allocation behavior.
Preserve source delimiters when rendering string-literal AST nodes so literal text cannot be confused with missing-expression markers.
Remove the single-use trapOnFalse wrapper while preserving the existing checked runtime-status control flow.
Separate solved Array and Table lowering from the shared AST entry point, using guard clauses for array strategy selection.
Rely on the solver invariant that every valid bracket-literal cell has exactly one cached output type.
Use the solver invariant that valid literals cannot mix scalar and array-valued cells, avoiding a redundant full-cell scan during lowering.
Compile immediate rank-one literals through compileFixedArrayLiteral directly. Rank-one values do not store dimensions, so remove the unused dimension calculation and slice-returning adapter.
Generalize shortest-input zip semantics to rank-N element-wise operations by selecting each result dimension independently and mapping result coordinates through each source's row-major strides. Element-wise operations no longer abort on unequal inner dimensions; stacking and concatenation retain their rectangular shape checks. Document conditional outcomes as conceptual value/yield lanes for the future PIR lowering.
Rank-one length is stored by the runtime vector rather than in the LLVM array value. Skip rebuilding that discarded metadata in scalar array operations and concatenation while preserving rank-N dimensions.
Let ArrayLen supply zero for empty and unresolved element types during concatenation. Keep concrete-type checks only around operations that emit typed element access.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Arrayto a homogeneous leaf type plus rank, replacing the separateMatrixtype with rectangular rank-N arrays.[Empty]arrays so[]can cross function boundaries, refine through concatenation, and reset an established array without changing its type or rank.Inference and shape rules
[][Empty]Arrays use one flat row-major buffer with runtime dimensions; rows are not separately allocated. Ragged literals and statically known shape/rank mismatches are compile errors. Runtime-dependent stacking, concatenation, and element-wise operations validate inner dimensions before proceeding. Empty operands are shape-neutral for runtime operations; concatenation adopts the nonempty operand's inner shape.
Indexing removes the outer dimension. A range is still an iteration driver rather than a slice:
[matrix[rows]]collects and stacks selected rows. Out-of-bounds higher-rank cells produce zero-filled children with the source's inner dimensions, preserving rectangular shape. Shape-preserving collection across multiple indexed dimensions is deliberately deferred.Table headers require at least one name. The preferred layout uses a two-space hanging
:marker (: Name Score) so the first header aligns with the first value below it; whitespace remains non-semantic. Header-only tables are valid zero-row values whose printed form retains the header; their columns remain unresolved until data establishes their types. Projected columns are independent array values, including when the source table is temporary. Range-driven table cells are rejected because table rows require statically sized cells.User-visible behavior
[[1 2] [3 4]]and the equivalent two-row scalar literal produce the same rank-2 value.[]can be passed to functions and can refine through concatenation.[]to a concrete array empties it while preserving its established leaf type and rank.[[]], retain their dimensions while adopting an established leaf type.Implementation
{ data, dim0, ..., dim(N-1) }, backed by one flat runtime vector.Validation
go test -race ./lexer ./parser ./compilergo vet ./...with the documentedbyollvmenvironmentpython3 test.py(60/60 passed)python3 test.py --leak-check tests/array(13/13 passed, no leaks)