Repository navigation
Conversation
…xorigin#29544) ## What type of PR is this? - [x] BUG - [x] Improvement - [ ] API-change - [ ] Documentation - [ ] Feature - [ ] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: Fixes matrixorigin#29542 ## What this PR does / why we need it: Arithmetic predicates must preserve rounding, NULL and checked-error semantics, including after data is flushed. Inverse real-number algebra and invalid metadata bounds can silently remove matching rows or suppress genuine errors. ### Design and existing ownership - Remove unsafe equality arithmetic transposition and its four sole-use helpers. Reuse the existing comparison normalizer; native column equality/range normalization remains available. - Keep arithmetic bounds in the existing zone-map owner. Check signed/unsigned overflow and nonfinite values. Unknown, varying-denominator and unsupported widened bounds retain blocks for the existing row executor. - Consume already-materialized supported scalar Fold values in the existing evaluator; remove the ranges consumer's blanket filter discard. NULL, lists, malformed/unmaterialized and unsupported payloads remain unknown. Reset reused scratch so previous proofs cannot leak into later objects. - Repair `Decimal64.Mul` capped-scale handling at its existing owner, including the signed-boundary fallback. Use real objects and existing resource ownership in the CN shuffle test fixture. - Close the latest scalar-panic review finding at the generic metadata callback boundary. Independently combined column endpoints may overflow ROUND even when every actual row is valid. Only a callback `*moerr.Error` with `ErrOutOfRange` invalidates the exclusion proof. Returned errors also remain unknown. Temporary results and optional function state have one deferred cleanup on success, partial failure, allocation failure and panic. Internal/runtime/untyped and cleanup panics propagate; actual SQL row overflow still errors. No new execution owner, state machine, stored state, wire/catalog format or framework. ### Validation and independent QA The latest finding was reproduced before the fix through real SQL and the actual ROUND kernel: correlated rows returned `100` before flush but raised overflow after flush; the metadata callback left 16 bytes allocated after scratch cleanup. Disabled pruning and the nearby boundary passed. After the fix, valid controls return `100`, the synthetic proof becomes unknown, and memory returns to its baseline before process teardown. A real MaxInt64 row still raises ROUND overflow before/after flush and with pruning disabled. Current scalar-boundary validation: - Complete uncached `pkg/sql/colexec` unit suite passes on the final production code; the final test refinements are covered by the focused race run. Focused final `-race` covers actual signed/unsigned ROUND, disjoint → overflow → disjoint → matching → neighboring bounds, and success/returned-error/arithmetic/internal/runtime/string/cleanup-panic/allocation-failure paths. Cleanup occurs once; a failed allocation never runs the callback. - Repository-configured incremental golangci-lint: **0 issues**. Formatting/diff checks pass. Escape analysis confirms the new callbacks/deferred closures do not escape. - Rebuilt local service: **153 passed, 0 failed/abnormal, 1 existing ignored**, across arithmetic semantics (78), explain_check (35) and pushdown (40 passed/1 ignored). The two-row regression reuses the existing SQL fixture. EXPLAIN ANALYZE CHECK requires both `Block Filter Cond` and `outputRows=1`; genuine ROUND error checks reuse the existing overflow table. Still-valid preceding validation covers complete `pkg/container/types`, `pkg/vm/engine/tae/index` and `pkg/vm/engine/disttae` suites, the planner/filter suites, focused race/SCA, prepared rebinding and independent decimal scale/extrema oracles. Earlier CI's six EXPLAIN expectations were updated only after result/error and pruning checks confirmed the removal of unsafe inversion. Design and final independent overall QA use **gpt-6.1-sol xhigh**, as requested. Latest verdict: **ACCEPT**, with no remaining verified blocker, including the latest P2. Commit: `a0b46b0de29386ff42b10cf1796cf466b54b6928`. Reviewed diff SHA256: `6dfd348f3b746d75446eff7f7b6ca7d08e300de7d1d2034641ea5d71104e2779`. ### Performance, increment and limits On the same two flushed objects / 2,048 rows, safe addition/subtraction/multiplication read **1 block / 1,024 rows**, versus **2 blocks / 2,048 rows** on the original PR head, with identical results. Native equality still reads one row. Existing 10,000-row EXPLAIN checks require `inputBlocks=1 inputRows=8192`. Metadata arithmetic microbenchmarks remain allocation-free; these are not whole-workload throughput measurements. The latest correction adds **7 net production lines**, **207 test lines**, and no documentation files. It replaces repeated cleanup with one owned boundary. Tests distinguish real kernel/reuse behavior, injected failure ownership and public SQL behavior, reusing existing fixtures rather than duplicating setup. Full PR: **production −228**, **tests +777**, **documentation 0**, **total +549**; test growth is reviewed separately from the production reduction. Unknown proofs intentionally retain blocks. Constant-left subtraction currently has no planner block filter and retains both blocks in the probe. Local real-service evidence uses one CN; new remote multi-CN CI is not awaited or claimed passed. The preceding head's Coverage/CI Required failure is not represented as green. Old binaries retain old behavior during a mixed-version rollout. No whole-workload throughput equivalence is claimed. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…age (matrixorigin#29548) ## What type of PR is this? - [ ] API-change - [ ] BUG - [x] Improvement - [ ] Documentation - [ ] Feature - [x] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: Related to matrixorigin#29333. This addresses its local admission diagnostic gap, not the unconfirmed root cause of the historical TKE incident. It intentionally does not close that issue. ## What this PR does / why we need it: Persisted-expression authoring rejection previously reported only the required protocol. A supported binary waiting for its local authoring floor was therefore indistinguishable from insufficient local protocol support. The existing 20105 error now includes the CN identity, observed local protocol and authoring floor. The admission decision and diagnostic use the same captured values; missing/invalid values are reported without formatting arbitrary runtime objects. The existing message prefix, safe admission predicate and standalone missing-floor fallback are preserved. Remove the single-use boolean helper; introduce no state, RPC, log stream, worker or retry policy. Extend existing fixtures rather than add another cluster: - Planner controls cover numeric boundaries, invalid/missing state, absent process, standalone fallback, and independent read/authoring permission. - Inject a catalog fence failure into the existing CN fixture. Recovery at the same epoch advances the local fence but keeps authoring closed during Preparing; only the enabled/admitted/global-fenced snapshot opens it, even when Ready=false. - In the existing naturally admitted SQL fixture, temporarily lower the local runtime protocol solely to exercise the public diagnostic. Assert MySQL20105, CN/local state, absence of the rejected view, existing data health, then restore the value and create/query the same name successfully. This injection is not claimed as reproduction of the TKE root cause. ### Validation On macOS arm64 / Go1.26.4, native artifacts built with `make -j8 cgo`: - New diagnostic assertion fails against unchanged base production for the intended missing-message-fields reason, while the admission outcome stays20105. - Complete `pkg/sql/plan` and `pkg/cnservice` tests pass, `-count=1 -vet=all`. - Focused HAKeeper post-barrier activation controls pass. - Real SQL `TestIssue28909DirectFunctionWireMetadata` passes, including diagnostic, rejection atomicity and same-name recovery checks. - Incremental repository-configured golangci-lint passes for all three changed package directories. All Go validation uses the repository CGo wrapper. No TKE1TP+2AP incident replay was performed; the historical activation blocker remains unconfirmed. CI is pending and is not represented as passed. ### Performance / compatibility Successful admission retains two bounded runtime lookups and no diagnostic formatting. Extra formatting is on rejection only. No admission/barrier/readiness, wire, catalog, durable data or routing changes. No new production resources. Design and final overall review use gpt-6.1-sol with xhigh reasoning. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…rigin#29543) `INTERSECT ALL` must emit each key `min(left_count, right_count)` times, independently of DOP and CN placement. The existing parallel set-op compilation path broadcasts both complete inputs to each set-op worker and merges their outputs. Unlike DISTINCT operations, ALL cannot remove those duplicated worker outputs without also removing legitimate duplicates. Route `INTERSECT ALL` through the same existing single-multiplicity-owner compilation path as `EXCEPT ALL`/`MINUS ALL`. The input producers remain parallel and feed the one owner through the existing merge/connector operators. No new operator, hash partitioner, protocol, cache, or persistence format is introduced; DISTINCT set-op compilation and the multiset hash/count implementation remain unchanged. Keep the existing single-scope fast path before this fallback: already-single-worker inputs feed the owner directly, without extra consolidation merges or copying connector boundaries. Remove the replaced, unreachable parallel INTERSECT ALL branch. This is a correctness-first fallback: the ALL set-operation stage uses one owner rather than several workers doing duplicated work. Large multiset queries may benefit from a future correctly key-partitioned implementation; this patch does not introduce one or claim that serial set processing has no performance tradeoff. ### Validation - On unpatched main `7b3d0c3bcf`, the ownership regression fails: two set-op output owners are created when one is required. The identical test passes with this fix and verifies both parallel input lists are retained. - Complete `pkg/sql/compile`, `pkg/sql/colexec/intersectall`, `pkg/sql/colexec/minusall`, and `pkg/sql/colexec/dispatch` tests pass. - Focused race-mode compilation tests for INTERSECT ALL, EXCEPT ALL, and DISTINCT set operations pass. - Existing operator coverage includes multiple build batches, NULL multiplicity, new keys, reset/reuse, and memory cleanup. - Real public-SQL two-CN regression on the same persisted input bags: unpatched main fails with value `1` expected twice but returned four times; the patched version passes. The physical plan asserts both remote input scans and parallel scan work, not just the presence of two CNs. - The final two-CN SQL regression, including a single-worker literal control, passes with `-count=2` (43.147s), including reuse of the shared cluster after cleanup. - TP/AP single-worker topology controls prove both original inputs feed the owner directly; parallel controls cover multiple scopes and a single scope with Mcpu=2. Actual public EXPLAIN for `SELECT 1 INTERSECT ALL SELECT 1` confirms 3 scopes, 2 merges and 2 connectors. - Extend the existing set-operator BVT on its existing six-/five-row bags; independently assert duplicate counts, NULL, symmetry, empty input, and downstream COUNT/SUM without changing old expected results. Normal mo-tester comparison passes twice on the same freshly built local service, each 37/37 SQL statements with zero failures; the test tables are absent afterwards. These validations ran on a local macOS/arm64 embedded two-CN cluster and a separately built local BVT service, not on a production cluster or server 50. The original nightly observed 42 rows instead of 3, consistent with DOP-fold duplication. Its physical plan was not retained, so this PR does not claim that 14 physical workers were directly observed in that historical query or assign a recent introducing commit from pass/fail timing alone. Approved by: @XuPeng-SH
…trixorigin#29549) ## What type of PR is this? - [x] BUG - [x] Improvement - [ ] API-change - [ ] Documentation - [ ] Feature - [ ] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: Fixes matrixorigin#29492 ## What this PR does / why we need it: Terminal cleanup confused an unsuccessful channel enqueue with an unsuccessful terminal publication. The edge had already recorded the fatal cause and closed Done, and receivers could recover the terminal from durable edge state. Full channels and End after cancellation therefore produced misleading `timeout=30s` warnings and abnormal-cleanup metrics despite nonblocking publication. Use PipelineEdge's existing terminal state as the single owner of cleanup publication. PublishTerminal returns the effective terminal and whether durable publication is satisfied; existing Try/Send boolean behavior is preserved. Connector, Dispatch and remote-notify cleanup use that result. Remove duplicated terminal logic, ineffective enqueue retries, fallback sends and cleanup timers. Normal End preserves queued data and deferred spool reclamation. Error/Abort and End superseded by a fatal terminal abort with the original cause; allocation account finalization still waits for consumer quiescence. Missing edges/channels remain diagnosed, without calling a nonblocking failure a timeout. Dispatch reuses the existing resolver to prefer substantive errors over synthetic delivery fallout. No wire, stored-data, catalog or mixed-upgrade mechanism changes. ### Validation - Identical pre/post white-box probe: the unfixed production source recorded three abnormal-cleanup events for an already durable cancellation, in 0.00s; fixed source recorded zero. Permanent assertions extend existing full-queue, original-cause and spool-reclamation fixtures instead of retaining the probe. - Publication tests cover full/available queues, partial multi-sender End, Error/Abort first-cause preservation, duplicates, graceful completion, invalid input and reuse. Missing-register tests require diagnostics and reclamation. - Normal and race suites: process, connector, dispatch, compile and pipeline. Focused lifecycle regressions additionally pass 100 race repetitions each. - Real SQL test on an owned three-CN embedded cluster: 24,576 persisted rows, a successful percentile spill control (`SpillRows=49152`), server `KILL QUERY` at an observable execution barrier with live remote senders, expected cancellation, sender/FIN lifecycle gauges returning to zero, unchanged cleanup failure counters, and exact percentile results on every CN. Test body 18.97s. - Incremental vet-all and configured golangci-lint checks. The canceled black-box query is deliberately gated before aggregation; the successful spill control is not evidence that cancellation itself reached spill. Strict full-channel causality is proved by component tests. This is not a claim of reproducing the original TKE 10M-row scale or proving a historical memory leak. The reported 30s was a configured value, not a measured cleanup delay. ### Engineering and performance No new state machine, stored state, retry policy or production test hook. The production delta is a net reduction; normal data forwarding is unchanged and cleanup removes timers, retry bookkeeping and false warning/metric work. Tests reuse existing component fixtures; the separate SQL fixture is isolated because it explicitly requires three CNs and forced remote placement. Design and final overall review: gpt-6.1-sol / xhigh. CI remains pending; local validation is reported separately and no CI wait is implied. Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
## What type of PR is this? - [ ] API-change - [x] BUG - [ ] Improvement - [ ] Documentation - [ ] Feature - [ ] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: Fixes matrixorigin#29560. ## What this PR does / why we need it: `TestJoinCoordinatorStage` completed its five SQL subtests, but cluster cleanup timed out after 40 minutes. FIN held the receive-state mutex while waiting for synchronous send completion. The exiting writer's deferred session Close needed the same mutex before it could drain and fail the queued FIN Future. Server shutdown then waited forever for that writer. Claim FIN in the existing receive-state registry, release its mutex before cache access/write admission/send completion, and retire receive/send sequence state only after successful flush. The inline terminal claim rejects duplicate FIN and later requests on that stream while unrelated streams can progress. Close can independently drain pending sends; closed sessions cannot recreate receive state. Complete the same response ownership boundary: release pre-send FIN rejections, configure the pipeline server's existing MORPC drop-release callback, use the cached message ID after goetty encoding, and snapshot Future flags before completion can return them to their pool. Ordinary Future cancellation semantics remain unchanged to protect borrowed payload lifetime. No new queue, worker, timer, retry, wire format, catalog format, or persisted-data format. The terminal flag lives only in an existing active-stream entry. This removes network-length receive-state lock contention and the shutdown deadlock; no SQL throughput improvement is claimed. ### Validation - Deterministic white-box reproduction fails on clean production code at main `113d374b2e5167238d8411c117dfe879aef8bad6`; explicitly releasing the queued Future allows both blocked calls to finish. - Candidate tests cover queued FIN with no writer, real writer failure and Stopper completion, duplicate FIN/later-request exclusion, unrelated stream progress, invalid authority/pending cache, canceled admission with a full queue, exact message/Future releases, and zero retained state/queue gauges. - Each of eight focused FIN tests: independent `-race -count=100` runs passed; cleanup-adjusted cases were revalidated. - MORPC owning package: normal with `-vet=all` and race passed. - CN service owning package: normal with `-vet=all` and race passed. - Compile FIN lifecycle/ACK success/error/timeout consumers passed. - Real two-CN `TestJoinCoordinatorStage`: normal 18.012s and race 23.311s, including cluster teardown and port-lease release. All five SQL subtests retained. - Incremental golangci-lint for the affected packages and final test delta: 0 issues. - Local environment: Darwin arm64, Go 1.26.4, repository native build and CGo wrapper. This is not a claim of a Linux CI pass; CI remains pending and is not waited on. Design and final overall review completed using actual `gpt-6.1-sol` / `xhigh`. Final decision: PASS, no unresolved material code or validation blockers. Reviewed base `113d374b2e5167238d8411c117dfe879aef8bad6` plus all five final delivery files; source fingerprints match the validation ledger. Session `01a0f77c-c1ba-76e3-ae82-ba019248459d`.
…dex completeness (matrixorigin#29558) ## What type of PR is this? - [ ] API-change - [x] BUG - [x] Improvement - [ ] Documentation - [ ] Feature - [x] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: Fixes matrixorigin#29555 Fixes matrixorigin#29557 ## What this PR does / why we need it: An exact DECIMAL constant such as `id = 16001.0` promotes an integer primary key and loses native block filtering. Constants exposed by scalar-subquery pushdown have the same problem after binding. FORCE INDEX over a nullable UNIQUE key can silently omit NULL-containing rows because the hidden UNIQUE table does not store them. - Share exact Decimal256 divisibility and signed/unsigned range proofs across comparison and IN/BETWEEN coercion. Retain the executable peer and provenance; preserve fractional, out-of-range, approximate and explicit column-cast domains. - Complete integer-domain restoration in the existing numeric filter normalizer after constant exposure and before scan statistics. Remove only a proven lossless implicit column promotion, independently prove every static peer, and rebind the native integer overload. Scan filters and JOIN conditions share the same rule, including commuted ranges and eligible list/BETWEEN forms. Dynamic prepared sources remain under the existing specialization/cache contract. - Require every UNIQUE key component to be declared NOT NULL or proven non-NULL by a local filter before admitting hinted covering or backfill access. AND combines evidence; every OR arm must reject that component's NULL. Incomplete candidates retain the complete base scan, including forced JOIN backfill. - Keep safe UNIQUE and nonunique index access available. No storage, catalog or index encoding changes; existing indexes do not need rebuilding. - Close the shutdown regression’s setup-only system connection pool after account initialization, while its cluster generation is still running. This prevents final cleanup from sending COM_QUIT through a connection terminated by the intentional shutdown; preserve every commit, durability, restart-write and cleanup assertion. Validation on the final production changes: - Full `pkg/sql/plan` tests with `-vet=all`, plus the final test-only atomicity delta, passed. - Existing one-CN SQL frontend fixture passed: covering/backfill rowsets, nullable composite/OR controls, ORDER/GROUP, CTAS, rollback, direct/folded/scalar integer domains, real Block Filter predicates, signed/unsigned and Decimal256 boundaries, empty scalar subqueries, outer JOIN controls, and prepared scalar reuse. - Typed counterexamples cover narrowing/signedness, declared and physical decimal capacity (including Width=0), explicit casts, target mismatch, NULL/dynamic provenance, idempotence, and whole-list rejection without mutation. - Updated the existing `optimizer/index_cast_optimization` BVT oracle for integral `9.0`, preserving the `9.5` rejection control and all other statements. The whole 51-statement case passed twice through mo-tester on the same current-production one-CN instance, with teardown checked. - The shutdown regression passed twice in one process with `-vet=all`, and once under `-race -vet=all`, including final cleanup. - Incremental lint for both changed packages, gofmt and diff checks passed. Required gpt-6.1-sol/xhigh design and overall review completed with no unresolved material blocker. Baseline wrong results and lost Block Filter were reproduced. CI is pending. The pruning claim is verified by native predicates and Block Filter selection; no throughput percentage is claimed. Unsafe forced UNIQUE plans may use a slower complete base scan. Actual stopped 4.2-dev data-directory upgrade replay was not run; mixed-worker upgrades are outside scope. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…res (matrixorigin#29546) ## What type of PR is this? - [ ] API-change - [x] BUG - [x] Improvement - [ ] Documentation - [ ] Feature - [x] Test and CI - [x] Code Refactoring ## Which issue(s) this PR fixes: Fixes matrixorigin#28490. ## What this PR does / why we need it: Signed ROUND mishandled negative remainders, and speculative expression evaluation could publish unsafe metadata or surface diagnostics from inactive branches. This change restores signed rounding and keeps failed or unproven metadata conservative, with residual row execution owning SQL results and selected diagnostics. - Keep checked signed/integer/Decimal64/128 precision arithmetic in the scalar owners. Preserve typed overflow diagnostics and selection masks. - Centralize metadata admission by resolved overload, physical domain and original constant controls. Reuse comparison/list codecs, checked index arithmetic and timezone interval proofs. A current point-valued divisor may bound division while a point-valued column remains FLAT. Arithmetic scale checks apply to decimal operands. - Publish only validated endpoints. Errors, panics, warnings, NULL/nonfinite endpoints and uncertified temporal ranges retain the residual expression. Consume current Fold payloads through their producer/adapter and preserve independent filters. - Separate runtime row independence from speculative fold eligibility in the existing function-owned walker. Constant CASE retains ordinary index access; CASE with row-dependent or volatile children remains ineligible. Folding and metadata keep their lazy CASE restrictions. Remove the redundant planner volatility walk. - Let local constant-input vectors keep one cleanup owner and remove their second copy. Refuse unavailable bounds/nonpoint division before allocating a speculative child. Keep the fresh isolated warning destination and existing result/kernel cleanup owners. ### Design and overall review The prior approved s1+d1+p1+a1 documents are archived in [the formal design/review record](matrixorigin#29546 (review)). The bounded runtime-constness/ownership amendment and final overall review use actual **gpt-6.1-sol / xhigh**, session `01a0f822-7696-7662-8d51-ebe3e7561938`; [the accepted amendment and final overall PASS review](matrixorigin#29546 (review)) confirm no unresolved material blocker. Rebased onto main `32f54981401fd3888702f2dab8702d2e62c6997a` as requested. Conflict reconciliation retains the current point-divisor proof and validator-aware publication helper alongside main's lifecycle changes. ### Validation Current candidate `5359b63d19b48d5e730f8c8815ecd0844fa0acec`: - Full owning/consumer UT: function, planner, colexec, rule, readutil, disttae and compile pass. Focused race: 62 named tests/subtests, three terminal package passes. - Configured incremental vet/lint over all eight changed owning packages pass; lint reports 0 issues. Native prerequisites and CPU service build pass. - Real SQL validates forced-index versus base-scan NULL/range results, nested and reversed CASE, row dependence, prepared true/false/true rebinding, inactive versus selected overflow, and successful execution after a prepared error on flushed data. - Secondary-index BVT: two normal rounds on the same isolated instance, each 240/240 success, 0 failed/ignored/abnormal. Both constant CASE plans retain Index Table Scan and return `(3,'below-95')`, matching IGNORE INDEX. Teardown is empty and owned processes exit cleanly. - Historical scalar/domain evidence is retained where inputs are unchanged: exact legal-domain decimal oracle 15,300 cases; signed ROUND/selection, temporal/CHAR and pruning matrices; two original ROUND BVT rounds. These historical runtime runs used the earlier binary and are distinguished from current candidate evidence. ### Performance and limits Six alternating pairs of the same metadata microbench compare the pre-optimization and candidate binaries on the same machine: | Path | Before ns/op | After ns/op | Before B/allocs | After B/allocs | | --- | ---: | ---: | ---: | ---: | | Native comparison | 105.20 | 104.15 | 24 / 1 | 24 / 1 | | Generic ROUND | 2448.0 | 2401.5 | 1920 / 16 | 1920 / 16 | | Constant ABS | 1862.0 | 1486.5 | 1800 / 12 | 1312 / 10 | The constant path removes 488 bytes and two allocations per invocation in this fixture. Timings are local measurements under shared-machine load. The diagnostic probe's measured 8-byte allocation remains in generic ROUND; the earlier main comparison had 1912 bytes/15 allocations. Numeric widening CAST and Decimal256 metadata retain conservative fallback. Whole-SQL throughput is not inferred from these microbenchmarks. CI will run on the published revision; it is not awaited as requested.
…9547) Connect opt-in embedded Sirius SELECT execution to transaction-aware MO scan pipelines with admission-first reader creation, preserved scan parallelism and lease-bounded batch publication. Decode native results into checked borrowed MO vectors, retire unfinished readers after successful native EOF without hiding cancellation or execution errors, and record terminal GPU, input and result statistics. Preserve Flight behavior. Approved by: @XuPeng-SH, @fengttt
…gin#29556) ## What type of PR is this? - [ ] API-change - [x] BUG - [x] Improvement - [ ] Documentation - [ ] Feature - [ ] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: Closes matrixorigin#29550 Closes matrixorigin#29551 Closes matrixorigin#29552 Closes matrixorigin#29553 ## What this PR does / why we need it: Restore four SQL contracts at their existing conversion and binding owners: - DECIMAL accepts both exponent spellings and validates the complete exponent without scale wrap. The common numeric-prefix scanner supplies borrowed lexical facts; destination quantization uses at most 76 scratch bytes, copies only retained digits and rounds once. Physical digit arithmetic is shared with existing decimal parsers. There is no coefficient-string materialization, reparsing, integer-only fallback or exponent-sized allocation. Strict CAST rejects overflow; explicit CAST clamps only proven overflow, never malformed input. Numeric-prefix/assignment adapters retain their distinct admission contracts. - Complete in-range prepared integer text compares in the signed peer's actual integer domain, preserving adjacent BIGINT identities beyond 2^53 and DML scope. The existing bounded integer normalizer, parameter-source identity and value-dependent cache ownership are reused. Typed scan/singleton JOIN proofs retain FLOAT/ROUND and mixed-integer safety guards; the existing simple-projection normalization and statistics owners are reused rather than adding another interpreter. - New LAST_DAY expressions return nullable DATE vectors and metadata, including CTAS/views. Native DATE/DATETIME overloads avoid formatting/reparsing; invalid, zero and NULL inputs retain NULL behavior. Existing serialized VARCHAR overloads remain executable after a stopped upgrade. - Prepared EXPLAIN is classified by its underlying SELECT at the shared binding entry, preserving the wrapper and excluding explained DML from SELECT-only conversions. Superseded ROUND-specific and duplicate overflow branches are removed. Documentation describes final contracts rather than the review chronology. Runtime-filter BVT asserts the lookup scan's actual `inputBlocks=1 inputRows=1` using an operator-scoped regex, without pinning timings or ignoring the complete ANALYZE result. Existing malformed-DECIMAL goldens are aligned with lexical failure, not restored to blanket overflow clamping. ### Validation Current candidate: Darwin/arm64, Go 1.26.4, matching stamped repository CGo dependencies: - Full types/function/planner/colexec normal suites passed; full `vet` on changed conversion owners and incremental golangci-lint on the complete affected package closure passed, zero issues. - Independent exact `big.Rat` oracle: 1800 inputs / 3600 strict and explicit conversions, widths 1–76, varying scales, signs, long mantissas and positive/negative exponents. Constructor boundaries, lexical failures, rounding/carry, prefix and scientific contracts passed. Ordinary short-text DECIMAL64 conversion has a zero-allocation regression assertion. - Real public frontend: all four prepared numeric/temporal scenarios passed, including persisted decimal values, DML, DATE metadata/schema and EXPLAIN binding. - Real mo-tester: complete decimal (690), scientific notation (37) and runtime-filter (69) cases passed twice on the same instance: 1592 successes, zero failed/ignored/abnormal. Scan-fixture table teardown was asserted before replay. The counter oracle accepts the golden and rejects a scan-only `inputRows=20004` mutation while aggregate row counts remain unchanged. - Earlier exact-head focused race evidence remains valid for unchanged prepared binding/cache and persisted LAST_DAY ABI paths: seven tests ×100 iterations passed. This amendment introduces no shared state, goroutines or lifecycle changes. Matched non-race CAST-stage benchmarks, 8192 nonconstant VARCHAR rows, three 200ms samples, same machine/toolchain/native dependencies: | Conversion | Declared base | Final candidate | Allocations/batch | | --- | ---: | ---: | ---: | | strict DECIMAL(18,2) | 294µs | 295µs | 1 | | explicit DECIMAL(18,2) | 323µs | 298µs | 1 | | explicit DECIMAL(38,2) | 3214µs | 1221µs | 1 | The prior PR head imposed approximately 3× DECIMAL64 CAST-stage work and 16385 allocations/batch; that regression is removed. These are conversion-stage measurements, not whole-query throughput claims. Neighboring MySQL string-to-FLOAT conversion remains zero-allocation. Matched samples are approximately +2% for ordinary/scientific text and +6% for a long leading-zero prefix (about 0.6–2.7ns), rather than extrapolating the isolated scanner's percentage to the whole consumer. Design and final overall review used actual gpt-6.1-sol / xhigh. Final overall review: APPROVE, no unresolved material blocker, session `01a0f93e-b6aa-78d2-819d-f324ae4310c6`. Submitted commit `e66cdafe56817fa62881dc7d5131c75cacadb695` preserves the exact reviewed complete-diff fingerprint `6fcaa7282e0aca51117eaab635bddc13b958bc95500e6116c1ca49006afc4cc0`. Scope includes persisted data/serialized LAST_DAY compatibility; mixed-version rollout is excluded. Remote CI is not awaited and is not claimed to have passed.
…in#29561) ## What type of PR is this? - [x] Improvement - [x] Test and CI ## Which issue(s) this PR fixes: Related to matrixorigin#29562 — race UT CPU, allocation and fixture-cost optimization. The issue remains open until the actual CI acceptance target is demonstrated. ## What this PR does / why we need it: HAKeeper health checks repeatedly deep-copy thousands of configuration fields through reflection. Race profiles show this consumes substantial CPU and allocations throughout embedded tests. Implement the existing `deepcopy` hook for `ConfigData`, preserving detached maps/items/unknown protobuf bytes and nil/empty values. Keep the surrounding StateQuery ownership and execution path unchanged; add real-dispatch isolation tests and a benchmark populated through actual service configs and heartbeats. Move the private two-CN view restore/restart test from the upgrade package into the existing multicn package. It now uses normal exclusive fixture admission instead of keeping its two CNs live alongside the upgrade package's shared CN. Preserve its SQL, identity, prepared-plan, restore and restart assertions, including partial-start cleanup ownership. Reduce sequence-placement source data from 64 to two rows per UNION branch after a public physical-plan probe showed identical normalized scope structure on both coordinators. Preserve repeated per-row advancement, cross-CN continuity, every value/persisted-state assertion, and all separate concurrency/lock/cancellation tests. Correct the AP control comment to describe planning classification, without claiming a remote scan. ### Validation and performance - Race: full protobuf/logservice and HAKeeper packages; full issues package for the copy change (1,267 terminal test/subtest outcomes identical); full multicn and upgrade packages after the fixture move. - Race: sequence family including concurrent sessions, parallel UNION, lock handoff and lock-wait timeout; compiler placement/DOP/noncoordinator rejection and retry-quiescence tests; seven existing fixture cancellation/rollback/cleanup-generation tests. - Real StateQuery benchmark, Go 1.26.4 / amd64 v1 / race / 8 CPUs: median **22.73 ms → 0.85 ms**, **2,172,160 → 561,640 B/op**, **49,213 → 3,892 allocs/op** across three repetitions. These are component results. - Latest local issues pair: **368.7 → 350.2 seconds**, **623.8 → 556.1 CPU-seconds**. A brief sibling-core probe affected the control; this is directional evidence, not a precise attributable improvement or a CI extrapolation. - Preserve the fulltext stress test's 20 rounds and 240-second deadline. The targeted treatment run passed on HDD; a retained cumulative baseline deadline failure remains unexplained and is not a successful performance baseline. **Acceptance still pending: reduce the actual CI race-UT wall time by at least ten minutes while lowering overall CPU/memory. Local measurements do not prove this target.** Keep cache seeding disabled: the bounded local registry import timed out with zero entries published and completed cleanup. This PR adds no cache, scheduler, timeout extension, skipped tests or shortened stress rounds. Design and independent final implementation review use actual `gpt-6.1-sol` with `xhigh` reasoning. Final implementation review: **PASS, no blocking findings**. Incremental configured vet and golangci-lint v2.6.2 (built with Go 1.26.4): **PASS, 0 issues**.
…gin#29563) ## What type of PR is this? - [ ] API-change - [ ] BUG - [x] Improvement - [ ] Documentation - [ ] Feature - [x] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: issue matrixorigin#29562 (partial contribution; the several-minute CI goal remains open) ## What this PR does / why we need it: ESQL uses only Info and EsqlQuery, but constructing the full Elasticsearch client registers every generated API and keeps those closures in transitive consumer executables. Call the generated requests through the existing elastictransport client, retaining its authentication, retry/replay, endpoint selection and certificate-fingerprint behavior. Keep the existing private HTTP transport, CA handling, connection cache and cleanup owners. Preserve SDK request headers/configuration with an immutable adapter and independent comparisons to the pinned original SDK. Require a genuine 2xx Info response before publishing the connection. This also rejects HTTP 101, which previously admitted a connection before the SDK product check completed. No new product-check state, retries, cache or scheduler. Measured locally with Go 1.26.4, race/GOAMD64v1, 8 CPU/16 GiB, p6/link3, the same 477-package light selection and warmed Go caches: | Metric | Before | After | | --- | ---: | ---: | | Total CPU | 1437.94 s | 1349.68 s (-6.14%) | | Repeated link CPU | 948.1 s | 840.4 s (-11.4%) | | Cgroup peak memory, including file cache | 7.96 GiB | 6.65 GiB (-16.4%) | | Sampled anonymous peak | 5.55 GiB | 5.32 GiB (-4.1%) | | Unchanged isolated race executable | 321,304,640 B | 302,662,200 B (-5.8%) | | Constructor allocations | 18,679 B / 557 | 1,759 B / 29 | All 395 test-bearing light packages passed on both sides; no cases were removed. Wall time was 778 -> 419 s, but severe baseline disk contention prevents attributing that difference to CI. Actual CI savings are unverified; this PR does not claim a six-minute saving. The test-only SDK reference still compiles typedapi once; no cold-compiler saving claimed. The final six-line QA restoration was validated in its owning package after the broader runs; production/link inputs are unchanged. Validation: owning package normal/race; 100 race repeats each for info/query/stream cancellation, concurrent query retries and failed-handshake socket release; incremental vet/lint. Retain the original eight empty-401 idle-socket failures and add nonempty wrong/missing-product failures. Preserve private-CA, truncation, 8,192-byte error limit and cache admission/reuse/removal controls. Real SQL probe through a one-CN frontend/planner/operator/reader passed typed CSV/NULL, session reuse, bounded errors, truncated response and disconnect/101 rejection; completed teardown and exit 0. The probe uses a real HTTP fixture, not an external Elasticsearch deployment, and adds no cluster fixture to CI. Plan and final review use actual gpt-6.1-sol with xhigh reasoning. Evidence and profiles are retained outside the delivery diff. No merge/rebase of main.
…or (matrixorigin#29568) A synchronous FULLTEXT2/HNSW rebuild can report success but commit an empty index when its side-effecting build TVF runs on a remote scan worker. `IsSingle` limits each input scope's parallelism; it does not establish the transaction-owning coordinator as the writer. The remote process uses a mirror transaction with a separate workspace. Internal index INSERTs borrow that transaction, and SELECT pipeline results do not transfer those workspace writes back to the coordinator for commit. Declare coordinator-required placement in the existing table-function plugin registry, and opt in only `fulltext2_create` and `hnsw_create`. Both APPLY and child-bearing table-function compilation gather input through the existing coordinator merge scope before invoking the writer. Standalone TVFs are already local. Source scans retain their placement and parallelism; searches and ordinary table functions retain their existing behavior. There is no new algorithm-name dispatch in the compiler, protocol, persistence format, retry, worker, or transaction-commit behavior. Add registry and typed placement tests plus one real two-CN MySQL frontend regression. The SQL test persists three rows per algorithm using public DN flush, rebuilds from both coordinators, asserts indexed results and replacement metadata, and checks empty-input rebuild followed by transaction rollback. The existing scoped multi-CN plan hook bypasses the tiny query's classification threshold, not the builder, storage, transaction, or remote transport. Ordinary default scheduling is a passing control, not a standalone reproducer. On clean pre-fix `52eecb55`, the same public SQL assertions fail after successful ALTER: FULLTEXT2 returns no rows instead of IDs 1,3 and HNSW no rows instead of IDs 1,2,3. Pre-fix `b75086c6` also fails the typed writer-placement assertions while view/search controls pass. With this fix, both builder cases, metadata, source-row, empty-input, and rollback assertions pass. Scope and risk: this fixes the demonstrated remote-writer ownership defect. The original nightly did not retain its exact builder-CN/mirror trace, so this does not establish the cause of every nightly mismatch or an introducing commit. Gathering build input can increase network traffic and concentrate builder work; full-scale build performance was not benchmarked. Search runtime and placement are unchanged. No 4.2 backport is included. Validated final head `41238efc09`, based on `main@d50a2d4`, with Go 1.26.4, darwin/arm64, using `.agents/skills/mo-dev/scripts/mo-cgo-test`. Registry stress evidence was reused across the rebase because its source, tests, and dependencies were unchanged; the remaining final checks ran after the rebase: ```text -count=1 -vet=all -timeout=4m ./pkg/indexplugin/plan ./pkg/fulltext2/plugin/plan ./pkg/vectorindex/hnsw/plugin/plan ./pkg/sql/compile -race -count=1 -timeout=4m ./pkg/indexplugin/plan ./pkg/fulltext2/plugin/plan ./pkg/vectorindex/hnsw/plugin/plan ./pkg/sql/compile -count=1 -timeout=5m -v ./pkg/tests/sqlintegration/multicn -race -count=1 -timeout=5m -v -run '^TestIssue29566IndexRebuildCoordinator$' ./pkg/tests/sqlintegration/multicn -race -count=100 -timeout=120s -run '^TestTableFuncRegistrationPlacement$' ./pkg/indexplugin/plan -count=1 -timeout=4m -v -run '^(TestApply.*|TestOuterApply.*|TestHnswCreate.*|TestFulltext2Create.*)$' ./pkg/sql/colexec/apply ./pkg/sql/colexec/table_function ``` All completed with exit 0. Full multi-CN package: 61.806s; new SQL test: 19.06s normal / 20.48s race. Tests are local embedded clusters, not server 50 or the original TKE environment. No CI/BVT result is claimed before it runs. Approved by: @XuPeng-SH
…origin#29570) ## What type of PR is this? - [ ] API-change - [ ] BUG - [ ] Improvement - [ ] Documentation - [ ] Feature - [x] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: Related to matrixorigin#29515. The production optimization already landed in matrixorigin#29508. This PR repairs its regression coverage; it does not automatically close the issue or claim to explain the conflicting latest report. ## What this PR does / why we need it: The existing matrixorigin#29515 BVT ignores the entire EXPLAIN ANALYZE column, including the scan counters that distinguish the reported performance defect from a correct result. It also checks only COUNT, which cannot distinguish a stale safe binding from a new safe binding returning a different row. - Assert the target table scan's actual `inputBlocks=1 inputRows=8192` for direct, scalar, ABS, scalar ABS, singleton derived and derived ABS forms, plus the existing second-parameter case. Match the scan's immediately following Analyze row, excluding timings and memory statistics. - Reuse each prepared generation through safe → unsafe → NULL → negative → different safe → recovered safe bindings. Check COUNT and MIN separately from scan pruning, so same-type value reuse is observable. - Replace the first generated value with `0.10`, preserving the existing 100,000-row sorted fixture. Incorrectly narrowing DOUBLE `0.104` to the stored DECIMAL value now causes a result failure. - Keep existing 2^53 collision, nonconstant derived, CAST and malformed-text/warning controls. Do not freeze the unsafe fallback's scan volume. - Keep only result headers for regex-owned ANALYZE statements; the SQL regex checks actual output. Remove redundant volatile plan bodies. No new fixture, process, cache or production path. Implementation: 0 changed production lines. Tests: SQL +78 net lines, result +58 net lines; documentation: 0. Every added execution observes a distinct expression/binding transition. The existing fixture and topology are reused. ### Validation Linux/amd64, Go 1.26.4, matching native dependencies; independently built main `d59cc02ef6` release binary with SIMD, isolated one-CN/TN/LOG instance and DISK-V2 storage. Tested with the published mo-tester runtime jar from main `309a0ca` (the compose CI Dockerfile clones mo-tester main). - Original BVT: 163/163. Updated BVT: **213/213 twice on the same instance**, zero failed/ignored/abnormal; explicit catalog teardown checked before replay and after the final run. Reported case duration remained 1 second on this machine, not a throughput benchmark. - Actual mo-tester ScriptParser/Java regex QA: all **21** assertions accept healthy plans and reject scan-only 13-block/100,000-row mutations, missing counters and a different table name, with aggregate rows preserved. - Public SQL negative control: a scalar predicate that scans all 13 blocks retains all COUNT/MIN results; exactly its three ANALYZE assertions fail (210 other statements pass). This proves the new oracle detects the performance defect independently of result correctness. - Current-main public reproduction on DECIMAL(18,2), with and without manual flush: all six safe shapes read one block. Thirty-six SQL PREPARE binding transitions passed. - Independent binary-protocol QA: **80** comparisons matched explicit column-side DOUBLE comparison across fractional/negative/NULL/nonfinite values, positive and negative 2^53 collisions, and recovery. - Existing focused planner test `TestPreparedNumericPredicateFiltering` passed at the unchanged production revision. `git diff --check` passed. No Go files changed, so new Go SCA, full UT or race reruns are not required for this SQL/result-only closure. ### Review and remaining limitation Design and final overall review used actual **gpt-6.1-sol / xhigh**, session `01a0fca6-9b52-7d22-ae0e-518dd340da19`. Final decision: **APPROVE, no unresolved material blocker**. Commit `bf1cd887bd41c4ed8f20a089928480117f89f4c7` preserves the reviewed complete-diff SHA-256 `dce4fe300e9653534e469f209c7329b97ee4b210f6dab6a7ad493835c4c1b0b5`. The latest issue comment reports 13-block scans on `d50a2d4`; that behavior was not reproduced on this independently built current main. The relevant planner/binding/frontend sources are identical between those revisions. A rendered DOUBLE cast alone does not establish full scanning: current plans can display it while their actual scan reads one block. The comment's binary/configuration/execution discrepancy remains unexplained. This test-only change introduces no production runtime overhead. Local validation does not claim remote CI passed; CI is not awaited or rerun.
…atrixorigin#29567) ## What type of PR is this? - [x] BUG - [x] Improvement - [x] Test and CI ## Which issue(s) this PR fixes: Fixes matrixorigin#29564 Fixes matrixorigin#29565 ## What this PR does / why we need it: Conservative workspace estimates can change after ordinary UPDATEs even when the business row count is unchanged. Exact equality invalidated otherwise reusable plans, repeatedly rebuilding logical plans, parameter metadata and compiled state. Join ordering independently estimated intermediate nodes before attaching their crossing predicates, causing Q9 to join unfiltered lineitem/orders too early. ### Generation-anchored reoptimization Ordinary positive finite observations may drift by less than a factor of two relative to the installed generation. Doubling or halving requires rebuilding; intermediate observations never slide the baseline. Prepared admission checks the installed specialization rather than its template. Existing schema, source-domain, diagnostic, snapshot and replacement-publication owners retain precedence. Executable Cartesian and RIGHT DEDUP plans, background plans, zero/unavailable observations and conservative-maximum transitions retain exact admission. Factor two is a reoptimization tradeoff, not a memory guarantee or a measured optimum. Existing allocation limits and spill paths remain responsible for runtime capacity. ### Safe reusable physical execution Broader reuse exposed a latent integer hash-iterator bug. `Find` leaves encoded keys and byte offsets populated, but `Insert` previously cleared them only on exit. A following insert could encode a new key at the previous lookup's offset. UPDATE hashbuild looks up old unique keys and retains its iterator across executions, so a conflicting prepared UPDATE could miss the matching DEDUP bucket and commit duplicate UNIQUE values. Move the existing integer insertion initialization from exit to entry, at the encoding owner. This covers both same-map read/write transitions and cached-owner reuse without disabling DML reuse, adding special admission exceptions, or introducing another cache/state machine. It adds no allocation or storage format; the bounded initialization work remains one pass per insertion. ### Predicate-aware join construction Transfer applicable crossing predicates from the existing pending pool before estimating each intermediate join. Preserve composite/non-equality/multi-table scope and genuine Cartesian multiplication. Remove the superseded singleton-admission proof rather than maintaining a parallel execution path. ## Validation - Internal iterator regression fails the original implementation: `Find(21) -> Insert(11) -> fresh Find(11)` misses the inserted key. The correction passes real read/write, owner-change, retained-capacity regrowth, integer-width/composite, NULL and `DetectDup` cases; full hashmap tests and focused hashbuild reuse/DEDUP/IGNORE/REPLACE/resource-cleanup tests pass. - Real COM_STMT regression preserves the 128-row witness. Integer, bigint, string and composite UNIQUE updates reject conflicts with MySQL 1062. Verify surviving successful work, commit and rollback, fresh-session contents, index-versus-primary scan results, NULL, self-update and recovery using the same prepared statement. - Each UNIQUE subtest owns bounded rollback cleanup and its assertion helpers. A diagnostic `FailNow` after a transactional write proves the next subscenario observes the restored value before any DDL; the injected run intentionally exits nonzero, while remaining scenarios pass. - Existing frontend generation-identity/rebuild, material-growth, schema/domain/error and ordinary-dispatch evidence remains valid. Cartesian/RIGHT DEDUP resource-boundary controls and Q9 optimizer tests at SF1/100/1000 retain their distinct oracles. Real join semantics and deterministic parallel-intersection fixture/cleanup checks are unchanged. - Final focused SQL validation, incremental affected-package vet/lint, formatting and diff checks are recorded locally. Platform: Go 1.26.4, macOS/arm64, repository CPU CGo artifacts. Required CI at the preceding exact head `96196b4a9c` passed; CI for this follow-up commit is pending, not awaited or claimed green. ## Performance and limits Preserve the admission and physical-plan reuse improvements; do not recompile every DML to hide stale state. Matched three-sample first-insert microbenchmarks retain identical bytes and allocation counts. These are bounded mechanism checks, not proof of full TPCC throughput, reused-transition elapsed time, TPCH 1T or multi-CN recovery; those remain unmeasured. Fresh workspace observation still scans existing metadata, and strict RIGHT DEDUP plans can still invalidate on small changes. ## QA rationale Successful updates and stable plan identities do not establish constraint integrity. Reuse validation must include the actual constraint rejection, the state left after a failed statement, committed contents, and recovery. The complementary internal regression checks read-to-write encoding transitions with an independent fresh lookup rather than asserting scratch flags or relying on coverage counts.
…gin#29573) ## What type of PR is this? - [x] BUG - [x] Improvement ## Which issue(s) this PR fixes: Fixes matrixorigin#29571. ## What this PR does / why we need it: General `MergeSortBatches` calls `UnionOne` for each row and column. Although each varlen append copies its payload, `UnionOne` unconditionally discarded the destination's disjoint-area proof. Canonical marshal then rebuilt offsets and emitted descriptors individually, causing 8,233 allocations for an 8,192-row inline VARCHAR vector instead of 6. The ordinary fixed-width path also paid for absent metadata sidecars and redundant prefix processing. Preserve the existing proof only for a flat destination with uniquely owned descriptor and area buffers, after the appended descriptor/payload is initialized. Repeated selections and constant sources remain safe because each append copies independently. Allocation failures, borrowed buffers and preexisting aliases remain conservative. Owned NULL appends clear reused descriptors; shared windows retain their previous behavior without modifying their owner's descriptors. The ordinary metadata branch skips absent-sidecar length publication and inactive preflight finalization. It retains allocation admission and the all-NULL prefix's parameter-identity reset. No SQL semantics, wire format, planner, concurrency or workflow changes. Add focused coverage for mixed/repeated selections, constants, NULL/reset, borrowed and aliased storage, shared-window ownership, constant destinations, allocation failure and metadata transitions. The real merge consumer test uses overlapping key ranges to force `UnionOne`, checks sorted values and marshal roundtrip, and reuses the output buffer. That consumer test fails on the unmodified base and passes with this fix. ### Validation Linux/amd64, Go 1.26.4; independent `make cgo` from base `d59cc02ef6` with verified native provenance. Both benchmark binaries use that same native build. Latest main `401d153` changes only unrelated prepared-DECIMAL SQL/result tests. Passed: - Complete `pkg/container/vector`, `pkg/container/batch` and `pkg/objectio/mergeutil` suites (`-count=1 -timeout=180s`) via the repository CGo wrapper. - Focused objectio chunked/legacy-column tests and hashbuild recovery projection/area/runtime-filter marshal tests, covering other consumers of the same proof. - Changed-package `golangci-lint` using the repository configuration: **0 issues**. - `gofmt`, `git diff --check`, and independent complete-diff ownership/error/compatibility review. ### Before/after performance Same host, sequential runs, `GOMAXPROCS=1`, `-benchtime=300ms -count=5`; medians. Component probes use the issue's benchmark with its bug-expected assertion removed. The end-to-end merge benchmark is included in this PR. | Operation | Base | Fixed | Improvement | |---|---:|---:|---:| | 8,192-row VARCHAR marshal after UnionOne | 402.0 us / 8,233 allocs | 6.60 us / 6 allocs | 98.4% less time | | 8,192-row VARCHAR copy + marshal | 767.4 us | 272.7 us | 64.5% less time | | 8,192-row BIGINT copy + marshal | 204.8 us | 161.5 us | 21.2% less time | | Overlapping MergeSortBatches + marshal, 16,384 rows | 3.381 ms / 16,485 allocs | 1.468 ms / 31 allocs | 56.6% less time | For the real merge consumer, allocated bytes fall from approximately 934 KB/op to 9.8 KB/op. Its keys are deliberately interleaved, avoiding the separate disjoint-concatenation optimization. Reproduce the delivered consumer benchmark after `make cgo`: ```bash GOMAXPROCS=1 .agents/skills/mo-dev/scripts/mo-cgo-test \ -run '^$' -bench '^BenchmarkMergeSortBatchesOverlappingVarlenaMarshal$' \ -benchmem -benchtime=300ms -count=5 ./pkg/objectio/mergeutil ``` Limitations: an isolated, unchanged BIGINT marshal control measured approximately 0.25 us slower in an alternating recheck; its allocation count stays 6 and the combined copy+marshal workload still improves by 21%. Its cause is not assigned, and this PR does not claim every microbenchmark improves. These results establish the vector regression and improvement in its actual merge consumer; they do not quantify whole SQL/BigData wall time or promise a return to 4.2's total runtime. No cluster or full BigData acceptance run was launched. Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…9532) ## What type of PR is this? - [ ] API-change - [x] BUG - [x] Improvement - [x] Documentation - [ ] Feature - [x] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: Fixes matrixorigin#29399 on `main`; forward-ports the remaining fixes from matrixorigin#29431 (`4.2-dev`). ## What this PR does / why we need it: Repeated restore can mistake CLONE provenance or a view definition for legacy CHECK metadata, lose ownership/current grants, or preserve revoked authorization in an existing connection. Partial restore also treated a role name as identity: renaming an owner blocked restore, while reusing its old name could transfer DROP authority to a different role. - Preserve structured CHECK recovery; recognize CLONE provenance and exclude view definitions from legacy table CHECK parsing. - Recreate objects under historical creator/owner IDs. Partial restore validates and pins required live principals within their physical catalog generation. Role renames preserve identity; name reuse cannot acquire ownership. Table restore locks and retains an existing database without requiring its historical creator/owner. - Resolve partial restore's historical logical source once, including existing view dependency planning, before ownership checks and destructive DDL. Ownership lookup intersects that source after existing external-table and unservable-view omissions. Execution reuses the same objects and classifications; FK/master/publication guards can still reject or omit candidates. This prevents an irrelevant deleted view principal from blocking ordinary data restoration, without exempting servable views. Explicit external-table restore still rejects; skipped single-view restore preserves the live view. - Full account restore rebuilds principal catalogs and can rewind their auto-increment IDs. **Partial restore across rebuilt catalogs rejects ordinary custom principals before DROP, even if IDs/names match or full restore legitimately reconstructed them.** Only reserved bootstrap identities with verified IDs/names and required direct admin grants can cross that boundary. Use a point in the current catalog generation or full restore; documented in `docs/ai-skills/backup-restore.md`. - Keep principal pins and current scoped grant capture/rebinding in the existing restore transaction. DROP USER/ROLE use private pessimistic RC transactions to honor those locks even under ordinary optimistic SI. Partial restore preserves current grants; full/cross-account catalog rebuilding retains its mapping. - Apply existing DATA BRANCH authorization to ordinary-user CLONE, preserving administrator historical/cross-account behavior, subscription errors and recreated publisher identity mapping. - Recheck authorization in existing and text/binary prepared connections across CNs. Within one privilege evaluation, deduplicate only identical SQL misses while retaining ordered errors, early hits and role/view scope. An enum-bounded stack buffer satisfies SCA preallocation without suppression or extra persistent authorization state. The upstream merge at `54af0d38290c66b61ac78a46aa07a363eb576aae` is preserved, including matrixorigin#29563 ESQL and matrixorigin#29568 FULLTEXT2/HNSW coordinator placement. No force push, workflow change or merge-policy bypass. ### Validation Go 1.26.4 / macOS ARM64, verified native provenance. Latest source-selection correction (`93405cdc0ea`): - Authenticated public SQL regression covers snapshot/PITR, independently deleted view creator/owner, ordinary-table data recovery, skipped live-view ID/definition/result preservation, and servable-view missing-principal rejection before DROP. Existing external rejection, retained/recreated DB and unchanged principal-generation controls remain. - Ownership tests cover excluded raw physical metadata, required servable views/sequences and fail-closed missing ownership. FK lookup tests verify shared pointers and scope without fresh enumeration. - After preserving the upstream merge: full frontend **26.886 s**, all six `TestIssue29399*` public tests **31.475 s**, and configured incremental golangci-lint **0 issues**. The byte-identical correction also passed all six tests under race detection before the merge (**69.920 s**). Go vet and repository molint completed with exit 0; existing molint recover diagnostics outside changed code are retained. No linter suppression/configuration weakening. Unchanged evidence from the whole-PR review is retained: planner/compiler and V406 maintenance tests; prepared/cache/active-role controls; normal BVT comparisons including metadata (ownership/CLONE 125/125, builtin full-then-partial 213/213, cache 37/37, revoked-role 39/39). The inherited target merge separately passed compiler/index owning packages and two-CN FULLTEXT2/HNSW rebuild integration. Its initial `TestConnMeasuresDelayedBufferedFlush` timing failure remains recorded; isolated repetitions and subsequent full frontend passed without weakening assertions. ### Performance and limitations Partial source selection adds bounded name maps, uses existing view planning once and avoids repeated partial source enumeration. It adds no persistent state, catalog format, principal query per object or new lock. Current-dependent runtime guards can conservatively omit a prevalidated candidate; this change does not claim exact prediction of every dynamic FK skip. Existing matched public microbenchmark: reader p50 median **1.617 → 1.333 ms (~17.6% lower)**; admin **243.5 → 244.3 µs**, with comparable controls. These local measurements do not recover the old stale-cache roughly 80 µs latency or establish production throughput. Authorization code is unchanged by the latest source-selection correction. The latest correction's design and final overall review use actual **gpt-6.1-sol / xhigh**, reconciling the whole-PR evidence with the changed paths. The source-selection correction's remote CI remains distinct from the target synchronization below; skipped jobs are not passes. ### Prior target synchronization and CI Prior synchronization head: `b190162ef097987a71e9252a49266caaee810451`; target `main`: `401d153150d9076f84aaaf2a32cdcf880b478395`. Normal merge parents are the preserved live `93405cdc0eafd1d2ce15a65d75d5afe2d531b36a` and the new target. This closes the observed strict-baseline BEHIND condition without rebasing, force-pushing or changing Ready/base state. Only matrixorigin#29570's two prepared-DECIMAL SQL/result files change relative to the first parent, byte-identically to the target; every other tracked file is unchanged. The existing source selection, ownership, current-grant transaction, authorization implementation and disclosed performance limits are preserved. Fresh monitor-local exact-merge binary build passes. The new upstream prepared-DECIMAL case passes strict JDBC metadata validation **213/213**, with named database cleanup zero (macOS ARM64 / Go 1.26.4 / Java 20). Independent read-only mechanical merge review approves and whitespace/worktree checks are clean. The prior `93405cdc` frontend/public/race proof above is external local unchanged-Go evidence, not a fresh whole-suite execution or current-head remote CI pass. No source-branch cache-certificate or internal-executor fix is blindly copied to main. XuPeng-SH approved `93405cdc` at 14:21 UTC after whole-PR follow-up; this is an actual submitted review, not the local mechanical approval. Prior-head Connector/J run [37015828472](https://github.com/matrixorigin/matrixone/actions/runs/37015828472) attempt 1 fails during ONNX Runtime download (HTTP 504), before the server/test. Exactly one failed-job rerun request was accepted. Attempt 2 / [110870725085](https://github.com/matrixorigin/matrixone/actions/runs/37015828472/job/110870725085) downloads the dependency and actually passes pool-reset tests for all four Connector/J versions at 14:07:46 UTC. That success belongs to `93405cdc`, not this new merge head. No healthy jobs or whole ALL CI run were rerun/cancelled and no build/workflow/tester/business workaround was added. That head's Connector/J [37023553819](https://github.com/matrixorigin/matrixone/actions/runs/37023553819) actually succeeds at 15:01:50 UTC; ALL CI remains unfinished when it is superseded by the next target merge. This is historical evidence, not a result for the current head. ### Prior target synchronization and validation Prior head: `380bbce20601196b9c9b09ea48089bb666985857`; target `main`: `5a09e1ffbf40a59f50c55fe89e6f0c23eba143a3`. After main merges matrixorigin#29567, strict protection marks the previous head BEHIND again. A normal merge preserves parents `b190162ef097987a71e9252a49266caaee810451` and the new target, without conflicts, manual overwrites, rebasing or force push. Its 15-file delta matches the upstream commit exactly, including modes and blob IDs. All existing PR changes from the approved implementation remain intact. Inherited prepared runtime-plan statistics admission does not bypass schema invalidation or text/binary EXECUTE reauthorization; shared restore source selection, ownership preflight and current-grant transactions are unchanged. Fresh monitor-local exact-head proof (macOS ARM64 / Go 1.26.4): - Full frontend/plan/hashmap **-race passes: 34.982 s / 36.546 s / 3.344 s**. - All six authenticated `TestIssue29399*` tests pass (**29.908 s**), including the eight optimistic-SI privilege-removal arms, snapshot/PITR ownership and source-selection boundaries, principal pins and invalid-grant rollback. Upstream reordered-join semantics and prepared workspace visibility tests pass (**13.031 s**), including integer/bigint/string/composite unique controls. The test-only atomic output `[no statements]` is not production coverage. - Exact binary build and version `380bbce206` pass. Configured incremental lint across all five upstream changed package roots reports **0 issues**. Worktree and whitespace checks are clean. Independent read-only merge/integration review approves; it checks inheritance and interaction, not the general upstream statistics policy. A fresh OPEN/Ready/head guard was awaited and read before the accepted normal push. Latest current-head CI is newly triggered and not complete; no BVT, full coverage, required-context or merge-gate pass is claimed. Prior-head successes are not substituted for current-head results. No workflow, tester or source cache-certificate/internal-executor framework is copied into main. Existing performance limitations and the separate constant-view issue matrixorigin#29572 remain unchanged. ### Current target synchronization and validation Current head: `ae4460f455bafa98a5beb301906adbd0abc522b6`; target `main`: `9078a8bd87af49ea06f52e09d64e7740122aea61` (matrixorigin#29573 vector UnionOne/marshal). Strict-baseline BEHIND is addressed by a normal merge with first parent `380bbce20601196b9c9b09ea48089bb666985857`, without conflict, manual overwrite, rebasing or force push. The four-file delta is byte/mode-identical to the upstream delta; the original 43-file PR patch remains identical against the new target. Restore source selection, ownership/current-grant transactions, principal-generation restriction, protocol/DAG and authorization implementation are preserved. Independent exact-merge local proof (macOS ARM64 / Go 1.26.4, verified inherited native inputs and absolute CGo paths): - Complete vector/batch/mergeutil **-race passes (6.392 s / 3.393 s / 3.712 s)**; complete frontend/plan **-race passes (34.427 s / 34.374 s)**. - All six authenticated `TestIssue29399*` plus authenticated fixture configuration pass **under race (65.232 s)**, including all eight optimistic-SI privilege-removal arms. Another 28 objectio/hashbuild marshal, recovery/area and runtime-filter consumer tests pass **under race (2.722 s / 2.297 s)**. - Configured incremental lint in the two upstream changed package roots: **0 issues**. Exact-head binary build/version, gofmt, whitespace and worktree checks pass. Native/packaged binary artifacts are preserved; the exact-head validation binary is separate. - Independent read-only integration/mechanical review **APPROVE**, covering ownership/aliasing, NULL reuse, errors, marshal and restore/clone consumers. This is local correctness evidence for this merge, not a GitHub approval, full BVT/BigData, Linux/GPU proof or performance comparison. Test-only atomic `[no statements]` is not production coverage. A fresh OPEN/Ready/head/body guard is awaited and read before the accepted normal push. The source-only SQL-task principal follow-up is not blindly copied into main's unchanged adapter/authorization path. Existing performance limitations and separate constant-view issue matrixorigin#29572 remain. Current exact-head [ALL CI run 37034311275](https://github.com/matrixorigin/matrixone/actions/runs/37034311275), attempt 1, completes **SUCCESS** at **2026-10-02 17:51:19 UTC**. Actual Ubuntu UT, arm64 SCA, shared build, UT Coverage, PROXY/PESSIMISTIC multi-CN BVT, Coverage and CI Required all succeed. The [CI Required log](https://github.com/matrixorigin/matrixone/actions/runs/37034311275/job/110959138144) confirms `full` scope, generation `37034311275-1`, all five dependency groups successful and `coverage_ready=true` / `coverage_status=passed`. The [Coverage log](https://github.com/matrixorigin/matrixone/actions/runs/37034311275/job/110952754024) selects UT and both complementary BVT profiles from that same generation and publishes the merged result. Current-head Connector/J succeeds; Iceberg is **SKIPPED**, not a functional test pass. No failed-job or whole-run rerun/cancellation is performed this round. The PR remains **OPEN / Ready / MERGEABLE / BLOCKED / APPROVED**. The retained approval was submitted on the earlier `93405cdc` implementation; it is not a new review of this target-merge head. All four discussion threads remain unresolved, with no new human findings. Completed CI Required and test jobs do not establish that every review/protection requirement has been satisfied; no Ready/base/approval/merge or protection policy is changed. --------- Co-authored-by: XuPeng-SH <xupeng3112@163.com>
Embedded race tests repeatedly pay the normal HAKeeper check interval after bootstrap, registration, and admission evidence has already committed. Wake the existing checker on those state changes, and let CN startup reuse its existing admission notification before refreshing authoritative inventory. Stable heartbeats, the periodic fallback, bootstrap deadline, command delivery, and readiness requirements retain their existing contracts. The earlier expression-owner traversal optimization remains in this PR: skip scalar payloads while preserving traversal of exported expression-bearing fields. Neither optimization removes test scenarios or weakens existing assertions. ### Design and cost - Heartbeats return a local, nonpersistent NoOP proposal-result hint when accepted registration/readiness, replica count, capability, or existing barrier evidence changes. CommandBatch bytes and RPC fields are unchanged; older versions keep the periodic fallback. - A capacity-one nonblocking store notification feeds the existing serial ticker. No additional worker, cached readiness, state machine, or configuration is introduced. Late notifications remain safe after shutdown. - CN startup treats admission notifications as hints and still checks revocation, upgrade failure/cancellation, and fresh authoritative inventory. A notification alone cannot authorize startup. - Stable heartbeat Tick, resource, address, replica term and ordinary receipt changes do not generate checker notifications. Failed checks retain the configured periodic cadence and wall-clock bootstrap budget. ### Measured result Paired complete `pkg/tests/issues/isolated` race runs on the same eight pinned CPUs with runtime data on NVMe. The baseline is phase3 commit `ac340a83` on main `5a09e1ff`; the comparison adds this startup change, so subsequent upstream vector improvements are not attributed to this patch. | Metric | Baseline | Startup change | Change | |---|---:|---:|---:| | Wall | 222.90 s | 167.63 s | -24.80% / **55.27 s saved** | | CPU | 199.17 s | 167.98 s | -15.66% | | Peak RSS | 3,850,928 KiB | 3,719,880 KiB | -3.40% | | Cumulative allocation | 9,924.69 MiB | 9,212.12 MiB | -7.18% | | Twelve service starts | 146.93 s | 99.32 s | -32.41% | All 45 test/subtest PASS identities are identical in both runs. The CN-notification-only experiment did not improve wall time; the complete committed-state chain is necessary for the measured result. The [latest completed racing job](https://github.com/matrixorigin/matrixone/actions/runs/37029518305/job/110912618974) spent 62m46s in UT and reported 41 service starts totaling 10.64 minutes. These measurements suggest **approximately 2–4 minutes of CI savings**, not an observed CI result or proof that the CI target has been achieved. Keep matrixorigin#29562 open until comparable CI evidence confirms the result. ### Validation and review - Full normal and race owning packages: hakeeper, logservice, cnservice. - Full race consumers: isolated issues, embed, SQL integration, upgrade, and multi-CN integration; covers bootstrap/repair, WAL recovery, non-leader restore, restart, dynamic CN expansion, startup query readiness, CDC lifecycle/takeover, and two-CN view description. - Three new concurrency tests each pass 100 race repetitions: startup/in-check notification coalescing, shutdown safety, and admission-triggered authoritative refresh. Existing lost-response/ack/BatchID/command-ID assertions are retained. - New pure RSM cases cover stable unrelated fields, replica 1-to-2 reporting, barrier evidence re-observation, generation rollback before activation, rejected stale generations, and one-time de-admission on protocol downgrade. - Changed-package vet, repository-pinned incremental golangci-lint (zero issues), formatting, diff checks, and `make err-check` pass. - Design and final independent read-only review: **gpt-6.1-sol / xhigh, PASS, zero blockers**. - Mergify's existing main update is preserved. On the updated head, isolated and multi-CN race integration and mergeutil pass. Vector race passes under UTC; under America/New_York, the unchanged main TestRowToString assertion incorrectly formats a negative current timezone offset as an hour instead of converting the epoch date (expected 1970-01-01 -4:00:00 vs actual 1969-12-31 19:00:00). This pre-existing timezone limitation is recorded separately from this change. No agent main merge/rebase is performed. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…rigin#29576) Remote expression send validation independently walked the same pipeline and queried the same destination for each enabled feature. This change analyzes expression requirements once and shares one raw destination version observation within a send, using the existing compiler capability owner. The ordered gate preserves feature-specific version floors and diagnostics, checks live cancellation and coordinator protocol at each enabled gate, and releases responses on all paths. Receiver analysis, string provenance, vector stream handshakes and grouping consumer checks retain their independent responsibilities. Each subsequent send analyzes and probes afresh; no cache, new state machine or wire change is introduced. Deletes 11 obsolete per-feature wrappers: production changes are 203 additions / 599 deletions (396 fewer lines); test changes are 183 additions / 6 deletions, retaining existing end-to-end cases and adding ordered-floor, cancellation, rollout, unknown/stale destination and response ownership coverage. Design: [approved exact revision](https://github.com/matrixorigin/matrixone/blob/ffee1890cc6c0d19e76621507437273638d21dde/docs/design/remote-expression-send-validation.md). Related to matrixorigin#29562; this does not close the CI optimization goal. Validation: - Full compile normal and race: identical 2,784 PASS identities; new unhappy-path race tests repeated 100 times. - Complete issues race against base 529ee09: 1,285 PASS in each run; four dynamic PITR timestamp names normalized, all remaining identities exact. - Full multicn race passed; go vet, pinned incremental golangci-lint (0 issues), make err-check and formatting checks passed. - Read-only design, implementation and evidence review using gpt-6.1-sol / xhigh: PASS, zero blockers. Matched local measurements use prebuilt race binaries, eight pinned CPUs and NVMe runtime storage. Whole issues wall time: 530.55 → 508.88 s (-21.66 s / -4.08%); CPU: 935.39 → 889.61 s (-4.89%); cumulative allocations: 43,441.55 → 42,361.31 MiB (-2.49%, about 1.05 GiB less). Peak RSS increased 3.65% (3,502,900 → 3,630,868 KiB). This single pair demonstrates neither peak-memory reduction nor minutes saved in CI. Real send microbenchmark medians under race: mixed expressions -36.34%, wide expressions -37.85%; expression-destination probes 3 → 1 and allocations 39 → 19. These measurements use a zero-latency fake version client and are not extrapolated to CI duration.
…atrixorigin#29574) ## Latest checkpoint: 0a73284 (2026-10-03) **Non-final checkpoint. Overall PR readiness remains blocked: full function race FAILED.** Read-only **gpt-6.1-sol / xhigh** approved committing/pushing this narrow stage under the explicit staging request, with the failed gate left open. - Fixture-only +9/-3: release simulated atomic unlock blockers after assertions, then reuse the existing generation/reset/join owner. No production or dependency edit. - 74 related lock tests pass normally; all 74 lock cases pass within the failed full-race run. The two benefited tests each pass 100 individual race repetitions. - Six complete normal package runs pass. Three alternating serial pairs yield medians: wall 11.750s → 9.869s (-16.0%), CPU 2.081s → 2.451s (+17.8%), peak RSS 209.7MiB → 199.9MiB (-4.7%). No CPU-saving or whole-CI claim. - Configured incremental golangci-lint: PASS, 0 issues. MO-specific lint has no incremental findings; its two unchanged unsafe diagnostics match prior evidence. - Full function race fails `TestRegexp2EvaluationAdmissionRejectsUnboundedRunnerGrowth` with an engine timeout on the admitted positive control. Verified clean baseline full race passes. Separate dependency probes prove serial/concurrent clock defects; they do not establish causality for the full-package failure. Complete compatible clock backport and final race validation remain pending; v1.11.5 upgrade withdrawn. Review/evidence summary is committed in `docs/design/20261003-expression-quality-consolidation.md`. Refs matrixorigin#29249; continuous tracking remains open. --- ## What type of PR is this? - [x] Improvement - [x] Test and CI - [x] Code Refactoring ## Which issue(s) this PR fixes: Refs matrixorigin#29249. ## What this PR does / why we need it: Consolidates expression execution around the existing production owners and removes retired implementations with their exclusive tests. - Share checked calendar/time arithmetic across six temporal ADD/SUB adapter pairs. Preserve selection, NULLs, warning policy, timezone and precision. - Reduce TIMESTAMPADD DATE's eight wrapper-dependent loops to four batch-dispatched loops (constant/dynamic unit × DATE/DATETIME output). Keep unit admission, one result conversion, and existing calendar/diagnostic owners. A tiny NULL helper is inlined; error handling is shared on the failure path. No parsed-unit cache or persistent state. Return failed DATE→DATETIME allocation before accessing backing memory. - Validate literal source metadata before allocation; consolidate constructor failure cleanup while preserving non-NULL domain/binary behavior. - Retire unused Iceberg remote scaffolding while retaining the active local coordinator, catalog/credential checks and ObjectIO lifetime. - Retire test-only prepared/auth wrappers and obsolete schema/index/flat-ring geometry helpers. Retained tests exercise active scoped authorization, logical schema admission, contiguous index prefixes and geometry owners. - Consolidate repeated fixtures while retaining literal results, result metadata, folded residual execution, allocation baselines and failure controls. Against main `9078a8bd87af49ea06f52e09d64e7740122aea61`: 27 files. Implementation +257/-1189, tests +1890/-3936, design documents +161/-0. Counts describe each increment; they do not establish correctness or imply an audit of every recently changed module. ### Current optimization validation Local Go 1.26.4/Linux amd64 with fresh native CGo inputs, GOMAXPROCS=2, GOMEMLIMIT=2GiB, build concurrency1; heavy checks run sequentially on the shared machine. - Full function package count1 PASS (14.982s), including 2,304 other temporal ADD/SUB differential scenarios and 2,496 TIMESTAMPADD DATE scenarios against the original baseline, plus literal expected-value and denied-growth account/vector controls. - Focused DATE race PASS (2.386s). Strengthened admission-before-warning test rerun normally and with race PASS (1.139s); a later invalid unit must prevent earlier overflow warnings and metadata writes. Four loop modes preserve exactly one warning per selected overflowing row with constant operands. - Actual binder/executor reuse PASS (11 batches, sizes0..2048, masks, NULLs and precision transitions). - Single-CN SQL frontend replay of the existing `func_datetime_timestampadd.test/.result` PASS: all56 statements, exact headers and rows, DATE/DATETIME/TIMESTAMP, NULL, overflow and WHERE controls. Uses the existing integration fixture and task-private temporary files; no new cluster fixture is shipped. This is the targeted SQL BVT, not a full multi-CN rerun. - Incremental owning-package golangci-lint v2.6.2 PASS, no findings. Repository-pinned molint exits0 with the same two pre-existing unsafe diagnostics in unchanged files as the prior successful CI; no new diagnostics. Formatting/diff checks PASS. - All six shipped lightweight benchmark cases execute successfully. ### Expression CPU performance Exact baseline, pre-optimization PR body and candidate run in the same binary on reused2048-row vectors; caller bound type reset each batch. Primary cases measured in both orders, five500ms samples each, unchanged allocations. Reverse-order medians (ns/batch): | Case | Original baseline | Before optimization | Candidate | Candidate vs baseline | | --- | ---: | ---: | ---: | ---: | | Fixed DAY | 36,146 | 35,263 | 34,389 | -4.86% | | Fixed SECOND | 32,650 | 33,613 | 32,762 | +0.34% | | Dynamic DAY/HOUR | 158,729 | 161,642 | 160,331 | +1.01% | First-order candidate deltas: DAY-4.79%, SECOND-0.87%, dynamic+0.88%. MICROSECOND, MONTH and dynamic DATE-only controls remain close to baseline (-0.87%..+0.32%). Scalar dates have no allocation increase. Partial masks/NULLs retain small differences (SECOND+1.08%, dynamic+1.27%). Seven-row SECOND remains299.5→322.3ns (+22.8ns/+7.61%) versus baseline, slightly faster than pre-optimization324.9ns. These limits are disclosed; the change does not claim every input is faster. Shared-machine microbenchmarks do not establish whole-query or TPCC speedup. ### Overall review and historical evidence CI at `f13510d9524284347ace0dd47884a43f58680262` passed: SCA, normal UT, build, coverage and multi-CN proxy/pessimistic BVT (run 37079464383). Dedicated Iceberg E2E was skipped. This follow-up only deletes redundant tests; new-head CI remains pending after push and is not awaited. Actual gpt-6.1-sol/xhigh design and final overall review evidence: Design session `01a0fed4-73e0-72f2-a61c-a6e33e7cc7f6`; final review session `01a0ff00-0445-78f1-9eca-d5843f425059` (APPROVE, no unresolved material blocker). Actual model and effort verified from both session contexts; final reviewed diff SHA-256 `f93173c32478b36d47ac9e920564d1846c8e057b5f9cf00181e5f5108e7bb070`. ### Final test-quality cleanup Remove three obsolete branch-oriented temporal tests (72 lines) whose nonzero/optional-error assertions allowed incorrect identity behavior. Exact calendar values and scaling overflow remain covered by `TestCalendarIntervalArithmetic`; UTC conversion rejection and accepted neighbors remain covered by `TestTimestampArithmeticUtcLowerBound`; upper overflow, NULL and row isolation remain covered through the real adapters by `TestTemporalMicrosecondBoundaryOverflowIsNull`. No runtime code changes or replacement fixtures. These three retained suites passed once on Go 1.26.4/macOS arm64 with the native CGo wrapper (`-v -count=1`, 0.762s); diff checks passed. Existing mutation evidence demonstrates the weak tests accepted a broken implementation while the canonical table rejected it; those unchanged tests required no repeat mutation or benchmark run. Follow-up design session `01a0ff65-e236-74e1-9b13-725a5ffcaddf` and final overall review session `01a0ff69-39f6-7ec2-8cf6-01a242b577a9`, both verified actual gpt-6.1-sol/xhigh. Final review PASS, no actionable findings; the prior P3 is closed. ### Staged coverage and fixture-lifetime audit Refs matrixorigin#29249 (ongoing tracking issue; remains open). This stage preserves the remote test retirement and changes four existing test files plus the existing expression design document; no production behavior, framework, heavyweight fixture or additional test is introduced. Stage delta: +174/-55. - Register existing process/vector/result/account cleanup immediately, including partially initialized raw constant constructors and unexpected non-nil executors on error. Payload probes release before the constructor under test; account Seal and Finalize both execute before teardown failure reporting. Body assertions remain before fallback cleanup. - Publish all 43 retired top-level test names with retained named scenarios or explicit retirement rationale, plus transformed subcases and forwarding-wrapper ownership, in `docs/design/20261003-expression-quality-consolidation.md` (revision 2). - Current local Go 1.26.4/Linux amd64: all 9 affected tests pass normally and with `-race -tags=matrixone_test -count=1`; complete owning packages pass (function 14.120s, colexec 0.171s). Heavy runs are serialized with GOMAXPROCS=2, GOMEMLIMIT=2GiB and `-p=1`. - Configured owning-package golangci-lint: 0 issues. Pinned molint exits 0 with seven existing unchanged-source diagnostics (two unsafe imports and five recover calls), matching prior successful CI. No new diagnostic; formatting and diff checks pass. - Existing public SQL replay, real executor reuse and expression microbenchmarks remain semantically valid because this stage changes test cleanup and documentation only. No same-condition suite-cost baseline was measured; line counts/package times do not establish CPU, memory or CI-time savings. Cluster setup, admission wait, service startup and scenario costs are reported separately in the design document. - Known follow-up boundaries remain explicit: outer zonemap scratch-cleanup mutation previously survived; credential/local coordinator tests are not remote-policy E2E proof; dedicated Iceberg E2E was skipped. These are not relabeled as passes. Stage design: actual gpt-6.1-sol/xhigh session `01a0ff9d-f447-7d91-9776-444b7e87a846` (ACCEPT). Final stage/overall reconciliation: actual gpt-6.1-sol/xhigh session `01a0ffb1-e7d3-73e2-85a8-5e68591d3792` (APPROVE, no unresolved material blocker). Reviewed stage diff SHA-256 `c9c38795143cbcfb1a20a9deb7712efb26aaa48b4b286692ddc1fc9fbc814ee4`. New-head CI is not awaited or claimed green. ### Follow-up: zonemap scratch cleanup oracle Strengthen the two existing round/scalar overflow cleanup suites with pre-fallback Go-heap ownership baselines and explicit cleared-slot assertions (+10 test lines, +8 documentation lines; no production changes or new fixture). `CurrNB()` alone observes native bytes and missed Go-heap operand ownership. - Frozen overlay removing the outer cleanup: old scalar/tuple probes pass (0.039s); strengthened round and signed/unsigned scalar tests fail on 32 live Go-heap bytes (0.040s). - Independent overlay frees vectors but retains pointers: heap baselines pass, nil-slot oracles reject stale scratch pointers (0.044s). Both mutants fail in test bodies before fallback cleanup. - Normal full colexec package PASS (0.150s); both existing suites race PASS (1.147s), count1, controlled native wrapper and serialized runs. Incremental golangci: 0 issues; pinned molint exits0, changed external test package reports no diagnostics. Formatting/diff checks pass. - Actual gpt-6.1-sol/xhigh review session `01a0ffbe-f5b8-7890-adcc-c93a54702beb`: APPROVE, no concrete blocker. Reviewed diff SHA-256 `27741a5b9941e2e5f01f6c06588b300334e82f577a28ab0b4c91ab580c8e31e0`. The prior surviving-mutation boundary is now closed for these exact two errors; no arbitrary leak coverage, test-cost or new-head CI claim. matrixorigin#29249 remains open. ### DATE_FORMAT fixture compression with measured owning-package savings Replace seven obsolete initializers and repeated execution blocks with one explicit literal table and one DateFormat-specific adapter. `TestFormat`'s four duplicate pairs merge into `TestDateFormat/all_tokens`; six named format subtests retain all 24 original date/result pairs byte for byte. Correctness workload falls from 600 cases / 4,915,200 rows to six cases / 24 rows. Existing per-row format, NULL, masked selection and zero-date oracles retain their behavior, with cleanup-only changes. Inputs and result wrappers now release before their process; native and Go-heap ownership are checked after case cleanup. Six existing benchmark names, format assignments, 100×8192 population and per-case `BenchMarkRun()`→`Run()` sequence remain. Benchmark iteration/timing policy is unchanged; no production edit or new heavy fixture. The exact map and measurement limits are in the existing expression design document. Three serial, alternating measurements per compiled binary on identical runtime/native/dependency inputs, excluding compilation; all six complete function package runs PASS: | Owning function package | Before compression (`c161ffc580`) | After | Median change | | --- | --- | --- | --- | | Wall seconds | 13.625 | 11.846 | −13.1% | | CPU seconds (user+system) | 4.431 | 2.618 | −40.9% | | Peak RSS MiB | 487.2 | 208.5 | −57.2% | Wall ranges: before13.573–13.697s, after11.825–11.850s. CPU ranges: before4.428–4.643s, after2.407–2.620s. RSS ranges: before478.9–491.4MiB, after200.4–215.7MiB. Runtime uses monotonic timing and terminated-child `wait4` CPU/RSS. Source-derived input/result/bool backing arrays fall121.875MiB→624bytes; this excludes vector storage/allocator overhead and is distinct from process RSS. These results establish owning-package savings on this shared machine, not whole-CI or production-query speedup. Exact final source focused normal selection PASS; three focused suites race PASS1.165s including six format scenarios and unchanged edge suites. Configured incremental golangci:0issues; pinned molint exits0 with only two unchanged unsafe-import diagnostics; formatting/diff checks PASS. Six complete package cost runs are the normal full-package validation. No new BVT is required because public/product behavior is unchanged; existing real SQL evidence remains valid. Benchmark sources are compiled and shapes reviewed; their existing enormous iteration loop was not executed. Actual gpt-6.1-sol/xhigh design session `01a0ffc9-3e70-7eb2-a11c-e7c0e7dd6e7b`: ACCEPT. Final review session `01a0ffdd-3e50-7a40-aafe-0dc8d39e1548`: APPROVE, no verified material blocker. Exact reviewed stage diff SHA-256 `46fe5e0e6e19abd6b732366c7705e55a9ba54dd6b30dfffe6596c611b9e1f16e`. New-head CI is not awaited or claimed green. Refs matrixorigin#29249; tracking issue stays open.
) The transaction data-trace collector has a separate control, storage and lifecycle path that allocates queues and workers even when collection is disabled. CN transaction-client construction captures the service before bootstrap registers it, while dynamic probes still run, leaving incomplete tracing and hot-path diagnostic work. Remove the collector and its entire caller closure: workers/queues, CSV/S3/MO exporters, filters, CN registration/close/directory ownership, probes, trace-only sequence/action clocks and events, executor/client flags, session consumers, internal DSN parameters, control command, generated configuration defaults, exclusive fixtures/tests and operational docs. Remove trace-only lock-conflict Result fields from their producer and RPC schema together, including the extra locked waiter-queue traversal; deadlock dependency tracking and semantic lock results retain their existing owners. Preserve ordinary spans, metrics, Explain and reader summaries, actual commit/rollback/closed callback phases, lock/snapshot semantics and WAL/recovery paths. Keep only exact catalog DDL/seeds/bootstrap registrations and inert `Txn.Trace` TOML / `disable_txn_trace` session declarations for stopped-version replacement and rollback. Existing trace rows, controls and historical files are neither deleted, rewritten nor replayed. No runtime facade, collector replacement, compatibility branch or mixed deployment is supported. Dynamic positional launch templates switch with the source commit; external older six-slot templates must also be replaced. Existing service TOML files still parse in either version. Source comments track final removal after the rollback window excludes collector-bearing versions and deployed TOMLs/client initialization omit the retired settings. Catalog/bootstrap/upgrade/planner declarations and exclusive tests retire together; removing declarations never implicitly deletes historical data. Actual old/new/old binary switching was not executed; catalog/data/config preservation and same-version stopped restart were validated. Related to matrixorigin#29249. Net reduction: 6,459 lines (implementation including generated production code 4,837; tests including mocks 1,159; docs 447; config 16). Latest closure design and final overall review: gpt-6.1-sol/xhigh. Staged implementation retains independent shutdown/admission/UUID safety oracles, consolidates overlapping fixtures and adds minimal real SQL/restart and dynamic-template coverage. Validation: - Owning normal packages: CN, frontend, compile, table scan/lockop, disttae/logtailreplay, incrservice, bootstrap, embed, CLI, taskservice, SQL export, txn client/options, executor, error causes and control commands passed. - Real embedded SQL verifies fresh 9-table/5-disabled-control bootstrap, historical data/enabled-control/file preservation, inert enabled configuration and session settings, unsupported retired command, and full stopped restart. - CN and transaction-client owning race tests, focused admission/close race stress (100 iterations), configured SCA on the affected closure (0 issues), formatting and exact catalog SQL/seed/init-order comparison passed. - Lock Result retirement: exact-toolchain regeneration control is byte-identical before schema edits; surviving wire tags remain unchanged. Full lockservice normal/race executions reported only two incorrect newly added timestamp assertions, with no other failures or race reports. Corrected real remote no-conflict/commit/abort controls pass normal/race; unchanged suite evidence is reused. Lockop passes and affected configured SCA reports 0 issues. Existing forwarding, cancellation/deadlock and table-definition timestamp codec cases remain covered. - The restart test exposed a background cron auto-increment nil capture. The identical preinsert failure was reproduced on clean production baseline 529ee09; it is a pre-existing startup-order defect, not a log-clean restart claim. The retirement SQL/data assertions pass; the baseline probe validates ordinary SQL across the same stopped restart. No end-to-end throughput improvement percentage is claimed; deletion proves removal of collector work.
## What type of PR is this? - [ ] API-change - [x] BUG - [ ] Improvement - [ ] Documentation - [ ] Feature - [ ] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: issue matrixorigin#26879 Incremental follow-up to merged matrixorigin#29428. ## What this PR does / why we need it: Prepared common-value expressions still lose type context across unresolved nested results or IFNULL lowering, and a plain NULL before a DECIMAL peer incorrectly inherits the later DECIMAL domain. - Propagate the combined domain of **all** fixed numeric peers into marker-only common-value children before overload selection, preserving every DECIMAL scale constraint and reusing the existing numeric-prefix and precision/overflow binder. Fixed-peer order must not turn an exact overflow into a rounded DOUBLE. - Jointly normalize runtime marker shapes across unresolved nested/sibling results before child overloads are rebuilt; do not promote one marker's roundable scale into another marker's fixed-scale constraint. Root operands keep the existing spelling-sensitive inference gate; a concrete SQL PREPARE string boundary protects the **whole** region before any child conversion. - Bind IFNULL's result operands in their original SQL order without including its CASE control expression. - Preserve NULL-first inference ordering and concrete string, FLOAT, and explicit CAST boundaries. - Keep executable parameter references and mark value-derived bindings as value-dependent, rather than caching a value-specific plan by source type. Regression coverage includes DECIMAL64/128/256, both SQL PREPARE and COM_STMT, nested/reversed/deep common-value calls, typed/untyped NULL controls, same-type value changes, invalid-text recovery, ABS-derived results, and explicit CHAR/DOUBLE boundaries. Original comparison results were checked against MySQL 8.0.45. The review follow-up also covers mixed DECIMAL(65,30)/DECIMAL(38,0) peers in both orders, nested versus direct-marker controls, 65-digit overflow (error 1690), the Decimal256 46/47-integral-digit boundary at scale 30, and recovery on the same prepared statement. These precision cases use the existing direct-marker behavior and MatrixOne's 76-digit bound as their oracle. The second review follow-up covers heterogeneous marker values (65 nines and `1e-30`) in nested COALESCE, IFNULL, deep nesting, and separate big/big versus tiny/tiny sibling subtrees. All use the exact DECIMAL(76,11) domain rather than rejecting or falling back to DOUBLE; 77-digit overflow and recovery remain covered in both protocols. The root-string review follow-up covers LEAST/GREATEST with direct, COALESCE, IFNULL, deep and sibling operands. SQL PREPARE root values such as `abc`, `!` and the empty string retain lexical child spellings (`0002`) and do not spuriously reject 77-digit strings. Same-type string→numeric-prefix→string rebinding is value-dependent; COM_STMT retains its existing numeric-prefix policy. The new public matrix contains 200 executions. ### Validation - Go 1.26.4 with verified worktree-local CGo artifacts and the repository test wrapper. - Complete `pkg/sql/plan` tests in normal mode; focused changed-code coverage: 202/229 executable lines (88.2%). - Public two-protocol regressions and existing issue matrixorigin#27088 regressions. - Six frontend DECIMAL/source-binding/runtime-cache regressions. - Independent public-protocol probe: 182/182 executions passed. - mo-tester-generated and oracle-reviewed results; normal comparison twice against the same owned standalone instance: 208/208 each, with teardown checks. - `go vet`, incremental repository lint, matching gofmt, and `git diff --check`. Latest verified `mo/main` (`fcd9686e0d`, including the subsequent prepared numeric/temporal, peer-pruning and remote-expression changes) was merged before the review update; the final build and tests were rerun on that baseline. Local CLAUDE documents, logs, and caches are not included in the PR. Validation note: the initial delivery's full-package atomic coverage hit the unchanged trailing-zero allocation assertion (`2` allocations versus a limit of `1`). The complete package passed in normal mode, and the focused changed-code coverage run passed; no allocation assertion was relaxed. Incremental lint passed; the earlier full-package lint report contained only formatting findings in two untouched baseline test files. --------- Co-authored-by: XuPeng-SH <xupeng3112@163.com>
…igin#29584) ## What type of PR is this? - [x] BUG - [x] Improvement ## Which issue(s) this PR fixes: Fixes matrixorigin#29579 ## What this PR does / why we need it: ### Root cause Sysbench 1.1 `de18a03` normalizes its Lua `sql.type.INT` binding to an `int64_t`/BIGINT buffer before the MySQL driver maps it to `MYSQL_TYPE_LONGLONG` ([client implementation](https://github.com/akopytov/sysbench/blob/de18a03/src/lua/internal/sysbench.sql.lua#L388-L400)). The nightly schema nevertheless has an `INT` primary key. The prepared comparison binder currently allows safe wider-integer narrowing only in SELECT. For UPDATE/DELETE, this produces `cast(id AS BIGINT) = ?`. Storage's PK-filter extraction requires a direct column operand; the cast prevents it and leaves a residual full scan, including RowID materialization. This explains the issue's expensive RowID construction without requiring a RowID allocator rewrite. On an unchanged main build at `fcd9686e0d`, a real binary-protocol INT64 parameter against a persisted 100,000-row INT-PK table reproduced: - SELECT: 1 input block / 1 input row. - UPDATE: 13 input blocks / 100,000 input rows, `cast(id AS BIGINT)`. ### Minimal fix and preserved contracts - Extend normalization only for a wider signed parameter compared directly with a signed integer column of at most 32 bits in a non-SELECT statement. This includes UPDATE/DELETE; SELECT's prior rules remain unchanged. No new unsigned, mixed-sign, 64-bit-target, string or floating-point coercion policy. - Require the existing current-value exactness/range proof before narrowing the parameter, never the column. Preserve the executable ParamRef rather than embedding its present value. - Retain a deep-copied complete cast in the existing per-EXECUTE diagnostic guards, even if a later rewrite removes the predicate. A valid guard allows plan/compile reuse; an out-of-range parameter rejects cache reuse and falls back to the original wide comparison without exposing a new narrowing error. - Preserve other consumers' `ValueDependent` state. A value-dependent fallback does not replace the good cached plan. Existing NULL/type-category handling and assignment casts remain unchanged. - No new cache, worker, global state, retry or waiting mechanism. ### Verification Native libraries and local mo-service were built from this worktree. Tests use an owned local service/port/data directory, not a shared regression cluster. - Full `pkg/sql/plan` package: PASS (7.781s). - All 91 frontend `TestPrepared*` top-level tests: PASS, including the new UPDATE/DELETE/multiple-predicate cache contracts. - Related compile prepared-reuse/reader-cleanup tests: PASS. - Tests prove safe → safe plan/compile identity reuse; both signed bounds; unsafe → safe recovery; independently checked multiple parameters and multiple narrow domains; preserved unrelated value dependencies; NULL and protocol-width category separation; and assignment overflow still raised by the actual assignment executor. - Real MySQL binary-protocol UPDATE/DELETE controls match the base build's affected rows and errors for signed limits, outside-domain values, NULL, unsigned input, range comparisons, assignment overflow and rollback. Repeated candidate checks pass; fixture row counts and cleanup are verified. - `go vet`, incremental golangci-lint, gofmt and whitespace checks: PASS. ### Performance evidence and limits The same persisted UPDATE now reports **1 input block / 1 input row**, with a direct PK comparison and block filter. A 30-update prepared sample decreased from about 250ms to 74ms; a separate indexed-update loop's median fell from about 8.1ms to 2.5ms per request in the initial controlled runs. An additional real **Sysbench 1.0.20**, 100,000 rows / 4 threads / 10s, measured about **261 TPS on base vs 1451 TPS patched**, both with zero errors/reconnects. This is a local corroborating client, not the nightly's exact 1.1 binary or its 1000W/concurrency profile. Sequential mutable-fixture measurements and shared-host timing are not a formal end-to-end benchmark. We rejected an initial uncached narrowing prototype after it added roughly 20% latency to small-table DML. The submitted version preserves cache identity. Its guarded cache-admission microbenchmark costs about 15.9µs vs 3.4µs for the same-width control (about 12.5µs incremental). Repeated small-table/control runs do not show the initial persistent 20% penalty; timings vary with host load. The guards are intentionally retained rather than weakening correctness to remove this bounded cost. These results establish the scan-path fix and tested semantic/cache contracts. They do **not** establish production-scale nightly TPS recovery or zero risk for every workload; full nightly validation and normal PR CI remain required before merge. Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
## What type of PR is this? - [x] BUG - [x] Improvement ## Which issue(s) this PR fixes: Fixes matrixorigin#29580. ## What this PR does / why we need it: Replace the successful DATETIME/TIMESTAMP-to-DECIMAL128 per-row format-and-parse conversion with exact arithmetic. The existing session time-zone conversion remains unchanged. This is a 16-line production change; no planner, storage format, cache, background task, or shared state changes. ## Root cause and semantic boundaries The reported BigData profile spends most CPU in `packedDatetimeDecimal128`: it formats `YYYYMMDDHHMMSS.ffffff` and parses the resulting string for every row. The packed coefficient fits Decimal128, so we can construct it exactly and use the existing `Scale` rounding implementation. Overflow and unsupported internal width/scale/value inputs fall back to the original parser, preserving its exact error. Decimal64 is deliberately **not** changed: its existing 18-significant-digit parse followed by scale conversion can round twice. Replacing that path would change results. ## Validation Based on main `fcd9686e0de5c71b6cd2913f922ad54e7df0a324`. - 31,570 deterministic comparisons against the original text parser, including zero, boundaries, fractional scales, invalid internal inputs, overflow and exact error strings. - Fixed expected-value rounding controls, including unchanged Decimal64 double rounding. - Existing NewCast DATETIME/TIMESTAMP/NULL/time-zone tests and complete `pkg/sql/plan/function`: pass (15.475s). - `go vet` and incremental golangci-lint: pass. - Fresh native build with provenance and candidate `mo-service`: pass. - Real isolated SQL service: two existing temporal Golden fixtures replayed twice (84 SQL statements), plus 18 boundary/NULL/zero-date/time-zone/mixed-IN/cleanup assertions. Results matched; the original BigData mixed-IN shape also matched the unchanged base service. All test-owned SQL resources and process cleaned up. Same-mode microbenchmark medians from 3 runs: | Scale | Original parser | Arithmetic | Speedup | |---|---:|---:|---:| | 0 | 414.80 ns | 92.27 ns | 4.50x | | 4 | 429.80 ns | 74.83 ns | 5.74x | | 6 | 387.60 ns | 45.86 ns | 8.45x | | 18 | 394.00 ns | 49.35 ns | 7.98x | Successful conversion allocations decrease from 40 B / 3 allocations to 0 B / 0 allocations. These are helper benchmarks, **not** a claim that the full 1B-row BigData query improves by the same factor. No full BigData rerun has been performed. ## Risk review The old parser is the differential oracle and remains the fallback; no changes to Decimal64, SQL coercion rules, TIMESTAMP time-zone conversion, NULL handling, or error ownership. This bounded internal equivalence optimization does not introduce concurrency/lifecycle ownership changes. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: XuPeng-SH <xupeng3112@163.com>
…#29586) MORPC idle GC can close a healthy connection while an accepted unary RPC is still waiting for its response. Unary requests do not lock the backend, and heartbeats intentionally do not refresh business activity. In matrixorigin#29582, the CN received `backend connection closed` after approximately 59 seconds, while the TN completed the five-minute checkpoint later. Recheck idle eligibility under the existing backend state and Future-map locks. A live ordinary unary Future prevents idle sealing. Seal request/stream admission before scheduling existing bounded asynchronous cleanup, so a previously selected handle cannot admit work after GC. Preserve completed/canceled/deadline cleanup, stream locks, draining ownership and pool accounting when cleanup slots are full. No new occupancy counter, worker, timer, retry path or public interface is introduced; the Future scan runs only for aged candidates. The CI follow-up repaired the prepared-authorization fixture's stale call to a retired table-only helper using the active object-scoped helper with `objectTypeTable`. The latest main now owns the identical fixture correction, so it is no longer an additional file in this PR's diff. All authorization assertions remain, and the retired production wrapper remains removed. ISCP lease validation now uses the executor callback's logical row count across batches. A valid constant vector can represent two rows with one physical string: the old reader incorrectly accepted that as one runner. Reject multiple logical rows before copying data, copy only the single runner, and reject NULL/empty runners. Existing transaction/result cleanup and system-account execution remain unchanged. Consolidate the reader tests into nine orthogonal cases while reducing that test file by two lines. Approved by: @XuPeng-SH
… searches (matrixorigin#29585) ## What type of PR is this? - [x] Improvement - [x] Test and CI ## Which issue(s) this PR fixes: Refs matrixorigin#29249. The broader audit and external regexp2 deadline work remain open. ## What this PR does / why we need it: Regexp cache entries own metadata, while each evaluation owns its executable regexp and runner storage. Drop that executable and release its budget lease instead of performing additional nil-input searches during matching and cleanup. Extend the existing resource/offset test to verify repeated scans, retained captures, cache isolation and idempotent budget release. Existing deadline, callback-error and admission-exhaustion tests cover failure cleanup. Move fatal assertions out of lock-test mutex sections so failed assertions cannot deadlock fixture teardown. Replace a wall-delay-based backlog test with observation of the real full-capacity admission branch, then verify exactly one request transfers after freeing a slot. Cancel and join its actor before fixture reset; reuse the existing context probe for both sequence consumers. The previous CI UT, coverage and SCA jobs failed because the declared base still referenced a deleted frontend test helper. This PR did not change frontend at that revision; clean-base and original-head compilation reproduce the same failure. Merge main's existing helper migration rather than restore an obsolete shim. Remove this PR's 140 appended checkpoint-history lines from the design document; preserve the already merged document. ### Validation At final tree `328a932f89`, Go 1.26.4 / macOS arm64, supported native build and CGo test wrapper, `matrixone_test` tag: - Full function package normal: PASS (12.451s). - Full function package race: PASS (17.995s). - Full frontend and plan packages normal: PASS (26.644s / 13.118s), including main's prepared-binding and DATETIME integration. - Six focused regexp cleanup/admission, backlog transfer and sequence-publication tests: PASS, 10 repetitions each. - Repository-configured incremental golangci-lint across function/frontend/plan: PASS. Custom molint exits successfully; its existing unsafe-import/recover reports are confined to unchanged sources. - Formatting/diff checks: PASS. Existing regexp benchmarks, run serially against the clean base and final tree (3 samples, 500ms/sample), show stable RE2 controls and no slowdown in affected compatibility paths. The grapheme case drops from 221 to 217 allocations and from 15,880 to 15,608 bytes/op; other measured allocation counts are unchanged. Timing samples are local descriptive evidence, not a statistically established throughput claim. All checks used the final source tree; committing the documentation cleanup did not change tested executable or test inputs. CI at the updated head is pending, not claimed green. The old size-label TLS failure is infrastructure, not addressed with product changes. ### Scope and limits No regexp dependency, SQL result/error, data format, runtime lock path, native build or workflow change. `regexp2` remains v1.10.0. This cleanup removes extra searches but does not promise immediate garbage collection, production-query throughput gains, or resolution of the separate dependency clock defect. Existing broader fixture teardown limitations remain outside this bounded change. Keep this PR's existing draft status. ### Additional Linux verification and independent review At `328a932f89`, Go 1.26.4 / Linux amd64, with worktree-owned native artifacts and the supported CGo wrapper: complete frontend/function packages pass (24.867s / 8.945s, `-count=1`). These are current-head local results. Earlier focused authorization tests pass all ten subcases; changing only the mock object type to view rejects all six granted-execution cases, demonstrating that the fixture checks the real authorization query boundary. Those focused/mutation results precede the merge; their relevant helper and production authorization sources are unchanged. Actual `gpt-6.1-sol / xhigh` design, full review, and final-head reconciliation completed. Final decision at `328a932f89`: **APPROVE; no material blockers**. Final review covers all four PR files, document cleanup, and interactions with merged prepared-DML/DATETIME dependencies. Final-head review session: `01a101b1-a680-79b2-b1a7-dda59f3e1da8`. Independent review is read-only source/evidence review; it does not claim independent reruns of the author-reported macOS checks. Final net change versus merged main: tests +59 lines, production -10 lines, total +49. No duplicate frontend repair commit is needed: main's helper migration already covers it. Remote CI remains unverified; no explicit rerun was requested. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…n#29583) ## What type of PR is this? - [x] BUG - [x] Improvement ## Which issue(s) this PR fixes: Fixes matrixorigin#29578 ## What this PR does / why we need it: The exact-membership Top-K path already pins its filter-column IOVector, but then calls `MustVectorToCached`, whose validated snapshot clones the entire column. Profiles in matrixorigin#29578 identify this avoidable copy in IVF PRE filtering. This change removes that copy for a closed, read-only operation without weakening the existing arbitrary-callback boundary. ### Change and safety boundary - Add an ObjectIO-owned membership descriptor admitting only the existing Cbitmap, Sorted64 and CRoaring concrete read-only implementations, optionally intersected with the existing PK search descriptor. - Bind filter columns to their pinned cache backing only inside ObjectIO. The external tombstone callback receives owned row offsets, never a cache vector alias. - Keep the old snapshot path for unsupported filters and arbitrary callbacks. Membership lifetime remains owned by the reader; borrowed vectors are freed before the IOVector is released. - Preserve NULL handling, sorted/unsorted searches, tombstone application, Top-K materialization, row-ID semantics, and existing malformed-secondary-filter fallback. No new worker, retry, global cache, format or configuration. ### Validation Using native artifacts rebuilt from this worktree: - Full `pkg/objectio`, `pkg/vm/engine/readutil`, and `pkg/vm/engine/tae/blockio` tests: PASS, normal and race. - Shared-cache concurrency test repeated 100 times under race: PASS. - Persisted-object consumer tests cover all three membership implementations, sorted/unsorted input, dense/sparse/empty/NULL cases, PK intersection, missing base-PK predicate, tombstone failure, cancellation, and exactly-once release. - Counterexample: an arbitrary legacy callback mutates its snapshot, but cannot corrupt the sealed cache or subsequent output. - `go vet`, incremental golangci-lint, gofmt and diff whitespace checks: PASS. An 8192-row compound-key binding + membership microbenchmark (three repetitions): | Path | ns/op | B/op | allocs/op | | --- | ---: | ---: | ---: | | Snapshot | 78,014–79,936 | 411,520 | 4 | | Scoped | 48,498–48,966 | 10,112 | 3 | This reduces allocation bytes by about 97.5% and this microbenchmark's time by about 39%. It is **not** a claim of equivalent end-to-end IVF QPS recovery: object layout, merge state, membership probing and vector reads also contribute to the incident. A full nightly/TKE run was not executed for this PR. --------- Co-authored-by: XuPeng-SH <xupeng3112@163.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…matrixorigin#29587) ## What type of PR is this? - [ ] API-change - [ ] BUG - [ ] Improvement - [ ] Documentation - [ ] Feature - [x] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: Regression coverage for matrixorigin#29579, following the merged fix matrixorigin#29584. No production behavior change or additional issue-closing claim. ## What this PR does / why we need it: Correct prepared DML results alone did not detect the signed-INT PK pruning regression: widening the column preserves results but scans the persisted table. Consolidate duplicated tests at their responsibility owners and add the missing logical read-work oracle without large datasets or timing thresholds. | Owner | Distinguishing regression contract | | --- | --- | | Planner | Full signed/unsigned, boundary, approximate-domain and explicit-cast matrix; DELETE keeps native/fallback construction controls. Shared guard capture owns domain probing and recovery. | | Frontend | Each key ordinal observes fresh values. Positive overflow remains checked per position; one canonical UPDATE also checks negative INT32 overflow at cached admission. Safe→unsafe→safe rejects narrow compile reuse, prevents fallback admission, preserves the live cache and recovers the original plan/compile. NULL and source-width category controls run once. Independent assignment cast remains covered. | | Public protocol | One prepared statement for the indexed text matrix and one for heap predicates, retaining warning/result checks. Binary UPDATE/DELETE reuse verifies complete row state, overflow/NULL behavior, assignment error and subsequent recovery. | | Persisted reader | Three rows flushed into three objects/blocks. Literal and binary BIGINT-marker point predicates read one row/block; explicit column CAST reads three while producing the same write. Both UPDATE and DELETE are measured. | The persisted scenario reuses the existing single-CN fixture. Table-local merge pause is acknowledged, logtail visibility is awaited, and physical inventory is asserted before measuring. Writes roll back to preserve the layout. Two passes recreate the same table and verify teardown; failure cleanup uses fresh bounded contexts. Existing 64-bit identity, SELECT, decimal, temporal, arithmetic and diagnostic contracts stay in place. No new test framework or production accessor is introduced. ### Validation Go 1.26.4, Darwin arm64, verified CPU native provenance; `GOMAXPROCS=2`, `GOMEMLIMIT=2GiB`. Current frontend follow-up: - All three `TestPreparedSignedNarrowingDMLCacheGuard` subcases passed (2.385s package). - Fault sensitivity: a reviewer-only overlay falsely authorizing negative INT32 overflow fails at the new negative cached-compile rejection (1.951s, expected failure). Without the overlay, the same test passes. This demonstrates a regression detector, not a present production defect. The overlay is not committed. - Whole normal frontend package with `-vet=all` passed; incremental frontend golangci-lint, formatting and diff checks passed. - Follow-up design and final overall review used verified `gpt-6.1-sol / xhigh` sessions; final verdict: no unresolved material blocker, with the negative-overflow review finding closed. Semantically unchanged validation from the initial consolidation: - Whole normal planner and SQLintegration packages, and all subcases of `TestIssue27443BinaryPreparedDMLAndAggregate`, passed. - Focused `-race` persisted scenario passed, including both fixture passes. - `-vet=all` for all four owning packages passed. - Temporarily disabling signed-narrowing eligibility restored column widening: public results still passed, while the new work budget failed with three input rows/blocks instead of one. Both failure passes cleaned up successfully. The mutation was removed. The integration scenario body was about 0.79s; with the initial UPDATE work controls and two persisted passes it was about 3.98s. This is a small fixed fixture cost, not a product-performance improvement claim. The negative-overflow follow-up adds only two bind/execute steps in an existing fixture. There are no sleeps, retry-until-pass loops or benchmark-sized datasets. ### Scope and limitations Prepared `EXPLAIN ANALYZE` does not reuse the runtime cache. Its input counters prove fresh persisted-reader selectivity; ordinary binary results and frontend cache identity cover reuse separately. This PR does not claim direct work-counter proof for cached `Compile.Reset`, physical I/O/cache misses, intra-object block pruning, or mixed-version upgrades. No production, data-format or runtime-performance change. CI is pending and was not waited for. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…rigin#29589) ## What type of PR is this? - [x] Improvement - [x] Test and CI - [x] Code Refactoring ## Which issue(s) this PR fixes: Related to matrixorigin#29562. CI wall-time savings remain to be measured at the checked PR SHA. ## What this PR does / why we need it: Embedded race tests repeatedly exercise real bootstrap, restore, metadata classification and stream/object writers. Reduce their common owner costs while retaining the original serial runner, GC policy, test selection and lifecycle boundaries. Bootstrap emits one privilege INSERT per fixed role; restore relies on the existing CLONE schema/data owner and drops dead manual-create fallbacks. State-only HAKeeper assertions omit unused cluster snapshots. Object-name and restore-annotation classifiers preserve the old RE2 languages with stateless scans. Branch defaults use the existing CPU budget; event, logtail segment and object entry buffers allocate for actual payloads. Expression traversal reuses the struct field count without a cache. The prepared EXPLAIN test also uses the live authorization SQL owner after main removed its old forwarding helper; SQL and authorization assertions are unchanged. Validation before the subsequent test-only consolidation: both matched current-base/head canonical stages passed, with the same 651 test pass/skip and package terminal identities. All ten modified owner packages passed vet and incremental configured lint; make err-check passed. Full frontend race passed after the final annotation refinement; full owning/dependent normal/race evidence and cancellation/error/retry/cleanup, typed-nil/detached-state, byte-roundtrip and independent classifier oracles cover the changed contracts. Privilege catalog checks add coverage inside an existing real cluster fixture. Matched ten-package serial race stage on local base 6249e48: | Metric | Base | Optimized | Observed change | | --- | ---: | ---: | ---: | | Wall time | 1,463.66s | 1,300.52s | −163.14s (2m43s) | | Child CPU time | 1,037.29s | 947.30s | −8.68% | | RSS residency integral | 2.383e9 KiB·s | 2.164e9 KiB·s | −9.17% | | Mean sampled RSS (integral / wall) | 1,627,988KiB | 1,664,133KiB | +2.22% | | Sampled combined peak RSS | 3,851,908KiB | 3,884,692KiB | +0.85% | | Cgroup memory peak, including charged file cache | 4,308,996,096B | 4,619,218,944B | +7.20% | Go1.26.4, race/short/matrixone_test, UTC, GOMAXPROCS=8, affinity 0–15, CPU quota 800%, memory limit 16GiB, default GC, isolated HDD temp storage. Precompiled binaries used the byte-verified original serial runner and unchanged test selection. No competing owned compilation/test; foreign jobs were observed in both runs. These are local observations, not precise quiet measurements or confirmed CI savings; no peak-memory reduction is claimed. Older fcd9686 profiles are excluded from this gain calculation. Both local whole-issues attempts exhausted the selected 20m diagnostic budget (base 121/144 and head 128/144 top-level PASS); incomplete profiles are excluded from performance claims. The same 23 complete remaining top-level tests passed on both binaries. Prefix plus tail covers the identical 144 top-level tests and all their subtests; this is not a single-process whole-package PASS or a CI timeout change. Additional resource checks retain mixed outcomes: - Original-runner complete embed, reverse HEAD→BASE, both passed with 248 identical terminal records and no foreign builds observed: head wall+4.45%, CPU+1.85%, RSS-time+10.16%, peakRSS+2.83%. Anonymous memory at the peak rose about86MiB, while charged file cache fell; this cannot be attributed solely to file cache. - Isolated complete UPDATE peak-owner test, same binaries with profiles: both passed, head CPU+11.40%/wall+13.90%, peakRSS essentially unchanged. Initial bootstrap allocation still fell41.62→24.52MiB and privilege INSERTs35→2; other work also varied. The exact timing difference is unresolved, so this adverse sample is preserved. - Opposite-order BASE→HEAD complete embed attribution run, same binaries with CPU/heap profiles and GC trace: both passed with 247 identical test terminals, no foreign builds observed; head CPU−0.97%, RSS-time−1.06%, peakRSS−4.07%, total allocation 6691.19→6274.43MiB(−6.23%). This direct-binary diagnostic does not use the original runner and is not pooled with the preceding samples. It shows owner allocation reductions; outcomes vary across fixture history, order and profiling configurations, and the timing cause remains unresolved. It does not prove that every scenario avoids regression. All supplemental scopes recorded zero OOM or memory-limit events and remained below about3.3GB charged memory under the existing 16GiB limit. The engineering conclusion is lower observed total CPU/resident occupancy for the ten-package stage, with mixed local boundaries and adequate measured capacity; the stronger quiet/all-peaks resource gate is not claimed passed. Actual CI gains and the original issue's 10min target remain unconfirmed. Tradeoffs: reused small logtail segments pay about 19ns for the capacity guard in the deterministic normal warm benchmark. Event overflow above two args remains supported but the three-arg normal benchmark is about 67% slower with an extra allocation; current production call sites fit the inline budget. Wide 300-column/four-block object writes measured about0.95% slower under race with lower allocation bytes. No admission/scheduling/GC experiment is delivered. One clean-main MinIO setup timeout was preserved as invalid measurement; focused/full/original-runner controls passed without a retry/skip/deadline change. Test/document consolidation: shared event growth/cleanup coverage across log/discard paths; one explicit cluster-cleanup matrix preserves system-only behavior and both ordered DELETE failures; object writer cases share MemoryFS/mpool with distinct paths and scoped cleanup. Decoded column count excludes vacuous value checks. The 800-case RE2 differential matrix and fuzz cases remain; benchmark timing no longer contains a self-derived correctness oracle. Six design documents become one owner-contract document. Net diff falls from 1,350 to 1,191 lines (production +27, tests +1,046, documentation +118). Final affected normal/race tests and scoped vet/lint/err-check are rerun after consolidation. Existing stage/profile figures above predate this consolidation; no additional CI savings are attributed to it.
…9590) - Accept MySQL `utf32` as a DDL compatibility charset spelling. - Accept `utf32_general_ci` and `utf32_bin` and normalize them to MatrixOne's existing UTF-8 text identities. - Keep the compatibility contract explicit: MatrixOne does not claim native UTF-32 storage or collation semantics, and `SHOW CREATE TABLE` does not emit an unsupported `utf32` identity. - Add planner regressions for the JeecgBoot declaration, table defaults, binary collation, and normalized output. Tests: - `GOWORK=off ./.agents/skills/mo-dev/scripts/mo-cgo-test -count=1 -timeout=300s ./pkg/sql/plan` Approved by: @XuPeng-SH
… work (matrixorigin#29591) ## Purpose Refs matrixorigin#29249. Continuation after merged matrixorigin#29585, rebased onto main `7ff3eac2e8`. - Give the existing function fixture one rollback/release owner, migrate terminal and borrowed consumers to their actual lifetimes, and remove the duplicate evaluator type and test-only production declarations. - Share 23 identical fixed-type comparison loops while retaining float64 epsilon/NaN behavior, existing vector wrappers, NULL ordering, diagnostics and variable-length comparison paths. - Preserve real boundary oracles while reducing Sleep wall time through virtual time, payload admission map rebuilding, XML budget setup, and LIKE budget/NTT test work. - Keep literal result, metadata, raw geometry, failure, account and cleanup assertions. Extend the existing reuse fixture rather than adding a type-by-behavior matrix. Go code: net **312 lines removed**, counting the fixture move as movement rather than reduction. Design/coverage documentation: 107 added lines, net whole change 205 lines removed. ## Evidence - Actual Codex CLI `gpt-6.1-sol`, `xhigh`, read-only, multi-agent disabled: design and implementation APPROVE. Rebase range-diff shows both reviewed patches identical. - Full owning-package normal and race tests, incremental configured lint and go vet: rebased HEAD `9f0fb65aaf`: normal **7.024s**, race **12.782s**, lint/vet exit 0; all 221 Go source hashes unchanged. - Old-source real Abs/Cast probes independently establish exact float32, tolerant float64/paired NaN, NULL polarity and first/last-row diagnostic contracts. - Comparator equality and premature-NULL-skip mutations fail the selected real-entry test. Private constructor/input/result/helper release mutations fail exact allocation baselines; Sleep units, payload capacity, XML work spending and LIKE dispatch/budget mutations are also rejected. - Production-only compilation passed the ownership checkpoint; no native sources changed. No SQL behavior is introduced, so BVT was not run for this test-harness closure. Matched three-pair serial package measurements before comparator deduplication: wall **9.079s -> 6.281s**, CPU **2.397s -> 2.132s**, peak RSS **168544 -> 170472 KiB**. Compilation excluded. These local observations establish no CI-wide time saving or memory reduction. ## Remaining limits This is a draft continuation, not closure of matrixorigin#29249 or the 342-file audit. Formal regexp2 v1.10 has a deterministic stopped-clock/stale-deadline failure, consistent with a previously observed race failure; a private fix/probe exists but is unpublished and not silently substituted here. A subsequent green run does not erase that defect. Broader error-oracle cleanup, benchmark iteration accounting and the memory objective remain open. Detailed responsibility and coverage map: `docs/design/20261003-function-test-fixture-ownership.md`. ## Clock gate follow-up Independent dependency defect is tracked in matrixorigin#29592. The formal regexp2 dependency remains unchanged. Whole-package race failed again at `regexp_icu_test.go:370`; the recorded earlier green run does not erase it. Overall delivery remains blocked by that gate. A benchmark-contract follow-up is locally committed as `255303da69` but **not pushed to this PR head**. Full normal tests plus all 176 benchmarks at 1x, the four affected UTs plus all 176 benchmarks under race at 2x, actual B.N+1 entry-count probes, post-admission Cast input mutations, and incremental lint/vet are complete. Independent `gpt-6.1-sol/xhigh` review accepted the local implementation/evidence closure and explicitly withheld overall delivery approval. The full-package race failure is preserved; it was not retried as an acceptance check.
…30092) ## What type of PR is this? - [x] BUG - [x] Improvement - [x] Test and CI ## Which issue(s) this PR fixes: Fixes matrixorigin#30090. ## What this PR does / why we need it: A fresh pessimistic RC `CREATE OR REPLACE VIEW` previously took the database lock before nested DROP lifecycle admission. That manufactured retained ownership and caused FastFail 20706 behind a SNAPSHOT gate instead of waiting. The replacement now enters the existing C/G/D admission before taking the ordinary database lock. Real retained-lock transactions still FastFail; nested DROP retains domain/identity revalidation and definition-change fences. The replacement admission also preserves the existing no-default-database error: constant views without USE return MySQL 1046 before admission or engine/lock effects. The regression was reproduced on the previous PR implementation, where ordinary CREATE returned 1046 but OR REPLACE returned an internal lifecycle error. Scope: 16 production lines, 215 test lines, no standalone documentation changes. Deleted the PR's redundant six-port test: the existing test already covers nine rendered listener addresses, allocator range and uniqueness. The integration test registers holder rollback immediately after BeginTx so assertion failures cannot bypass cleanup. No lock-service policy, retry, timeout, catalog format, or new admission state is introduced. ## Validation Rebased without conflicts onto main `8aaa8ae9e6`; published head `76c1ab96c0`. Local Go 1.27.1 / darwin-arm64; repository CGo wrapper and freshly built source-matched native artifacts. ONNX cache checksum matches the repository's required artifact. - Complete compile package: normal PASS 10.704s; race PASS 12.754s. New typed NoDB check forbids engine Database, Lock, and transaction Snapshot side effects. - Actual two-CN public regression: NoDB error/control, gate release, cancellation/server transaction drain before releasing the holder, unchanged old view on cancel, subsequent replacement, binary prepared-reader rebind, and retained-lock FastFail. Normal and race PASS; three same-process race repetitions PASS (30s budget derived from first 7.73s case). - Latest-main logtail read-barrier/reconnect dependency: six focused cases plus boundary subcases, normal PASS 1.407s and race PASS 2.123s. - Retained listener-range/uniqueness test PASS; removed test assertions are subsumed by that stronger existing oracle. - Final changed-package gofmt, go vet, configured molint and golangci-lint v2.14.0: PASS; zero new golangci diagnostics and git diff --check clean. - Whole issues package: normal PASS 265.380s and race PASS 644.282s; both modes have no failures or skips. - Required gpt-6.1-sol/xhigh amended design and final overall review: PASS, with no unresolved material blocker or validation requirement. Actual model/effort recorded from session 01a13720-7c29-77a2-a3ec-8d83274b3035. Prior-head CI and author Shanghai50 evidence are historical, not new-head passes. New remote CI is not claimed passed before it executes. ## Performance and safety Additional work is bounded to pessimistic RC replacement DDL, including a registry/frontier admission pass. Plain CREATE VIEW and non-RC routing and SELECT/DML hot paths are unchanged. Replacement DDL is not claimed to have zero overhead or a measured throughput improvement. Locks stay transaction-owned through existing commit/rollback/cancellation. Tests observe a real queued waiter and cancelled server ownership disappearance, rather than using sleeps to establish phases. No worker, cache, queue, retry loop or unbounded retained state is added. Full TKE nightly acceptance is not claimed. This is separate from temporary DELETE (matrixorigin#30089/matrixorigin#30091) and prepared pruning (matrixorigin#29661). --------- Co-authored-by: XuPeng-SH <xupeng3112@163.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
## What this PR does / why we need it: NiFi/MySQL writers can emit an `ON DUPLICATE KEY UPDATE` assignment for the primary key even when the value is unchanged. For direct literals and prepared parameters, this change casts the candidate to the target key type, checks it against the primary key on the resolved conflicting row, and retains the stored key when they match. A changed or NULL key still fails. The existing `VALUES(pk)` path is unchanged. The private guard keeps the stored key's exact type and non-null result metadata, avoiding spurious action-row validation and CHAR-to-VARCHAR coercion. ## Validation - Rebased and validated against `main` at `64b7a438`. - `make build-with-prebuilt-native BIN_NAME=mo-service-issue29850-64b` passed. - CGo tests passed for `./pkg/sql/plan` and `./pkg/sql/plan/function`. - `TestIssue29850PreparedPrimaryKeyUpsertIdentityGuard` passed. - Planner tests cover signed numeric primary-key literals (`-1`, `+1`, and `-(1)`), while `-(1+0)` remains rejected; the issue integration test executes a negative-key no-op upsert and verifies that the payload updates. - Issue BVT passed 60/60 on the rebuilt service using an isolated endpoint on port 39870. - A MySQL protocol probe covered literal and prepared assignments, cast-equal values, changed/NULL rejection, rollback, and prepared-statement reuse. - Independent final review of the exact PR tree passed. - Local prepared-statement timing on the PR tree used alternating execution order. For 100 pairs × 2,000 executions per arm, the primary-key-only no-op reference (`id=VALUES(id)`) had a median of 206 µs/op; the newly supported direct-parameter guard (`id=?`, same key value) had a median of 230 µs/op (paired median ratio 1.100; moving-block bootstrap 95% interval 1.037–1.142). Per-pair timings remain noisy, while both execution orders show similar median ratios (~1.09–1.10). This compares a new direct-assignment path with the existing `VALUES(pk)` path, since direct primary-key assignments were rejected before this PR; it is not a regression on an existing accepted form. For payload-updating statements, 20 pairs × 500 executions per arm had a median ratio of 1.016 (95% bootstrap interval 0.905–1.069), so this sample does not show a reliable payload-update slowdown. The `VALUES(pk)` implementation remains unchanged. The full repository BVT, multi-CN validation, and a real NiFi end-to-end run were not performed. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…ixorigin#30138) ## What type of PR is this? - [x] BUG - [x] Test and CI ## Which issue(s) this PR fixes: Closes matrixorigin#30094 ## What this PR does / why we need it: Follow-up to matrixorigin#30095 for the same external-table NULLIF rebind issue. matrixorigin#30022 changed InitNullMap to skip non-NULLIF assignments and continue collecting later NULLIF rules. After synchronizing main, two table rows in matrixorigin#30095 still expected the old stop-at-first-unsupported behavior, causing four external/LOAD subcases to fail in UT and coverage, including matrixorigin#29963 CI. Update the explicit expected maps and case names to the current contract. Keep all repeated-binding assertions, add the leading non-NULLIF boundary, and remove the obsolete early return that exempted these cases from JSON round-trip validation. Every parsed external/LOAD case now validates persisted rules after twenty rebinds. The existing real BuildPlan regression remains intact. Only one test file changes (+3/-8). No production logic, runtime overhead, SQL behavior, CI gate, or dependency changes. ## Validation - Clean latest main f02952f: reproduced the exact four CI assertion failures locally; the six other parsed-AST controls and real BuildPlan control passed. - Fixed source: full owning planner normal suite PASS (20.350s); focused race PASS (2.407s). All twelve external/LOAD subcases and the real BuildPlan control passed under normal and focused race, count=1. - Incremental gofmt, go vet, configured molint and golangci-lint on pkg/sql/plan, plus git diff --check: PASS, zero new golangci diagnostics. - Local Go 1.27.1 / darwin-arm64 using the repository CGo wrapper. Source-matched native build and provenance recording passed with the prebuilt ONNX dependency passed through the environment. The reproducer and verification are local executions. New PR CI is not claimed passed before it completes. This PR does not change merge protections or investigate how matrixorigin#30095 was merged before its updated CI completed.
…gin#29963) Accept `ALTER TABLE ... ADD [COLUMN] (column_definition, ...)` and lower each definition to the existing ordered `AlterAddCol` path. This supports the Debezium JDBC statement reported in matrixorigin#29851 while preserving the existing scalar ALTER productions. Parenthesized indexes and constraints are outside this change. This PR integrates official main commit `dd5bdb7a896398a161a93734a8073fe1ccc031b3` to resolve the parser conflict while retaining its parser changes. Relative to that base, the PR changes only the MySQL grammar, generated parser, and focused parser tests. The generated parser was produced by the repository generator; its tables were not hand-edited. Diff against the integrated main tree: grammar +23/-1, generated parser +7,978/-7,786, parser tests +32. The generated diff preserves all 1,974 existing action bodies/stack offsets and adds four actions; generation reported zero shift/reduce or reduce/reduce conflicts. Approved by: @XuPeng-SH, @aunjgr, @aptend
…atrixorigin#29994) ## What type of PR is this? - [ ] API-change - [x] BUG - [ ] Improvement - [ ] Documentation - [ ] Feature - [x] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: Fixes matrixorigin#29954 Related to matrixorigin#29910: the production fix is already on main in matrixorigin#27799; this PR adds regression coverage. ## What this PR does / why we need it: Deleting a parent with no matching children can panic when an indexed CASCADE child table contains unrelated rows. Direct DELETE appends these recursive branches after planner tag remapping. Use a matched-only index lookup and local input positions throughout untagged recursive deletes, and keep runtime filters disabled to avoid a feedback wait cycle. Carry the tagged self-referencing CASCADE source into its recursive context before createQuery, so indexed REPLACE keeps the planner tags needed by index scans, locks, and DELETE. Positional direct-DELETE actions still use the matched-only local layout. Add planner assertions and SQL regressions for empty and matching cascades, single/composite/unique indexes, NULL keys, prepared deletes, rollback, unrelated-row preservation, and RESTRICT controls. Cover indexed self-referencing REPLACE with secondary/unique indexes, recursive descendants, empty actions, forced-index reads, key reuse, and subsequent direct DELETE. Extend the existing ALTER regression to cover populated self references during VARCHAR resizing, including dropping and re-adding the foreign key (matrixorigin#29910). Validation: - Review follow-up: the indexed self-referencing REPLACE planner regression reproduces a panic at the previous PR head. After propagating the descendant source tag, focused regressions and full `pkg/sql/plan` tests pass (9.479s); package vet and configured incremental golangci-lint pass. - Updated indexed-cascade SQL regression passes normal mo-tester comparison twice (128/128 each), with database teardown independently checked before/between/after runs. Secondary-index EXPLAIN confirms that the forced-index control uses an Index Table Scan. - Rebased onto main `7af631bdf7`; full `pkg/sql/plan` tests pass with repository CGo setup, Go 1.27.1, GOAMD64=v3, GOEXPERIMENT=simd. - gofmt, diff checks, package vet, and configured incremental golangci-lint pass. - Before rebase, normal mo-tester comparisons passed: ALTER-integrity 81/81 statements; indexed cascade 102/102 twice; six related FK cases 681/681. The main update does not change the FK planner/ALTER paths; its result-metadata evidence hook is opt-in and disabled in these tests. - Actual COM_STMT_EXECUTE checks passed for single/composite indexes, empty and matching deletes, rollback, and second-session isolation. - On the clean pre-fix baseline, the original SQL reproduces the panic and both new planner subtests fail. This PR targets main; it does not include a v4.2.5 backport. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…origin#29830) ## What type of PR is this? - [ ] API-change - [x] BUG - [ ] Improvement - [ ] Documentation - [ ] Feature - [x] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: matrixorigin#29095 ## What this PR does / why we need it: `information_schema.columns` could expose base-table columns hidden by an active row-column rewrite rule. This change applies rewrite-aware visibility filtering when binding the `information_schema.COLUMNS` view. Direct projections are allowlisted, stacked rules intersect their visible columns, and unsupported projections fail closed while preserving existing object authorization and caller predicates. The role-rule regression covers `SELECT *`, `information_schema.columns`, an unrelated hidden table, hint-on/off behavior, and complete cleanup. Validation: - Focused and owning `pkg/sql/plan` tests passed. - Public mo-tester BVT ran twice on the same clean service instance; each run passed 224/224 statements with 0 failures and 0 abnormal results. - The issue should be verified by the reporter after this PR is merged. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…efresh (matrixorigin#29195) ## What type of PR is this? - [ ] API-change - [x] BUG - [ ] Improvement - [ ] Documentation - [ ] Feature - [ ] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: Fixes matrixorigin#29148 ## What this PR does / why we need it: A prepared statement can retain an old row-rule policy after cache invalidation, including a failed SET ROLE, allowing execution outside the current tenant scope. This change: - checks prepared-policy generations at publication, cache reload, execution, and migration, including failed role switches and mandatory rules with optional rewrite disabled; - preserves the harmless disabled-to-disabled control and requires fresh PREPARE for stale active or unknown policies; - fails closed with ER_NEED_REPREPARE before binding, rebuild, execution, or cursor fetch; - discards invalidated LONG_DATA uploads without a response and keeps explicit close/deallocate available; - prevents invalidated or unvalidated stale handles from being replayed during migration; - adds frontend unit coverage and a distributed regression for direct versus prepared execution. Affected clients must explicitly PREPARE the statement again after the policy refresh. ## Validation Local Go 1.26.4, Darwin/arm64, native prerequisites built from this checkout: - frontend focused regressions and full package with `mo-cgo-test -vet=all -count=1 -timeout=600s ./pkg/frontend` - frontend full package with `-race`; 5040 passing test/subtest events, no race warnings - adaptive race stress: `TestConcurrentRuleCacheAccess` passed 100 repetitions - authenticated protocol lifecycle regression and full `pkg/tests/issues` with `-short -tags matrixone_test -count=1 -timeout=900s` - `pkg/common/moerr`, incremental golangci-lint (0 issues), gofmt, and diff checks - `cmd/mo-service` build; `role_rule_prepared.sql` passed 26/26 twice on the same clean local instance, with teardown checked after each round --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
## What type of PR is this? - [ ] API-change - [ ] BUG - [ ] Improvement - [ ] Documentation - [ ] Feature - [ ] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: issue matrixorigin#27943 ## What this PR does / why we need it: feat: enable cost-based window hash partitioning --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
) ## What type of PR is this? - [ ] API-change - [x] BUG - [ ] Improvement - [ ] Documentation - [ ] Feature - [ ] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: issue matrixorigin#29151 ## What this PR does / why we need it: Preserve SQL `NULL` as Starlark `None` for procedure arguments, `mo.getvar()`, and cells returned by `mo.sql()`. Previously nullable arguments failed conversion, while SQL NULL cells became indistinguishable from empty strings. Non-NULL result cells retain their existing string conversion, and outbound `None` continues to produce SQL NULL. Add focused conversion/builtin tests and a self-contained SQL regression for nullable numeric/string inputs, INOUT, empty strings and zero values, mixed rows and columns, and outbound NULL. Test both result-cell error paths without publishing partial results, and update an existing result mock for NULL inspection. Validation (Linux CPU/CGo, unchanged frozen content; evidence imported and verified on Mac): - Base: `5c959556431225711a33d1af5fe9a5b6d4060122`; head: `cd79b0b8db345526f005b40bd366db3e529bd16e`; tree: `091bcbb9d3a38750f7a89c0d24c8b0c65fd772d3`. - Base behavior: real SQL reproduction confirms rejected nullable IN arguments / getvar and NULL cells returned as empty strings. - PR-head behavior: CGo-enabled focused tests and complete `pkg/frontend` passed (exit 0); formatting, vet, configured incremental lint, official service build and native verification passed. - BVT: `test/distributed/cases/procedure/starlark_null.sql` and `.result`; canonical `mo-tester -n -g -o -p` normal comparison passed twice on the same instance, 33/33 each, no failed/ignored/abnormal cases. Explicit database/procedure catalog counts were zero between and after runs; the owned service was stopped. - Validation was run at original cloud commit `8171a59977b67229c2b92e9c8f3c5da236c58dc0`; the current commit corrects author/committer identity for CLA with an identical tree and empty source diff. - Mac import: archive SHA256, bundle/source/patch/evidence hashes, exact head/tree/diff, clean worktree and `git diff --check` verified. The machine move did not change relevant validation inputs. QA required: yes — user-visible SQL/Starlark NULL semantics change. Automation covers real CALL, session variables, mixed result cells and reverse NULL conversion; release/client and deployment compatibility remain external QA work. Non-NULL cell typing and unrelated Starlark types are outside this change. Use the issue reference above to keep QA acceptance separate from merge. Independent semantic review completed with a fresh gpt-6-astra / medium reviewer: Stage A independently reviewed the original issue, formal contract and frozen source before seeing author evidence; immutable A found no concrete in-scope defects. Stage B then reconciled raw execution, source identity, failure recovery and cleanup evidence and found no mandatory semantic gaps. No code/test corrections were required. Ready for external review under the requested policy. External CI/CR, CLA confirmation and QA acceptance remain pending; none is claimed as passed. The original cloud commit used an unlinked identity; author/committer metadata has been corrected to the verified contributor with the same code tree, and the CLA service result still requires confirmation. Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Sirius's integration branch was renamed to `perf/embedded-mo-scan`. Update the embedded build guide and bridge instructions so their Git fetch and `SIRIUS_MERGED_REF` commands resolve the current branch. Align the active integration targets in the design documents while preserving the historical merged commit identities. Approved by: @XuPeng-SH
## What this PR does / why we need it: - Tidy the nested Iceberg Go adapter module (pkg/iceberg/adapter/iceberggo) with Go 1.27.1, matching the MatrixOne root module. - Add a failing-on-drift PR check that runs go mod tidy -diff in the nested module. Include root go.mod and go.sum in the Iceberg workflow path filters so root dependency and toolchain updates exercise this guard. - Keep the existing Iceberg E2E job disabled as configured; the new module-sync check is independent. ## Why The 129 Family Connector job in [run 38151944315](https://github.com/matrixorigin/mo-auto-test/actions/runs/38151944315/job/114606608683) fails immediately during go test -tags iceberggo -count=1 ./... with "go: updates to go.mod needed". optools/iceberg_ci.bash runs that command from the nested module. The nested module was added on October 4 with Go 1.26.4. The root module was upgraded to Go 1.27.1 on October 8, but the nested module was not synchronized. Its existing Iceberg E2E PR job is explicitly disabled, and its path filter omitted root go.mod/go.sum; dependency changes therefore were not caught before nightly. ## Original validation - Reproduced the module-update error before the change with Go 1.27.1. - go mod tidy -diff is clean after the change. - go list -mod=readonly -tags iceberggo ./... passes. - actionlint passes on the changed workflow. - Full local go test reaches native compilation but cannot complete on macOS because MO C headers (usearch.h, xxhash.h, jemalloc.h, roaring.h) are not available. Linux CI should verify the complete test. ## 2026-10-14 follow-up Merged official main `2ce4e58af6f03b943ee288ffad81012766cf830a` to include the corrected logservice port fixtures behind the previous unrelated UT compilation failure. The diff against main remains the three Iceberg workflow/module files; this adds no separate kernel fix. On merge commit `cc5ce3d4b7`: - `go mod tidy -diff` in `pkg/iceberg/adapter/iceberggo`: passed. - `actionlint .github/workflows/iceberg-connector.yml`: passed. - `git diff --check`: passed. Full Iceberg/native integration testing was not rerun. The existing disabled E2E configuration is unchanged. No external environment or manual workflow was started; remote CI still needs to validate the new head. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
…9969) ## What type of PR is this? - [ ] API-change - [x] BUG - [ ] Improvement - [ ] Documentation - [ ] Feature - [x] Test and CI - [ ] Code Refactoring ## Related issue(s) - matrixorigin#29844 The issue stays open for the reporter to verify after merge. ## What this PR does / why we need it: External LOAD could persist TIME values beyond ±838:59:59.000000, and textual inputs beyond the internal duration range (for example `2562047788:00:00`) failed early in the decoder even when native INSERT clipped them. CSV/JSONLINE now retain the original TIME text until shared assignment; Parquet STRING applies the same policy within each decoded page. Strict assignments reject with error 1690 and the original value; permissive/IGNORE assignments clamp to the signed endpoint with warning 1264. Both LOAD planners validate supplied TIME before dependent defaults. Fractional conversion honors TIME_TRUNCATE_FRACTIONAL once. Typed Parquet TIME/timestamp sources retain their duration/timezone behavior, including extreme signed values and INT64_MIN diagnostics. LOCAL without REPLACE retains its scoped TIME assignment policy. The global statement IGNORE profile requires explicit IGNORE, preserving strict LOAD LOCAL rejection of invalid DATE/DATETIME/TIMESTAMP and integer inputs. Ordinary, prepared and internal profile tests verify reset and decoder behavior. To keep the new text staging and assignment on the same implementation during rolling upgrades, supplied TIME inputs in CSV/JSONLINE/Parquet LOAD execute their readers and assignment on the coordinator CN. Local file, row-group and byte-offset parallelism still respects max_dop, and writer distribution uses its existing path. This conservatively includes typed Parquet TIME because schema is not yet known at routing; S3 reader throughput for these loads is limited to one CN. Arrow, foreign scans and omitted TIME default columns retain their existing placement. No new protocol fields, protocol version or storage migration are introduced. Validation on macOS arm64 / Go 1.27.1 with verified CPU native artifacts: - Six complete owning packages passed through the repository CGo wrapper; final external and compile packages were revalidated after their last changes. - Incremental go vet and repository-pinned golangci-lint passed; gofmt and diff checks are clean. - Fourteen real native INSERT/LOAD/LOCAL probes verified signed endpoints, 1690/1264 diagnostics, strict rollback and LOCAL REPLACE behavior. - The TIME BVT (128 statements) and unchanged strict LOAD LOCAL regression issue_25366 (59 statements) passed **187/187 statements twice** in normal comparison mode, with both databases dropped after each round and the owned service joined successfully. Before the scoped-policy correction, the exact PR head reproduced the same 10 issue_25366 failures seen in Compose PROXY CI; no expected results were relaxed. - Scope regressions verify coordinator placement, assignment attachment, complete file/row-group coverage, byte-offset/max_dop and unaffected route controls. Both real LOAD planners also verify assignment/global-policy separation through statement reuse and the existing LOCAL Parquet rejection. - gpt-6-astra medium design and final overall review passed without blocking findings. Public SQL validation used a dedicated 1CN/1TN/1LOG instance. Mixed-version routing is supported by actual scope tests and execution-path review; a multi-CN mixed-version integration run and throughput benchmark were not performed locally. New CI results remain pending after delivery. --------- Co-authored-by: XuPeng-SH <xupeng3112@163.com>
## What type of PR is this? - [ ] API-change - [x] BUG - [ ] Improvement - [ ] Documentation - [ ] Feature - [ ] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: issue matrixorigin#22108 ## What this PR does / why we need it: `BuildJsonPlan` currently copies a live `StatsInfo` value while a background statistics refresh can still atomically update its S3 counters. The ordinary struct read races with those writes. Add an owner-side `StatsInfo.Snapshot` that explicitly loads atomic counters and copies stable owner-managed fields. Use that detached value for JSON plan capture, preserving diagnostic values and the JSON schema. Each counter is loaded independently; this is not a transactionally consistent cross-counter snapshot. Regression coverage checks all diagnostic fields, nil/zero controls, snapshot independence, concurrent counter updates, and the real `BuildJsonPlan`/JSON serialization boundary. ### Validation and review provenance Validated source: `f69e6d5a0d29186a19c6497ef22591630666af91`, tree `8f6efe64df0cbb41afe1a284f0be06fcccdd4889`, base `1b66ff135c64ef1c642a297105bf599cd3173dee`. The unchanged source was validated on Linux/amd64 with Go 1.27.1 and the repository-native CGo wrapper. Publication from macOS reuses these source-bound execution results; no Mac test rerun is claimed. - Base counterexample: `TestBuildJsonPlanConcurrentStats` fails under `-race` at the ordinary StatsInfo copy, matching the conflicting access pair in matrixorigin#22108. The controlled writer uses the actual S3 counter mutator; a live GlobalStats worker/debug BVT workload was not run. - Fixed focused race run: all three new tests pass. - `TestBuildJsonPlanConcurrentStats` and `TestStatsInfoSnapshotConcurrentCounters` each pass 100 repetitions under `-race`, separately selected in one process per test. - Both owning-package suites pass in normal and race modes: 420 named test/subtest passes per mode (51 statistic, 369 explain), zero skips. - gofmt, go vet, `git diff --check`, and incremental repository-pinned golangci-lint v2.14.0 pass. - Independent Astra medium source-only Stage A review and subsequent raw-evidence Stage B reconciliation approve this exact source with no mandatory blockers. The reviewed archive and every manifest checksum were verified before publication. - Full repository tests and remote CI are separate gates; no full CI pass is claimed by these local results. Remote CI and maintainer CR are pending at publication. ### BVT applicability BVT: N/A — the changed contract is internal diagnostic-read synchronization. Normal SQL result comparison cannot observe atomic-versus-ordinary reads or deterministically arrange the asynchronous S3 writer at plan capture. The race-instrumented regression directly exercises the real serialization boundary and asserts final counter values and detachment. No SQL result, plan transformation, protocol metadata, or JSON schema changes. SQL BVT was not executed. ### QA decision and scope QA required: yes — this is a concurrency race fix involving background statistics updates. Production entry: frontend execution-plan recording through `BuildJsonPlan`. Automated terminal coverage: race-instrumented snapshot capture and JSON serialization, final exact S3 counters, field/JSON preservation, and detached-value independence. Remaining QA: deployed debug/race-instrumented execution-plan capture while background statistics refresh runs, with no race at the repaired BuildJsonPlan counter-read boundary. These package tests do not claim live multi-CN or full debug/BVT workload acceptance. Keep matrixorigin#22108 open for QA acceptance. Phase timings, PermissionAuth, WaitActiveCost, and whole `StatsInfo.Reset` remain owner-managed/quiescent operations. Existing writer-versus-Reset/reuse lifetime and attribution concerns, and other live StatsInfo readers, remain separate pre-existing limitations. This PR does not claim total statistics concurrency safety. --------- Co-authored-by: XuPeng-SH <xupeng3112@163.com>
## What type of PR is this? - [ ] API-change - [x] BUG - [ ] Improvement - [ ] Documentation - [ ] Feature - [ ] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: issue matrixorigin#29467 ## What this PR does / why we need it: CTAS builds a follow-up internal `INSERT ... SELECT` after the `CREATE TABLE` plan is built. The generated statement must retain the source query's effective row-rewrite policy so hidden rows cannot be materialized. This change: - attaches the validated rewrite policy to the CTAS source AST; - carries that typed, frontend-owned `RewriteOption` through the scoped CTAS compiler context; - attaches the policy to the generated INSERT source before internal planning; - keeps execution independent of diagnostic SQL redaction and removes the obsolete raw leading-hint reparse path; - covers SQL-mode literal handling, diagnostic-marker literals, remapped policy keys and bodies, and the actual CTAS population boundary. The generated target remains outside the source rewrite scope, and the existing privilege-check boundary is preserved. Validation: - `GOWORK=off go test -mod=readonly -count=1 -timeout 300s ./pkg/sql/compile` - `GOWORK=off go test -mod=readonly -count=1 -timeout 180s ./pkg/sql/parsers` - `GOWORK=off go vet -mod=readonly ./pkg/sql/compile ./pkg/sql/parsers` - Existing access-control CTAS regression and exact-head public-session validation cover the user-visible row-policy result. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Owner
Author
|
Closing this fork-local draft because the intended pull request is in matrixorigin/matrixone: matrixorigin#30183. The branch and commits are preserved for that upstream draft. |
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
Which issue(s) this PR fixes:
Related to matrixorigin#25107. This is the public row-locking compatibility slice and does not close the umbrella issue.
What this PR does / why we need it:
This PR supports:
FOR UPDATE NOWAITandFOR SHARE NOWAIT, with request-local fast-fail locking and MySQL lock-conflict error handling.FOR SHARE SKIP LOCKED, including skipping conflicting candidates, retaining shared locks for returned rows, refillingLIMIT, and preserving refresh/retry outcomes across multi-target lock attempts.Nonblocking row-lock methods are admitted at MORPC v113. Peers without those methods fail closed before lock side effects; existing v112 WEEK/DAYOFYEAR contracts remain unchanged. The implementation keeps lock conflicts, retries, fatal binding errors, and owned-lock cleanup distinct. Async callbacks snapshot pooled request state before the request can be reset and reused.
FOR UPDATE SKIP LOCKEDis not part of this PR. This candidate contains only the already-published lock work; parser compatibility changes are handled in the separate matrixorigin#30024 parser PR. The two split PRs use the same main base and have no source dependency on each other.Validation
Candidate HEAD:
08e550d03d776a3e82a511b59043d271e1d9e9d2, based on main071a79b77f82a2d2f8e8270eaef3c27a29fab4d1.Exact-head checks passed using the repository's CGo test wrapper and verified native inputs:
GOWORK=off .agents/skills/mo-dev/scripts/mo-cgo-test -mod=readonly -vet=all -p=1 -count=1and passed for./pkg/lockservice,./pkg/sql/colexec/lockop,./pkg/sql/parsers/dialect/mysql,./pkg/txn/client,./pkg/queryservice,./pkg/sql/plan,./pkg/sql/compile,./pkg/common/moerr,./pkg/defines,./pkg/pb/lock,./pkg/pb/pipeline, and./pkg/pb/plan.GOWORK=off .agents/skills/mo-dev/scripts/mo-cgo-test -race -count=1 -v -p=1 -timeout=10m -run '^(TestLockOpSkipLockedRetryIsReturnedBeforePublishingPrefix|TestLockOpSkipLockedResetReleasesBorrowedOutput|TestLockOpSkipLockedFreeReleasesBorrowedOutput)$' ./pkg/sql/colexec/lockop. Lockservice command:GOWORK=off .agents/skills/mo-dev/scripts/mo-cgo-test -race -count=1 -v -p=1 -timeout=10m -run '^(TestNoWaitFatalBindFenceRetainsReadOnlyCleanupRoute|TestRemoteFirstReadOnlyTryLockConflictStillRegistersCleanupRoute)$' ./pkg/lockservice. All five named tests passed. The full lockservice package ran without-race.make build-with-prebuilt-nativepassed. Canonical MySQL parser generation (GOWORK=off make -B mysql_sql.gofrom the parser directory) and lock/plan/pipeline protobuf generation with protoc 3.21.1, gogofast 1.3.2, goimports, and the two repository postprocessors each ran twice; both passes were byte-stable.FOR UPDATE NOWAITandFOR SHARE NOWAITagainst a locked row returned MySQL error 3572; SQLStateHY000was observed (the harness asserts the error number, not SQLState). The same explicit transaction remained usable afterward. With a held candidate,FOR SHARE SKIP LOCKED LIMIT 2refilled to[2,3]; a separate NOWAIT probe confirmed row 2 was locked (row 3 was not individually probed). When all candidates were locked the query returned no rows. The reachable preparedSET @v=(SELECT ... FOR SHARE SKIP LOCKED)route skipped a row held by another session.The SQL BVT used one local CN cluster with multiple SQL sessions; no cross-CN SQL BVT was run. Split-head CI has not run yet and will be checked after publication. Validation is limited to the observed row-2 lock probe and SQLState response above; the tests do not assert a row-3 lock probe or SQLState.
Delivery notes
This is a Draft PR containing only the published lock feature split. It is independent of the parser changes in matrixorigin#30024 and is
Related to #25107, notCloses #25107.