Repository navigation
fix(data): publish migration and cursor corrections (successor to #185) - #195
PeterGuy326 wants to merge 8 commits into
Conversation
|
Independent automated preflight: HOLD on deployment order at The source tree has migrations 1–23 and 26, without 24/25. The runner uses Goose without WithAllowMissing, so deploying this candidate before the lower numbered migrations makes their later introduction invalid. This is source-confirmed, not a newly executed PostgreSQL upgrade test. An enforced deployment sequence or coordinated, verified-unreleased numbering is required before this PR can be treated as independently deployable. The actual cursor, populated uniqueness, and bounded-query fixes passed independent source/compile checks; the implementation worker's real PostgreSQL evidence remains separately attributed. Latest 20/20 successful checks do not prove the cross-PR upgrade sequence. Follow-up is assigned, with no migration-policy waiver or deployed-history rewrite authorized. |
Code ReviewOverall code quality is good and test coverage is solid. Here are a few specific issues and suggestions: 1. [Medium] Migration 0026's DELETE deduplication may be slow on large datasetsDELETE FROM embeddings_text
WHERE id NOT IN (
SELECT DISTINCT ON (file_id, chunk_index) id
FROM embeddings_text
ORDER BY file_id, chunk_index, id
);
Suggestion: Add a pre-check to skip the DELETE if there are no duplicates: DO $$
BEGIN
IF EXISTS (
SELECT 1 FROM embeddings_text
GROUP BY file_id, chunk_index
HAVING count(*) > 1
LIMIT 1
) THEN
DELETE FROM embeddings_text
WHERE id NOT IN (
SELECT DISTINCT ON (file_id, chunk_index) id
FROM embeddings_text
ORDER BY file_id, chunk_index, id
);
END IF;
END $$;This way, on the vast majority of production tables that have no duplicates, the migration only needs a lightweight GROUP BY check to pass. 2. [Low] Two valuable comments were removedThese two comments were removed from // The anchor memory must exist, be visible under the caller's allowed
// paths, and not be forgotten — mirroring Get — so listing relations never
// leaks the existence of a memory the caller cannot read.// BFS from target following supersedes/corrects edges in their forward
// direction: if we ever reach sourceID, there is a cycle.The first explains the existence-leakage guard for the security check — this is critical context for understanding why the anchor check is necessary. The second explains the BFS strategy of 3. [Low] Three unrelated fixes bundled in one migrationThe PR description says "each is a single DDL statement and none warrants its own schema version." However, Item 3 (the FK cascade change) is semantically the heaviest — it drops and recreates three foreign key constraints. If a deployed environment has manually adjusted these FKs (e.g., temporarily changed to DEFERRABLE), this migration would overwrite them. Consider documenting this assumption in the PR description or as a migration comment. 4. [Nit]
|
waterbro-8
left a comment
There was a problem hiding this comment.
Code review(draft,给作者的清单,非批准)
一、keyset 分页:形状是对的,而且做对了一件容易漏的事
我特意核了排序与游标谓词是否自洽,因为这是这类改动最常错的地方:
- 谓词
(r.created_at < $t OR (r.created_at = $t AND r.id > $id)),而 SQL 的排序是ORDER BY r.created_at DESC, r.id(id升序)。两者一致,不会出现"同一时间戳的行被跳过或重复"。这条我对照了 main 上的原实现与这条分支的实现,排序没有被改动,新增的谓词是配得上它的。 - 取
q.Limit+1行来探测是否还有下一页,len(out) <= q.Limit时不出next_cursor。✓ - 游标取自裁剪后那一页的最后一行(
result.Relations[len-1]),不是out[q.Limit]——这点很容易写错,这里是对的。 - 游标绑定了过滤条件的哈希(workspace + memory + direction + relationType 四项参与 sha256)。这是我认为最值得记一笔的地方:它让一个游标无法被拿到另一组过滤条件下重放。很多 keyset 实现会漏掉这个约束,导致跨查询的游标串味。
- 非法游标走
ErrInvalidCursor→ handler 映射成 400invalid_cursor,不是 500。✓
二、0026 迁移:三条都是能站住的,而且写清楚了取舍
- Item 1(唯一性):先按
DISTINCT ON (file_id, chunk_index) ... ORDER BY file_id, chunk_index, id去重,再加UNIQUE (file_id, chunk_index)。注释明说"保留 UUID 最小的那一行是为了确定性选择,UUID 顺序不等于插入顺序"——这句话很关键,它承认了去重规则是人为约定而非"保留原始那行"。我不认为这是缺陷(写入路径本来就是先删后插,理论上不会有重复),但这个诚实度值得保持。 - Item 2(部分索引):
idx_memories_source_file_id与idx_memories_created_by_user_id都是WHERE ... IS NOT NULL的部分索引,正是ON DELETE SET NULL反查需要的形状,而且部分索引不会为空值浪费空间。顺带一提:source_file_id这个索引正好也是我在 #210 上提出的那条守卫缺陷(只查memories.path、不查source_file_id)如果要修,所需要的底层支撑——两条 PR 在这点上是有协同的。 - Item 3(级联对齐):
memory_relations的三个 FK 改为与memories.workspace_id一致的ON DELETE CASCADE。理由写得很实在:memories会被 workspace 删除连带清掉,而memory_relations不级联的话,删 workspace 会因为还残留引用它的边而失败。这是真的会咬人的不一致。 Down把三处 FK 还原成无级联、删掉索引、删掉约束,顺序与 Up 相反。✓
三、非阻塞,但建议补一句的两点
- 这次迁移在生产库上不是无感的,而且它碰的是最大的那张表。 Item 1 的
DELETE ... WHERE id NOT IN (...)加ADD CONSTRAINT UNIQUE,都要全表扫描embeddings_text并加锁(约束建立期间对写入是排他的)。这不是缺陷,但docs/MIGRATION_SEQUENCE.md目前只讲了顺序与严格性,没讲迁移窗口。考虑到 0024(重写files+ 建索引)、0025(在三个embeddings_*上建 HNSW)、0026(又是embeddings_text)是连着跑的,建议在序列文档里补一句"这三次迁移会依次对最大的几张表加锁,请预留窗口"。 - Item 1 的
Down是不保数据的(被去重删掉的行不会回来),迁移注释里没提。这是 down 迁移的常态,我不反对;只是既然注释对其他取舍都写得很细,这里补一句"down 不恢复被去重的行"会更完整。
四、接口变更的处理是完整的
ListRelations 的 Go 签名从 ([]Relation, error) 改成 (*ListRelationsResult, error),涉及三处:api.go 的 MemoryService 接口、handlers_memory.go 的实际调用、以及 handlers_memory_test.go 里的 stub——三处都改了,没有留下编译不过的悬空实现。HTTP 层面是向后兼容的:relations 字段仍在,只是新增了可选的 next_cursor 与可选的 cursor 查询参数。
五、我没有声称验证的事
本机没有 PostgreSQL,我没有跑过 migrations_test.go、也没有跑过分页的集成测试。以上结论来自逐行读迁移、SQL 与分页逻辑,不是运行时观测。
另外需要点明:这条 PR 的 CI 绿不等于它可以先合。docs/MIGRATION_SEQUENCE.md 已经写明它是链上第 3 环(#195 HOLD behind #197),我同意这个约束——0026 声明的前驱是 schema 25,跳序部署会让 Goose 正确地在启动时失败。
(本次为独立评审:我不是本 PR 作者,也不是其 head 的推送者。draft 状态下我不投批准票。)
fdccb43 to
0a48f84
Compare
) ## Linked draft successor — original #183 remains open Refs #176. ## Current follow-up: `c0421168bd145077bf164f91c0d2b454a79760ef` - Fixed the remaining exported MCP route enum: `tools/list` now includes `lexical`. The in-process MCP regression reproduced the missing enum before the fix and now verifies both schema and `tools/call` HTTP forwarding. All 9 MCP tests pass on Linux; affected-package vet passes. - Added a strict, populated PostgreSQL sequential-upgrade regression and a DB-free embedded-migration continuity guard. Actual PostgreSQL 16.14 / pgvector 0.8.2 advances 23 → 24, verifies lexical backfill and full history, and accepts ordinary production startup afterward. No `AllowMissing` option or migration renumbering. - Cumulative source/base order is #194 → #197 → #195. #197 remains HOLD for text-query planner acceptance; #195 remains HOLD behind it. See `docs/MIGRATION_SEQUENCE.md` for deployment and existing-gap recovery boundaries. - Tests ran at `2f2e965b6794863d7f5a38a748262da4691beb35`; this final head only corrects the documented verification command to `scripts/verify.sh integration`. Fresh exact-head CI and independent review remain required. The earlier evidence below is historical, not a fresh-head approval. Preserves @sun-970 / liyuanyang's complete authored chain through `13ebe9efa6ba3c733199d374e8db3d107f598ca2` for #176. This draft makes the bounded reviewed corrections accessible; it does not replace independent review, CI, or human approval. Do not close #183 before a replacement is verified and merged. The original fork's Git-data write returned HTTP 404 and its repository permissions report `push:false`. No ACL override was attempted. This canonical branch preserves original commit identities and hashes. Added blobs, trees, and commits were checked against local Git hashes; no force update was used. ## Corrections - `scripts/verify.sh`: expect migration 24, matching this branch's lexical migration. The former expectation of 23 caused the PostgreSQL validation failure. - Refresh audited Web development dependencies, traceable to #192 commit `11e02e21ef2c3dbd2dae26e4376872e54e78ecb5` via a separate `cherry-pick -x` commit. Audit threshold unchanged. - No additional search/ranking changes beyond original #183. ## Validation at head `3d885a8427e06fd7175011ef0188a06f32028a3f` - PASS: migration to 24 on PostgreSQL 16.14 / pgvector 0.8.2 and PostgreSQL 17.10 / pgvector 0.8.3. - PASS: `TestLexicalSearchWithoutWorker`, five cases: lexical without worker, text/auto fail closed, CJK trigram, path restriction. Go tests cross-compiled for Linux and executed against real disposable PostgreSQL with `GOMAXPROCS=2` and serial package builds. - PASS: audited lockfile check; shared fix from #192 remains explicit, not attributed to this feature. - NOT CLAIMED: provider-backed vector retrieval or production benchmark quality. - REQUIRED: fresh CI on this exact head and independent review. Original-head CI/review is not approval of this draft. Original feature scope: model-free file-corpus lexical route, migration 0024, API/CLI/docs, managed-provider bypass. Scope excludes tokenizer, generation lifecycle, and ranking redesign.
平台组 Review(崔泽生,assigned by @冯浩然)一、总体评价结构清晰、文档充分的 PR,解决 #178 五项数据面卫生中的三项(migration 0026),同时修复关系分页游标方向 bug、index generation listing N+1 查询。CI 22/22 全绿。PR body 明确标注"非独立可部署"和 HOLD behind #197。 二、Migration 0026 正确性Item 1:
Item 2:
Item 3:
Up/Down 对称性:
三、memory/relation.go 游标分页——核心 bug 修复正确排序
集成测试验证了游标跨 filter 失效、同时间戳分页不重复不遗漏 ✅ 被删除的注释:同意 sun-970 的判断——anchor memory 安全不变量说明和 wouldCycle BFS 策略说明是"WHY is non-obvious"注释,建议恢复。 四、indexgeneration/service.go N+1 修复——干净
五、词法搜索路由(附加变更)
六、依赖链Goose 严格模式,跳序部署会正确失败。三个 PR 连续对大表加锁,生产部署需预留维护窗口。 七、建议级问题
八、核心判断代码质量高,变更正确,测试覆盖充分。依赖链管理严格,文档诚实。Draft 状态下不投批准票。等待 #197 先行解决、conflict 解决后,建议合并。 |
4536392 to
95af9fe
Compare
…178) 1. Add UNIQUE (file_id, chunk_index) on embeddings_text with defensive deduplication before the constraint. 2. Add partial indexes on memories(source_file_id) and memories(created_by_user_id) to avoid seq scans on ON DELETE SET NULL. 3. Add ON DELETE CASCADE to memory_relations FKs to align with the memories cascade and prevent orphan-edge failures on workspace delete. 4. Fix N+1 in indexgeneration List: batch-fetch generations for all builds in one query instead of per-build. 5. Add keyset cursor to ListRelations reusing the existing encodeListCursor/decodeListCursor helpers from memory/list.go. Includes migration 0025 with Down blocks and an integration test proving the embeddings_text uniqueness constraint rejects duplicates.
d1a3a95 to
ac4a3c9
Compare
waterbro-8
left a comment
There was a problem hiding this comment.
复审 exact head ac4a3c9512ec9066ce034b138c4f7b6bd0eb5182。migration 0026、N+1 batch、relation keyset 的实现形状均正确;排序 created_at DESC, id ASC 与 cursor 谓词 created_at < / id > 一致,filter hash 也绑定了 workspace/memory/direction/type。
当前仍缺 #178 的显式 Acceptance,暂不能批准:
- #178 要求 item 2 在 populated DB 上验证,并记录
ON DELETE SET NULL路径加索引前后的EXPLAIN。当前新增的migrations_test.go只验证 item 1 的去重/唯一约束;PR/issue 也没有任何 EXPLAIN 证据。请补可复现的 populated-DB 验证与 before/after plan(测试或验证 ledger 均可,但须能对应 source_file_id,且说明 created_by_user_id 是否同样验证)。 - #178 当前仍为
status:needs-triage。仓库 AGENTS.md 要求 material change 在status:ready后才能合入,需 requirement owner 先完成 ready 决策。
此前该 head 只有 PR Policy,并非完整 CI;我已在 exact branch 手动触发 CI、Validate Agent memory、Security Baseline,运行中。待上述 AC/治理项补齐且新 head checks 通过后再复审。
|
补充 exact-head 验证:我手动触发的三套 workflow 已全部结束。
Runs: https://github.com/bytefolk/mem/actions/runs/35425019338 · https://github.com/bytefolk/mem/actions/runs/35425019201 · https://github.com/bytefolk/mem/actions/runs/35425019538 这解决了 current head 原本缺失的运行验证,但不替代上一条 review 所列 #178 EXPLAIN / populated-index evidence 与 ready 决策。 |
xiaocui-big
left a comment
There was a problem hiding this comment.
平台组正式 Approve(崔泽生)
Re-review on head ac4a3c9。此前 9/18 的详细 review 结论不变:migration 0026 三项变更正确、游标分页谓词与排序一致、filter hash 绑定防跨条件重放、N+1 修复干净。
当时未投批准票是因为 PR 处于 Draft 且 HOLD behind #197。现在:
- #197 已被 #219 取代并合入 main
- 本 PR 已 rebase 到最新 main,只保留 #178 hygiene 切片
- CI 全绿(含 Go race/coverage、migration、PostgreSQL integration、Security Baseline)
- mergeable: MERGEABLE
代码层面的判断没有变化——实现正确,测试覆盖充分,依赖链管理严格。
关于 waterbro-8 提出的验收项
- #178
status:ready决策 — 同意这是治理项而非代码缺陷。维护者需完成 triage 决策。 - Populated DB EXPLAIN 证据 — 合理要求,但不阻塞代码正确性判断。可在合并后由 follow-up 补齐,或由 requirement owner 在 triage 时一并记录。
代码已就绪,批准。
|
Populated-database verification for #178 item 2 is now recorded against exact head Environment and resultPostgreSQL 17.10, pgvector 0.8.3, pg_trgm 1.6, disposable local database. Applied the unmodified Up SQL from migrations 0001–0025 at this head, populated the fixture, captured the baseline, then applied the unmodified 0026 Up and repeated the same checks. Each migration ran in its own transaction through psql; this was SQL validation, not a new Goose/Go acceptance run. Fixture: 100,000 memories; each tested FK has 10,000 non-NULL values, including 100 references to the target parent, and 90,000 NULLs. Planner scan options remained enabled; no forced-index setting was used.
These are child scan-node buffers, not total DELETE/UPDATE buffers. All four DELETE cases preserved the 100,000 memories, nulled exactly 100 matching references, and restored those references on rollback. ReproductionUse an isolated test database with the extensions above. Extract each migration from this exact Git commit and take only the text before INSERT INTO users(id,email,password_hash) VALUES
('10000000-0000-0000-0000-000000000001','owner@fixture.invalid','fixture'),
('10000000-0000-0000-0000-000000000002','target@fixture.invalid','fixture');
INSERT INTO workspaces(id,name,resource_owner_user_id) VALUES
('20000000-0000-0000-0000-000000000001','fixture','10000000-0000-0000-0000-000000000001');
INSERT INTO files(id,user_id,name,path,size,sha256,storage_key) VALUES
('30000000-0000-0000-0000-000000000001','10000000-0000-0000-0000-000000000001','target','/target',0,repeat('1',64),'fixture:target'),
('30000000-0000-0000-0000-000000000002','10000000-0000-0000-0000-000000000001','other','/other',0,repeat('2',64),'fixture:other');
INSERT INTO memories(id,workspace_id,created_by_user_id,source_file_id,
kind,content,path,source_type,idempotency_key_sha256,request_sha256,content_sha256)
SELECT md5('mem195-'||i::text)::uuid,
'20000000-0000-0000-0000-000000000001',
CASE WHEN i<=100 THEN '10000000-0000-0000-0000-000000000002'::uuid
WHEN i<=10000 THEN '10000000-0000-0000-0000-000000000001'::uuid END,
CASE WHEN i<=100 THEN '30000000-0000-0000-0000-000000000001'::uuid
WHEN i<=10000 THEN '30000000-0000-0000-0000-000000000002'::uuid END,
'note','synthetic local FK fixture '||i::text,'/fixture','test',
repeat(md5(i::text),2),repeat('0',64),repeat('0',64)
FROM generate_series(1,100000) AS i;
INSERT INTO embeddings_text(id,file_id,chunk_index,chunk_text) VALUES
('40000000-0000-0000-0000-000000000001','30000000-0000-0000-0000-000000000002',0,'original'),
('40000000-0000-0000-0000-000000000002','30000000-0000-0000-0000-000000000002',0,'legacy duplicate');
ANALYZE;Enable nested-plan capture in the same privileged local test session: LOAD 'auto_explain';
SET auto_explain.log_min_duration=0;
SET auto_explain.log_nested_statements=on;
SET auto_explain.log_analyze=on;
SET auto_explain.log_buffers=on;
SET auto_explain.log_timing=off;
SET auto_explain.log_format=json;
SET auto_explain.log_level=notice;
BEGIN;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF, FORMAT JSON)
DELETE FROM files WHERE id='30000000-0000-0000-0000-000000000001';
SELECT count(*), count(source_file_id) FROM memories; -- 100000, 9900
ROLLBACK;
BEGIN;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF, FORMAT JSON)
DELETE FROM users WHERE id='10000000-0000-0000-0000-000000000002';
SELECT count(*), count(created_by_user_id) FROM memories; -- 100000, 9900
ROLLBACK;Apply 0026 Up on this populated fixture, run UPDATE ONLY "public"."memories" SET "source_file_id" = NULL
WHERE $1 OPERATOR(pg_catalog.=) "source_file_id";
UPDATE ONLY "public"."memories" SET "created_by_user_id" = NULL
WHERE $1 OPERATOR(pg_catalog.=) "created_by_user_id";The four observed scan excerpts, with unrelated JSON fields omitted: [
{"case":"before/source_file_id","Node Type":"Seq Scan","Actual Rows":100,"Actual Loops":1,"Rows Removed by Filter":99900,"Shared Hit Blocks":5920,"Shared Read Blocks":0},
{"case":"after/source_file_id","Node Type":"Index Scan","Index Name":"idx_memories_source_file_id","Actual Rows":100,"Actual Loops":1,"Rows Removed by Index Recheck":0,"Shared Hit Blocks":7,"Shared Read Blocks":2},
{"case":"before/created_by_user_id","Node Type":"Seq Scan","Actual Rows":100,"Actual Loops":1,"Rows Removed by Filter":99900,"Shared Hit Blocks":5925,"Shared Read Blocks":0},
{"case":"after/created_by_user_id","Node Type":"Index Scan","Index Name":"idx_memories_created_by_user_id","Actual Rows":100,"Actual Loops":1,"Rows Removed by Index Recheck":0,"Shared Hit Blocks":17,"Shared Read Blocks":0}
]Additional migration checks and limits0026 Up succeeded on populated data. The duplicate chunk fixture retained UUID This establishes the two requested FK lookup plans on the stated fixture. It does not establish production latency, concurrency/locking behavior, or PostgreSQL 16 performance; repository CI uses PG16, separately from this local PG17 run. A complete user DELETE still triggered another sequential scan for No implementation change was made. #178 still needs the maintainer's readiness decision, and the existing change request needs the original reviewer's re-review. This evidence and another reviewer's approval do not substitute for either decision. The old #197 dependency is no longer the blocker. |
waterbro-8
left a comment
There was a problem hiding this comment.
复审 exact head ac4a3c9512ec9066ce034b138c4f7b6bd0eb5182(无新 commit)。不是验收。未合。
技术阻断(#178 item 2 populated EXPLAIN)已解除: 评论里的 10 万行 fixture、真实父 DELETE 触发的两条 FK UPDATE、加索引前后 plan 对应 source_file_id 与 created_by_user_id(Seq Scan → 对应 Index Scan,100 行 / 99900 filtered)。这是 SQL 校验,不是 Goose/Go 新验收,也不是 PG16 CI 或生产时延;forgotten_by_user_id 仍是 Seq Scan,已如实写出。可以对应这项 AC。
仍不能 Approve 为可合入:
- #178 仍是
status:needs-triage。AGENTS.md 要求 material change 在status:ready后才能合。#178 ready 是维护者决策,评论证据和另一人的 Approve 都不能替代。
合入是你。我不合。
waterbro-8
left a comment
There was a problem hiding this comment.
Fresh review on exact head 71db9ab67acb22e25f26e958059b2ed8058e0624. The five #178 changes remain scoped and technically sound. Latest main is integrated without rewriting published history; main owns migration 0026, so data-plane hygiene is correctly sequenced as 0027. TestMigrationUpgradeSequence now verifies schema 26 producer-agent indexing before schema 27 deduplication/unique rejection. The issue is status:ready, populated PostgreSQL EXPLAIN evidence is recorded, the branch is mergeable, and all 21 hosted checks pass. Approving for squash merge; this is not a production-latency claim.
waterbro-8
left a comment
There was a problem hiding this comment.
Fresh formal review on exact head 71db9ab67acb22e25f26e958059b2ed8058e0624.
No merge-blocking finding:
- latest main owns migration 0026 and this branch correctly sequences hygiene as 0027;
TestMigrationUpgradeSequenceverifies 26 producer-agent indexing before 27 dedupe/unique rejection; - 0027 Up deterministically deduplicates
(file_id, chunk_index), adds the unique constraint and two partial FK indexes, and aligns all three relation FKs with cascade; Down reverses constraints/indexes in safe order (deduplicated rows are intentionally not reconstructible); - populated migration/duplicate
23505coverage and recorded EXPLAIN evidence satisfy #178 item 1/2 evidence; - relation keyset uses
created_at DESC, id ASCwith the matching< timestamp / > idpredicate, takes limit+1, emits cursor from the retained page tail, and binds workspace/memory/direction/type via the existing cursor codec; - index-generation List is two queries regardless of build count and maps each generation back by build/workspace, with integration query-count coverage.
#178 is now status:ready; all 21 hosted checks on this exact head are green. APPROVED for the configured squash auto-merge. This is not a production-latency claim.
|
@Bindy-lbb 最后一项 branch-policy gate:请对 exact head |
sun-970
left a comment
There was a problem hiding this comment.
Reviewed the diff from outside — I do not hold write access on this repository, so this is a comment rather than an approval. A maintainer with write access still needs to give the formal verdict.
Pagination is correct. At head, the query orders by r.created_at DESC, r.id (line 337), which exactly matches the new predicate (r.created_at < $t OR (r.created_at = $t AND r.id > $i)); next_cursor is built from the last row of the trimmed page after fetching limit+1. No skip/duplicate at equal timestamps. Binding the cursor to a sha256 fingerprint of (workspace, memory, direction, relationType) is a good touch — a cursor cannot be replayed against a different filter, and mismatch is rejected as ErrInvalidCursor → 400.
Also good: the response stays backwards compatible ({relations} → {relations, next_cursor?}), a nil slice still serializes as [], and the N+1 in indexgeneration.Service.List is replaced by one = ANY($2::uuid[]) batch query plus in-memory grouping — same result, one round trip.
Migration 0027:
- The dedup before
ADD CONSTRAINTkeeps the smallest id per (file_id, chunk_index) with a deterministic ORDER BY, and the comment is honest that UUID order is not insert order. Correct. - Aligning the
memory_relationsFKs toON DELETE CASCADEmatches the stated root cause (workspace delete cascades memories, then fails on the relation edges), and the Down body re-adds the FKs without an action — the right inverse. - One operational note:
ALTER TABLE … ADD CONSTRAINT UNIQUEholds ACCESS EXCLUSIVE while the index builds (CONCURRENTLYis not available for a constraint added this way). On a largeembeddings_textthis briefly blocks writes — worth a line in the migration comment if deploy windows matter.
Bindy-lbb
left a comment
There was a problem hiding this comment.
Reviewed exact head 71db9ab67acb22e25f26e958059b2ed8058e0624 against current main 73c316a (14/14 changed files inspected; OCR selected 7 production files, and I also inspected the 7 excluded test/documentation files).
One merge-blocking finding: migration version 0027 now collides with main. Main commit 8c4080a (#235) added server/internal/db/migrations/0027_vault_sync.sql, while this PR adds 0027_data_plane_hygiene.sql. The files do not create a Git filename conflict, so MERGEABLE and the old 21 green checks do not establish migration compatibility.
I reproduced this without a database by assembling the unchanged SQL migrations from current main plus this PR's hygiene migration and calling the repository-pinned Goose v3.22.1 CollectMigrations. Goose rejects the set with goose: duplicate version 27 detected, naming both files. Startup uses Goose over all embedded migrations.
Please integrate current main and move the unmerged hygiene migration to the next available version (currently 0028), updating EXPECTED_MIGRATION_HEAD, migration-sequence assertions, test fixture filenames, and migration documentation together. Rerun migration/upgrade and PostgreSQL integration checks on the updated head. Preserve already released migration identities; privately applied draft versions still need an operator-owned recovery plan.
Local validation on this PR head: go test ./internal/api ./internal/memory ./internal/indexgeneration ./internal/db passed. I did not rerun live PostgreSQL integration or production performance checks. The batching and cursor changes have no other blocking finding from this review. Withholding approval until the migration collision is resolved.
main commit 8c4080a (#235) added 0027_vault_sync.sql, colliding with this branch's 0027_data_plane_hygiene.sql. Goose v3.22.1 CollectMigrations rejects the combined set with "goose: duplicate version 27 detected", which blocks database migration; the previous 21 green checks did not cover this new combination. Integrate current main and move the unmerged hygiene migration to the next available version, 0028: - rename migrations/0027_data_plane_hygiene.sql -> 0028_data_plane_hygiene.sql - update migrations_test.go fixture path - update TestMigrationUpgradeSequence version gates (>= 28) so the upgrade sequence still asserts 26 producer-agent indexing, 27 vault_sync, then 28 dedupe/unique rejection - update docs/MIGRATION_SEQUENCE.md: predecessor schema 27, head 24 through 28, and record why the version moved twice (0026 -> 0027 -> 0028) Already released migration identities are preserved; 0027 now belongs to main's vault_sync. Verification on this head (Go 1.27.0, Windows): - goose.CollectMigrations over the assembled tree: 28 migrations, no duplicate version, head 0028 (was: duplicate version 27 panic) - go build ./... clean - go test ./internal/db/... ./internal/memory/... ./internal/indexgeneration/... ./internal/api/... pass (excluding the pre-existing Windows-only TestWorkspaceImportSpoolsSecureTemporaryFile 0600-mode failure, which also fails on clean main)
Blocking finding reproduced; fix is in #237@Bindy-lbb I reproduced the migration collision with the same method and confirmed your root cause. Main Assembling the unmodified SQL from current main plus this PR's hygiene migration and calling the repository-pinned Goose v3.22.1 Your two points both hold: it blocks migration, and the previous 21 green checks did not cover this combination. MERGEABLE and the filename-level diff never established migration compatibility. FixBecause I only hold read access on
Released migration identities are preserved - Verification on the fixed head (Go 1.27.0, Windows)
The only excluded test is Not claimedNo live PostgreSQL, no production-latency run. |
PostgreSQL integration on the previous head upgraded the populated sequence through 28 successfully, then failed the runner assertion with "ERROR: expected migration version 27, got 28" (run 38694417311, PostgreSQL integration job 116287630741), which also kept the rollback/forward and race stages from running. The runner reads one constant, so this is the whole fix: - scripts/verify.sh: EXPECTED_MIGRATION_HEAD 27 -> 28. All nine assert_migration_version calls and the up-to target already reference the variable, so no other line changes. Mode stays 100755. - docs/TESTING.md: the same stale "declared current head (currently 27)" is updated to 28 so the documented expectation matches the runner. Checked there is no remaining hardcoded 27 migration assertion: grep -rn 'EXPECTED_MIGRATION_HEAD=27' and the TESTING.md phrase both come back empty. bash -n scripts/verify.sh is clean. Not claimed: no local PostgreSQL run. The deterministic mismatch this repairs is the assertion, not the migration path; the PostgreSQL integration and race stages still need a rerun on the updated head.
Bindy-lbb
left a comment
There was a problem hiding this comment.
Fresh final review on exact head 8263a012e905e1bcff32608538438cd18b5857d3 after full integration of #237. APPROVED: both findings from my previous reviews are resolved, with no remaining blocking finding.
The complete fix branch was fast-forwarded into codex/fix-pr-185, preserving main ancestry and d7a16a7; #237 is recorded as merged. Migration 0027 remains main's vault migration, and hygiene is a byte-identical rename to 0028. Runner assertions, fixture references, upgrade-sequence gates and docs now agree on head 28. scripts/verify.sh is mode 100755 with LF line endings.
Review coverage: all 15 paths in the final PR delta are accounted for across the original review and the six-file integration re-review (15 reviewed, 0 skipped, 100%). Existing cursor ordering/filter binding and batched generation listings retain the original reviewed behavior; no unrelated upstream changes remain in the final delta against main.
Validation on this exact head:
- Local: pinned Goose v3.22.1 collects 28 migrations, and db/memory/indexgeneration/api package tests pass.
- #195's newly triggered CI run 38701995887 succeeds, including Go race/coverage.
- #195's newly triggered Validate Agent memory run 38701995948 succeeds; both Migration and PostgreSQL regression and PostgreSQL race regression explicitly pass.
- All current hosted checks are completed SUCCESS, including Security Baseline and PR Policy.
The live PostgreSQL evidence is from CI; I am not claiming a local PostgreSQL or production-latency run. The prior migration-collision thread is resolved. Approved for the repository's configured squash merge of #195; no protection changes or admin bypass.
|
Status: ready to merge — blocked only by the last-push approval rule. Head A squash merge attempt returns:
Note the earlier approval on |
Refs #178.
After #197 closed
#197 was superseded by #219 (0025 HNSW on main). This PR no longer stacks on
codex/fix-pr-180. It is rebased onto currentmainand keeps only the #178 data-plane hygiene slice:memory_relationswith workspace deletionHead:
ac4a3c9(3 cherry-picks: 5cbd05b → rename 0026 → cursor/populated verification). Lexical/HNSW files that were already on main were dropped.Not claimed: live production latency, or that #197 merged.