diff --git a/AGENTS.md b/AGENTS.md index 2d811ae..b7c9fad 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -28,7 +28,7 @@ tests/ # pytest；按 audio/image/storage/workflow 分目录 ## 硬性架构约定 1. **媒体隔离**：audio 和 image 各自拥有独立的 package 和 workflow 入口，**不要**把共享 Stage 参数化成"通用管线"。可共享的只有 `storage/` 层 infra 和 `config.py` 的辅助函数。 -2. **数据 API 优先级**：所有数据操作优先用 **Daft**；Daft 做不到再用 **lance-ray**；**pylance 直接调用是最后手段**（目前锁在 <8，blob compaction 被禁用）。 +2. **数据 API 优先级**：所有数据操作优先用 **Daft**；Daft 做不到再用 **lance-ray**；**pylance 直接调用是最后手段**。当前固定为 pylance 8.0.0；blob compaction 仍需在单独改动中完成 lance-ray 兼容性和对象存储回归验证后再启用。 3. **行不丢弃**：analyze 阶段 manifest 里每个条目对应输出一行，失败用 `status` 列标记（`download_failed` / `decode_failed` / ...），分数结论置 null。合规场景必须能区分"内容有问题"和"根本没处理"。 4. **配置只走环境变量**，经 `config.py` 的 `env_int`/`env_bool`/`env_choice` 等辅助函数读取。注意：Daft 原生只读少数 `DAFT_*` 环境变量，新增 Daft 执行参数必须在 `storage/io.py::configure_daft_runner()` 里显式传给 `daft.set_execution_config()`，否则是死配置；S3 参数同理要传进 `daft_io_config()` 的 `S3Config`。 5. **写出受控**：`write_lance` 必须传 `LANCE_MAX_ROWS_PER_FILE` / `LANCE_MAX_BYTES_PER_FILE`；JSONL 写出前用 `coalesce_for_write()` 收敛分区，避免小文件。 diff --git a/README.md b/README.md index 1d8150c..e1abd7b 100644 --- a/README.md +++ b/README.md @@ -410,8 +410,8 @@ python -m multimodal_toolkit.workflow.manage \ |------|------|------| | Daft | 0.7.15 | 主执行引擎 | | daft-lance | 0.4.0 | `read_lance`、`write_lance`、`take_blobs`、`create_scalar_index`、`compact_files` | -| pylance | 7.0.0 | Lance dataset、blob v2、ANN scanner、delete、cleanup | -| lance-ray | 0.4.2 | 向量索引创建；写回路径暂缓 | +| pylance | 8.0.0 | Lance dataset、blob v2、ANN scanner、delete、cleanup | +| lance-ray | 0.4.2 | 向量索引创建；与 pylance 8 的分布式提交兼容性见下方限制 | | Ray | 2.55.1 | 由 lance-ray 引入；除非 `USE_RAY=1`，否则 Daft 走 native runner | Daft 默认 runner 是 `native`（本地多线程）。设置 `USE_RAY=1` 可把 Daft 相关步骤切到 Ray。 @@ -455,8 +455,12 @@ confidence 与 reason 字段在本地行中为 null。VLM 调用失败时仍保 **本地 Lance URI 已做端到端验证。** S3 上的 Lance 表读写由底层库覆盖，但在本 POC 中应作为一个独立的验证项对待。 -**blob v2 资产表的 compaction 暂时禁用。** -当前 `lance-ray 0.4.2` 的向量索引创建依赖 pylance 7.x，而 pylance 8.0.0 虽然修复了 blob v2 -compaction，却改动了 `lance-ray 0.4.2` 仍在调用的分布式索引 commit API。项目暂时锁定 -`pylance<8.0.0`，Stage 5 在删除行之后跳过 compaction。等升级到 pylance 8.x 并配套兼容的 -lance-ray 版本后再重新开启 blob v2 compaction。 +**pylance 8.0.0 与 lance-ray 0.4.2 存在已知的分布式索引限制。** +pylance 8.0.0 修复了 blob v2 compaction，但移除了 `lance-ray 0.4.2` 在分布式向量索引提交阶段 +调用的 `create_index_segment_builder()`。Stage 3 会捕获该错误并回退到 pylance 本地建索引， +所以索引功能仍可用，但大表会失去 Ray 分布式建索引的资源扩展能力。等配套兼容的 lance-ray +版本可用后，应移除这条回退限制。 + +**blob v2 资产表的 compaction 仍暂时禁用。** +本次依赖升级不改变 Stage 5 的表管理行为；删除行之后仍跳过 compaction。后续启用时需单独覆盖 +lance-ray、本地 Lance URI 和 MinIO/S3 的 Blob v2 回归验证，避免把版本对齐和行为变更混在一起。 diff --git a/docs/daft-tuning.md b/docs/daft-tuning.md index a4b4803..1990b52 100644 --- a/docs/daft-tuning.md +++ b/docs/daft-tuning.md @@ -112,7 +112,9 @@ CPU/网络 在下载下一批图片 | `LANCE_MAX_BYTES_PER_FILE` | `512MB` | 单个 Lance 数据文件最大字节数。 | | `DAFT_JSON_TARGET_FILESIZE` | `128MB` | JSONL 单文件目标大小。 | -为什么要管：碎文件会拖慢后续所有读取，而我们的 pylance 7 禁用了 blob compaction，写坏了没有事后补救，只能在写入侧控制。默认值一般不用动。 +为什么要管：碎文件会拖慢后续所有读取。虽然 pylance 已升级到 8.0.0，Stage 5 仍要等 +lance-ray 和对象存储路径完成独立回归验证后才会开启 blob compaction；在此之前仍应从写入侧 +控制文件布局。默认值一般不用动。 ### 3.7 观测（长任务建议必开） diff --git a/multimodal_toolkit/workflow/manage.py b/multimodal_toolkit/workflow/manage.py index 1a6091e..47b1631 100644 --- a/multimodal_toolkit/workflow/manage.py +++ b/multimodal_toolkit/workflow/manage.py @@ -1,15 +1,15 @@ -"""Stage 5: manage the lance asset table — delete rows by ingest_time range. +"""Stage 5：按 ingest_time 范围删除 Lance 资产表中的行。 -API priority (lance_ray preferred for table management): - delete → pylance ds.delete() (no Daft/lance-ray equivalent) - compact → disabled for blob v2 tables until pylance 8 is supported - cleanup → pylance ds.cleanup_old_versions() (no alternative) +表管理仍遵循项目的数据 API 优先级： + delete → pylance ds.delete() （Daft/lance-ray 没有等价 API） + compact → 暂不执行；升级 pylance 后仍需单独验证 lance-ray 和对象存储路径 + cleanup → pylance ds.cleanup_old_versions() （没有替代 API） - --before DATE delete rows where ingest_time < DATE - --after DATE delete rows where ingest_time > DATE + --before DATE 删除 ingest_time < DATE 的行 + --after DATE 删除 ingest_time > DATE 的行 -DATE format: ISO 8601, e.g. 2025-01-01 or 2025-01-01T00:00:00 -At least one bound must be provided; both can be combined for a date range. +DATE 使用 ISO 8601 格式，例如 2025-01-01 或 2025-01-01T00:00:00。 +至少传一个边界；两个边界可以组合成日期范围。 """ from __future__ import annotations @@ -35,16 +35,17 @@ def delete_by_date( clauses.append(f"ingest_time > timestamp '{after}'") filter_str = " AND ".join(clauses) - # delete: pylance only (no Daft/lance-ray equivalent) + # 删除目前只有 pylance 提供 API；这里保留直接调用，不能为了统一入口而绕过 + # Lance 自身的事务提交语义。 ds = lance.dataset(lance_uri, storage_options=lance_storage_options(lance_uri)) ds.delete(filter_str) print(f"[ok] deleted rows where: {filter_str}") - # Blob v2 tables cannot be compacted safely on pylance 7.x - # (lance-format/lance#7071). Keep compaction disabled until the project - # can move to pylance 8.x together with a compatible lance-ray release. - print("[warn] compaction skipped; enable after upgrading to pylance 8.x") + # pylance 8.0.0 已修复 Blob v2 compaction 的解码问题，但版本升级本身不等于 + # 分布式 compaction 链路已经验证。这里继续跳过，待后续改动同时覆盖 lance-ray、 + # 本地 Lance URI 和 MinIO/S3 后再开启，避免依赖升级悄悄扩大表管理行为。 + print("[warn] compaction skipped; enable after validating lance-ray and object storage") ds.cleanup_old_versions() diff --git a/pyproject.toml b/pyproject.toml index 44316a1..b4f614a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -19,7 +19,7 @@ dependencies = [ "pillow", "psutil", "pyarrow", - "pylance<8.0.0", + "pylance==8.0.0", "python-dotenv", "ray", "soundfile", diff --git a/tests/workflow/test_index_manage.py b/tests/workflow/test_index_manage.py index 8f67849..47b8589 100644 --- a/tests/workflow/test_index_manage.py +++ b/tests/workflow/test_index_manage.py @@ -1,9 +1,8 @@ -"""Tests for workflow/index.py and workflow/manage.py — local lance tables. +"""在本地 Lance 表上验证 workflow/index.py 和 workflow/manage.py。 -build_embedding_index is marked `ray` and excluded from the default run (see -pyproject) — run it explicitly with `pytest -m ray`. delete_by_date does not -compact while the project is pinned to pylance 7.x, so those tests stay in the -default suite. +build_embedding_index 的真实 Ray 路径不属于默认测试；需要时用 `pytest -m ray` +显式执行。pylance 已升级到 8.0.0，但 delete_by_date 仍有意不执行 compaction， +因此删除测试继续留在默认测试集。 """ from __future__ import annotations diff --git a/uv.lock b/uv.lock index cef60ab..0abe688 100644 --- a/uv.lock +++ b/uv.lock @@ -691,19 +691,19 @@ wheels = [ [[package]] name = "lance-namespace" -version = "0.7.7" +version = "0.8.6" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "lance-namespace-urllib3-client" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/06/5c/9822af615fc1bd3ee1073994696c739aecde377be32435ec3303aed1bc5d/lance_namespace-0.7.7.tar.gz", hash = "sha256:d00b525f2e26993a6c61668e798bca6c808605ab8a79f29f86a1a1af92d91ae2", size = 10754, upload-time = "2026-05-20T17:32:59.45Z" } +sdist = { url = "https://files.pythonhosted.org/packages/af/12/f7ab93b29be3edbf5fc3610714bf2d06088e7f4524bfb38dfd6852458b08/lance_namespace-0.8.6.tar.gz", hash = "sha256:18232e721c8188145f4ec9389cc2dfbeeabf54a619d94885ea1b3375bee9f4af", size = 11529, upload-time = "2026-06-12T17:36:41.651Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/11/43/186acc1156da20c351db196e2b6241b2453b16dc1b4cc8e0a626667ca471/lance_namespace-0.7.7-py3-none-any.whl", hash = "sha256:477a7ca6b5e1f673a2c9ba52f42d6e8e3ff7c27a601392a21eb90fba98d0309b", size = 12581, upload-time = "2026-05-20T17:32:57.389Z" }, + { url = "https://files.pythonhosted.org/packages/a0/1b/5b1668ee2dc8910965f390640359112a31157092fcf8e000b89c79b58708/lance_namespace-0.8.6-py3-none-any.whl", hash = "sha256:571eae34f9aad70e5b05020416c2860889b9ec82993ccd0eb015e7b39c3ea309", size = 13383, upload-time = "2026-06-12T17:36:43.456Z" }, ] [[package]] name = "lance-namespace-urllib3-client" -version = "0.7.7" +version = "0.8.6" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "pydantic" }, @@ -711,9 +711,9 @@ dependencies = [ { name = "typing-extensions" }, { name = "urllib3" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/07/95/38ab81ccc1e09beeecd8ddfc61b8bc73831dc5053db1e3f9021f64a4896b/lance_namespace_urllib3_client-0.7.7.tar.gz", hash = "sha256:4d8c066628c17c6a10cf643b51a7f7ae1bfb8a614d9cc54a5af38a4ba2b4b102", size = 202930, upload-time = "2026-05-20T17:32:58.308Z" } +sdist = { url = "https://files.pythonhosted.org/packages/c7/80/fb224b4a89c1c1638cde949cb6cce6c3aca7759effbfea46a3d9c3960b21/lance_namespace_urllib3_client-0.8.6.tar.gz", hash = "sha256:b6fb1d306e74a7576e5309919020be744527de484a63dbf5eed10f8b368548df", size = 228772, upload-time = "2026-06-12T17:36:42.609Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/35/96/5483e48e40433b1d078183c15a92c99e59a156041b0260e7f18ee34e7c08/lance_namespace_urllib3_client-0.7.7-py3-none-any.whl", hash = "sha256:9221c3e00fd89f0c811953d94b32d2ea527765280460a174f5872dc8a74c0ed6", size = 334767, upload-time = "2026-05-20T17:32:55.883Z" }, + { url = "https://files.pythonhosted.org/packages/c5/90/1e27de15cd1b16785a1c7312beb0a59e75c8344a815f600f58173a565bd1/lance_namespace_urllib3_client-0.8.6-py3-none-any.whl", hash = "sha256:9d78249c3fb15aa3d15d668f78f04a275af3d08d800a7027492f37996ac4968b", size = 369950, upload-time = "2026-06-12T17:36:40.438Z" }, ] [[package]] @@ -927,7 +927,7 @@ requires-dist = [ { name = "pillow" }, { name = "psutil" }, { name = "pyarrow" }, - { name = "pylance", specifier = "<8.0.0" }, + { name = "pylance", specifier = "==8.0.0" }, { name = "python-dotenv" }, { name = "ray" }, { name = "soundfile" }, @@ -1473,7 +1473,7 @@ wheels = [ [[package]] name = "pylance" -version = "7.0.0" +version = "8.0.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "lance-namespace" }, @@ -1481,12 +1481,12 @@ dependencies = [ { name = "pyarrow" }, ] wheels = [ - { url = "https://files.pythonhosted.org/packages/ac/ad/2f64921bf346e7075aef24a72595db44821724a3d89a9a92dd24e79632aa/pylance-7.0.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:98422021975be76e72b1572f41b8c9abb3bee5bdc9bfa5e9ce731110a65ed4d1", size = 62134146, upload-time = "2026-05-27T21:59:37.459Z" }, - { url = "https://files.pythonhosted.org/packages/73/1c/c5a01bee0160b55d9a98895cbd33091d038f0a0995b121ab72e629008d02/pylance-7.0.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4bec86ee5b6fbd8bfc493e653f0a1fba0303cfe5492b9b46fc25ab908edc7183", size = 65373684, upload-time = "2026-05-27T22:04:01.584Z" }, - { url = "https://files.pythonhosted.org/packages/eb/da/1fe8b8f7dbfe734d76af76acc994fc360a0d0c79a4874ef69f5a72a58fe3/pylance-7.0.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:881491432c53184e52f8d1db8d5f872f39a03f36fb104bec77b33d379519d8b5", size = 69458555, upload-time = "2026-05-27T22:16:50.567Z" }, - { url = "https://files.pythonhosted.org/packages/76/f0/dd505cf3fd0226ab9d94759acd713125af1d3bfacfd80bbd52e3b9f89509/pylance-7.0.0-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:18453999e7fff4f76b16d6b7882c9df0628bd142ff95e2461bd7dd5ee3fe0af3", size = 65394430, upload-time = "2026-05-27T22:05:30.923Z" }, - { url = "https://files.pythonhosted.org/packages/17/ba/2357b81034f28eb00790e258ed140289a6a887a7468ca9df6349fd186b27/pylance-7.0.0-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:04a58051d408c60fe76d41a220dcaf8fea8fb6d1aa0ca78a709b60bc3cc8d19a", size = 69473470, upload-time = "2026-05-27T22:17:18.935Z" }, - { url = "https://files.pythonhosted.org/packages/1f/ec/5c00b6303a67d787f9475141832cbdc513d674ac3dcaeef8a7b169905e65/pylance-7.0.0-cp39-abi3-win_amd64.whl", hash = "sha256:467d4864af047eaab4e1370e2f1e88e2c6f507c079874421116cb41d78bc3629", size = 74792863, upload-time = "2026-05-27T22:19:23.875Z" }, + { url = "https://files.pythonhosted.org/packages/3b/99/03db9f4f723353502d2b5f5d8a0eb9a30f0dc41a8cc036ac5fa486471223/pylance-8.0.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:769cb38e43a533fdffb87a804043f6a37793eb77d4acbc14cffe9e45dc8bd90b", size = 67235437, upload-time = "2026-07-01T15:09:09.519Z" }, + { url = "https://files.pythonhosted.org/packages/48/af/8f8f8b9c70200e3ab99d8785c26ba691d0ff475cae27f7620618dfc8f3b6/pylance-8.0.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:597277a993d159dfcf58230958f1c5f6edebca8693fbd3502129be951ec6927e", size = 70558570, upload-time = "2026-07-01T15:13:50.95Z" }, + { url = "https://files.pythonhosted.org/packages/ca/4c/b10b02a7891753ec1d4a846b8cc001fbafb4bf7c48c2bda5303cf57c1b8a/pylance-8.0.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2c8823134a09ff311b0a963227bdfc160c686910944a373af357cd5b3fa70b5c", size = 74747275, upload-time = "2026-07-01T15:27:09.817Z" }, + { url = "https://files.pythonhosted.org/packages/4c/56/09602b664d8036aa0105ffdb94189c23f9fde4ce71963f4a98f05fda526d/pylance-8.0.0-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:28c98c6ae99edc76622e2af331e99da4d57c9f44c32ebba2872a8e60dee9f7f8", size = 70578246, upload-time = "2026-07-01T15:14:53.143Z" }, + { url = "https://files.pythonhosted.org/packages/4b/1a/9f49850dfee7b1e763f358ff7477a1aa27228d1e40f907020f13ea13b3e1/pylance-8.0.0-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:0b779a7465424de04975cedc774b1eadff8638e0f96f6f19245e9bea332e1e62", size = 74708654, upload-time = "2026-07-01T15:27:54.326Z" }, + { url = "https://files.pythonhosted.org/packages/77/5d/1876b43e680b9afb0d827af032bd72d3f70a9eaf68cb8f5f4286b20a613e/pylance-8.0.0-cp39-abi3-win_amd64.whl", hash = "sha256:ea2e35722f546eb2e056eafb1cc39ced5f353157653516a1472f1127bfa92eec", size = 80511928, upload-time = "2026-07-01T15:28:56.53Z" }, ] [[package]]