Restore retained checkpoints across compatible nodes

contracts/agents-api/node-generation-protocol.md

@@ -73,9 +73,9 @@ Node startup and generation loading validate complete Provider operation declara
 
 ## Generation control
 
-A node without generation management serves only its enrolled generation, with fixed configuration, and receives no preparation or retention frames. A generation-managing node prepares the target generation Core announces in `welcome` and `heartbeat_ack` and keeps serving its durable serving generation while it does; target preparation is independent of the serving provider's readiness.
+A node without generation management serves only its enrolled generation, with fixed configuration, and receives no preparation or retention frames. It can report that exact generation in `Health.Generations`; a checkpoint-capable provider must do so, including its verified checkpoint qualification. `provider_ready` and the reported state must agree. A generation-managing node prepares the target generation Core announces in `welcome` and `heartbeat_ack` and keeps serving its durable serving generation while it does; target preparation is independent of the serving provider's readiness.
 
-Its `hello` and heartbeats carry at most eight generation observations. Each names a positive signed-64-bit generation, its lowercase SHA-256 specification digest, a `ready`, `preparing` or `failed` state and an optional fixed diagnostic. The target and serving generations come first; other records rotate fairly. Eight bounds one message, not the number of generations a node may keep. An omitted observation never authorizes deletion or implies absence.
+Its `hello` and heartbeats carry at most eight generation observations. Each names a positive signed-64-bit generation, its lowercase SHA-256 specification digest, a `ready`, `preparing` or `failed` state and an optional fixed diagnostic. The target and serving generations come first; other records rotate fairly. Eight bounds one message, not the number of generations a node may keep. An omitted observation never authorizes deletion or implies absence. A ready checkpoint-capable generation includes `checkpoint` with nonempty opaque `artifact_domain` and `execution_class` tokens, each at most 256 bytes. Unsupported providers omit it; preparing or failed generations never advertise it. Core persists the declaration with the authenticated connection and generation, and uses it only while that generation is ready on the current online connection. The [Provider contract](../../docs/sandbox-provider.md#checkpoint-transfer) defines archive and execution compatibility.
 
 Retention uses its own bounded exchange. A `retention` request names at most eight local `(generation, specification_digest)` references, a UUID, a sequence that increases by one, the current `connection_id` and the owner epoch. The `retention_ack` must match the complete pending request, entry order and identity included, and give an explicit boolean `keep` for every entry. One exchange is pending per connection, and a disconnect discards it. An unsolicited, replayed, stale, partial or mixed acknowledgement deletes nothing. Retention traffic never uses the Provider request queue.
 

contracts/agents-api/sandbox-deployment.md

@@ -142,7 +142,7 @@ A node is online while it is connected under the current owner epoch with a hear
 
 Each node adds `rollout: {state, ready_generation, diagnostic?}`, where `ready_generation` is the nullable durable serving pin and `diagnostic` a fixed code for the target generation; allocation items add `deployment_generation`. Poll every five seconds only while `rollout.state` is `preparing` or `reset` is not null; old Sessions and failed, update-required or offline nodes alone do not keep polling active.
 
-Node-backed creation validates the target deployment and commits the Session, pending Environment and any initial input without reserving compute. A full, offline or preparing fleet leaves that accepted work waiting; an unconfigured deployment, reset or unsupported combination still rejects admission. The common scheduler uses bounded rotating scans of unplaced demand, with pages ordered by recorded time and Environment ID and a fixed time boundary for each scan so continuous arrivals cannot prevent it from revisiting older work. It checks online presence, exact serving-generation readiness, address, shared capacity and the selected generation's Harness/filesystem compatibility, then prefers the newest qualifying pin. Placement is immutable once reserved. A bounded scan skips temporarily unavailable demand and resumes after a restart. Existing suspended allocations restore on their original node through the same capacity lock; this is not a global fairness guarantee across hot restores and unplaced work. Input retains its [original five-minute deadline](./environments.md#reservations), including time waiting for capacity.
+Node-backed creation validates the target deployment and commits the Session, pending Environment and any initial input without reserving compute. A full, offline or preparing fleet leaves that accepted work waiting; an unconfigured deployment, reset or unsupported combination still rejects admission. The common scheduler uses bounded rotating scans of unplaced demand, with pages ordered by recorded time and Environment ID and a fixed time boundary for each scan so continuous arrivals cannot prevent it from revisiting older work. It checks online presence, exact serving-generation readiness, address, shared capacity and the selected generation's Harness/filesystem compatibility, then prefers the newest qualifying pin. Placement is immutable once reserved until confirmed release or a settled checkpoint transfer. A bounded scan skips temporarily unavailable demand and resumes after a restart. Existing suspended allocations reserve eligible checkpoint capacity through the same lock under the [checkpoint transfer contract](../../docs/sandbox-provider.md#checkpoint-transfer); this is not a global fairness guarantee across hot restores and unplaced work. Input retains its [original five-minute deadline](./environments.md#reservations), including time waiting for capacity.
 
 ## Reset
 

contracts/agents-api/zh/node-generation-protocol.md

@@ -1,7 +1,7 @@
 ---
 title: "沙箱节点协议"
 source: contracts/agents-api/node-generation-protocol.md
-source_hash: 3fa1d9257acefa2d144f6e76ac5c2d3ff1d074ebb2b21fb48a2f79ef58ad6073
+source_hash: fb446e91c3df3c9e397e5e9abb794245b0ae11d81176879207718680e8ab4173
 ---
 
 沙箱节点在其主机上运行 Docker 或 microsandbox Provider，并通过一个 WebSocket 与 Core 相连。Core 通过该连接发送 Provider 操作；节点针对本地 Provider 执行这些操作，并报告就绪状态、主机测量值及其持有的部署代次。Core 始终是唯一的生命周期所有者：节点绝不重试变更操作或调度工作。帧和校验器位于 [`services/core/internal/sandbox/node`](https://github.com/MiniMax-AI/OpenAgentCore/tree/main/services/core/internal/sandbox/node)（`wire.go`、`generation_wire.go`）；节点用于注册和读取配置的 HTTP 路由位于[机器连接 API](machine-api.md#node-routes)。
@@ -75,9 +75,9 @@ Core 发送包含以下内容的 `request` 帧：
 
 ## 代次控制 {#generation-control}
 
-未启用代次管理的节点只服务其登记的代次，配置固定，并且不会收到准备或保留帧。支持代次管理的节点会准备 Core 在 `welcome` 和 `heartbeat_ack` 中通告的目标代次，并在此期间继续服务其持久化的服务代次；目标代次的准备独立于服务 Provider 的就绪状态。
+未启用代次管理的节点只服务其登记的代次，配置固定，并且不会收到准备或保留帧。它可以在 `Health.Generations` 报告该精确代次；支持检查点的 Provider 必须报告，并包含已验证的检查点资格。`provider_ready` 必须与报告的状态一致。支持代次管理的节点会准备 Core 在 `welcome` 和 `heartbeat_ack` 中通告的目标代次，并在此期间继续服务其持久化的服务代次；目标代次的准备独立于服务 Provider 的就绪状态。
 
-节点的 `hello` 和心跳最多携带八条代次观察记录。每条记录指定一个正值的有符号 64 位代次编号、其小写 SHA-256 规范摘要、`ready`、`preparing` 或 `failed` 状态，以及可选的固定诊断信息。目标代次和服务代次的记录排在前面，其余记录公平轮换。八条记录限制的是单条消息，而不是节点可保留的代次数量。省略某条观察记录绝不会授权删除，也不会暗示不存在。
+节点的 `hello` 和心跳最多携带八条代次观察记录。每条记录指定一个正值的有符号 64 位代次编号、其小写 SHA-256 规范摘要、`ready`、`preparing` 或 `failed` 状态，以及可选的固定诊断信息。目标代次和服务代次的记录排在前面，其余记录公平轮换。八条记录限制的是单条消息，而不是节点可保留的代次数量。省略某条观察记录绝不会授权删除，也不会暗示不存在。 支持检查点的 ready 代次携带 `checkpoint`，包含非空且不透明的 `artifact_domain` 和 `execution_class` token，各最多 256 字节。不支持的 Provider 省略该字段；preparing 或 failed 代次不声明它。Core 将声明与已认证连接和代次一起持久化，仅在当前在线连接的该代次仍 ready 时使用。[Provider 契约](../../../docs/zh/sandbox-provider.md#checkpoint-transfer) 定义归档与执行兼容性。
 
 保留使用独立且有界的交换。`retention` 请求最多指定八个本地 `(generation, specification_digest)` 引用、一个 UUID、一个每次递增 1 的 `sequence`、当前 `connection_id` 和所有者 epoch。`retention_ack` 必须与完整的待处理请求匹配，包括条目顺序和身份信息，并为每个条目给出显式布尔值 `keep`。每条连接只能有一个交换处于待处理状态，断连会将其丢弃。任何未经请求、重放、过期、不完整或混合的确认都不会删除任何内容。保留流量从不使用 Provider 请求队列。
 

contracts/agents-api/zh/sandbox-deployment.md

@@ -1,7 +1,7 @@
 ---
 title: "沙箱部署"
 source: contracts/agents-api/sandbox-deployment.md
-source_hash: 245f1791cf2e42b3a40c91aa0df4f390a44709db5e0ffbcae1e12dc4170344fe
+source_hash: 537ae957a77d625b3adb7828156352a661ae09270b57b174406c6db159a957bd
 ---
 
 沙箱部署为 Core 管理的 `openai_hosted` 执行选择 Sandbox Provider、每个沙箱的资源以及不可变的 Runtime 发行版。PostgreSQL 为每个安装维护一个当前有效选择；Web 和 Core API 写入同一配置。节点文件保存其已安装副本和特定于主机的路径，且不能覆盖其资源或 Runtime。该选择独立于 Harness。部署可以保持未配置状态，没有节点；此时它拒绝托管准入。
@@ -145,7 +145,7 @@ POST 会在持久保存候选配置之前对其进行验证，并且不会创建
 
 每个节点会添加 `rollout: {state, ready_generation, diagnostic?}`，其中 `ready_generation` 是可为 null 的持久服务固定状态，`diagnostic` 是目标代次的固定代码；分配项会添加 `deployment_generation`。仅当 `rollout.state` 为 `preparing` 或 `reset` 非 null 时，才每五秒轮询一次；旧 Session 以及失败、需要更新或离线的节点本身均不会使轮询保持活动状态。
 
-节点模式创建先验证目标部署，提交 Session、pending Environment 及初始输入，但不预留计算资源。节点全部已满、离线或准备中时，已接受的工作继续等待；未配置部署、reset 或不支持的组合仍拒绝准入。共同调度器对尚未放置的需求进行有界循环扫描，分页按需求记录时间和 Environment ID 排序，每轮使用固定时间上界，避免持续的新需求阻止重新访问较早的工作。它检查在线状态、精确服务代次就绪情况、地址、共享容量和所选代次的 Harness/文件系统兼容性，然后优先选择最新的合格固定代次。预留后的 placement 保持不可变。有界扫描跳过暂时不可调度的需求，并在重启后继续。已有 suspended allocation 通过同一容量锁在原节点恢复；这不构成热恢复和未放置工作之间的全局公平性保证。输入保留[原始五分钟期限](./environments.md#reservations)，等待容量的时间也计入其中。
+节点模式创建先验证目标部署，提交 Session、pending Environment 及初始输入，但不预留计算资源。节点全部已满、离线或准备中时，已接受的工作继续等待；未配置部署、reset 或不支持的组合仍拒绝准入。共同调度器对尚未放置的需求进行有界循环扫描，分页按需求记录时间和 Environment ID 排序，每轮使用固定时间上界，避免持续的新需求阻止重新访问较早的工作。它检查在线状态、精确服务代次就绪情况、地址、共享容量和所选代次的 Harness/文件系统兼容性，然后优先选择最新的合格固定代次。预留后的 placement 保持不可变，直到确认释放或已结清的检查点转移。有界扫描跳过暂时不可调度的需求，并在重启后继续。已有 suspended allocation 按[检查点转移契约](../../../docs/zh/sandbox-provider.md#checkpoint-transfer)，通过同一容量锁预留符合条件的检查点容量；这不构成热恢复和未放置工作之间的全局公平性保证。输入保留[原始五分钟期限](./environments.md#reservations)，等待容量的时间也计入其中。
 
 ## 重置 {#reset}
 

deploy/node/node_generations.py

@@ -239,6 +239,9 @@ def validate_preparation_plan(root, plan, base, installer):
         if not any(paths == [release / name for name in installer.MICRO] for release in (root, root / "releases" / source)):
             raise installer.InstallError("Preparation artifacts are outside their immutable release")
         expected_home = generation_home(root, plan, base, installer)
+        if micro["checkpoint_root"] != base["native"]["checkpoint_root"]:
+            raise installer.InstallError("Preparation checkpoint store differs")
+        installer.prepare_checkpoint_root(Path(micro["checkpoint_root"]), initialize=False)
         if Path(micro["runtime_home"]) != expected_home:
             raise installer.InstallError("Preparation native store differs")
         for path in paths + [expected_home]:
@@ -399,6 +402,7 @@ def prepare(args, installer):
                 raise installer.RuntimeDownloadError("Runtime release provenance differs") from error
             if args.provider == "microsandbox":
                 args.runtime_home = Path(value["native"]["runtime_home"]) if value else generation_home(root, args.configuration, base, installer)
+                args.checkpoint_root = Path(base["native"]["checkpoint_root"])
             if value is None:
                 value = installer.provider_config(root / "releases" / runtime["source_commit"], args, runtime["image_id"])
             if preparation is None:

deploy/node/node_install.py

@@ -279,6 +279,49 @@ def micro_home(installation_id):
     return directory
 
 
+def checkpoint_root(args):
+    return getattr(args, "checkpoint_root", None) or Path.home() / ".oac/checkpoints" / hashlib.sha256(args.installation_id.encode()).hexdigest()[:12]
+
+
+def prepare_checkpoint_root(directory, *, initialize=True):
+    directory = Path(directory)
+    if not directory.is_absolute() or directory.resolve() != directory:
+        raise InstallError("Checkpoint root must be a canonical absolute directory")
+    if not initialize and not directory.is_dir():
+        raise InstallError("Retained checkpoint root is missing")
+    safe_directory(directory)
+    marker = directory / ".oac-checkpoint-store"
+    if not existing_file(marker):
+        if not initialize:
+            raise InstallError("Retained checkpoint store identity is missing")
+        if any(directory.iterdir()):
+            raise InstallError("Checkpoint root contains unowned state")
+        try:
+            descriptor = os.open(marker, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
+        except FileExistsError:
+            pass
+        else:
+            with os.fdopen(descriptor, "w") as output:
+                output.write(str(uuid.uuid4()) + "\n")
+                output.flush()
+                os.fsync(output.fileno())
+    existing_file(marker)
+    value = marker.read_text()
+    try:
+        identity = uuid.UUID(value.rstrip("\n"))
+    except ValueError:
+        raise InstallError("Invalid checkpoint store identity") from None
+    if identity.int == 0 or value != str(identity) + "\n":
+        raise InstallError("Invalid checkpoint store identity")
+    with marker.open("rb") as source:
+        os.fsync(source.fileno())
+    descriptor = os.open(directory, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW)
+    try:
+        os.fsync(descriptor)
+    finally:
+        os.close(descriptor)
+
+
 def provider_config(root, args, runtime_image):
     result = {"installation_id": args.installation_id, "provider": args.provider, "core_url": args.core_url + "/api/v1",
               "specification": args.configuration["specification"], "generation": args.configuration["generation"]}
@@ -294,6 +337,7 @@ def provider_config(root, args, runtime_image):
         result["native"] = {
             "helper_path": str(root / MICRO[0]), "runtime_path": str(root / MICRO[1]), "firmware_path": str(root / MICRO[2]),
             "runtime_home": str(getattr(args, "runtime_home", micro_home(args.installation_id))),
+            "checkpoint_root": str(checkpoint_root(args)),
             "network": {"default_egress": "deny", "default_ingress": "deny", "rules": core_rules + [
                 {"action": "allow", "direction": "egress", "destination": "public"},
                 {"action": "allow", "direction": "egress", "destination": "host", "protocol": "udp", "port": "53"},
@@ -348,8 +392,17 @@ def configure_node(root, args, token):
     args.configuration = node_spec.fetch(args, token, retained, open_request, allow_enrollment=not (root / "registered.json").exists())
     args.provider = args.configuration["provider"]
     preflight(args.provider)
+    if args.provider != "microsandbox" and getattr(args, "checkpoint_root", None) is not None:
+        raise InstallError("Checkpoint root applies only to microsandbox nodes")
     if args.provider == "microsandbox":
+        retained_provider = private_json(root / "provider.json")
+        if retained_provider:
+            retained_root = Path(retained_provider["native"]["checkpoint_root"])
+            if getattr(args, "checkpoint_root", None) not in (None, retained_root):
+                raise InstallError("Retained checkpoint root differs; preserve its state")
+            args.checkpoint_root = retained_root
         runtime_home = micro_home(args.installation_id)
+        prepare_checkpoint_root(checkpoint_root(args), initialize=retained_provider is None)
         safe_directory(runtime_home)
         owner = runtime_home / "oac-installation.json"
         if not owner.exists() and any(runtime_home.iterdir()):
@@ -1244,6 +1297,7 @@ def main(argv=None):
     parser.add_argument("--core-url", type=origin)
     parser.add_argument("--provider", choices=("docker", "microsandbox"), help="Optional assertion; Core owns provider selection")
     parser.add_argument("--installation-id", required=True)
+    parser.add_argument("--checkpoint-root", type=Path, help="Private microsandbox checkpoint directory, shared across compatible nodes when configured")
     parser.add_argument("--enrollment-token-stdin", action="store_true", help="Read the one-time enrollment token from standard input")
     parser.add_argument("--generation-action", choices=("prepare", "collect"), help=argparse.SUPPRESS)
     parser.add_argument("--generation", type=int, help=argparse.SUPPRESS)
@@ -1272,7 +1326,7 @@ def main(argv=None):
     if os.geteuid() != 0:
         raise InstallError("Node installation and removal require root. Run this command with sudo.")
     if args.uninstall:
-        if args.source_url or args.bundle or args.core_url or args.provider or args.enrollment_token_stdin:
+        if args.source_url or args.bundle or args.core_url or args.provider or args.enrollment_token_stdin or args.checkpoint_root:
             parser.error("--uninstall takes only --installation-id and --force")
         uninstall_system(args)
         return