Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions apps/desktop/loopx-control-plane/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,55 @@ cannot qualify a broken executable. Core also qualifies non-editable wheel
fingerprints: a local reuse fence, not publisher or source attestation. Older
CLIs without that readback can use the automatic App-owned fallback.

When the App selects a different runtime, the standard macOS status and Chat
LaunchAgents must load that executable too. Restarting an unchanged LaunchAgent
only starts its previous runtime again. Before opening the workspace, the App
uses Core's existing `update activate` owner to align the pair, preserving their
registry, runtime root, accounts, tool arguments and log configuration. The
user's CLI command and installation remain unchanged. Already matching services
are reused without another activation command or restart.

The transition holds the existing Chat ingress fence and waits for accepted
work before changing either service. It verifies the loaded manager arguments,
qualifies the selected executable, reloads the pair and checks both HTTP runtime
identities. A failed start restores the original configuration; a private,
locked handoff journal recovers interrupted transitions. Custom shell effects,
changed manager configurations and unrelated service labels are not rewritten.
Archive CLI updates recognize the runtime binding and move the same pair to the
new CLI runtime; other managed services retain their existing restart behavior.

版本选择与系统服务启动使用同一个运行时。已有请求会继续执行;只有请求完成且原服务
来源可验证时才切换。切换保留原数据、账号和启动参数,并确认两个 HTTP 服务都使用
目标版本;失败恢复原配置,中断由本地记录恢复。已匹配的服务直接复用,不反复重启。

If alignment cannot complete, the native supervisor automatically retries with
increasing delays (2, 4, 8, 16, 32, then at most 60 seconds between attempts).
The cooldown starts after an attempt finishes; it does not expire while a slow
attempt is still running. Each attempt rediscovers the runtime and reuses the
same manager/Chat fences and interrupted-handoff recovery. There is no permanent
retry-count latch. Once the services qualify, the workspace opens without a
click or reload. The startup screen shows the next automatic retry; waiting is
visually idle, with details available in **Recovery & updates**. **Retry now**
is optional and resets the delay through the native supervisor, preserving the
selected runtime. Pending Chat work retains its existing automatic retry.
Before any stop or write, alignment verifies the original literal launcher,
the actual `serve-status`/`chat` command position and its loopback endpoint;
a standard label alone never qualifies a custom service. Interrupted handoffs
apply the same role and unchanged-data-route checks before restoring services.

恢复是默认流程:暂时切换失败会自动退避重试,中断的切换会先恢复原服务。服务就绪后
自动进入工作区,无需点击按钮;「立即重试」只是跳过等待的可选入口。自定义来源仍需
可验证,已有请求仍受保护;检查更新、安装重启和显式运行时选择保留原有边界。旧 Core
尚不支持对齐时会继续检测可用运行时,不回退到反复终止旧服务。

开发诊断可使用 `loopx --format json update activate --service all
--runtime-executable /absolute/path/to/loopx`:这是会切换服务的操作,保留原工作区和
请求保护,不安装软件或更改终端 CLI。结果中的 `runtime_identity` 与
`readiness_verified` 分别表示目标身份与真实 HTTP 验证;安装检查成功不等于工作区
资源齐全。`service_runtime_route_unverified` 表示配置来源无法验证,
`service_runtime_alignment_failed` 表示切换失败并尝试恢复,
`service_runtime_rollback_failed` 表示仍需恢复服务。不要删除工作区数据来处理这些错误。

Wheel qualification has a separate ten-second subprocess deadline because it
starts Core, checks the TypeScript runtime and hashes package contents. The
one-second environment/version probes remain unchanged. A slow valid wheel can
Expand Down Expand Up @@ -623,6 +672,26 @@ disposable registry does not isolate machine-wide usage reporting; this
synthetic run must not contribute adoption or installation telemetry. The
opt-out applies only to this test invocation and leaves user consent unchanged.

The managed-service regression additionally exercises real launchd and both
HTTP backends, switching an explicitly selected older Core to the current
checkout after `npm ci && npm run build:chat` in the dashboard directory:

```sh
LOOPX_USAGE_PING=0 LOOPX_TEST_OLD_RUNTIME=/absolute/path/to/older/loopx \
uv run --extra test python -m pytest tests/test_managed_service_runtime.py -q -s
```

Run from the checkout root. Only test service labels are redirected to unique
disposable labels; the manager, Core processes, HTTP and stores are real. The
fixture uses temporary registry/account directories and ports, removes its own
services afterwards, and never unloads the machine's standard services. It
checks the target identity, original registry and repeated alignment without
PID changes, including a real manager failure followed by rollback and successful
recovery. Unit cases cover busy Chat, changed configuration, unsupported shells,
failed startup/rollback and interrupted handoff. Native tests cover exact
identity/readiness receipts, automatic recovery without user input, capped
backoff after slow attempts, and preservation of update admission.

This installs only in the disposable root. The second test uses an explicit
local launch pin to route the production supervisor's two services to its own
fresh empty registry and data directory. It checks both artifact identities,
Expand Down
3 changes: 3 additions & 0 deletions apps/desktop/loopx-control-plane/src-tauri/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,9 @@ fn maintenance_origin(url: &Url) -> String {
// echo verbatim into the boot surface, where it names the recovery panel's
// actionable diagnostics instead of a misleading fixed message.
fn boot_failure_message(error: &str) -> String {
if services::runtime_alignment_error(error) {
return "正在自动恢复本地服务,完成后会打开工作区。".to_string();
}
if error == "chat_work_pending" {
return "管家正在处理请求,完成后会自动连接工作区。".to_string();
}
Expand Down
159 changes: 157 additions & 2 deletions apps/desktop/loopx-control-plane/src-tauri/src/maintenance.rs
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ pub struct Maintenance {
pending: Mutex<Option<(String, Update)>>,
last_failure: Mutex<Value>,
runtime_retry: Mutex<RuntimeRetry>,
service_retry: Mutex<ServiceRetry>,
install_journal_discarded: AtomicBool,
environment_cache: Mutex<Option<(Instant, Value)>>,
startup_started: std::sync::OnceLock<Instant>,
Expand All @@ -53,6 +54,26 @@ struct RuntimeRetry {
last_attempt: Option<Instant>,
}

#[derive(Default)]
struct ServiceRetry {
failures: u32,
next_attempt: Option<Instant>,
}

impl ServiceRetry {
fn failed(&mut self, now: Instant) {
let delay = Duration::from_secs((2_u64 << self.failures.min(5)).min(60));
self.failures = self.failures.saturating_add(1);
self.next_attempt = Some(now + delay);
}

fn remaining(&self, now: Instant) -> Duration {
self.next_attempt
.map(|next| next.saturating_duration_since(now))
.unwrap_or_default()
}
}

impl RuntimeRetry {
fn admit(&mut self, now: Instant) -> bool {
if self.attempts >= 3
Expand All @@ -72,7 +93,17 @@ impl Maintenance {
if self.incomplete_app_installation.load(Ordering::Acquire) {
json!({"phase":"error", "details":{"code":"app_install_incomplete"}})
} else {
self.snapshot.lock().unwrap().clone()
let mut snapshot = self.snapshot.lock().unwrap().clone();
if snapshot["phase"] == "service_error"
&& crate::services::runtime_alignment_error(
snapshot["details"]["code"].as_str().unwrap_or(""),
)
{
let retry = self.service_retry.lock().unwrap();
snapshot["details"]["retry_after_ms"] =
json!(retry.remaining(Instant::now()).as_millis() as u64);
}
snapshot
}
}

Expand Down Expand Up @@ -395,6 +426,14 @@ impl Maintenance {
fn reconcile_services<T>(
&self,
start: impl FnOnce() -> Result<T, String>,
) -> Result<Option<T>, String> {
self.reconcile_services_with_clock(Instant::now, start)
}

fn reconcile_services_with_clock<T>(
&self,
now: impl Fn() -> Instant,
start: impl FnOnce() -> Result<T, String>,
) -> Result<Option<T>, String> {
if self.busy.load(Ordering::Acquire)
|| self.incomplete_app_installation.load(Ordering::Acquire)
Expand All @@ -419,8 +458,21 @@ impl Maintenance {
if matches!(phase.as_str(), "available" | "restart_required") {
return Ok(None);
}
// A failed alignment owes automatic recovery, not a user click. Bound
// repeated attempts without removing the Core's route/Chat fences.
if phase == "service_error" {
let retry = self.service_retry.lock().unwrap();
if !retry.remaining(now()).is_zero() {
return Ok(None);
}
if retry.next_attempt.is_some() {
drop(retry);
self.publish("connecting", json!({}));
}
}
match start() {
Ok(services) => {
*self.service_retry.lock().unwrap() = ServiceRetry::default();
self.publish("ready", json!({}));
Ok(Some(services))
}
Expand All @@ -436,9 +488,25 @@ impl Maintenance {
"chat_work_pending" => "chat_work_pending",
"chat_state_unverified" => "chat_state_unverified",
"chat_service_unverified" => "chat_service_unverified",
"service_runtime_alignment_unavailable" => {
"service_runtime_alignment_unavailable"
}
"service_runtime_alignment_failed" => "service_runtime_alignment_failed",
"service_runtime_route_unverified" => "service_runtime_route_unverified",
"service_runtime_rollback_failed" => "service_runtime_rollback_failed",
_ => "service_start_failed",
};
self.publish("service_error", json!({"code":code}));
let mut details = json!({"code":code});
if crate::services::runtime_alignment_error(code) {
let failed_at = now();
let mut retry = self.service_retry.lock().unwrap();
retry.failed(failed_at);
details["retry_after_ms"] =
json!(retry.remaining(failed_at).as_millis() as u64);
} else {
*self.service_retry.lock().unwrap() = ServiceRetry::default();
}
self.publish("service_error", details);
}
Err(error)
}
Expand Down Expand Up @@ -657,6 +725,19 @@ pub async fn desktop_update(
action: String,
channel: String,
) -> Result<Value, String> {
if action == "reconnect" {
let state = app.state::<Maintenance>();
let _guard = state.acquire()?;
let _supervision = state.supervision.lock().await;
allow_action(
state.snapshot.lock().unwrap()["phase"]
.as_str()
.unwrap_or(""),
&action,
)?;
*state.service_retry.lock().unwrap() = ServiceRetry::default();
return Ok(state.publish("connecting", json!({})));
}
if cfg!(dev) || !app_update_platform_supported() {
return Err("platform_update_not_supported".into());
}
Expand Down Expand Up @@ -2536,6 +2617,80 @@ mod tests {
}
}

#[test]
fn failed_alignment_recovers_automatically_after_backoff() {
for code in [
"service_runtime_alignment_unavailable",
"service_runtime_route_unverified",
"service_runtime_alignment_failed",
"service_runtime_rollback_failed",
] {
let state = Maintenance::default();
let clock = std::cell::Cell::new(Instant::now());
for delay in [2, 4, 8, 16, 32, 60, 60] {
assert_eq!(
state.reconcile_services_with_clock::<()>(|| clock.get(), || Err(code.into())),
Err(code.into())
);
assert_eq!(
state.snapshot.lock().unwrap()["details"]["retry_after_ms"],
delay * 1000
);
clock.set(clock.get() + Duration::from_secs(delay) - Duration::from_millis(1));
assert!(state
.reconcile_services_with_clock::<()>(
|| clock.get(),
|| panic!("cooldown must not touch services")
)
.unwrap()
.is_none());
clock.set(clock.get() + Duration::from_millis(1));
}
// No reconnect, reload or user action between failure and success.
assert_eq!(
state
.reconcile_services_with_clock(|| clock.get(), || Ok(42))
.unwrap(),
Some(42)
);
assert_eq!(state.startup_snapshot()["phase"], "ready");
assert!(state.service_retry.lock().unwrap().next_attempt.is_none());
}
}

#[test]
fn recovery_cooldown_starts_after_attempt_and_keeps_update_gates() {
let state = Maintenance::default();
let clock = std::cell::Cell::new(Instant::now());
assert!(state
.reconcile_services_with_clock::<()>(
|| clock.get(),
|| {
clock.set(clock.get() + Duration::from_secs(70));
Err("service_runtime_alignment_failed".into())
}
)
.is_err());
assert!(state
.reconcile_services_with_clock::<()>(
|| clock.get(),
|| panic!("no immediate retry after slow IO")
)
.unwrap()
.is_none());
clock.set(clock.get() + Duration::from_secs(2));
for phase in ["available", "restart_required"] {
state.publish(phase, json!({}));
assert!(state
.reconcile_services_with_clock::<()>(
|| clock.get(),
|| panic!("automatic recovery must preserve update admission")
)
.unwrap()
.is_none());
}
}

#[test]
fn only_verified_previous_apps_keep_the_safe_restart_promise() {
// Verified App bundle + pairing runtime: safe-restart class, journal
Expand Down
Loading
Loading