diff --git a/apps/ai-providers/host-service/MEDIA.md b/apps/ai-providers/host-service/MEDIA.md index b04a5012b..0a910f7dd 100644 --- a/apps/ai-providers/host-service/MEDIA.md +++ b/apps/ai-providers/host-service/MEDIA.md @@ -54,6 +54,8 @@ one place only. Prompts and text are UTF-8 bytes for input limits. | `model.video.cancel` | `job` | Confirmed cancellation of a queued remote job, or an error | | `model.capabilities` | `{}` | `configured` flags, limits and lifecycle note | | `model.budget` | `{}` | Existing call/token budget plus `media` resource usage and limits | +| `model.transcribe_capability` (Windows) | `{}` | `schema:1`, `supported`, `configured`; no provider identity or credential | +| `model.transcribe` (Windows, foreground) | `audio:{app_path}`, a WAV inside the caller's app storage | Trimmed `text`, `duration_seconds` | Image sizes are `512x512`, `1024x1024` (default), `1536x1024`, and `1024x1536`. The OpenAI route rejects `512x512` and `seed` before submission; @@ -61,6 +63,12 @@ the MiniMax route supports both. Only one image per request is supported. Generated PNG/JPEG dimensions must match the request. Audio output is MP3; apps should disclose that the voice is AI-generated. Duration is omitted when the provider does not report it; the host does not invent a value. +Transcription accepts only a 16 kHz mono PCM16 WAV of at most 1 MiB and +30 seconds, from the calling app's own storage. The host selects a configured +MiniMax/MiniMax CN/MiniMaxI route and uploads the WAV once. Any installed +Windows app with the model grant can probe this method; the catalog does not +promise a configured provider. The provider must return nonempty text and a +valid duration. The app receives neither the provider body nor credentials. Embeddings must be finite numeric vectors with consistent dimensions, at most 4096 dimensions. Provider batch indices are validated and reordered. @@ -102,6 +110,7 @@ model call/rate budget. Media also reserves its own daily units before HTTP: | --- | --- | | Image output pixels | 8,388,608 | | Speech input characters | 10,000 | +| Transcription input seconds | 300 | | Video duration seconds | 12 | | Embedding input UTF-8 bytes | 100,000 | diff --git a/apps/ai-providers/host-service/MEDIA.zh-CN.md b/apps/ai-providers/host-service/MEDIA.zh-CN.md index 96c00155d..cb543fa82 100644 --- a/apps/ai-providers/host-service/MEDIA.zh-CN.md +++ b/apps/ai-providers/host-service/MEDIA.zh-CN.md @@ -50,11 +50,17 @@ | `model.video.cancel` | `job` | 远端排队任务确认取消,或返回错误 | | `model.capabilities` | `{}` | `configured` 标志、上限和生命周期说明 | | `model.budget` | `{}` | 现有调用/令牌预算,加上 `media` 资源使用量和上限 | +| `model.transcribe_capability`(Windows) | `{}` | `schema:1`、`supported`、`configured`;不返回提供方身份或凭证 | +| `model.transcribe`(Windows 前台) | `audio:{app_path}`,调用应用存储中的 WAV | 去首尾空白后的 `text`、`duration_seconds` | 图片尺寸为 `512x512`、`1024x1024`(默认)、`1536x1024` 和 `1024x1536`。 OpenAI 路由会在提交前拒绝 `512x512` 和 `seed`;MiniMax 路由支持两者。 每次请求只生成一张图片。PNG/JPEG 的实际尺寸必须匹配请求。语音输出为 MP3;应用 应说明声音由 AI 生成。供应商不提供时长时不返回该字段,宿主不会编造时长。 +转写只接受调用应用自身存储中的 16 kHz 单声道 PCM16 WAV,最多 1 MiB、30 秒。 +宿主选择已配置的 MiniMax/MiniMax CN/MiniMaxI 路由,仅上传一次。任何获准使用模型的 +Windows 已安装应用都可探测此方法;方法目录不保证提供方已配置。提供方必须返回 +非空文字和有效时长;应用不会收到提供方原始响应或凭证。 向量必须由有限数值组成、维数一致且不超过 4096;供应商批量索引经过校验和排序。 H3 提供分辨率级别和宽高比,不提供精确像素尺寸、帧率或种子选项。 @@ -88,6 +94,7 @@ MiniMax H3 需要按量计费 API 权益;仅有 M Plan/聊天账户并不足 | --- | --- | | 图片输出像素 | 8,388,608 | | 语音输入字符 | 10,000 | +| 转写输入秒数 | 300 | | 视频时长秒数 | 12 | | 向量输入 UTF-8 字节 | 100,000 | diff --git a/apps/ai-providers/host-service/README.md b/apps/ai-providers/host-service/README.md index 3098ee7ed..64c95fff6 100644 --- a/apps/ai-providers/host-service/README.md +++ b/apps/ai-providers/host-service/README.md @@ -264,3 +264,27 @@ code now reports denied caller admission, not a missing declaration. Tests use a fake transport: `cargo test --locked -p octosense-llm-service --test complete`. A live check against DeepSeek (costs a fraction of a cent): `DEEPSEEK_API_KEY=… cargo run --locked -p octosense-llm-service --example model_complete_live`. + +## App-local speech transcription on Windows + +The local Windows host exposes `model.transcribe_capability({})` and +`model.transcribe({"audio":{"app_path":"/agent/voice.wav"}})` to admitted +installed apps with the model grant. The transcription call additionally requires foreground. +Capability returns only `schema`, `supported` +and `configured`; transcription returns only `text` and `duration_seconds`. +Other platforms do not advertise this bridge. The host rechecks the caller's +grant and current provider before returning a result. + +Transcription reads a nonempty, mono 16 kHz PCM16 WAV of at most 1 MiB from +the calling app's own storage. The host chooses the first usable MiniMax provider +in the person's configured order, keeps its key in the host, and sends one +bounded multipart upload using `asr-1.0`. Kimi-only configurations report +`configured: false`. The upload is charged to the existing per-app call/rate +budget and a separate 300-second daily transcription quota. It is never +automatically retried; failures expose stable error categories without provider +response bodies. No speech model settings are added. Fake-transport coverage is in +`cargo test --locked -p octosense-llm-service --test media`. On 2026-10-13, +the task-owned Windows Pantry candidate recorded real speech and returned a +MiniMax transcript to its editable input after manual and automatic Stop. +That live check used a person-configured host provider; it did not test other +apps, providers or platforms. diff --git a/apps/ai-providers/host-service/README.zh-CN.md b/apps/ai-providers/host-service/README.zh-CN.md index cb3330fc3..f6e46923d 100644 --- a/apps/ai-providers/host-service/README.zh-CN.md +++ b/apps/ai-providers/host-service/README.zh-CN.md @@ -83,3 +83,22 @@ cargo test --locked -p octosense-llm-service --test complete 模型测试使用假传输,覆盖省略能力声明后的调用、缺少宿主准入时拒绝、输出校验、预算和取消。媒体测试还检查账号切换、准入撤销、并发限制及迟到回复。这些测试不验证真实模型供应商或手机。 真实供应商示例 `model_complete_live` 会产生真实请求和费用,只有明确进行该项验收时才运行。macOS 钥匙串测试也默认忽略,需显式选择 `-- --ignored keychain`。仓库只使用根目录统一固定的依赖及 `.sources/`,不增加个人机器路径或本地 patch。 + +## Windows 应用本地语音转写 + +本地 Windows 宿主向获准使用模型的已安装应用开放 +`model.transcribe_capability({})` 和 +`model.transcribe({"audio":{"app_path":"/agent/voice.wav"}})`。能力探测仅返回 +`schema`、`supported`、`configured`,转写仅返回 `text`、`duration_seconds`。 +实际转写还要求应用处于前台。其他平台不列出这条桥;回复前会重新检查调用应用的准入及供应商配置。 + +宿主只读取调用应用自身存储中非空、单声道 16 kHz PCM16、最多 1 MiB 的 WAV。 +它按配置顺序选首个可用 MiniMax,用宿主密钥发送一次固定 `asr-1.0` multipart +请求。只配置 Kimi 时 `configured` 为 false。上传共用应用调用次数及速率预算, +并单独计入每天 300 秒的转写限额, +不会自动重试;失败只提供稳定错误类别,不暴露供应商响应正文。 +本阶段不增加语音模型设置。 +`cargo test --locked -p octosense-llm-service --test media` 使用假传输验证。 +2026-10-13 在任务专用 Windows Pantry 候选中,真实语音经手动停止与自动停止后, +MiniMax 转写均返回到可编辑输入框。该现场检查使用用户在宿主配置的提供方, +不证明其他应用、提供方或平台的支持。 diff --git a/apps/ai-providers/host-service/src/complete/media/mod.rs b/apps/ai-providers/host-service/src/complete/media/mod.rs index 262a1dceb..15482219b 100644 --- a/apps/ai-providers/host-service/src/complete/media/mod.rs +++ b/apps/ai-providers/host-service/src/complete/media/mod.rs @@ -13,6 +13,7 @@ use std::time::Duration; use url::Url; mod wire; +mod transcribe; pub const RESPONSE_MAX: usize = 4 * 1024 * 1024; pub const ASSET_MAX: usize = 2 * 1024 * 1024; @@ -29,6 +30,7 @@ const JOBS: usize = 64; pub struct Limits { pub image_pixels_per_day: u64, pub audio_characters_per_day: u64, + pub transcription_seconds_per_day: u64, pub video_seconds_per_day: u64, pub embedding_bytes_per_day: u64, } @@ -37,6 +39,7 @@ impl Default for Limits { Self { image_pixels_per_day: 8 * 1024 * 1024, audio_characters_per_day: 10_000, + transcription_seconds_per_day: 300, video_seconds_per_day: 12, embedding_bytes_per_day: 100_000, } @@ -47,6 +50,7 @@ impl Limits { match kind { "image" => self.image_pixels_per_day, "audio" => self.audio_characters_per_day, + "transcription" => self.transcription_seconds_per_day, "video" => self.video_seconds_per_day, "embeddings" => self.embedding_bytes_per_day, _ => 0, @@ -64,6 +68,16 @@ pub trait Transport: Send + Sync { headers: &[(String, String)], body: Option<&str>, ) -> Result<(u16, Vec), String>; + + fn request_bytes( + &self, + _method: &str, + _url: &str, + _headers: &[(String, String)], + _body: &[u8], + ) -> Result<(u16, Vec), String> { + Err("Binary media requests are unsupported by this transport.".into()) + } } pub struct Http; impl Transport for Http { @@ -100,6 +114,36 @@ impl Transport for Http { } Ok((status, bytes)) } + + fn request_bytes( + &self, + method: &str, + url: &str, + headers: &[(String, String)], + body: &[u8], + ) -> Result<(u16, Vec), String> { + let agent = ureq::AgentBuilder::new() + .timeout(Duration::from_secs(120)) + .redirects(0) + .build(); + let mut request = agent.request(method, url); + for (name, value) in headers { + request = request.set(name, value); + } + let response = match request.send_bytes(body) { + Ok(response) | Err(ureq::Error::Status(_, response)) => response, + Err(_) => return Err("The media provider could not be reached; the request may already have been accepted.".into()), + }; + let status = response.status(); + let mut bytes = Vec::new(); + response.into_reader().take(RESPONSE_MAX as u64 + 1) + .read_to_end(&mut bytes) + .map_err(|_| "The media provider response was interrupted.".to_string())?; + if bytes.len() > RESPONSE_MAX { + return Err("The media provider response exceeded the host byte limit.".into()); + } + Ok((status, bytes)) + } } #[derive(Clone)] @@ -166,7 +210,7 @@ impl State { if value["apps"].as_object().unwrap().values().any(|app| { app.as_object().is_none_or(|counts| { counts.iter().any(|(kind, count)| { - !matches!(kind.as_str(), "image" | "audio" | "video" | "embeddings") + !matches!(kind.as_str(), "image" | "audio" | "transcription" | "video" | "embeddings") || !count.is_u64() }) }) @@ -183,7 +227,7 @@ impl State { match Self::load_budget(root, now) { Ok(value) => { let mut result = json!({"resets_at":(now/DAY_MS+1)*DAY_MS/1000}); - for kind in ["image", "audio", "video", "embeddings"] { + for kind in ["image", "audio", "transcription", "video", "embeddings"] { result[kind] = json!({"used":value["apps"][app][kind].as_u64().unwrap_or(0),"limit":self.limits.for_kind(kind)}); } result @@ -437,6 +481,8 @@ fn run_call(context: &Context) -> Result { context.check()?; context.host.attach(&context.call.host_dir); match context.call.method() { + "transcribe_capability" => transcribe::capability(context), + "transcribe" => transcribe::run(context), "complete" => { let request = super::Request::from_args(&context.call.args)?; context @@ -500,6 +546,17 @@ fn run(context: &Context, request: wire::Request) -> Result { fn reserve(context: &Context, kind: &str, units: u64, now: u64) -> Result<(), Refusal> { // Both shared call/rate policy and modality resources must admit before any // billable request. Failed reservations are conservative, never refunded. + admit_call(context, now)?; + context.host.media.reserve( + &context.call.host_dir, + &context.call.app_id, + now, + kind, + units, + ) +} + +fn admit_call(context: &Context, now: u64) -> Result<(), Refusal> { { let mut ledger = context.host.ledger.lock().unwrap(); ledger.attach(context.call.host_dir.join("model/ledger.json")); @@ -514,13 +571,7 @@ fn reserve(context: &Context, kind: &str, units: u64, now: u64) -> Result<(), Re ) })?; } - context.host.media.reserve( - &context.call.host_dir, - &context.call.app_id, - now, - kind, - units, - ) + Ok(()) } fn video_start( @@ -755,12 +806,31 @@ fn public_url(raw: &str) -> Result { } pub(super) fn catalog() -> Vec { - wire::schemas() + let mut methods: Vec<_> = wire::schemas() .into_iter() .map(|(name, summary, args, result)| { HostApiMethod::new(name, 1, "model", summary, args, result) .with_platforms(&["macos", "android", "windows", "linux", "ios", "openharmony"]) .with_agent_access(AgentAccess::Allowed) }) - .collect() + .collect(); + methods.push(HostApiMethod::new("model.transcribe_capability", 1, "model", + "Check whether this Windows host can transcribe the calling app's WAV", + json!({"type":"object","additionalProperties":false}), + json!({"type":"object","required":["schema","supported","configured"], + "properties":{"schema":{"const":1},"supported":{"type":"boolean"}, + "configured":{"type":"boolean"}},"additionalProperties":false})) + .with_platforms(&["windows"]) + .with_agent_access(AgentAccess::Allowed)); + methods.push(HostApiMethod::new("model.transcribe", 1, "model", + "Transcribe a bounded WAV from the calling app's storage", + json!({"type":"object","required":["audio"],"additionalProperties":false, + "properties":{"audio":{"type":"object","required":["app_path"], + "additionalProperties":false,"properties":{"app_path":{"type":"string","minLength":1}}}}}), + json!({"type":"object","required":["text","duration_seconds"], + "additionalProperties":false,"properties":{"text":{"type":"string","minLength":1}, + "duration_seconds":{"type":"number","exclusiveMinimum":0}}})) + .with_platforms(&["windows"]) + .with_agent_access(AgentAccess::ForegroundOnly)); + methods } diff --git a/apps/ai-providers/host-service/src/complete/media/transcribe.rs b/apps/ai-providers/host-service/src/complete/media/transcribe.rs new file mode 100644 index 000000000..1397e825d --- /dev/null +++ b/apps/ai-providers/host-service/src/complete/media/transcribe.rs @@ -0,0 +1,246 @@ +//! Windows app-local WAV to MiniMax speech-to-text bridge. +use super::{wire, Candidate, Class, Code, Context, Refusal, RESPONSE_MAX}; +use serde_json::{json, Value}; +use std::io::Read; +use std::path::Path; + +const MAX_WAV: usize = 1024 * 1024; +const MAX_TEXT: usize = 4096; + +fn unavailable() -> Refusal { + Refusal::new( + Code::Capability, + "Speech transcription is unavailable to this app on this host.", + ) +} + +fn invalid_audio() -> Refusal { + Refusal::new( + Code::BadRequest, + "The app audio is missing, invalid, or over 1 MiB.", + ) +} + +fn provider_error() -> Refusal { + Refusal::new( + Code::Provider, + "The transcription provider failed; the upload was not retried.", + ) +} + +pub(super) fn supported() -> bool { + supported_on(std::env::consts::OS) +} + +fn supported_on(platform: &str) -> bool { + platform == "windows" +} + +fn route(candidates: &[Candidate]) -> Result { + let minimax: Vec<_> = candidates + .iter() + .filter(|candidate| { + matches!( + candidate.provider.family.to_ascii_lowercase().as_str(), + "minimax" | "minimax-cn" | "minimaxi" + ) + }) + .cloned() + .collect(); + wire::route(&minimax, "audio", Class::Fast).map_err(|_| { + Refusal::new( + Code::NoProvider, + "Configure a supported speech provider in AI providers.", + ) + }) +} + +pub(super) fn capability(context: &Context) -> Result { + if context.call.args.as_object().is_none_or(|o| !o.is_empty()) { + return Err(Refusal::new(Code::BadRequest, "The capability request takes no arguments.")); + } + context.check()?; + let configured = route(&context.providers()?).is_ok(); + Ok(json!({"schema":1,"supported":supported(),"configured":configured && supported()})) +} + +fn load_audio(host_dir: &Path, app_id: &str, args: &Value) -> Result, Refusal> { + if app_id.is_empty() || app_id == "." || app_id == ".." + || app_id.contains(['/', '\\', ':', '\0']) + { + return Err(invalid_audio()); + } + let fields = args.as_object().ok_or_else(invalid_audio)?; + if fields.len() != 1 { + return Err(invalid_audio()); + } + let audio = fields + .get("audio") + .and_then(Value::as_object) + .ok_or_else(invalid_audio)?; + if audio.len() != 1 { + return Err(invalid_audio()); + } + let raw = audio + .get("app_path") + .and_then(Value::as_str) + .ok_or_else(invalid_audio)?; + let path = raw.strip_prefix('/').unwrap_or(raw); + if path.is_empty() + || path.contains(['\\', ':', '?', '#', '\0']) + || path + .split('/') + .any(|part| part.is_empty() || matches!(part, "." | "..")) + || !path.to_ascii_lowercase().ends_with(".wav") + { + return Err(invalid_audio()); + } + let root = host_dir + .parent() + .ok_or_else(invalid_audio)? + .join(app_id) + .canonicalize() + .map_err(|_| invalid_audio())?; + let file = root + .join(path) + .canonicalize() + .map_err(|_| invalid_audio())?; + if !file.starts_with(&root) + || !file.is_file() + || file.metadata().map_err(|_| invalid_audio())?.len() > MAX_WAV as u64 + { + return Err(invalid_audio()); + } + let mut bytes = Vec::new(); + std::fs::File::open(file) + .map_err(|_| invalid_audio())? + .take(MAX_WAV as u64 + 1) + .read_to_end(&mut bytes) + .map_err(|_| invalid_audio())?; + if bytes.len() <= 44 + || bytes.len() > MAX_WAV + || &bytes[..4] != b"RIFF" + || &bytes[8..12] != b"WAVE" + || u32::from_le_bytes(bytes[4..8].try_into().unwrap()) as usize + 8 != bytes.len() + || &bytes[12..16] != b"fmt " + || u32::from_le_bytes(bytes[16..20].try_into().unwrap()) != 16 + || u16::from_le_bytes(bytes[20..22].try_into().unwrap()) != 1 + || u16::from_le_bytes(bytes[22..24].try_into().unwrap()) != 1 + || u32::from_le_bytes(bytes[24..28].try_into().unwrap()) != 16_000 + || u32::from_le_bytes(bytes[28..32].try_into().unwrap()) != 32_000 + || u16::from_le_bytes(bytes[32..34].try_into().unwrap()) != 2 + || u16::from_le_bytes(bytes[34..36].try_into().unwrap()) != 16 + || &bytes[36..40] != b"data" + || u32::from_le_bytes(bytes[40..44].try_into().unwrap()) as usize + 44 != bytes.len() + || (bytes.len() - 44) % 2 != 0 + { + return Err(invalid_audio()); + } + Ok(bytes) +} + +fn form(wav: &[u8]) -> (String, Vec) { + let boundary = format!("octosense-{}", uuid::Uuid::new_v4().simple()); + let mut body = Vec::with_capacity(wav.len() + 512); + for (name, value) in [ + ("model", "asr-1.0"), + ("response_format", "json"), + ("stream", "false"), + ] { + body.extend_from_slice(format!("--{boundary}\r\nContent-Disposition: form-data; name=\"{name}\"\r\n\r\n{value}\r\n").as_bytes()); + } + body.extend_from_slice(format!("--{boundary}\r\nContent-Disposition: form-data; name=\"file\"; filename=\"voice.wav\"\r\nContent-Type: audio/wav\r\n\r\n").as_bytes()); + body.extend_from_slice(wav); + body.extend_from_slice(format!("\r\n--{boundary}--\r\n").as_bytes()); + (boundary, body) +} + +pub(super) fn run(context: &Context) -> Result { + if !supported() { + return Err(unavailable()); + } + if !context.call.may_prompt || context.call.from_sheet { + return Err(unavailable()); + } + context.check()?; + let wav = load_audio(&context.call.host_dir, &context.call.app_id, &context.call.args)?; + let route = route(&context.providers()?)?; + let seconds = ((wav.len() - 44) as u64 + 31_999) / 32_000; + super::reserve(context, "transcription", seconds, (context.host.clock)())?; + context.check()?; + if !context + .providers()? + .iter() + .any(|candidate| wire::binding(candidate) == route.binding) + { + return Err(super::cancelled()); + } + let (boundary, body) = form(&wav); + let headers = [ + ("Authorization".into(), format!("Bearer {}", route.key)), + ( + "Content-Type".into(), + format!("multipart/form-data; boundary={boundary}"), + ), + ]; + let (status, bytes) = context + .host + .media + .transport + .request_bytes( + "POST", + &format!("{}/v1/speech_to_text", route.base), + &headers, + &body, + ) + .map_err(|_| provider_error())?; + context.check()?; + if !context + .providers()? + .iter() + .any(|candidate| wire::binding(candidate) == route.binding) + { + return Err(super::cancelled()); + } + if bytes.len() > RESPONSE_MAX { + return Err(Refusal::new( + Code::TooLarge, + "The transcription response is too large.", + )); + } + if status == 429 { + return Err(Refusal::new( + Code::Rate, + "The transcription provider is rate limited.", + )); + } + if !(200..300).contains(&status) { + return Err(provider_error()); + } + let answer: Value = serde_json::from_slice(&bytes).map_err(|_| provider_error())?; + let text = answer + .get("text") + .and_then(Value::as_str) + .map(str::trim) + .filter(|text| !text.is_empty() && text.len() <= MAX_TEXT) + .ok_or_else(provider_error)?; + let duration = answer + .get("duration") + .and_then(Value::as_f64) + .filter(|n| n.is_finite() && *n > 0.0 && *n <= 35.0) + .ok_or_else(provider_error)?; + Ok(json!({"text":text,"duration_seconds":duration})) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn only_windows_is_supported() { + assert!(supported_on("windows")); + for platform in ["macos", "android", "linux", "ios"] { + assert!(!supported_on(platform)); + } + } +} diff --git a/apps/ai-providers/host-service/src/vault.rs b/apps/ai-providers/host-service/src/vault.rs index 702863678..ab5f25157 100644 --- a/apps/ai-providers/host-service/src/vault.rs +++ b/apps/ai-providers/host-service/src/vault.rs @@ -319,6 +319,7 @@ mod tests { assert!(!valid_account("")); } + #[cfg(unix)] #[test] fn a_secrets_dir_keeps_keys_owner_only() { let dir = std::env::temp_dir().join(format!("llm-vault-{}", std::process::id())); @@ -355,6 +356,11 @@ mod tests { fn profile_json(core: &Path) -> serde_json::Value { serde_json::from_slice(&std::fs::read(octosense_llm_config::profile::profile_path(core)).unwrap()).unwrap() } + fn seed_secret(core: &Path, account: &str, value: &str) { + let dir = core.join("secrets"); + std::fs::create_dir_all(&dir).unwrap(); + std::fs::write(dir.join(account), value).unwrap(); + } /// HarmonyOS: a key an earlier build kept in the secrets folder, which /// the embedded kernel never reads, moves into the profile; the folder, @@ -362,15 +368,25 @@ mod tests { #[test] fn keys_in_the_secrets_folder_move_into_the_profile() { let core = core_dir("move"); - write_profile(&core, serde_json::json!({"DEEPSEEK_API_KEY": "keychain:", "GROQ_API_KEY": "keychain:GROQ_API_KEY::_main"})); - let folder = SecretsDir::new(core.join("secrets")); - folder.put("DEEPSEEK_API_KEY", "sk-test-move-0001").unwrap(); - folder.put("GROQ_API_KEY::_main", "gsk-test-move-0002").unwrap(); - folder.put("MOONSHOT_API_KEY", "sk-test-left-0003").unwrap(); - assert_eq!(move_secrets_into_profile(&core).unwrap(), 2); + let env = if cfg!(unix) { + serde_json::json!({"DEEPSEEK_API_KEY": "keychain:", "GROQ_API_KEY": "keychain:GROQ_API_KEY::_main"}) + } else { + serde_json::json!({"DEEPSEEK_API_KEY": "keychain:"}) + }; + write_profile(&core, env); + seed_secret(&core, "DEEPSEEK_API_KEY", "sk-test-move-0001"); + // Windows cannot create a file with ':' in its name; the unscoped + // migration remains covered there, while Unix covers scoped keys too. + if cfg!(unix) { + seed_secret(&core, "GROQ_API_KEY::_main", "gsk-test-move-0002"); + } + seed_secret(&core, "MOONSHOT_API_KEY", "sk-test-left-0003"); + assert_eq!(move_secrets_into_profile(&core).unwrap(), if cfg!(unix) { 2 } else { 1 }); let profile = profile_json(&core); assert_eq!(profile["config"]["env_vars"]["DEEPSEEK_API_KEY"], "sk-test-move-0001"); - assert_eq!(profile["config"]["env_vars"]["GROQ_API_KEY"], "gsk-test-move-0002"); + if cfg!(unix) { + assert_eq!(profile["config"]["env_vars"]["GROQ_API_KEY"], "gsk-test-move-0002"); + } assert_eq!(profile["config"]["llm"]["primary"]["context_window"], 1048576, "the selection is kept as it was"); assert!(!core.join("secrets").exists(), "the folder and the leftover key are gone"); #[cfg(unix)] @@ -389,7 +405,7 @@ mod tests { let path = octosense_llm_config::profile::profile_path(&core); std::fs::create_dir_all(path.parent().unwrap()).unwrap(); std::fs::write(&path, b"{ not json").unwrap(); - SecretsDir::new(core.join("secrets")).put("DEEPSEEK_API_KEY", "sk-test-keep-0001").unwrap(); + seed_secret(&core, "DEEPSEEK_API_KEY", "sk-test-keep-0001"); assert!(move_secrets_into_profile(&core).is_err()); assert_eq!(SecretsDir::new(core.join("secrets")).get("DEEPSEEK_API_KEY").unwrap().as_deref(), Some("sk-test-keep-0001")); let _ = std::fs::remove_dir_all(&core); diff --git a/apps/ai-providers/host-service/tests/media.rs b/apps/ai-providers/host-service/tests/media.rs index 45a738138..ff2ade60d 100644 --- a/apps/ai-providers/host-service/tests/media.rs +++ b/apps/ai-providers/host-service/tests/media.rs @@ -46,6 +46,7 @@ struct Seen { url: String, headers: Vec<(String, String)>, body: Value, + bytes: Option>, } #[derive(Default)] struct Fake { @@ -89,6 +90,7 @@ impl media::Transport for Fake { body: body .map(|b| serde_json::from_str(b).unwrap()) .unwrap_or(Value::Null), + bytes: None, }); let gate = self.gate.lock().unwrap().clone(); if let Some(gate) = gate { @@ -100,6 +102,25 @@ impl media::Transport for Fake { .pop_front() .ok_or_else(|| "synthetic no response".into()) } + + fn request_bytes( + &self, + method: &str, + url: &str, + headers: &[(String, String)], + body: &[u8], + ) -> Result<(u16, Vec), String> { + self.seen.lock().unwrap().push(Seen { + method: method.into(), + url: url.into(), + headers: headers.to_vec(), + body: Value::Null, + bytes: Some(body.to_vec()), + }); + if let Some(gate) = self.gate.lock().unwrap().clone() { gate.wait(); } + self.answers.lock().unwrap().pop_front() + .ok_or_else(|| "synthetic no response".into()) + } } impl complete::Transport for Fake { fn post( @@ -137,6 +158,9 @@ impl Rig { Self::limits(family, media::Limits::default()) } fn limits(family: &str, limits: media::Limits) -> Self { + Self::call_limit(family, limits, 1000) + } + fn call_limit(family: &str, limits: media::Limits, calls_per_day: u32) -> Self { let serial = SERIAL.lock().unwrap_or_else(|e| e.into_inner()); let root = std::env::temp_dir().join(format!("model-media-test-{}", uuid::Uuid::new_v4())); std::fs::create_dir_all(root.join(".host")).unwrap(); @@ -154,7 +178,7 @@ impl Rig { .grants(move |_, _| grant.load(Ordering::SeqCst)) .limits(complete::ledger::Limits { per_minute: 100, - calls_per_day: 1000, + calls_per_day, tokens_per_day: 100_000, }) .clock(move || clock.load(Ordering::SeqCst)); @@ -175,6 +199,9 @@ impl Rig { } } fn submit(&self, app: &str, method: &str, args: Value) -> usize { + self.submit_with_foreground(app, method, args, false) + } + fn submit_with_foreground(&self, app: &str, method: &str, args: Value, foreground: bool) -> usize { let heap = NEXT.fetch_add(1, Ordering::Relaxed); dispatch( ServiceCall { @@ -182,7 +209,7 @@ impl Rig { service: format!("model.{method}"), args, from_sheet: false, - may_prompt: false, + may_prompt: foreground, host_dir: self.root.join(".host"), }, heap, @@ -204,6 +231,9 @@ impl Rig { fn call(&self, method: &str, args: Value) -> Result { self.answer(self.submit(APP, method, args)) } + fn call_front(&self, app: &str, method: &str, args: Value) -> Result { + self.answer(self.submit_with_foreground(app, method, args, true)) + } fn start_video(&self) -> String { self.fake.json(200, json!({"task_id":"task_123"})); self.call("video", json!({"prompt":"Synthetic paper kite"})) @@ -236,6 +266,212 @@ fn png() -> String { STANDARD.encode(bytes) } +#[cfg(target_os = "windows")] +fn app_wav(rig: &Rig, app: &str, name: &str, bytes: &[u8]) { + let dir = rig.root.join(app).join("agent"); + std::fs::create_dir_all(&dir).unwrap(); + std::fs::write(dir.join(name), bytes).unwrap(); +} + +#[cfg(target_os = "windows")] +fn short_wav() -> Vec { + let mut bytes = Vec::new(); + bytes.extend_from_slice(b"RIFF"); + bytes.extend_from_slice(&38u32.to_le_bytes()); + bytes.extend_from_slice(b"WAVEfmt "); + bytes.extend_from_slice(&16u32.to_le_bytes()); + bytes.extend_from_slice(&1u16.to_le_bytes()); + bytes.extend_from_slice(&1u16.to_le_bytes()); + bytes.extend_from_slice(&16_000u32.to_le_bytes()); + bytes.extend_from_slice(&32_000u32.to_le_bytes()); + bytes.extend_from_slice(&2u16.to_le_bytes()); + bytes.extend_from_slice(&16u16.to_le_bytes()); + bytes.extend_from_slice(b"data"); + bytes.extend_from_slice(&2u32.to_le_bytes()); + bytes.extend_from_slice(&0i16.to_le_bytes()); + bytes +} + +#[cfg(target_os = "windows")] +#[test] +fn pantry_transcribe_is_admitted_and_capability_hides_provider_details() { + let rig = Rig::new("moonshot"); + assert_eq!(octosense_appstore::host_api::available_versions().get("model.transcribe"), Some(&1)); + assert_eq!(octosense_appstore::host_api::available_versions().get("model.transcribe_capability"), Some(&1)); + assert_eq!(rig.answer(rig.submit("pantry-steward", "transcribe_capability", json!({}))).unwrap(), + json!({"schema":1,"supported":true,"configured":false})); + assert_eq!(rig.answer(rig.submit(APP, "transcribe_capability", json!({}))).unwrap(), + json!({"schema":1,"supported":true,"configured":false})); + assert!(rig.answer(rig.submit("pantry-steward", "transcribe_capability", json!({"provider":"minimax"}))) + .unwrap_err().starts_with("bad_request:")); + *rig.config.0.lock().unwrap() = vec![candidate("moonshot"), candidate("minimax-cn")]; + assert_eq!(rig.answer(rig.submit("pantry-steward", "transcribe_capability", json!({}))).unwrap(), + json!({"schema":1,"supported":true,"configured":true})); + assert_eq!(rig.fake.count(), 0); +} + +#[cfg(target_os = "windows")] +#[test] +fn pantry_transcribe_uses_only_bounded_wav_and_fixed_minimax_multipart() { + let rig = Rig::new("moonshot"); + *rig.config.0.lock().unwrap() = vec![candidate("moonshot"), candidate("minimax-cn"), candidate("minimax")]; + let wav = short_wav(); + app_wav(&rig, "pantry-steward", "voice.wav", &wav); + app_wav(&rig, APP, "voice.wav", &wav); + rig.fake.json(200, json!({"text":" 今天吃豆腐 ","duration":0.25,"trace_id":"secret-trace"})); + let request = json!({"audio":{"app_path":"/agent/voice.wav"}}); + let background_error = rig.answer(rig.submit("pantry-steward", "transcribe", request.clone())).unwrap_err(); + assert!(background_error.contains("unavailable to agents/background surfaces"), + "background must be refused: {background_error}"); + rig.fake.json(200, json!({"text":"今天吃豆腐","duration":0.25})); + assert_eq!(rig.call_front(APP, "transcribe", request.clone()).unwrap()["text"], "今天吃豆腐"); + let result = rig.call_front("pantry-steward", "transcribe", request).unwrap(); + assert_eq!(result, json!({"text":"今天吃豆腐","duration_seconds":0.25})); + let seen = rig.fake.seen.lock().unwrap(); + assert_eq!(seen.len(), 2); + let call = &seen[1]; + assert_eq!(call.method, "POST"); + assert_eq!(call.url, "https://api.minimaxi.com/v1/speech_to_text"); + assert!(call.headers.iter().any(|(key,value)| key == "Authorization" && value == &format!("Bearer {KEY}"))); + let content_type = call.headers.iter().find(|(key,_)| key == "Content-Type").unwrap().1.as_str(); + assert!(content_type.starts_with("multipart/form-data; boundary=octosense-")); + let body = call.bytes.as_ref().unwrap(); + assert!(body.windows(wav.len()).any(|chunk| chunk == wav)); + let form = String::from_utf8_lossy(body); + for field in ["model", "file", "response_format", "stream"] { + assert!(form.contains(&format!("name=\"{field}\""))); + } + assert_eq!(form.matches("Content-Disposition: form-data; name=\"").count(), 4); + assert!(form.contains("asr-1.0") && form.contains("\r\njson\r\n") && form.contains("\r\nfalse\r\n")); + for forbidden in ["timestamp_level", "language", "speaker", "subtitle"] { + assert!(!form.contains(forbidden)); + } + assert!(!result.to_string().contains(KEY)); +} + +#[cfg(target_os = "windows")] +#[test] +fn pantry_transcribe_rejects_paths_and_malformed_audio_before_upload() { + let rig = Rig::new("minimax"); + app_wav(&rig, "pantry-steward", "valid.wav", &short_wav()); + app_wav(&rig, "pantry-steward", "empty.wav", b""); + app_wav(&rig, "pantry-steward", "broken.wav", b"RIFFbad"); + app_wav(&rig, "pantry-steward", "oversize.wav", &vec![0; 1024 * 1024 + 1]); + for path in ["../valid.wav", "C:/valid.wav", "agent\\valid.wav", "https://host/valid.wav", + "data:audio/wav;base64,AAAA", "//agent/valid.wav", "/agent/empty.wav", + "/agent/broken.wav", "/agent/oversize.wav", "/agent/missing.wav"] { + let err = rig.call_front("pantry-steward", "transcribe", json!({"audio":{"app_path":path}})).unwrap_err(); + assert!(err.starts_with("bad_request:"), "{path}: {err}"); + } + for args in [json!({}), json!({"audio":"/agent/valid.wav"}), + json!({"audio":{"app_path":"/agent/valid.wav","url":"https://host"}}), + json!({"audio":{"app_path":"/agent/valid.wav"},"model":"other"})] { + assert!(rig.call_front("pantry-steward", "transcribe", args).unwrap_err().starts_with("bad_request:")); + } + assert_eq!(rig.fake.count(), 0); +} + +#[cfg(target_os = "windows")] +#[test] +fn pantry_transcribe_charges_call_budget_and_never_retries_or_leaks_provider_errors() { + let rig = Rig::call_limit("minimax", media::Limits::default(), 0); + app_wav(&rig, "pantry-steward", "voice.wav", &short_wav()); + assert!(rig.call_front("pantry-steward", "transcribe", json!({"audio":{"app_path":"/agent/voice.wav"}})) + .unwrap_err().starts_with("budget:")); + assert_eq!(rig.fake.count(), 0); + drop(rig); + + let rig = Rig::new("minimax"); + app_wav(&rig, "pantry-steward", "voice.wav", &short_wav()); + rig.fake.json(403, json!({"error":format!("{KEY} https://private.example/trace") })); + let err = rig.call_front("pantry-steward", "transcribe", json!({"audio":{"app_path":"/agent/voice.wav"}})).unwrap_err(); + assert!(err.starts_with("provider:")); + assert!(!err.contains(KEY) && !err.contains("private.example")); + assert_eq!(rig.fake.count(), 1); +} + +#[cfg(target_os = "windows")] +#[test] +fn transcription_seconds_are_reserved_before_upload() { + let mut limits = media::Limits::default(); + limits.transcription_seconds_per_day = 0; + let rig = Rig::limits("minimax", limits); + app_wav(&rig, "pantry-steward", "voice.wav", &short_wav()); + let err = rig.call_front("pantry-steward", "transcribe", + json!({"audio":{"app_path":"/agent/voice.wav"}})).unwrap_err(); + assert!(err.starts_with("budget:"), "{err}"); + assert_eq!(rig.fake.count(), 0); +} + +#[cfg(target_os = "windows")] +#[test] +fn minimaxi_alias_is_a_configured_transcription_route() { + let rig = Rig::new("minimaxi"); + assert_eq!(rig.call("transcribe_capability", json!({})).unwrap()["configured"], true); +} + +#[cfg(target_os = "windows")] +#[test] +fn pantry_transcribe_rejects_empty_or_invalid_provider_text_without_leaking_response() { + let rig = Rig::new("minimax"); + app_wav(&rig, "pantry-steward", "voice.wav", &short_wav()); + let args = json!({"audio":{"app_path":"/agent/voice.wav"}}); + for reply in [ + json!({"text":" ","duration":0.25,"trace_id":KEY}), + json!({"text":"hello","duration":0,"trace_id":KEY}), + json!({"text":"hello","trace_id":KEY}), + json!({"text":"x".repeat(4097),"duration":0.25,"trace_id":KEY}), + ] { + rig.fake.json(200, reply); + let error = rig.call_front("pantry-steward", "transcribe", args.clone()).unwrap_err(); + assert!(error.starts_with("provider:"), "{error}"); + assert!(!error.contains(KEY)); + } + assert_eq!(rig.fake.count(), 4); +} + +#[cfg(target_os = "windows")] +#[test] +fn pantry_transcribe_drops_a_late_result_after_provider_change() { + let rig = Rig::new("minimax"); + app_wav(&rig, "pantry-steward", "voice.wav", &short_wav()); + let gate = Arc::new(Gate::default()); + *rig.fake.gate.lock().unwrap() = Some(gate.clone()); + rig.fake.json(200, json!({"text":"迟到回复","duration":0.25})); + let heap = rig.submit_with_foreground("pantry-steward", "transcribe", + json!({"audio":{"app_path":"/agent/voice.wav"}}), true); + rig.fake.wait_calls(1); + *rig.config.0.lock().unwrap() = vec![candidate("moonshot")]; + gate.release(); + assert!(rig.answer(heap).unwrap_err().starts_with("capability:")); + assert_eq!(rig.fake.count(), 1); +} + +#[cfg(target_os = "windows")] +#[test] +fn pantry_transcribe_requires_a_current_app_grant_at_start_and_delivery() { + let rig = Rig::new("minimax"); + app_wav(&rig, "pantry-steward", "voice.wav", &short_wav()); + let args = json!({"audio":{"app_path":"/agent/voice.wav"}}); + rig.allowed.store(false, Ordering::SeqCst); + assert!(rig.answer(rig.submit("pantry-steward", "transcribe_capability", json!({}))) + .unwrap_err().starts_with("capability:")); + assert!(rig.call_front("pantry-steward", "transcribe", args.clone()) + .unwrap_err().starts_with("capability:")); + assert_eq!(rig.fake.count(), 0); + + rig.allowed.store(true, Ordering::SeqCst); + let gate = Arc::new(Gate::default()); + *rig.fake.gate.lock().unwrap() = Some(gate.clone()); + rig.fake.json(200, json!({"text":"late","duration":0.25})); + let heap = rig.submit_with_foreground("pantry-steward", "transcribe", args, true); + rig.fake.wait_calls(1); + rig.allowed.store(false, Ordering::SeqCst); + gate.release(); + assert!(rig.answer(heap).unwrap_err().starts_with("capability:")); + assert_eq!(rig.fake.count(), 1); +} + #[test] fn embeddings_use_the_real_adapter_hide_provider_and_reorder_batch_results() { let rig = Rig::new("openai"); @@ -676,6 +912,39 @@ fn actual_http_transport_refuses_redirects_and_oversized_streams() { } } +#[test] +fn actual_http_transport_sends_binary_body_without_following_redirect() { + use media::Transport; + use std::io::{Read, Write}; + use std::net::TcpListener; + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let address = listener.local_addr().unwrap(); + let server = std::thread::spawn(move || { + let (mut connection, _) = listener.accept().unwrap(); + connection.set_read_timeout(Some(Duration::from_secs(5))).unwrap(); + let mut request = Vec::new(); + let mut buffer = [0u8; 1024]; + while !request.windows(4).any(|part| part == b"\r\n\r\n") { + let count = connection.read(&mut buffer).unwrap(); + assert!(count > 0); + request.extend_from_slice(&buffer[..count]); + } + let body_start = request.windows(4).position(|part| part == b"\r\n\r\n").unwrap() + 4; + while request.len() - body_start < 5 { + let count = connection.read(&mut buffer).unwrap(); + assert!(count > 0); + request.extend_from_slice(&buffer[..count]); + } + assert!(String::from_utf8_lossy(&request[..body_start]).starts_with("POST /fixture HTTP/1.1")); + assert_eq!(&request[body_start..], b"a\0b\xffc"); + write!(connection,"HTTP/1.1 302 Found\r\nLocation: http://127.0.0.1:1/must-not-follow\r\nContent-Length: 0\r\nConnection: close\r\n\r\n").unwrap(); + }); + let result = media::Http.request_bytes("POST", &format!("http://{address}/fixture"), + &[("Content-Type".into(), "application/octet-stream".into())], b"a\0b\xffc"); + assert_eq!(result.unwrap().0, 302); + server.join().unwrap(); +} + #[test] fn an_anthropic_only_route_does_not_reuse_its_key_for_media() { let rig = Rig::new("minimax"); diff --git a/crates/shell/src/audio_service/README.md b/crates/shell/src/audio_service/README.md index bfbe38914..1b8697d00 100644 --- a/crates/shell/src/audio_service/README.md +++ b/crates/shell/src/audio_service/README.md @@ -3,10 +3,16 @@ English | [简体中文](README.zh-CN.md) Installed apps can record a short voice clip and play an audio file through -OctoSense's native audio devices. These APIs currently support macOS and Android. -They do not transcribe speech, synthesize speech, capture system audio, download -media, or continue after the app leaves the foreground. Hardware acceptance is -**unverified**; synthetic tests never activate a microphone or speaker. +OctoSense's native audio devices. Recording supports macOS, Android and Windows; +playback is advertised on macOS and Android. +They do not themselves transcribe speech, synthesize speech, capture system audio, download +media, or continue after the app leaves the foreground. On 2026-10-13, a +task-owned Windows desktop host recorded real microphone input, saved a mono +16 kHz WAV after manual Stop and after the 30-second automatic limit, and +discarded an active recording when the app panel or host closed. Speaker +playback hardware acceptance on Windows is **unverified**, so Windows does not +advertise `audio.play/status/stop`; synthetic tests do +not activate a microphone or speaker. | Method | Arguments | Result | | --- | --- | --- | @@ -14,7 +20,7 @@ media, or continue after the app leaves the foreground. Hardware acceptance is | `microphone.record_status` | `session` | Current state and eventual output path/error | | `microphone.record_stop` | `session` | `stopping`; poll until `saved` or `failed` | | `microphone.record_cancel` | `session` | Stop and discard, without saving | -| `audio.play` | `path` | Host session and `starting` after bounded decoding and device setup | +| `audio.play` (macOS/Android) | `path` | Host session and `starting` after bounded decoding and device setup | | `audio.status` | `session` | Current state, including `playing`, `completed`, `stopped`, or `failed` | | `audio.stop` | `session` | `stopping`; poll for completion | @@ -40,6 +46,12 @@ the current OS grant; it cannot open a permission prompt or borrow another app's consent. The OS microphone indicator remains active while native input is open. Provide a visible recording state, Stop and Cancel in the app using these APIs. +Windows microphone methods are included in the host catalog and use the same app consent +and native capture route. The pinned Makepad Windows permission adapter reports +the OS grant as granted without querying the Windows privacy setting; if that +setting blocks capture, the device open/frames path reports failure. This is a +platform adapter limitation, not evidence of an OS grant. + Sessions belong to the host-assigned app identity and live Splash heap. Another app or a reopened instance cannot operate them. The shell supplies the focused installed app or expanded Glance owner, and the runtime verifies its foreground diff --git a/crates/shell/src/audio_service/README.zh-CN.md b/crates/shell/src/audio_service/README.zh-CN.md index 9f3bfc966..47ce8cdc9 100644 --- a/crates/shell/src/audio_service/README.zh-CN.md +++ b/crates/shell/src/audio_service/README.zh-CN.md @@ -3,8 +3,11 @@ [English](README.md) | 简体中文 已安装的应用可通过 OctoSense 的原生设备录制短语音片段和播放音频文件。 -目前支持 macOS 和 Android。这些 API 不提供语音转文字、语音合成、系统声音 -采集、媒体下载或后台播放。硬件验收仍为 **未验证**;合成测试不会开启麦克风或扬声器。 +录音支持 macOS、Android 和 Windows;播放仅在 macOS 和 Android 列入方法目录。这些录放音 API 本身不提供语音转文字、语音合成、系统声音 +采集、媒体下载或后台播放。2026-10-13 在任务专用 Windows 桌面宿主中,真实麦克风 +手动停止及 30 秒自动停止均保存了单声道 16 kHz WAV;录音中关闭应用面板或宿主后 +临时录音被清理。Windows 扬声器播放的硬件验收仍为 **未验证**,因此 Windows 不列出 `audio.play/status/stop`;合成测试不会开启 +麦克风或扬声器。 | 方法 | 参数 | 返回 | | --- | --- | --- | @@ -34,6 +37,10 @@ 不能借用其他应用的许可。原生输入打开期间,OS 麦克风指示器保持显示。 应用应使用这些 API 提供清晰的录音状态、停止和取消入口。 +Windows 麦克风方法已列入宿主目录,沿用同一套应用同意和原生采集路径。当前固定的 +Makepad Windows 权限适配器未查询 Windows 隐私设置,直接报告系统权限已授予; +若隐私设置阻止采集,设备打开或帧回调路径会报错。这个状态不能当作系统授权证明。 + 会话绑定宿主确认的应用身份和存活的 Splash 堆。其他应用或重新打开的实例 不能操作它。Shell 提供当前聚焦的已安装应用或展开的 Glance 所有者,运行时 检查其前台标记。切换应用、回到 Home、关闭堆、改变存储或账户、宿主进入后台、 diff --git a/crates/shell/src/audio_service/mod.rs b/crates/shell/src/audio_service/mod.rs index ee5b4d013..2f44c0cc8 100644 --- a/crates/shell/src/audio_service/mod.rs +++ b/crates/shell/src/audio_service/mod.rs @@ -131,6 +131,9 @@ fn parse(family: &str, method: &str, args: &Value) -> Result } } fn supported() -> bool { + cfg!(any(target_os = "macos", target_os = "android", target_os = "windows")) +} +fn playback_supported() -> bool { cfg!(any(target_os = "macos", target_os = "android")) } fn descriptor(name: &str, family: &str, description: &str, input: Value) -> HostApiMethod { @@ -147,7 +150,11 @@ fn descriptor(name: &str, family: &str, description: &str, input: Value) -> Host "error":{"type":["string","null"]},"frames":{"type":"integer","minimum":0,"description":"Device frames received or rendered; not delivery or audibility confirmation"}, "format":{"enum":["wav",null]}}}), ) - .with_platforms(&["macos", "android"]) + .with_platforms(if family == "microphone" { + &["macos", "android", "windows"] + } else { + &["macos", "android"] + }) .with_agent_access(AgentAccess::ForegroundOnly) } fn session_schema() -> Value { @@ -184,7 +191,7 @@ impl HostService for AudioService { "audio" } fn api_methods(&self) -> Vec { - if !supported() { + if !playback_supported() { return vec![]; } vec![ @@ -208,10 +215,13 @@ pub(crate) fn call(call: ServiceCall, reply: Replier) { let result = (|| { if !supported() { return Err( - "unsupported_platform: Native audio sessions currently support macOS and Android" + "unsupported_platform: Native audio sessions currently support macOS, Android and Windows" .into(), ); } + if call.service.starts_with("audio.") && !playback_supported() { + return Err("unsupported_platform: Audio playback is not available on this host".into()); + } if !call.may_prompt || call.from_sheet { return Err("foreground_required: Open the installed app to use audio".into()); } @@ -376,7 +386,7 @@ impl Default for State { background: false, // A desktop launch may start behind another application's window. // Only the native focus event can authorize this window initially. - window_unfocused: cfg!(target_os = "macos"), + window_unfocused: cfg!(any(target_os = "macos", target_os = "windows")), input: None, output: None, timer: Timer::default(), diff --git a/crates/shell/src/audio_service/tests.rs b/crates/shell/src/audio_service/tests.rs index 429eec11d..2332fcd74 100644 --- a/crates/shell/src/audio_service/tests.rs +++ b/crates/shell/src/audio_service/tests.rs @@ -128,8 +128,8 @@ fn native_sessions_follow_foreground_heap_identity_and_account_storage_without_a }; assert_eq!( state.window_unfocused, - cfg!(target_os = "macos"), - "macOS must wait for its first native focus event" + cfg!(any(target_os = "macos", target_os = "windows")), + "desktop capture must wait for its first native focus event" ); state.sessions.insert(id.clone(), session); maintain(&mut cx, &mut state, &Event::WindowLostFocus(WindowId(0, 0))); @@ -148,6 +148,17 @@ fn native_sessions_follow_foreground_heap_identity_and_account_storage_without_a "focus gain never resumes capture" ); + // The bounded lifetime still terminates a stalled recorder after focus + // returns; a late worker result cannot turn it into a saved file. + let session = state.sessions.get_mut(&id).unwrap(); + session.state = "recording"; + session.ended = None; + session.started = Instant::now() - Duration::from_secs(41); + session.shared = Arc::new(Shared::new()); + maintain(&mut cx, &mut state, &Event::Signal); + assert_eq!(state.sessions[&id].state, "failed"); + assert!(state.sessions[&id].shared.stop.load(Ordering::Acquire)); + // Queue an explicit Cancel before the UI sees a worker's ready file. // Native-only dispatch creates the normal pending reply; it does not // activate a device or grant an app any production capability. @@ -180,6 +191,7 @@ fn native_sessions_follow_foreground_heap_identity_and_account_storage_without_a let session = state.sessions.get_mut(&id).unwrap(); session.state = "stopping"; session.ended = None; + session.started = Instant::now(); session.shared = Arc::new(Shared::new()); let staged = session .storage @@ -275,10 +287,14 @@ fn bounded_capture_queue_reports_overflow_without_growing() { fn discovery_never_advertises_background_capture_or_unsupported_platforms() { let mut methods = microphone_methods(); methods.extend(AudioService.api_methods()); - assert_eq!(methods.len(), if supported() { 7 } else { 0 }); + assert_eq!(methods.len(), if playback_supported() { 7 } else if supported() { 4 } else { 0 }); for method in methods { method.validate().unwrap(); assert_eq!(method.agent_access, AgentAccess::ForegroundOnly); - assert_eq!(method.platforms, vec!["macos", "android"]); + assert_eq!(method.platforms, if method.name.starts_with("microphone.") { + vec!["macos", "android", "windows"] + } else { + vec!["macos", "android"] + }); } } diff --git a/crates/shell/src/platform_services/README.md b/crates/shell/src/platform_services/README.md index b6856cd3e..af19f5850 100644 --- a/crates/shell/src/platform_services/README.md +++ b/crates/shell/src/platform_services/README.md @@ -8,7 +8,7 @@ An installed script app can ask a compatible OctoSense host for camera, micropho - **App consent.** The person allows this app on a native host sheet. Consent covers this app on all of its accounts, and no other app. - **OS permission.** The operating system grants it to OctoSense as a whole, so on its own it authorizes no installed app. -This module's device service (`DeviceService`) implements permission status, request and revoke on Android and macOS, bounded fresh location sampling on Android and macOS, and reading the legacy last-known location on Android. On other platforms, API discovery (`runtime.list` and `runtime.describe`) does not list these methods: `status` answers `os_permission: "unsupported"`, and `request` and `location.get` fail with `unsupported_platform`. A permission adds no calendar, file picker, camera capture or background location service. To show the camera, use the existing `CameraPreview` widget. +This module's device service (`DeviceService`) implements camera and location permission status, request and revoke on Android and macOS, and microphone permission status, request and revoke on Android, macOS and Windows. Fresh location sampling is available on Android/macOS and the legacy last-known location on Android. Discovery omits unsupported methods on other platforms. Windows microphone status reports the host consent state, but the pinned Windows adapter cannot query the OS privacy setting; capture may still fail when the OS blocks the device. A permission adds no calendar, file picker, camera capture or background location service. To show the camera, use the existing `CameraPreview` widget. ## Declare what the app uses diff --git a/crates/shell/src/platform_services/README.zh-CN.md b/crates/shell/src/platform_services/README.zh-CN.md index 56d3dc27a..03c5e7c19 100644 --- a/crates/shell/src/platform_services/README.zh-CN.md +++ b/crates/shell/src/platform_services/README.zh-CN.md @@ -8,7 +8,7 @@ - **应用授权。** 用户在宿主的原生面板上允许这个应用。授权覆盖该应用的所有账户,不覆盖其他应用。 - **系统权限。** 操作系统授予 OctoSense 本身的权限。单凭它,任何已安装应用都无权使用设备。 -本模块的设备服务(`DeviceService`)在 Android 和 macOS 上实现权限的查询、申请和撤销,支持 Android/macOS 有时限的新鲜位置采样,并保留 Android 最近已知位置读取。在其他平台上,API 发现接口(`runtime.list` 和 `runtime.describe`)不列出这些方法:`status` 返回 `os_permission: "unsupported"`,`request` 和 `location.get` 以 `unsupported_platform` 失败。获得权限并不会多出日历、文件选择器、摄像头拍摄或后台定位服务。要显示摄像头画面,请使用已有的 `CameraPreview` 控件。 +本模块的设备服务(`DeviceService`)在 Android 和 macOS 上提供摄像头、位置权限的查询、申请和撤销;麦克风权限的这三项操作还支持 Windows。Android/macOS 支持有时限的新鲜位置采样,Android 保留最近已知位置读取。API 发现接口不列出不支持的方法。Windows 麦克风状态反映宿主的应用授权,但当前固定的 Windows 适配器不能查询系统隐私设置;系统阻止设备时录音仍可能失败。获得权限并不会多出日历、文件选择器、摄像头拍摄或后台定位服务。要显示摄像头画面,请使用已有的 `CameraPreview` 控件。 ## 声明应用要用的能力 diff --git a/crates/shell/src/platform_services/mod.rs b/crates/shell/src/platform_services/mod.rs index 07309e6b3..e06b8e753 100644 --- a/crates/shell/src/platform_services/mod.rs +++ b/crates/shell/src/platform_services/mod.rs @@ -116,11 +116,12 @@ fn permission(family: &str) -> Option { _ => None, } } -fn permission_supported() -> bool { +fn permission_supported(family: &str) -> bool { cfg!(any(target_os = "android", target_os = "macos")) + || (cfg!(target_os = "windows") && family == "microphone") } -fn prompt_supported() -> bool { - cfg!(any(target_os = "android", target_os = "macos")) +fn prompt_supported(family: &str) -> bool { + permission_supported(family) } fn status_name(status: PermissionStatus) -> &'static str { match status { @@ -131,10 +132,10 @@ fn status_name(status: PermissionStatus) -> &'static str { } } fn status(work: &Work, os: &str) -> Value { - json!({"capability": work.family, "supported": permission_supported(), "app_policy_granted": work.policy_allows(), + json!({"capability": work.family, "supported": permission_supported(work.family), "app_policy_granted": work.policy_allows(), "app_consent": work.current_grant().map(|g| g.allowed).unwrap_or(false), "os_permission": os, "location_read_supported": work.family == "location" && cfg!(target_os = "android"), - "location_sample_supported": work.family == "location" && permission_supported(), + "location_sample_supported": work.family == "location" && permission_supported(work.family), "scope": "host_api_v1_device_access", "background_permission": false}) } fn queue(work: Work) { @@ -180,7 +181,7 @@ impl HostService for DeviceService { json!({"type":"object","required":["abi","defaults","options","library_export_supported","library_export_requires_foreground","microphone_consent_required"]})) .with_agent_access(AgentAccess::Allowed)] } else { vec![] }; - if !permission_supported() { + if !permission_supported(self.family) { return capture_intent; } let input = json!({"type":"object","additionalProperties":false}); @@ -197,7 +198,7 @@ impl HostService for DeviceService { input.clone(), output.clone(), ) - .with_platforms(&["android", "macos"]) + .with_platforms(if self.family == "microphone" { &["android", "macos", "windows"] } else { &["android", "macos"] }) .with_agent_access(AgentAccess::Allowed), HostApiMethod::new( format!("{}.permission.request", self.family), @@ -207,7 +208,7 @@ impl HostService for DeviceService { input.clone(), output.clone(), ) - .with_platforms(&["android", "macos"]) + .with_platforms(if self.family == "microphone" { &["android", "macos", "windows"] } else { &["android", "macos"] }) .with_agent_access(AgentAccess::ForegroundOnly), HostApiMethod::new( format!("{}.permission.revoke", self.family), @@ -218,7 +219,7 @@ impl HostService for DeviceService { json!({"type":"object","required":["app_consent","os_permission_unchanged","scope"],"properties":{ "app_consent":{"const":false},"os_permission_unchanged":{"const":true},"scope":{"const":"host_api_v1_device_access"}}}), ) - .with_platforms(&["android", "macos"]) + .with_platforms(if self.family == "microphone" { &["android", "macos", "windows"] } else { &["android", "macos"] }) .with_agent_access(AgentAccess::Allowed), ]; methods.extend(capture_intent); @@ -311,7 +312,7 @@ impl HostService for DeviceService { return; } if self.family == "location" && call.method() == "sample.cancel" { - if !permission_supported() { + if !permission_supported(self.family) { reply.send(Err( "unsupported_platform: Location sampling is unavailable".into(), )); @@ -363,7 +364,7 @@ impl HostService for DeviceService { requesting: false, sample: sample_options.map(|options| location::Sample { options, fix: None }), }; - if !permission_supported() { + if !permission_supported(self.family) { if operation == Operation::Status { work.reply.clone().send(Ok(status(&work, "unsupported"))); } else { @@ -405,7 +406,7 @@ impl HostService for DeviceService { ); return; } - if !prompt_supported() { + if !prompt_supported(self.family) { work.fail( "unsupported_platform", "Trusted device consent is unavailable on this platform", diff --git a/crates/shell/src/platform_services/tests.rs b/crates/shell/src/platform_services/tests.rs index 2c35b29c5..87d44fd59 100644 --- a/crates/shell/src/platform_services/tests.rs +++ b/crates/shell/src/platform_services/tests.rs @@ -141,7 +141,7 @@ fn glance_devices_use_host_consent_for_declared_and_legacy_apps() { let replies = services::take_replies_for(&[heap]); assert_eq!(replies.len(), 1); let error = replies[0].2.as_ref().unwrap_err(); - if permission_supported() { + if permission_supported("camera") { assert_eq!( error, "camera.permission.request is unavailable to agents/background surfaces" @@ -257,7 +257,7 @@ fn descriptors_do_not_claim_unimplemented_location_or_background_authorization() for family in ["camera", "microphone", "location"] { let service = DeviceService { family }; let methods = service.api_methods(); - if !permission_supported() { + if !permission_supported(family) { assert_eq!(methods.len(), usize::from(family == "camera")); continue; } @@ -266,7 +266,7 @@ fn descriptors_do_not_claim_unimplemented_location_or_background_authorization() .find(|m| m.name.ends_with("permission.request")) .unwrap(); assert_eq!(request.agent_access, services::AgentAccess::ForegroundOnly); - assert!(!request.platforms.contains(&"windows".into())); + assert_eq!(request.platforms.contains(&"windows".into()), family == "microphone"); assert!(request.input_schema["additionalProperties"] == false); assert_eq!( methods.iter().any(|m| m.name == "location.get"), @@ -361,7 +361,7 @@ fn location_samples_validate_bounds_age_accuracy_and_platform_data() { } let methods = DeviceService { family: "location" }.api_methods(); let sample = methods.iter().find(|m| m.name == "location.sample"); - assert_eq!(sample.is_some(), permission_supported()); + assert_eq!(sample.is_some(), permission_supported("location")); if let Some(sample) = sample { assert_eq!(sample.agent_access, services::AgentAccess::ForegroundOnly); assert_eq!(sample.platforms, ["android", "macos"]); @@ -374,7 +374,7 @@ fn location_samples_validate_bounds_age_accuracy_and_platform_data() { /// events. A child process owns global registries; no OS/device is opened. #[test] fn location_sampling_broker_lifecycle() { - if !permission_supported() { + if !permission_supported("location") { return; } const CHILD: &str = "OCTOSENSE_LOCATION_TEST_CHILD"; diff --git a/docs/adr/0012-app-host-api-discovery.md b/docs/adr/0012-app-host-api-discovery.md index 2a034d6fc..a09938729 100644 --- a/docs/adr/0012-app-host-api-discovery.md +++ b/docs/adr/0012-app-host-api-discovery.md @@ -145,8 +145,12 @@ Embedded backend login is implemented on macOS/Android; Windows/Linux keep the separate external-browser authentication path, with platform execution still requiring acceptance. The subsequent [desktop browser extension](../desktop-embedded-browser.md) adds ordinary `WebReader` on Windows and Linux X11/XWayland; native Wayland and missing -engines fail visibly. This does not enable embedded backend authentication. Google Android login remains unsupported. The new device -adapter does not advertise Windows/Linux/iOS support. No broad OS access, +engines fail visibly. This does not enable embedded backend authentication. Google Android login remains unsupported. The device +adapter now advertises Windows only for microphone permission and recording in +the Issue #3 host extension. The Windows consent adapter cannot verify the OS +privacy setting; a successful consent request is not proof of audio frames. +Camera and location remain unsupported there. Linux/iOS are not advertised. +No broad OS access, arbitrary Rust/native library execution, or Wasm loading is added. ## Evidence and remaining acceptance diff --git a/docs/adr/0012-app-host-api-discovery.zh-CN.md b/docs/adr/0012-app-host-api-discovery.zh-CN.md index f99e30960..89cce4bc2 100644 --- a/docs/adr/0012-app-host-api-discovery.zh-CN.md +++ b/docs/adr/0012-app-host-api-discovery.zh-CN.md @@ -72,8 +72,10 @@ flowchart TD macOS/Android 实现嵌入式后端登录;Windows/Linux 保留独立的外部浏览器认证 路径,仍需平台执行验收。后续[桌面浏览器扩展](../desktop-embedded-browser.zh-CN.md)为 Windows 和 Linux X11/XWayland 提供普通 `WebReader`;原生 Wayland 或引擎缺失时明确报错。 -这不启用嵌入式后端认证。Google Android 登录仍不受支持。新设备适配器不宣称支持 -Windows/Linux/iOS。不增加广泛系统访问、任意 Rust/原生动态库执行或 Wasm 加载。 +这不启用嵌入式后端认证。Google Android 登录仍不受支持。设备适配器在 Issue #3 +宿主扩展中仅为 Windows 增加麦克风授权与录音;Windows 授权适配器不能核实系统隐私 +设置,授权成功不代表已收到音频帧。摄像头和位置在 Windows 仍不受支持;Linux/iOS +仍不列入。不增加广泛系统访问、任意 Rust/原生动态库执行或 Wasm 加载。 ## 证据与待完成验收 diff --git a/docs/architecture.md b/docs/architecture.md index 23bda75e1..a8dceb97f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -364,7 +364,7 @@ flowchart TB **Mail sending** never goes through the router. Mail's own composer (`mail.review_send`) and its reply cards end in one host-owned review of the exact message, drawn by the shell inside the card (`crates/shell/src/mail_review.rs`); Mail's agent can only propose a send (`mail.propose_send`). Only a trusted press and release of the review's Approve & Send control sends, and only a physical press is trusted: a tap on Android or a click on macOS. Developer mode and standing rules cannot authorize a send, and `mail.send` answers only `approval_required`. [Composed Mail cards](mail-composable-cards.md) describes the flow. -Two reviewed Makepad patches establish that trust (`tools/runtime-patches/`): `makepad-trusted-user-input.patch` for Android touchscreens, and `makepad-desktop-trusted-input.patch` for macOS pointer events that come from the HID source and that no other process posted. Synthetic and remote input are refused, and so is a native callback that runs inside their dispatch. Input on Windows or Linux, and accessibility input, cannot approve a send. The macOS path is **unverified**: no real message has been sent from a Mac. +The Makepad input patches (`tools/runtime-patches/`) mark Android touchscreen and macOS HID pointer events as trusted. The Windows extension marks hardware-class pointer input as trusted too; Linux input remains untrusted. A Windows UIAccess injector may be classified as hardware by the OS, so this signal must not be described as proof that a person pressed the control. Approval sheets still require the host review and a live press/release. The macOS path is **unverified**: no real message has been sent from a Mac. **Connected-account writes and sends** do not go through the router either. A GitHub commit (`github.review_save`) or a Google Calendar write (`gcalendar.review_save`) opens a host review of the exact change, and only its Approve & Save control saves it. A Gmail send (`gmail.draft.review`) opens the host's review of the exact reply, and only its Approve & Send control sends. On `main`, all three reviews are native (`crates/shell/src/connected_review.rs`) and accept only a trusted physical press and release, and each approval works once (`crates/oauth-service/src/host_api.rs`, `host_inbox.rs`). `desktop-v0.1.0-beta.2` checks for a physical press only on the Gmail send; its GitHub and Calendar saves use a host sheet that does not check how Approve & Save was pressed. An agent cannot open any of these screens: its tool calls reach the services with `may_prompt: false` (`host_tools/script_apps.rs`). The platform table in the [OAuth service README](../crates/oauth-service/README.md#current-delivery-boundary) shows where the physical press is supported and what is verified. diff --git a/docs/host-os-api-status.md b/docs/host-os-api-status.md index f175cb90f..654e4998c 100644 --- a/docs/host-os-api-status.md +++ b/docs/host-os-api-status.md @@ -15,7 +15,7 @@ This page describes the **OS and public API integration** in [Desktop RC2](https | Device calendars | `device_calendar.*` adds app/account-scoped consent, calendar selection, bounded event reads and physically reviewed writes; it is separate from `calendar.*` and `gcalendar.*`. | EventKit on macOS and CalendarProvider in Android Home. Actual OS-calendar interaction is pending; see the [device-calendar contract](../crates/shell/src/device_calendar/README.md). | | Public Mail drafts | `mail.compose@1` and `mail.compose_status@1` keep versioned drafts for the caller's app/account. `mail.review_send@1` and the `mail.send@1` compatibility route require native review; neither gives an agent direct SMTP access. | Send review is implemented on macOS/Android. Linux/Windows can compose/read status but have no physical-send adapter. Real SMTP acceptance is unverified; see the [Mail contract](../apps/mail/host-service/README.md). | | Photo selection and text sharing | `files.pick_photo@1` imports a signature-checked PNG/JPEG/WebP through an image-filtered document chooser; it does not grant photo-library access. `files.share@1` hands off bounded text, with no attachments. | Photo selection follows document-adapter availability. Text sharing is Android-only and reports chooser handoff, not delivery. Native chooser acceptance is pending. | -| Standalone audio | `microphone.record_start/record_status/record_stop/record_cancel@1` and `audio.play/status/stop@1` provide short, foreground-only, app-isolate-owned sessions using app storage. Recording requires the app and OS microphone grants. | macOS/Android source implementations. Real recording and playback acceptance is pending; these APIs do not add ASR/TTS or background streaming. See [audio sessions](../crates/shell/src/audio_service/README.md). | +| Standalone audio and transcription | `microphone.record_start/record_status/record_stop/record_cancel@1` records a bounded app-local WAV; `audio.play/status/stop@1` is the separate speaker path. `model.transcribe_capability@1` probes configured MiniMax ASR and `model.transcribe@1` sends a foreground app-local WAV through a host-owned provider route. | Recording: macOS, Android, Windows. Playback: macOS, Android; Windows speaker support is not advertised. Transcription: Windows only, with a separate 300-second daily quota and no speech output. Windows microphone consent cannot verify the OS privacy setting; real capture is verified only in the task-owned host. See [audio sessions](../crates/shell/src/audio_service/README.md) and [model media](../apps/ai-providers/host-service/MEDIA.md). | | Video widget controls | `video.playback_controls@1` advertises the Splash `Video` widget ABI: preparation, playback, pause/resume, seek, volume/rate and state/error inspection. It is **not callable through `host.request`**. | Native macOS playback passed 16 checks; OnePlus 6 / Android 15 passed 20 callback/decoder checks. The [separate fixture](../tools/fixtures/video-api-lab/README.md) uses only a synthetic local MP4; other codecs/backends remain unverified. | File transfers keep the whole-jail quota and 256-entry bounds. `files.import` takes a document of up to 64 MiB (16 MiB on Android, `max_import_bytes`): a worker stages it beside the jail and the UI links it in between script turns, so its bytes never pass through the script heap. Photo picks and exports keep the 1 MiB per-file bound. Import refuses overwrite and checks ownership, traversal and symlinks. With live app storage, `files.status` also reports `used_bytes` (the host's measure of the app's jail) and `quota_bytes` (the calling isolate's storage quota), so an app can show its storage use without counting its own files. Dialogs and callbacks retain the authenticated app/isolate identity; grants and lifetime are checked again on completion. Closing an app invalidates pending work, but cannot roll back an export whose native write has already started. Android document picks do not retain persistent URI grants. `files.pick_photo` adds an image-filtered document selection path, not a dedicated photo-library grant or large-blob API. Its 1 MiB bound explicitly refuses many original phone photos. See [file-transfer contract](../crates/shell/src/files_service/README.md), `FilesService::api_methods`, `Work::storage` and `handle_event` in [files_service/mod.rs](../crates/shell/src/files_service/mod.rs). diff --git a/docs/host-os-api-status.zh-CN.md b/docs/host-os-api-status.zh-CN.md index e0bbe37b4..0e9292674 100644 --- a/docs/host-os-api-status.zh-CN.md +++ b/docs/host-os-api-status.zh-CN.md @@ -15,7 +15,7 @@ | 设备日历 | `device_calendar.*` 增加应用/账户范围的授权、日历选择、有界事件读取及需要亲手批准的写入;它与 `calendar.*`、`gcalendar.*` 分开。 | macOS EventKit、Android Home CalendarProvider。真实 OS 日历交互仍待验收,见[设备日历契约](../crates/shell/src/device_calendar/README.zh-CN.md)。 | | 公共邮件草稿 | `mail.compose@1`、`mail.compose_status@1` 为调用应用/账户保留版本化草稿。`mail.review_send@1` 与兼容入口 `mail.send@1` 均需要原生审阅,不给 Agent 直接 SMTP 发送权限。 | macOS/Android 实现发送审阅;Linux/Windows 可编辑草稿和读取状态,没有亲手批准发送的适配器。真实 SMTP 接受结果未验证,见[邮件契约](../apps/mail/host-service/README.zh-CN.md)。 | | 选择照片、分享文本 | `files.pick_photo@1` 通过按图片过滤的文档选择器导入经文件签名检查的 PNG/JPEG/WebP,不授予照片库访问权。`files.share@1` 交接有大小限制的文本,不支持附件。 | 照片选择沿用文档适配器的平台范围;文本分享仅支持 Android,返回选择器交接结果,不证明送达。原生选择器验收仍待完成。 | -| 独立音频 | `microphone.record_start/record_status/record_stop/record_cancel@1`、`audio.play/status/stop@1` 提供使用应用存储、归属应用 isolate、仅限前台的短时会话。录音需要应用和 OS 麦克风授权。 | macOS/Android 源码实现;真实录音和播放仍待验收。不新增 ASR/TTS 或后台流媒体,见[音频会话](../crates/shell/src/audio_service/README.zh-CN.md)。 | +| 独立音频与转写 | `microphone.record_start/record_status/record_stop/record_cancel@1` 录制有界应用本地 WAV;`audio.play/status/stop@1` 是独立扬声器路径。`model.transcribe_capability@1` 探测 MiniMax ASR 配置,`model.transcribe@1` 将前台应用本地 WAV 经宿主提供方路由转写。 | 录音支持 macOS、Android、Windows;播放仅列 macOS、Android,Windows 扬声器尚未列为支持;转写仅限 Windows,每应用每天另有 300 秒限额,不提供语音播报。Windows 麦克风授权不能核实系统隐私设置;真实采集只在任务专用宿主验收。见[音频会话](../crates/shell/src/audio_service/README.zh-CN.md)与[模型媒体](../apps/ai-providers/host-service/MEDIA.zh-CN.md)。 | | Video 控件操作 | `video.playback_controls@1` 表示 Splash `Video` 控件 ABI,包含准备、播放、暂停/继续、定位、音量/速度和状态/错误查询;**不能通过 `host.request` 调用**。 | macOS 原生播放通过 16 项检查,OnePlus 6 / Android 15 通过 20 项回调/解码器检查。[单独测试](../tools/fixtures/video-api-lab/README.zh-CN.md) 只使用本地合成 MP4;其他编解码器及后端仍未验证。 | 文件传输沿用整个 jail 的字节配额及 256 条目录项限制。`files.import` 可导入最大 64 MiB 的文档(Android 为 16 MiB,见 `max_import_bytes`):工作线程把它暂存在 jail 旁,UI 在脚本轮次之间把它链接进来,字节不经过脚本堆。图片选择和导出仍使用单文件 1 MiB 上限。导入拒绝覆盖,并检查归属、路径越界和符号链接。有可用的应用存储时,`files.status` 还返回 `used_bytes`(宿主测量的应用 jail 占用)和 `quota_bytes`(调用方隔离实例的存储配额),应用无需自行统计文件即可显示存储用量。对话框及回调使用经认证的应用、isolate 身份,完成时再次检查授权与生命周期。关闭应用使等待中的请求失效,但无法回滚已经开始写入原生目标的导出。Android 文档选择不保留持久 URI 授权。`files.pick_photo` 新增按图片过滤的文档选择路径,不是专用照片库授权或大文件句柄 API。其 1 MiB 上限会明确拒绝很多手机原始照片。详见[文件传输契约](../crates/shell/src/files_service/README.zh-CN.md),以及 [files_service/mod.rs](../crates/shell/src/files_service/mod.rs) 中的 `FilesService::api_methods`、`Work::storage`、`handle_event`。 diff --git a/runtime-patches.lock.json b/runtime-patches.lock.json index 35933d1d4..357e2886e 100644 --- a/runtime-patches.lock.json +++ b/runtime-patches.lock.json @@ -4,7 +4,7 @@ "base_revision": "d653bc4493619ab979ccff165cf48c0c63b77d19", "patch": "tools/runtime-patches/makepad-settings.patch", "sha256": "3a79cddddc89dc4d064a9f0bdd17dc8111dcab9c796a334d95f61ecb9d5011aa", - "tree": "d047e32fdfb91aa26b95a62464645020bfa6a4f7", + "tree": "6a9d14109fa8c8d1e85a78e807ed2432be9a5f8c", "product_fixes": [ "Settings integration: Android IME synchronization, accessibility and reduced motion, GLES program-cache validation, obscured-gesture cancellation and scroll-position observation.", "Contained-app isolation and secret-input restrictions come from the pinned upstream runtime; the Settings overlay preserves those gates.", @@ -48,8 +48,11 @@ "A native host may stage up to 64 MiB into an app's jail (MAX_IMPORT_BYTES; prepare_import and commit_import), so files.import can take documents above the 1 MiB script write limit. Staged bytes never enter the script heap; the jail's byte quota and entry cap still apply, and script writes, photo picks and exports keep 1 MiB.", "An OS resume restarts only the Video playback its pause stopped, including a start it cancelled; a player paused by the person, or paused or stopped by its app during the pause, stays paused (#406).", "Camera capture and recording use explicit per-call boolean audio/library options, defaulting to silent app-local output. Permission availability never opts into recording audio or exporting captures; async microphone checks retain capture intent and background provenance. Unsupported library export fails before capture.", + "Windows user32 hardware-class pointer messages carry scoped input provenance to host review sheets. Posted and ordinary injected messages remain untrusted; UIAccess injection can be classified as hardware, and remote-desktop behavior is unverified. Approval eligibility changes across host services, not only camera consent.", "A hidden Android native composer remembers animated IME insets without requesting a root layout every frame; showing it applies the latest inset before its first visible frame.", - "Launch-input hardening: the Studio connection and the OCTOS_PROXY app HTTP proxy, which Makepad itself acts on from an Android Intent extra, are honoured only when the build opts into launch-time development input (debuggable, or the manifest meta-data dev.makepad.LAUNCH_STUDIO). A shipped build ignores them and drops any Studio host a development build saved. The exported activity would otherwise let another installed app point it at a host or a proxy." + "Launch-input hardening: the Studio connection and the OCTOS_PROXY app HTTP proxy, which Makepad itself acts on from an Android Intent extra, are honoured only when the build opts into launch-time development input (debuggable, or the manifest meta-data dev.makepad.LAUNCH_STUDIO). A shipped build ignores them and drops any Studio host a development build saved. The exported activity would otherwise let another installed app point it at a host or a proxy.", + "A contained TextInput exposes take_key_focus to its own Splash script so an explicit transcription can return keyboard focus to the editable Agent draft.", + "Windows WM_ACTIVATE treats mouse click activation as focus gain, preserving foreground device access when a person enters the app by clicking its window." ], "stacked": [ { @@ -188,6 +191,10 @@ "patch": "tools/runtime-patches/makepad-camera-capture-intent.patch", "sha256": "d056b76d2eb6b55aa9c96447a49fc90d32a9a235b6bc9a1ff125902a73c28208" }, + { + "patch": "tools/runtime-patches/makepad-windows-trusted-input.patch", + "sha256": "a72d973f7d7a2d629a87bb83df85e5b91b2acd6d8a322796a2637d5b8006a102" + }, { "patch": "tools/runtime-patches/makepad-hidden-composer-insets.patch", "sha256": "75935d129cfe4adf09555d3a94b0a7ad387f2ea97cf357f9d60c728d87239037" @@ -195,6 +202,14 @@ { "patch": "tools/runtime-patches/makepad-launch-input-hardening.patch", "sha256": "8eaafef3ef0591bec2a25c05d5187564d5dfb87a32e6b937b6d9c4e9f6fa5b84" + }, + { + "patch": "tools/runtime-patches/makepad-text-input-focus.patch", + "sha256": "cfd6d14537ab6125bdc53f42a9133ee8f92bd47b14b6df215c35522df8a19c03" + }, + { + "patch": "tools/runtime-patches/makepad-windows-click-focus.patch", + "sha256": "e4658270c78162ce71949fdb5fcd60d51e9e2eaca7216ca277fc5555cd55fd2e" } ] } diff --git a/tools/runtime-patches/makepad-text-input-focus.patch b/tools/runtime-patches/makepad-text-input-focus.patch new file mode 100644 index 000000000..ad925a660 --- /dev/null +++ b/tools/runtime-patches/makepad-text-input-focus.patch @@ -0,0 +1,14 @@ +diff --git a/widgets/src/text_input.rs b/widgets/src/text_input.rs +--- a/widgets/src/text_input.rs ++++ b/widgets/src/text_input.rs +@@ -2403,6 +2403,10 @@ impl Widget for TextInput { + method: LiveId, + args: ScriptValue, + ) -> ScriptAsyncResult { ++ if method == live_id!(take_key_focus) { ++ vm.with_cx_mut(|cx| self.take_key_focus(cx)); ++ return ScriptAsyncResult::Return(NIL); ++ } + if method == live_id!(text) { + let str_val = vm.bx.heap.new_string_from_str(&self.text); + return ScriptAsyncResult::Return(str_val.into()); diff --git a/tools/runtime-patches/makepad-windows-click-focus.patch b/tools/runtime-patches/makepad-windows-click-focus.patch new file mode 100644 index 000000000..9b8d64ee7 --- /dev/null +++ b/tools/runtime-patches/makepad-windows-click-focus.patch @@ -0,0 +1,41 @@ +diff --git a/platform/src/os/windows/win32_window.rs b/platform/src/os/windows/win32_window.rs +index 8cd0ceb..1f7620e 100644 +--- a/platform/src/os/windows/win32_window.rs ++++ b/platform/src/os/windows/win32_window.rs +@@ -112,6 +112,27 @@ + }, + }; + ++// WA_CLICKACTIVE is 2 in WinUser.h, but this pinned windows-rs binding omits it. ++const WA_CLICKACTIVE_VALUE: usize = 2; ++ ++fn window_activation_gained(wparam: usize) -> bool { ++ let state = wparam & 0xffff; ++ state == WA_ACTIVE as usize || state == WA_CLICKACTIVE_VALUE ++} ++ ++#[cfg(test)] ++mod activation_tests { ++ use super::*; ++ ++ #[test] ++ fn mouse_and_programmatic_activation_gain_focus_but_deactivation_loses_it() { ++ assert!(window_activation_gained(WA_ACTIVE as usize)); ++ assert!(window_activation_gained(WA_CLICKACTIVE_VALUE)); ++ assert!(window_activation_gained((1 << 16) | WA_CLICKACTIVE_VALUE)); ++ assert!(!window_activation_gained(0)); ++ } ++} ++ + /// Whether a screen coordinate survives the conversion `CreateWindowExW` and `MoveWindow` + /// take: a real number inside `i32`, and not the `CW_USEDEFAULT` sentinel that `i32::MIN` + /// would be read as. +@@ -795,7 +816,7 @@ unsafe fn imm_get_composition_string( + let window = &mut (*(user_data as *mut Win32Window)); + match msg { + WM_ACTIVATE => { +- if wparam.0 & 0xffff == WA_ACTIVE as usize { ++ if window_activation_gained(wparam.0) { + window.do_callback(Win32Event::WindowGotFocus(window.window_id)); + } else { + if window.is_popup { diff --git a/tools/runtime-patches/makepad-windows-trusted-input.patch b/tools/runtime-patches/makepad-windows-trusted-input.patch new file mode 100644 index 000000000..0797ba82f --- /dev/null +++ b/tools/runtime-patches/makepad-windows-trusted-input.patch @@ -0,0 +1,115 @@ +diff --git a/platform/src/os/windows/win32_event.rs b/platform/src/os/windows/win32_event.rs +index 8a02d73..3299138 100644 +--- a/platform/src/os/windows/win32_event.rs ++++ b/platform/src/os/windows/win32_event.rs +@@ -30,8 +30,8 @@ pub enum Win32Event { + primary: bool, + }, + +- MouseDown(MouseDownEvent), +- MouseUp(MouseUpEvent), ++ MouseDown(MouseDownEvent, bool), ++ MouseUp(MouseUpEvent, bool), + MouseMove(MouseMoveEvent), + MouseLeave(MouseLeaveEvent), + Scroll(ScrollEvent), +diff --git a/platform/src/os/windows/win32_window.rs b/platform/src/os/windows/win32_window.rs +index 01a356d..3c79a78 100644 +--- a/platform/src/os/windows/win32_window.rs ++++ b/platform/src/os/windows/win32_window.rs +@@ -1864,6 +1864,26 @@ pub fn send_focus_lost_event(&mut self) { + self.do_callback(Win32Event::WindowLostFocus(self.window_id)); + } + ++ fn os_reported_pointer_message() -> bool { ++ // This pinned windows-rs shim lacks Win32::UI::Input's message-origin ++ // bindings. Query user32 directly while processing the native message. ++ #[repr(C)] ++ #[derive(Default)] ++ struct InputMessageSource { device_type: i32, origin_id: i32 } ++ #[link(name = "user32")] ++ unsafe extern "system" { ++ fn GetCurrentInputMessageSource(source: *mut InputMessageSource) -> i32; ++ } ++ let mut source = InputMessageSource::default(); ++ let received = unsafe { GetCurrentInputMessageSource(&mut source) != 0 }; ++ received && Self::trusted_pointer_source(source.device_type, source.origin_id) ++ } ++ ++ fn trusted_pointer_source(device_type: i32, origin_id: i32) -> bool { ++ matches!(device_type, 2 | 4 | 8 | 16) // mouse, touch, pen, touchpad ++ && origin_id == 1 // IMO_HARDWARE also includes UIAccess-injected input ++ } ++ + pub fn send_mouse_down(&mut self, button: MouseButton, modifiers: KeyModifiers) { + if self.mouse_buttons_down == 0 { + unsafe { +@@ -1878,7 +1898,7 @@ pub fn send_mouse_down(&mut self, button: MouseButton, modifiers: KeyModifiers) + abs: self.last_mouse_pos, + time: self.time_now(), + handled: Cell::new(Area::Empty), +- })); ++ }, Self::os_reported_pointer_message())); + } + + pub fn send_mouse_up(&mut self, button: MouseButton, modifiers: KeyModifiers) { +@@ -1896,7 +1916,7 @@ pub fn send_mouse_up(&mut self, button: MouseButton, modifiers: KeyModifiers) { + window_id: self.window_id, + abs: self.last_mouse_pos, + time: self.time_now(), +- })); ++ }, Self::os_reported_pointer_message())); + } + + pub fn send_mouse_move(&mut self, pos: Vec2d, modifiers: KeyModifiers) { +@@ -2085,3 +2105,19 @@ pub fn virtual_key_to_key_code(wparam: WPARAM) -> KeyCode { + } + } + } ++ ++#[cfg(test)] ++mod input_source_tests { ++ use super::Win32Window; ++ ++ #[test] ++ fn only_os_hardware_class_pointer_messages_are_trusted() { ++ for device in [2, 4, 8, 16] { ++ assert!(Win32Window::trusted_pointer_source(device, 1)); ++ assert!(!Win32Window::trusted_pointer_source(device, 2)); ++ assert!(!Win32Window::trusted_pointer_source(device, 4)); ++ } ++ assert!(!Win32Window::trusted_pointer_source(0, 1)); ++ assert!(!Win32Window::trusted_pointer_source(1, 1)); ++ } ++} +diff --git a/platform/src/os/windows/windows.rs b/platform/src/os/windows/windows.rs +index fb1dd91..4d1a501 100644 +--- a/platform/src/os/windows/windows.rs ++++ b/platform/src/os/windows/windows.rs +@@ -293,11 +293,11 @@ fn win32_event_callback( + let time_now = with_win32_app(|app| app.time_now()); + self.paint_tick(time_now, true, d3d11_cx, d3d11_windows); + } +- Win32Event::MouseDown(mut e) => { ++ Win32Event::MouseDown(mut e, trusted) => { + self.dpi_override_scale(&mut e.abs, e.window_id); + self.fingers.process_tap_count(e.abs, e.time); + self.fingers.mouse_down(e.button, e.window_id); +- self.call_event_handler(&Event::MouseDown(e.into())) ++ self.call_event_handler_with_input_trust(&Event::MouseDown(e.into()), trusted) + } + Win32Event::MouseMove(mut e) => { + self.dpi_override_scale(&mut e.abs, e.window_id); +@@ -305,10 +305,10 @@ fn win32_event_callback( + self.fingers.cycle_hover_area(live_id!(mouse).into()); + self.fingers.switch_captures(); + } +- Win32Event::MouseUp(mut e) => { ++ Win32Event::MouseUp(mut e, trusted) => { + self.dpi_override_scale(&mut e.abs, e.window_id); + let button = e.button; +- self.call_event_handler(&Event::MouseUp(e.into())); ++ self.call_event_handler_with_input_trust(&Event::MouseUp(e.into()), trusted); + self.fingers.mouse_up(button); + self.fingers.cycle_hover_area(live_id!(mouse).into()); + }