Add foreground Windows app voice transcription

apps/ai-providers/host-service/MEDIA.md

@@ -54,13 +54,21 @@ one place only. Prompts and text are UTF-8 bytes for input limits.
 | `model.video.cancel` | `job` | Confirmed cancellation of a queued remote job, or an error |
 | `model.capabilities` | `{}` | `configured` flags, limits and lifecycle note |
 | `model.budget` | `{}` | Existing call/token budget plus `media` resource usage and limits |
+| `model.transcribe_capability` (Windows) | `{}` | `schema:1`, `supported`, `configured`; no provider identity or credential |
+| `model.transcribe` (Windows, foreground) | `audio:{app_path}`, a WAV inside the caller's app storage | Trimmed `text`, `duration_seconds` |
 
 Image sizes are `512x512`, `1024x1024` (default), `1536x1024`, and
 `1024x1536`. The OpenAI route rejects `512x512` and `seed` before submission;
 the MiniMax route supports both. Only one image per request is supported.
 Generated PNG/JPEG dimensions must match the request. Audio output is MP3;
 apps should disclose that the voice is AI-generated. Duration is omitted
 when the provider does not report it; the host does not invent a value.
+Transcription accepts only a 16 kHz mono PCM16 WAV of at most 1 MiB and
+30 seconds, from the calling app's own storage. The host selects a configured
+MiniMax/MiniMax CN/MiniMaxI route and uploads the WAV once. Any installed
+Windows app with the model grant can probe this method; the catalog does not
+promise a configured provider. The provider must return nonempty text and a
+valid duration. The app receives neither the provider body nor credentials.
 Embeddings must be finite numeric vectors with consistent dimensions, at
 most 4096 dimensions. Provider batch indices are validated and reordered.
 
@@ -102,6 +110,7 @@ model call/rate budget. Media also reserves its own daily units before HTTP:
 | --- | --- |
 | Image output pixels | 8,388,608 |
 | Speech input characters | 10,000 |
+| Transcription input seconds | 300 |
 | Video duration seconds | 12 |
 | Embedding input UTF-8 bytes | 100,000 |
 

apps/ai-providers/host-service/MEDIA.zh-CN.md

@@ -50,11 +50,17 @@
 | `model.video.cancel` | `job` | 远端排队任务确认取消，或返回错误 |
 | `model.capabilities` | `{}` | `configured` 标志、上限和生命周期说明 |
 | `model.budget` | `{}` | 现有调用/令牌预算，加上 `media` 资源使用量和上限 |
+| `model.transcribe_capability`（Windows） | `{}` | `schema:1`、`supported`、`configured`；不返回提供方身份或凭证 |
+| `model.transcribe`（Windows 前台） | `audio:{app_path}`，调用应用存储中的 WAV | 去首尾空白后的 `text`、`duration_seconds` |
 
 图片尺寸为 `512x512`、`1024x1024`（默认）、`1536x1024` 和 `1024x1536`。
 OpenAI 路由会在提交前拒绝 `512x512` 和 `seed`；MiniMax 路由支持两者。
 每次请求只生成一张图片。PNG/JPEG 的实际尺寸必须匹配请求。语音输出为 MP3；应用
 应说明声音由 AI 生成。供应商不提供时长时不返回该字段，宿主不会编造时长。
+转写只接受调用应用自身存储中的 16 kHz 单声道 PCM16 WAV，最多 1 MiB、30 秒。
+宿主选择已配置的 MiniMax/MiniMax CN/MiniMaxI 路由，仅上传一次。任何获准使用模型的
+Windows 已安装应用都可探测此方法；方法目录不保证提供方已配置。提供方必须返回
+非空文字和有效时长；应用不会收到提供方原始响应或凭证。
 向量必须由有限数值组成、维数一致且不超过 4096；供应商批量索引经过校验和排序。
 
 H3 提供分辨率级别和宽高比，不提供精确像素尺寸、帧率或种子选项。
@@ -88,6 +94,7 @@ MiniMax H3 需要按量计费 API 权益；仅有 M Plan/聊天账户并不足
 | --- | --- |
 | 图片输出像素 | 8,388,608 |
 | 语音输入字符 | 10,000 |
+| 转写输入秒数 | 300 |
 | 视频时长秒数 | 12 |
 | 向量输入 UTF-8 字节 | 100,000 |
 

apps/ai-providers/host-service/README.md

@@ -264,3 +264,27 @@ code now reports denied caller admission, not a missing declaration.
 Tests use a fake transport: `cargo test --locked -p octosense-llm-service --test complete`.
 A live check against DeepSeek (costs a fraction of a cent):
 `DEEPSEEK_API_KEY=… cargo run --locked -p octosense-llm-service --example model_complete_live`.
+
+## App-local speech transcription on Windows
+
+The local Windows host exposes `model.transcribe_capability({})` and
+`model.transcribe({"audio":{"app_path":"/agent/voice.wav"}})` to admitted
+installed apps with the model grant. The transcription call additionally requires foreground.
+Capability returns only `schema`, `supported`
+and `configured`; transcription returns only `text` and `duration_seconds`.
+Other platforms do not advertise this bridge. The host rechecks the caller's
+grant and current provider before returning a result.
+
+Transcription reads a nonempty, mono 16 kHz PCM16 WAV of at most 1 MiB from
+the calling app's own storage. The host chooses the first usable MiniMax provider
+in the person's configured order, keeps its key in the host, and sends one
+bounded multipart upload using `asr-1.0`. Kimi-only configurations report
+`configured: false`. The upload is charged to the existing per-app call/rate
+budget and a separate 300-second daily transcription quota. It is never
+automatically retried; failures expose stable error categories without provider
+response bodies. No speech model settings are added. Fake-transport coverage is in
+`cargo test --locked -p octosense-llm-service --test media`. On 2026-10-13,
+the task-owned Windows Pantry candidate recorded real speech and returned a
+MiniMax transcript to its editable input after manual and automatic Stop.
+That live check used a person-configured host provider; it did not test other
+apps, providers or platforms.

apps/ai-providers/host-service/README.zh-CN.md

@@ -83,3 +83,22 @@ cargo test --locked -p octosense-llm-service --test complete
 模型测试使用假传输，覆盖省略能力声明后的调用、缺少宿主准入时拒绝、输出校验、预算和取消。媒体测试还检查账号切换、准入撤销、并发限制及迟到回复。这些测试不验证真实模型供应商或手机。
 
 真实供应商示例 `model_complete_live` 会产生真实请求和费用，只有明确进行该项验收时才运行。macOS 钥匙串测试也默认忽略，需显式选择 `-- --ignored keychain`。仓库只使用根目录统一固定的依赖及 `.sources/`，不增加个人机器路径或本地 patch。
+
+## Windows 应用本地语音转写
+
+本地 Windows 宿主向获准使用模型的已安装应用开放
+`model.transcribe_capability({})` 和
+`model.transcribe({"audio":{"app_path":"/agent/voice.wav"}})`。能力探测仅返回
+`schema`、`supported`、`configured`，转写仅返回 `text`、`duration_seconds`。
+实际转写还要求应用处于前台。其他平台不列出这条桥；回复前会重新检查调用应用的准入及供应商配置。
+
+宿主只读取调用应用自身存储中非空、单声道 16 kHz PCM16、最多 1 MiB 的 WAV。
+它按配置顺序选首个可用 MiniMax，用宿主密钥发送一次固定 `asr-1.0` multipart
+请求。只配置 Kimi 时 `configured` 为 false。上传共用应用调用次数及速率预算，
+并单独计入每天 300 秒的转写限额，
+不会自动重试；失败只提供稳定错误类别，不暴露供应商响应正文。
+本阶段不增加语音模型设置。
+`cargo test --locked -p octosense-llm-service --test media` 使用假传输验证。
+2026-10-13 在任务专用 Windows Pantry 候选中，真实语音经手动停止与自动停止后，
+MiniMax 转写均返回到可编辑输入框。该现场检查使用用户在宿主配置的提供方，
+不证明其他应用、提供方或平台的支持。

apps/ai-providers/host-service/src/complete/media/mod.rs

@@ -13,6 +13,7 @@ use std::time::Duration;
 use url::Url;
 
 mod wire;
+mod transcribe;
 
 pub const RESPONSE_MAX: usize = 4 * 1024 * 1024;
 pub const ASSET_MAX: usize = 2 * 1024 * 1024;
@@ -29,6 +30,7 @@ const JOBS: usize = 64;
 pub struct Limits {
     pub image_pixels_per_day: u64,
     pub audio_characters_per_day: u64,
+    pub transcription_seconds_per_day: u64,
     pub video_seconds_per_day: u64,
     pub embedding_bytes_per_day: u64,
 }
@@ -37,6 +39,7 @@ impl Default for Limits {
         Self {
             image_pixels_per_day: 8 * 1024 * 1024,
             audio_characters_per_day: 10_000,
+            transcription_seconds_per_day: 300,
             video_seconds_per_day: 12,
             embedding_bytes_per_day: 100_000,
         }
@@ -47,6 +50,7 @@ impl Limits {
         match kind {
             "image" => self.image_pixels_per_day,
             "audio" => self.audio_characters_per_day,
+            "transcription" => self.transcription_seconds_per_day,
             "video" => self.video_seconds_per_day,
             "embeddings" => self.embedding_bytes_per_day,
             _ => 0,
@@ -64,6 +68,16 @@ pub trait Transport: Send + Sync {
         headers: &[(String, String)],
         body: Option<&str>,
     ) -> Result<(u16, Vec<u8>), String>;
+
+    fn request_bytes(
+        &self,
+        _method: &str,
+        _url: &str,
+        _headers: &[(String, String)],
+        _body: &[u8],
+    ) -> Result<(u16, Vec<u8>), String> {
+        Err("Binary media requests are unsupported by this transport.".into())
+    }
 }
 pub struct Http;
 impl Transport for Http {
@@ -100,6 +114,36 @@ impl Transport for Http {
         }
         Ok((status, bytes))
     }
+
+    fn request_bytes(
+        &self,
+        method: &str,
+        url: &str,
+        headers: &[(String, String)],
+        body: &[u8],
+    ) -> Result<(u16, Vec<u8>), String> {
+        let agent = ureq::AgentBuilder::new()
+            .timeout(Duration::from_secs(120))
+            .redirects(0)
+            .build();
+        let mut request = agent.request(method, url);
+        for (name, value) in headers {
+            request = request.set(name, value);
+        }
+        let response = match request.send_bytes(body) {
+            Ok(response) | Err(ureq::Error::Status(_, response)) => response,
+            Err(_) => return Err("The media provider could not be reached; the request may already have been accepted.".into()),
+        };
+        let status = response.status();
+        let mut bytes = Vec::new();
+        response.into_reader().take(RESPONSE_MAX as u64 + 1)
+            .read_to_end(&mut bytes)
+            .map_err(|_| "The media provider response was interrupted.".to_string())?;
+        if bytes.len() > RESPONSE_MAX {
+            return Err("The media provider response exceeded the host byte limit.".into());
+        }
+        Ok((status, bytes))
+    }
 }
 
 #[derive(Clone)]
@@ -166,7 +210,7 @@ impl State {
         if value["apps"].as_object().unwrap().values().any(|app| {
             app.as_object().is_none_or(|counts| {
                 counts.iter().any(|(kind, count)| {
-                    !matches!(kind.as_str(), "image" | "audio" | "video" | "embeddings")
+                    !matches!(kind.as_str(), "image" | "audio" | "transcription" | "video" | "embeddings")
                         || !count.is_u64()
                 })
             })
@@ -183,7 +227,7 @@ impl State {
         match Self::load_budget(root, now) {
             Ok(value) => {
                 let mut result = json!({"resets_at":(now/DAY_MS+1)*DAY_MS/1000});
-                for kind in ["image", "audio", "video", "embeddings"] {
+                for kind in ["image", "audio", "transcription", "video", "embeddings"] {
                     result[kind] = json!({"used":value["apps"][app][kind].as_u64().unwrap_or(0),"limit":self.limits.for_kind(kind)});
                 }
                 result
@@ -437,6 +481,8 @@ fn run_call(context: &Context) -> Result<Value, Refusal> {
     context.check()?;
     context.host.attach(&context.call.host_dir);
     match context.call.method() {
+        "transcribe_capability" => transcribe::capability(context),
+        "transcribe" => transcribe::run(context),
         "complete" => {
             let request = super::Request::from_args(&context.call.args)?;
             context
@@ -500,6 +546,17 @@ fn run(context: &Context, request: wire::Request) -> Result<Value, Refusal> {
 fn reserve(context: &Context, kind: &str, units: u64, now: u64) -> Result<(), Refusal> {
     // Both shared call/rate policy and modality resources must admit before any
     // billable request. Failed reservations are conservative, never refunded.
+    admit_call(context, now)?;
+    context.host.media.reserve(
+        &context.call.host_dir,
+        &context.call.app_id,
+        now,
+        kind,
+        units,
+    )
+}
+
+fn admit_call(context: &Context, now: u64) -> Result<(), Refusal> {
     {
         let mut ledger = context.host.ledger.lock().unwrap();
         ledger.attach(context.call.host_dir.join("model/ledger.json"));
@@ -514,13 +571,7 @@ fn reserve(context: &Context, kind: &str, units: u64, now: u64) -> Result<(), Re
             )
         })?;
     }
-    context.host.media.reserve(
-        &context.call.host_dir,
-        &context.call.app_id,
-        now,
-        kind,
-        units,
-    )
+    Ok(())
 }
 
 fn video_start(
@@ -755,12 +806,31 @@ fn public_url(raw: &str) -> Result<String, Refusal> {
 }
 
 pub(super) fn catalog() -> Vec<HostApiMethod> {
-    wire::schemas()
+    let mut methods: Vec<_> = wire::schemas()
         .into_iter()
         .map(|(name, summary, args, result)| {
             HostApiMethod::new(name, 1, "model", summary, args, result)
                 .with_platforms(&["macos", "android", "windows", "linux", "ios", "openharmony"])
                 .with_agent_access(AgentAccess::Allowed)
         })
-        .collect()
+        .collect();
+    methods.push(HostApiMethod::new("model.transcribe_capability", 1, "model",
+        "Check whether this Windows host can transcribe the calling app's WAV",
+        json!({"type":"object","additionalProperties":false}),
+        json!({"type":"object","required":["schema","supported","configured"],
+            "properties":{"schema":{"const":1},"supported":{"type":"boolean"},
+                "configured":{"type":"boolean"}},"additionalProperties":false}))
+        .with_platforms(&["windows"])
+        .with_agent_access(AgentAccess::Allowed));
+    methods.push(HostApiMethod::new("model.transcribe", 1, "model",
+        "Transcribe a bounded WAV from the calling app's storage",
+        json!({"type":"object","required":["audio"],"additionalProperties":false,
+            "properties":{"audio":{"type":"object","required":["app_path"],
+                "additionalProperties":false,"properties":{"app_path":{"type":"string","minLength":1}}}}}),
+        json!({"type":"object","required":["text","duration_seconds"],
+            "additionalProperties":false,"properties":{"text":{"type":"string","minLength":1},
+                "duration_seconds":{"type":"number","exclusiveMinimum":0}}}))
+        .with_platforms(&["windows"])
+        .with_agent_access(AgentAccess::ForegroundOnly));
+    methods
 }