Repository navigation
feat(ai): run the stack's AI on another machine or a hosted provider - #110
Merged
Merged
Conversation
When a hook or command saved a setting (`ctx.cfg(key, value)`), the writer replaced every quoted `key = "..."` in the file. Sections were ignored and comments were not skipped, so saving `[ai] default` also rewrote the commented-out model alternatives and any `default` key in another section. A key whose value was not a quoted string was appended again instead of replaced, which left a file TOML refuses to parse. The writer now changes the one assignment inside the named section, keeps any trailing comment on that line, and escapes the value.
The agent refused to start unless the local AI stacklet was set up, although it only ever talks to `[ai] openai_url`. A household running its models on another machine on the network, or with a hosted OpenAI-compatible provider, could not use the agent at all. The agent now requires only `messages`. Before it starts it checks the AI server instead: - no `[ai] openai_url` stops the start and names both ways to get one, `./stack up ai` or `./stack ai connect <url>` - a server that does not answer is a warning; the agent starts and replies once the server is back - a configured model the server does not list is named in a warning
Voice messages always went to `[ai] whisper_url`, which defaulted to the local Whisper on port 42062. On an instance without the ai stacklet that address serves nothing, and no key was ever passed on, so a hosted provider refused every voice message. Transcription is the OpenAI `/audio/transcriptions` call, which the hosted providers answer too, and oMLX with a speech model loaded. So: - with `[ai] whisper_url` unset, voice messages go to `[ai] openai_url` with `[ai] openai_key`: one server, one key - with it set, they go to that dedicated speech server with `[ai] whisper_key`, never the AI key - `[ai] whisper_model` names the speech model to ask for, `whisper-1` by default; other servers name theirs and refuse any other - `stack up ai` sets the local Whisper as the speech server when it builds it, and the installer no longer writes one for everyone Core, memory and chatai now receive the key and the model. Existing instances keep the `whisper_url` their installer wrote and see no change.
`./stack ai connect <url>` points the stack at an OpenAI-compatible server on another machine on the network, or a hosted provider, without installing anything on this Mac. `./stack ai connect local` switches back to the engine `./stack up ai` installed. Before, the only way was the "external endpoint" answer inside `./stack up ai`, which still installed Whisper and TTS locally and was labelled unsupported. The command checks the server answers, keeps the configured model if the server has it, takes the only chat model it lists otherwise, and asks for `--model <id>` when there are several. Nothing is written when the server does not answer or wants a key it was not given (`--key`). Voice messages: - `--whisper <url>` sets a dedicated speech server, which then gets voice messages whatever the AI server offers; the command checks it answers the transcription call - without one, the AI server gets them, and can only transcribe with a speech-to-text model loaded; the command looks for one in its model list, records it as `[ai] whisper_model`, and warns when there is none - `--whisper ai` removes the speech server, and is refused when the AI server lists no speech model A server outside the home network gets a warning naming what is sent there: document text, notes, chat questions and voice messages. Loopback, private and Tailscale addresses count as home. Nothing restarts; the command names the running stacklets to restart. CLI plugins can now save settings through `config["set_cfg"]`, the plugin-side twin of a hook's `ctx.cfg(key, value)`.
oMLX runs as a Homebrew service that the first `./stack up ai` starts. Nothing started it again: once it stopped, by hand, a brew upgrade or a crash, `./stack up ai` left it stopped, and every AI request failed until it was started by hand or at the next login. `./stack up ai` now starts the service when the local engine is the provider and it does not answer. A remote provider starts nothing on this Mac.
After `./stack ai connect <url>`, bringing up the ai stacklet means the AI engine and speech-to-text on this Mac take over from that server. That is a different setup, not a restart, so `./stack up ai` now says which endpoint is configured and asks first. No is the default and leaves the configuration as it was; without a terminal it stops instead of asking. Yes switches chat and voice to this Mac together and names the default model, which a hosted setup may have set to one this Mac does not have. The first `./stack up ai` no longer offers an external endpoint of its own. That path recorded a remote server and still installed Whisper and TTS here. `./stack up ai` is the local engine; a server elsewhere is `./stack ai connect <url>`, which the question now points to.
`./stack install --full` ran the first installer, which asked for every stacklet up front. It had drifted: it still offered a `bots` stacklet that no longer exists, and an "external endpoint" for AI that recorded a remote server while installing Whisper and TTS locally anyway. The messages-first installer is the only one now, and takes the name `installer.py`. It keeps the one thing it borrowed from the old one, the summary shown when a configuration already exists, which now points to `./stack ai connect` for another AI server.
After `./stack ai connect <url>`, `./stack list` still showed the ai
stacklet as `online localhost:42060` whenever its local containers ran:
its LLM check probes the configured address, and the remote server
answered it. The row claimed the engine on this Mac was serving when
nothing used it. On an instance that never installed it, the row said
`available`, as if there were no AI at all.
The row now reads `remote` with the server's host, set up locally or
not, and is degraded, naming the address, when that server does not
answer. The JSON output carries `remote` with the address.
A stacklet declares this in its manifest: `remote = "<address>"`, a
template that renders empty when the stacklet serves itself. While it
is remote only the health checks marked `remote = true` run; the others
probe this Mac. The ai stacklet uses the new `{ai_remote_url}`, the AI
address when the provider is external.
Switching to a remote server with `./stack ai connect <url>` left the local engine running: oMLX, Whisper and the speech container kept their memory, and `./stack restart` offered to restart the ai stacklet, which now asks about switching back to local mode. `./stack ai connect` now says the local engine is no longer used and that `./stack down ai` stops it. That stop reached oMLX only when famstack had installed it, so an instance first set up with a remote endpoint kept oMLX running after `./stack up ai` had started it. `./stack up ai` now records that famstack manages oMLX whenever the local engine is the provider.
With several models on the server and none chosen, `./stack ai connect` asked for `--model <id>` without saying which ids exist. The list was in the JSON output, but a terminal only shows the error line. The error now lists them.
When a hook asked the admin something and got a no, `./stack up` printed "Hook on_start.py error", then "on_start hook failed" twice, and told the family room the stacklet had failed to start. A hook now raises `stack.hooks.Cancelled` for that. `./stack up` prints its message once, exits 1 and tells nobody. The ai stacklet uses it for a no to setting up the local engine and to replacing a remote AI server, and when there is no terminal to ask in.
The local Whisper listened on loopback only, whatever the stack's mode. A second Mac pointed at this one with `./stack ai connect <url> --whisper <this Mac>:42062` was refused, and so was a reverse proxy reaching it over the LAN. It now follows the rule every other service binds by: all interfaces in port mode, loopback in domain mode. Like the other services in port mode it answers anyone on the network, without a password. The next `./stack up ai` rewrites the LaunchAgent and reloads it.
The admin guide gets a section on `./stack ai connect`: pointing the stack at another machine or a hosted provider, switching back with `local`, where voice messages go and how `--whisper` changes it, what a hosted provider receives, and what `./stack up ai` does on a remote setup. The first-run steps no longer mention the external endpoint `./stack up ai` used to offer. `stack.example.toml` explains the speech server and its key and model, and the operator command map and troubleshooting table cover the new command, the `remote` state in `./stack list`, and voice messages that fail on a remote setup.
famstack never set where oMLX listens, so it kept its own default, loopback, in every mode. Another machine pointed at this Mac with `./stack ai connect <this Mac>:42060` was refused. `./stack up ai` now sets it by the rule every other service binds by: all interfaces in port mode, loopback in domain mode. It checks on every start and restarts a running oMLX when the address changes, since oMLX reads its settings at start only. In port mode the engine answers anyone on the network who has its API key, `[ai] openai_key`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
./stack ai connect <url>, andlocalto switch back. The agent no longer requires the local ai stacklet.--whisper <url>), otherwise to the AI server if it lists a speech model.stack listshows the AI asremote,connectwarns when an address is outside the home network, andstack up aiasks before replacing a remote setup.stack up airestarts a stopped oMLX, a declined hook question is a cancel rather than a failure, and the--fullinstaller is removed.Commits are coherent on their own; rebase merge.