diff --git a/CHANGELOG.md b/CHANGELOG.md index 0454595..4f07685 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,8 @@ ### Added +- **Built-in MCP server** — `dexter mcp` serves the index to AI agents over the Model Context Protocol (stdio, or streamable HTTP with `--listen`), modeled on `gopls mcp`. Ten tools cover workspace overview, fuzzy symbol search, definitions with docs and specs, references (including use-chain injected call sites), module API summaries, file outlines, behaviour/protocol implementations, call hierarchy, reindexing, and workspace-wide rename with the same on-disk semantics as the editor rename. Like `dexter lsp` and the CLI, `dexter mcp` is a frontend of the workspace daemon: it opens no index and starts no watcher of its own, and every tool runs in the daemon against the same index, watchers, and caches as the editor, so an agent and an editor never index the same tree twice. Definitions and references use the same name navigation as go-to-definition, find-references, and `dexter lookup`. A tool call waits for a cold index for a short time, then answers with a note that the index is still building; a rebuilding or degraded index is stated in the answer, and a rename is refused until the index is complete. Each session's workspace root comes from MCP roots and resolves like the CLI's (with the launch directory as the fallback; a root that is not a project, or is the home directory, is refused); an explicit path argument (`dexter mcp `) or `--root` fixes it. Tools read the newest unsaved editor buffer of a file when an attached editor holds one, and say so. `--listen` serves only a loopback address unless `--listen-unsafe` is given. `dexter mcp --instructions` prints an agent-facing usage guide + - **Go-to-definition reaches the line that declared a generated function** — a function a macro generated used to resolve to the top of its module. Dexter now reads the line from the compiled module's debug info, which is standard compiler output, so no framework is special-cased. A generator that expands each function at the line of the call that declared it, or stamps it with `@file {file, line}`, sends definition, call hierarchy, the references declaration, and `dexter lookup` (including `--strict`) to that line. The line is used only when it was compiled from the file being opened; a BEAM older than the source still gives its line, because Dexter cannot compile the project and the last compile's line is closer than the module line; only edits to the declaring file move it, and the next compile makes it exact. A function whose only recorded line is the module line, such as one a `@before_compile` hook made, goes to the call in its module that declares it by name. When several calls spell the name, as an Ash action and the code interface that runs it do, the macro whose calls name the most of the module's generated functions wins, and a tie keeps the module line. A function with a clause per DSL call goes to every clause. A line past the end of the file is never returned. A module compiled without debug info falls back to its Docs chunk annotation, which is often, but not always, the same line. A generated module with no source of its own, such as one `Module.create` made or a Spark DSL entity, goes to the file it was compiled from, rebased onto the project when it was built elsewhere, and so does go-to-definition on its name. A module a macro made with `defmodule` and a name it computed records no line of its own, so its name goes to its first function's line. A bare call to a generated function of an imported module now resolves. Ash code interfaces go to their `define` line with released Ash, and from the recorded line with an Ash release that includes [ash-project/ash#2971](https://github.com/ash-project/ash/pull/2971) ([#108](https://github.com/remoteoss/dexter/issues/108)) - **Failures and degraded states are shown in the editor** — before, almost every problem went only to the log, so the editor showed a language server that did nothing. Now each condition that stops Dexter from working, or makes it work with less, is a `window/showMessage` in every attached editor: an index written by a newer or older Dexter build, a damaged index that is being rebuilt, an index that another process holds locked or that cannot be opened for another cause, an index that cannot be used, a fast build that fell back to the slow path, files that could not be indexed (one aggregate message), a root that is the home directory or not a project, file watching that is unavailable or does not cover some directories, FSEvents falling back to fsnotify, a workspace with no Elixir standard library, a session with no `mix` (told to that editor only), a Mix project whose formatter cannot run (each project on its own), an Elixir/OTP mismatch that stops the fast formatter (told once; Dexter then formats through `mix format` and does not start the failing BEAM again on each save), and a rename that could not change some files. Each message says what happened, what Dexter does about it, and what to do. Conditions that stop say so, and an editor that attaches later receives the conditions that are still active. Cold builds, rebuilds, and large incremental passes show LSP work-done progress where the editor supports it. `lookup`, `references`, and `reindex` state a rebuilding or unusable index on stderr, and `workspace/status` lists every active condition @@ -18,7 +20,7 @@ - **`dexter stop`** — stops a workspace daemon on demand instead of hunting for its pid. It finds the process by workspace, reports whether one was running, and is a no-op when nothing was. A plain stop refuses while editors or CLI clients are attached, so a shared workspace is not yanked out from under them; `--force` skips the handshake and locates the daemon process directly, escalating to a kill if it will not exit, which is the manual way out for a daemon that is stuck or built by an older version -- **One workspace daemon, shared by every frontend** — the editor and the CLI now attach to a single per-workspace process that owns the index, the file and Git watchers, and the language caches, instead of each keeping its own copy of all three and indexing the same tree twice. It is also the foundation an MCP frontend will attach to, so MCP can drop its own store, watchers, and caches rather than index the same tree a second time. The daemon belongs to the workspace rather than to whichever frontend started it, and it starts on demand: it listens on a socket under `/tmp/dexter-` and exits after 15 minutes with no clients; `DEXTER_DAEMON_IDLE_TIMEOUT` changes that (`0` keeps it forever). Ownership is an advisory kernel lock held for the process lifetime, so a crash or `kill -9` releases it at once — there is no stale lock to clear and no PID file to go wrong. The socket binds before the first index pass, so opening an editor never waits on a cold build, and a CLI call reuses whatever caches the editor already warmed. The handshake carries a contract version: when an upgrade leaves a daemon from an older build owning the workspace, the first current-build frontend replaces it automatically — the old daemon shuts itself down when it understands the contract, or is signaled by the pid its refusal carries when it predates it — and a frontend older than the daemon is refused with a message telling its user to restart it. `IndexVersion` remains the store's own rebuild trigger, bumped together with the contract. See `docs/daemon.md` +- **One workspace daemon, shared by every frontend** — the editor and the CLI now attach to a single per-workspace process that owns the index, the file and Git watchers, and the language caches, instead of each keeping its own copy of all three and indexing the same tree twice. The daemon belongs to the workspace rather than to whichever frontend started it, and it starts on demand: it listens on a socket under `/tmp/dexter-` and exits after 15 minutes with no clients; `DEXTER_DAEMON_IDLE_TIMEOUT` changes that (`0` keeps it forever). Ownership is an advisory kernel lock held for the process lifetime, so a crash or `kill -9` releases it at once — there is no stale lock to clear and no PID file to go wrong. The socket binds before the first index pass, so opening an editor never waits on a cold build, and a CLI call reuses whatever caches the editor already warmed. The handshake carries a contract version: when an upgrade leaves a daemon from an older build owning the workspace, the first current-build frontend replaces it automatically — the old daemon shuts itself down when it understands the contract, or is signaled by the pid its refusal carries when it predates it — and a frontend older than the daemon is refused with a message telling its user to restart it. `IndexVersion` remains the store's own rebuild trigger, bumped together with the contract. See `docs/daemon.md` - **Completion and hover for macro-generated functions** — Dexter now reads compiled BEAM exports and documentation to surface public functions and macros that do not exist in source. This includes generated functions and macros in application modules, entirely generated application modules such as Phoenix route helpers (including `alias ..., as: Routes`), Oban constructors hidden from generated documentation, exported introspection APIs such as `__schema__`, and generated dependency DSLs. Source indexing remains authoritative and compilation remains optional: stale BEAMs can contribute genuinely generated names, while an absent BEAM leaves the existing source-only behavior unchanged. Generated Spark/Ash DSL macros are resolved from persisted extension attributes and narrowed to the modules in scope at the cursor's nested block path: section macros at module level, an entity's macro inside its section body, and an entity's own option macros inside its body — including when a language form such as `for` or `if` sits in the block path. Hover on any of them renders the compiled signature. Both the OTP 24–27 and OTP 28+ atom-table layouts are supported. In a monorepo whose libraries are compiled as path dependencies of another Mix project, Dexter finds the workspace's builds and looks for each library's application in them, so a library without a `_build` of its own, and modules generated beneath its modules, still resolve; when a library is compiled in more than one build, the most recently compiled BEAM is used @@ -32,6 +34,8 @@ ### Fixed +- **Grouped aliases in a module rename** — moving one member of `alias Old.{A, B}` to another namespace rewrote the shared prefix, so the other members named modules that do not exist; the moved member now leaves the group and gets its own `alias`. A group whose members continue on the next lines (`alias Old.{` on its own line) kept the old prefix when the prefix module was renamed; it is now rewritten + - **A worktree moved into the project, or a directory that cannot be read, is no longer indexed by mistake** — `git worktree move` renames the directory and then writes its `.git` file again in place, so a watcher or a walk could find the file empty and index the whole worktree. Such a directory is now treated as a worktree until git is done, by the watchers and by both index walks, and each check reads the `.git` file once: two reads could see the empty file and then the complete one, and answer that a worktree was neither a worktree nor still being written. This happened with git 2.48 and later, which write the file again after the move; older git does not. A directory that the watcher cannot read (for example when the process has no file descriptor left) used to be skipped silently: it was not watched, not retried, and its files were reported as if it were a plain directory. It is now marked as not covered, so the coverage report says so and the retry reads it again, and its files are indexed only once Dexter knows what it is - **A def inside a macro's `quote` is no longer indexed as a function of the macro's module** — `defmacro route(...) do quote do def handle(...) end end` made the index say that the DSL module defines `handle/2`, which it does not. A consumer that imports the DSL then resolved `Consumer.handle` into the macro's body. Such a def is now skipped, so the call goes to the line in the consumer that declared it, from the compiled BEAM. What `__using__` injects, and a quote in a helper function, are still indexed as before. The index is rebuilt once after the upgrade diff --git a/README.md b/README.md index 5d0745b..4ca6181 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,7 @@ A fast, full-featured Elixir LSP optimized for large Elixir codebases. - [Look up definitions](#look-up-definitions) - [Find references](#find-references) - [Reindexing files manually](#reindexing-files-manually) +- [MCP server](#mcp-server) - [Hover documentation](#hover-documentation) - [Cursor-position-aware resolution](#cursor-position-aware-resolution) - [Rename](#rename) @@ -476,6 +477,31 @@ When running as an LSP server, dexter automatically: - Runs an incremental reindex on startup - Watches `.git/HEAD` for branch switches and reindexes when detected +## MCP server + +Dexter includes a built-in [Model Context Protocol](https://modelcontextprotocol.io) server, modeled on `gopls mcp`, so AI agents can navigate Elixir codebases through the index instead of grep. Tools cover symbol search, definitions with docs and specs, references, module API summaries, file outlines, behaviour/protocol implementations, call hierarchy, incremental reindexing, and workspace-wide rename. + +Register it with your MCP client. For Claude Code: + +```sh +claude mcp add dexter -- dexter mcp +``` + +Any client that speaks MCP over stdio works the same way: point it at `dexter mcp`. The server obtains its workspace from the client through MCP roots and resolves it the way the CLI does, so it binds the project the client is working in rather than the directory it was launched from. Clients that provide no roots get the launch directory (when it is a project), and an explicit path argument (`dexter mcp `) overrides negotiation entirely. One session serves one workspace: when a client gives several roots, the session uses the first one that is a usable project (an existing directory inside an Elixir project, not the home directory) and ignores the others. In `--listen` mode each resolved root gets its own workspace, so sessions from different projects can share one server. + +`dexter mcp` is a frontend of the workspace daemon, like `dexter lsp` and the CLI: it starts the daemon when necessary and keeps no index of its own. The tools answer from the same index, watchers, and caches as the editor, so edits made directly by an agent are indexed by the daemon's file watcher, and a `dexter_reindex` tool forces an immediate update if a lookup ever seems stale. A tool that answers from an index that is still building or degraded says so in its answer. The tools read file text the way the user sees it: when an editor attached to the same daemon holds a file open with changes it has not saved (and the file on disk has not changed since), outlines, definition snippets, and reference lines come from that buffer (with its line numbers), and the answer says so. The rename tool writes on disk and does not yet look at editor buffers, so save your editor's changes before an agent renames. + +Useful variants: + +```sh +# Serve over streamable HTTP instead of stdio (loopback addresses only; the +# server has no authentication, so another address needs --listen-unsafe) +dexter mcp --listen localhost:8092 + +# Print the agent-facing usage guide (save as context for clients that want it) +dexter mcp --instructions +``` + ## Hover documentation Dexter serves hover docs (`textDocument/hover`) for functions, modules, and types. When you hover over a symbol, it looks up the definition in the index and reads the `@doc`, `@moduledoc`, `@typedoc`, or `@spec` annotations from the source file. @@ -599,7 +625,7 @@ dexter init . One background process per project owns the index, so your editor and the CLI see the same fresh index instead of each maintaining their own. -The first `dexter lsp`, `dexter lookup`, `dexter references`, or `dexter reindex` +The first `dexter lsp`, `dexter mcp`, `dexter lookup`, `dexter references`, or `dexter reindex` starts it, and it exits on its own after 15 minutes with no clients. Nothing about editor configuration changes: `dexter lsp` still speaks LSP over stdio, it just proxies to the daemon, and the protocol bytes are copied rather than @@ -608,8 +634,8 @@ re-parsed. The daemon belongs to the workspace, not to whichever frontend started it. No editor owns the index for another: a second editor and a lookup in a shell both attach to the same daemon and share its index and resolution caches, so neither -can see a different, staler answer. It is also the process an MCP frontend will -attach to, so every frontend answers from one index. +can see a different, staler answer. `dexter mcp` attaches to it too, so every +frontend answers from one index. The daemon owns the SQLite index in `.dexter/` (one writer), the file watchers and `.git/HEAD` poll that drive incremental reindexes, stdlib and dependency @@ -660,7 +686,7 @@ dexter init --force ~/code/my-elixir-project If the issue persists, enable debug mode to get verbose logs. You can do this in two ways: 1. Set the `debug` option in your editor's LSP `initializationOptions` (see [LSP options](#lsp-options)). It applies to that editor session as soon as it connects. -2. Or set the `DEXTER_DEBUG=true` environment variable for the editor or CLI command that starts the workspace daemon. The daemon reads it when it starts, so if one is already running, run `dexter stop` first. This is also how to debug CLI commands such as `dexter lookup`. +2. Or set the `DEXTER_DEBUG=true` environment variable for the editor or CLI command that starts the workspace daemon. The daemon reads it when it starts, so if one is already running, run `dexter stop --force` first (a plain stop is refused while an editor or an MCP session is attached). This is also how to debug CLI commands such as `dexter lookup`. Debug mode logs timing and resolution details for every definition, hover, references, and rename request. Each editor receives the lines for its own requests in its LSP log (in Neovim usually `~/.local/state/nvim/lsp.log`, in VS Code Output > Dexter). Every editor and CLI command for a workspace shares one daemon, and all of its lines, including those for CLI commands, also go to the daemon's log file: `.log` in its runtime directory (`/tmp/dexter-` on macOS and Linux; see [docs/daemon.md](docs/daemon.md)). The first line `dexter lsp` writes to your editor's log names that file. diff --git a/cmd/main.go b/cmd/main.go index 4fe4629..b750d49 100644 --- a/cmd/main.go +++ b/cmd/main.go @@ -5,6 +5,8 @@ import ( "errors" "fmt" "log" + "net" + "net/http" "os" "os/signal" "path/filepath" @@ -15,6 +17,7 @@ import ( "github.com/remoteoss/dexter/internal/daemon" "github.com/remoteoss/dexter/internal/indexer" "github.com/remoteoss/dexter/internal/lsp" + dexter_mcp "github.com/remoteoss/dexter/internal/mcp" "github.com/remoteoss/dexter/internal/stdlib" "github.com/remoteoss/dexter/internal/store" "github.com/remoteoss/dexter/internal/version" @@ -153,6 +156,30 @@ func main() { }, } + var mcpListen string + var mcpListenUnsafe bool + var mcpInstructions bool + mcpCmd := &cobra.Command{ + Use: "mcp [path]", + Short: "Start the MCP server (stdio)", + Args: cobra.MaximumNArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + if mcpInstructions { + fmt.Print(dexter_mcp.Instructions) + return nil + } + projectRoot, err := resolvePath(args, 0) + if err != nil { + return err + } + cmdMCP(projectRoot, mcpListen, mcpListenUnsafe, len(args) > 0 || rootDir != "") + return nil + }, + } + mcpCmd.Flags().StringVar(&mcpListen, "listen", "", "Serve MCP over streamable HTTP on this loopback address instead of stdio") + mcpCmd.Flags().BoolVar(&mcpListenUnsafe, "listen-unsafe", false, "Allow --listen on an address that other machines can reach (the server has no authentication)") + mcpCmd.Flags().BoolVar(&mcpInstructions, "instructions", false, "Print the MCP instructions file and exit") + var daemonIdleTimeout time.Duration daemonCmd := &cobra.Command{ Use: "daemon ", @@ -184,7 +211,7 @@ func main() { }, } - rootCmd.AddCommand(initCmd, reindexCmd, lookupCmd, referencesCmd, stopCmd, lspCmd, daemonCmd, versionCmd) + rootCmd.AddCommand(initCmd, reindexCmd, lookupCmd, referencesCmd, stopCmd, lspCmd, mcpCmd, daemonCmd, versionCmd) if err := rootCmd.Execute(); err != nil { os.Exit(1) @@ -244,6 +271,20 @@ func findProjectRootWithMissing(path string, allowMissing bool) string { if err != nil { fatal(err) } + return projectRootIn(path, info) +} + +// projectRootFor is findProjectRoot for a frontend that must not exit on an +// error, such as the MCP server resolving a root that a client gave. +func projectRootFor(path string) (string, error) { + info, err := os.Stat(path) + if err != nil { + return "", err + } + return projectRootIn(path, info), nil +} + +func projectRootIn(path string, info os.FileInfo) string { if !info.IsDir() { path = filepath.Dir(path) } @@ -279,16 +320,9 @@ func requireProjectRoot(dir string, allowNonProject bool) { if allowNonProject { return } - if store.IsHomeDir(dir) { - if store.HasIndex(dir) { - return - } - fatal(fmt.Errorf("refusing to use %s as a workspace: it is your home directory, not a project\nhint: run from a project, pass --root , or pass -y/--yes if you really mean it", dir)) + if err := store.NonProjectRootError(dir); err != nil { + fatal(fmt.Errorf("%w\nhint: run from a project, pass --root , or pass -y/--yes to index it anyway", err)) } - if store.LooksLikeProject(dir) { - return - } - fatal(fmt.Errorf("refusing to use %s as a workspace: no mix.exs, .git, or Dexter database found, so it does not look like an Elixir project\nhint: run from a project, pass --root , or pass -y/--yes to index it anyway", dir)) } // defaultIdleTimeout resolves the daemon idle timeout. DEXTER_DAEMON_IDLE_TIMEOUT @@ -718,3 +752,99 @@ func fatal(err error) { fmt.Fprintf(os.Stderr, "Error: %v\n", err) os.Exit(1) } + +// mcpConfig resolves the MCP frontend's workspace roots. The fallback root and +// the roots that clients give go through the same project-root search as the +// CLI, so every frontend reaches the same daemon for one directory. +func mcpConfig(launchDir string, explicitRoot bool) (dexter_mcp.Config, error) { + root, err := projectRootFor(launchDir) + if err != nil { + return dexter_mcp.Config{}, err + } + cfg := dexter_mcp.Config{Root: root, Fixed: explicitRoot, ResolveRoot: mcpClientRoot} + if !explicitRoot { + // The launch directory is only a guess at the workspace. Starting a + // daemon on a directory that is not a project would index all of it. + if err := store.NonProjectRootError(root); err != nil { + cfg.FallbackErr = fmt.Errorf("%w. The MCP client gave no workspace root; configure the server with the project path (`dexter mcp `) or start it in the project", err) + } + } + return cfg, nil +} + +// mcpClientRoot resolves a directory that an MCP client gives as a root, with +// the same refusal as the launch directory: a client root that is not a +// project, or that is the home directory, would start a daemon that indexes +// all of it. The MCP frontend treats a refused root as unusable. +func mcpClientRoot(dir string) (string, error) { + root, err := projectRootFor(dir) + if err != nil { + return "", err + } + if err := store.NonProjectRootError(root); err != nil { + return "", err + } + return root, nil +} + +// cmdMCP serves MCP to an agent. Like `dexter lsp`, it is a frontend of the +// shared workspace daemon: it opens no index and starts no watcher, and every +// tool call is answered by the daemon of the session's workspace, which it +// starts when necessary. Logs go to stderr; stdout belongs to the MCP stdio +// transport. +// +// With an explicit path (or --root) every session uses that workspace. +// Without one, each session's root is negotiated through MCP roots and +// resolved like the CLI resolves its own, with the launch directory as the +// fallback for clients that give no root. +func cmdMCP(projectRoot string, listen string, listenUnsafe bool, explicitRoot bool) { + log.SetOutput(os.Stderr) + if listen != "" { + if err := dexter_mcp.CheckListenAddr(listen, listenUnsafe); err != nil { + fatal(err) + } + } else if listenUnsafe { + fatal(fmt.Errorf("--listen-unsafe needs --listen")) + } + cfg, err := mcpConfig(projectRoot, explicitRoot) + if err != nil { + fatal(err) + } + root := cfg.Root + frontend := dexter_mcp.NewFrontend(cfg) + defer frontend.Close() + if explicitRoot { + log.Printf("Dexter MCP v%s starting (root: %s, daemon log: %s)", version.Version, root, daemonLogPath(root)) + } else { + log.Printf("Dexter MCP v%s starting (workspace roots negotiated per session; fallback root: %s)", version.Version, root) + } + + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + + if listen != "" { + ln, err := net.Listen("tcp", listen) + if err != nil { + fatal(err) + } + log.Printf("MCP server listening on %s", ln.Addr()) + if err := dexter_mcp.CheckListenAddr(listen, false); err != nil { + log.Printf("WARNING: --listen-unsafe: the MCP server on %s has no authentication; anyone who can reach it can read this code and rename symbols", ln.Addr()) + } + httpSrv := &http.Server{Handler: dexter_mcp.HTTPHandler(frontend), ReadHeaderTimeout: 10 * time.Second} + go func() { + <-ctx.Done() + shutdownCtx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + _ = httpSrv.Shutdown(shutdownCtx) + }() + if err := httpSrv.Serve(ln); err != nil && err != http.ErrServerClosed { + fatal(err) + } + return + } + + if err := dexter_mcp.RunStdio(ctx, frontend); err != nil && ctx.Err() == nil { + fatal(err) + } +} diff --git a/cmd/main_test.go b/cmd/main_test.go index 84dea2d..46b2b7e 100644 --- a/cmd/main_test.go +++ b/cmd/main_test.go @@ -123,3 +123,76 @@ func TestFileFailureDoesNotHideTheBuildingNote(t *testing.T) { t.Errorf("the building note is hidden: %q", got) } } + +// Regression: the MCP fallback root and the roots that MCP clients give were +// resolved by two different searches (one treated mix.exs as a marker, one did +// not), so a session with roots and a session without them could start two +// daemons for one directory. Both now use the CLI's search. +func TestMCPRootsResolveLikeCLI(t *testing.T) { + project := t.TempDir() + if err := os.WriteFile(filepath.Join(project, "mix.exs"), []byte("defmodule App.MixProject do\nend\n"), 0o644); err != nil { + t.Fatal(err) + } + sub := filepath.Join(project, "lib") + if err := os.MkdirAll(sub, 0o755); err != nil { + t.Fatal(err) + } + + cfg, err := mcpConfig(sub, false) + if err != nil { + t.Fatal(err) + } + want := findProjectRoot(sub) + if cfg.Root != want { + t.Errorf("fallback root = %q, want the CLI root %q", cfg.Root, want) + } + negotiated, err := cfg.ResolveRoot(sub) + if err != nil { + t.Fatal(err) + } + if negotiated != want { + t.Errorf("negotiated root = %q, want the CLI root %q", negotiated, want) + } + if cfg.FallbackErr != nil { + t.Errorf("a Mix project was refused as the fallback root: %v", cfg.FallbackErr) + } +} + +// A launch directory that is not a project is not indexed when the MCP client +// gives no root. +func TestMCPFallbackRefusesNonProject(t *testing.T) { + cfg, err := mcpConfig(t.TempDir(), false) + if err != nil { + t.Fatal(err) + } + if cfg.FallbackErr == nil || !strings.Contains(cfg.FallbackErr.Error(), "does not look like an Elixir project") { + t.Errorf("FallbackErr = %v, want a refusal", cfg.FallbackErr) + } + explicit, err := mcpConfig(t.TempDir(), true) + if err != nil { + t.Fatal(err) + } + if explicit.FallbackErr != nil { + t.Errorf("an explicit root was refused: %v", explicit.FallbackErr) + } +} + +// Regression: a client root that was not a project (or was the home +// directory) started a daemon that indexed all of it. It is refused like the +// launch directory, so the frontend skips it as unusable. +func TestMCPClientRootRefusesNonProject(t *testing.T) { + cfg, err := mcpConfig(t.TempDir(), true) + if err != nil { + t.Fatal(err) + } + plain := t.TempDir() + if _, err := cfg.ResolveRoot(plain); err == nil || !strings.Contains(err.Error(), "does not look like an Elixir project") { + t.Errorf("ResolveRoot(%s) error = %v, want a refusal", plain, err) + } + + home := t.TempDir() + t.Setenv("HOME", home) + if _, err := cfg.ResolveRoot(home); err == nil || !strings.Contains(err.Error(), "home directory") { + t.Errorf("ResolveRoot(home) error = %v, want a refusal", err) + } +} diff --git a/docs/architecture.md b/docs/architecture.md index 3b0e63a..606d89e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -14,6 +14,7 @@ Dexter is a fast Elixir LSP server. It indexes module and function definitions f - `internal/daemon/` — the per-workspace daemon and its local transport: endpoint and lock derivation (`endpoint.go`), handshake and framing (`protocol.go`), connection handling and the built-in control methods (`server.go`), the client and the stdio proxies (`client.go`), the adapter registries (`registry.go`), and the per-platform ownership lock. `docs/daemon.md` has the ownership, lifecycle, and extension contract. - `internal/notify/` — the one path that tells the user about failures, degraded states, and long work. A `Reporter` logs each report and sends it to every attached editor as `window/showMessage` or work-done progress, and keeps the active conditions so it can replay them to an editor that attaches later. `notifytest` has a fake client that records what an editor would show. - `internal/treesitter/` — Tree-sitter integration for scope-aware variable rename and go-to-references. +- `internal/mcp/` — Model Context Protocol frontend (`dexter mcp`). The frontend negotiates each session's root and forwards every tool call to the workspace daemon as the `mcp/tool` control method; the tool bodies (one file per tool, gopls-style) run in the daemon. Tools are name-based (module/function, not file+position) and use the store, the name navigation in `internal/lsp/name_navigation.go`, and the exported surface in `internal/lsp/api.go`. ## String interpolation (`TokenResult.Interp`) @@ -202,9 +203,11 @@ A module rename also moves files whose names follow the module naming convention `protocol.WorkspaceEdit` from `go.lsp.dev/protocol` types `documentChanges` as `[]TextDocumentEdit` and cannot carry resource operations, so `internal/lsp/workspace_edit.go` defines the wire types and `renameHandler` answers `textDocument/rename` ahead of the generated dispatcher. A client that understands `documentChanges` ignores `changes` entirely, so once one file moves, every edit in the reply goes through `documentChanges`. +The MCP rename tool runs in the workspace daemon on its headless language service (`RenameFunction`/`RenameModule` in `internal/lsp/api.go`). That service holds no editor buffers, so the builders write every affected file and move closed files themselves, exactly as for closed files above, and the summary lists the files changed, moved, and not written. Renames run one at a time across the workspace: `IndexCoordinator.renameSerial`, which editor and MCP renames both take, is held from the first read of the affected files to the end of the writes, so two renames cannot write over each other's edits. A rename that takes it first waits until the index shows the previous rename, so its check that the new name is free sees that rename. These writes do not yet look at buffers that an editor attached to the same daemon has open; the MCP rename will adopt the shared rule for frontends without an editor (never write over unsaved editor work) when it lands in `internal/lsp`. + ### Grouped aliases -`alias Old.{A, B}` (and the `require`/`import` forms) names the module once, as the prefix, while the index records one reference per member — so a member's full name never appears on the line. `findGroupedAliasEdits` handles both directions: renaming the prefix rewrites the prefix, renaming a member rewrites that member inside the braces. Since every member on the line resolves to the same prefix edit, `applyEdits` drops TextEdits that overlap one already emitted for that line; the on-disk path rewrites the line as it goes and never sees the second match. +`alias Old.{A, B}` (and the `require`/`import` forms) names the module once, as the prefix, while the index records one reference per member — so a member's full name never appears on the line. `findGroupedAliasEdits` handles both directions: when the renamed module is the prefix or one of its ancestors, the prefix is rewritten (also on the opening line of a group whose members continue on the next lines, where the index records every member); renaming a member rewrites that member inside the braces, and a member that moves to another namespace leaves the group and gets its own `alias` line, so the other members keep their prefix. Since every member on the line resolves to the same prefix edit, `applyEdits` drops TextEdits that overlap one already emitted for that line; the on-disk path rewrites the line as it goes and never sees the second match. ## Indexing throughput diff --git a/docs/daemon.md b/docs/daemon.md index 7078503..2b47a18 100644 --- a/docs/daemon.md +++ b/docs/daemon.md @@ -2,19 +2,19 @@ Dexter runs one daemon per workspace. The daemon is the only process that opens the workspace index for normal operation and the only owner of filesystem and Git -watchers. LSP and ordinary CLI processes are frontends that connect to it over a -local socket; the MCP frontend will attach the same way. +watchers. LSP, MCP, and ordinary CLI processes are frontends that connect to it over a +local socket. -Implementation status: the daemon serves the LSP frontend and the `lookup`, -`references`, `reindex`, and `stop` commands. MCP is designed for but not -migrated yet — see [The MCP frontend](#the-mcp-frontend). +Implementation status: the daemon serves the LSP frontend, the MCP frontend +(`dexter mcp`, see [The MCP frontend](#the-mcp-frontend)), and the `lookup`, +`references`, `reindex`, and `stop` commands. `dexter init` remains an offline maintenance command. It acquires the same workspace ownership lock as the daemon, asks an idle daemon to exit, and refuses to run while a daemon with attached clients owns the workspace. There is no in-process editor mode: the daemon belongs to the workspace, and an editor that -served the index itself would be one more owner competing with the CLI — and, -once it lands, MCP — for the same caches and the same writer. +served the index itself would be one more owner competing with the CLI and +MCP for the same caches and the same writer. ## Goals @@ -33,7 +33,7 @@ once it lands, MCP — for the same caches and the same writer. ```text editor <-> dexter lsp --stdio proxy --\ local socket <-> workspace daemon -MCP frontend (planned) -------------/ | +MCP frontend (dexter mcp) ----------/ | CLI lookup/references/reindex -------/ +-- SQLite store +-- mutation queue +-- file/Git watchers @@ -77,6 +77,15 @@ responses carry an id, and the daemon may push notifications (a method, no id) between them. One connection therefore multiplexes concurrent calls and subscriptions, and a slow reindex cannot block a lookup. +A connection runs at most 64 requests at once; one more is refused at once +instead of blocking the reader. Each request has its own context, derived from +the connection's. A client that stops waiting for a request sends +`{"id":0,"method":"$/cancel","params":{"id":N}}`, the pattern of LSP's +`$/cancelRequest`: the reader handles it outside the request limit and cancels +request `N`, so its index waits end and its slot is free again. The canceled +request still answers; the client has dropped it. A subscription made by +`workspace/watch` belongs to the connection, not to the request that made it. + Each message is one line of at most 16 MiB, newline included. A writer refuses a longer line before sending anything, so the stream stays in step: a result that is too large fails only its own call, and a `workspace/changed` @@ -289,71 +298,83 @@ Built-in control surface: | `workspace/references` | semantic references through the shared language service | | `workspace/reindex` | whole workspace or one path, returning after the barrier | | `workspace/watch`, `workspace/unwatch` | subscribe to coalesced index changes, pushed as `workspace/changed` notifications | +| `$/cancel` | cancel one in-flight request of this connection; sent with id 0, no response | ## The MCP frontend -MCP is not migrated onto the daemon yet. Its server still carries its own copy of -workspace ownership: a store handle, a headless `lsp.Server`, stdlib discovery, -a filesystem watcher, a Git HEAD poll, an initial index pass, and an index barrier. -The daemon exists so those can be deleted rather than duplicated a second time. -The table below is the mapping that migration applies, and the sections after it -are the design it should follow. - -| MCP-side concept | Daemon equivalent | -|---|---| -| `binding.init`, `openStore` recovery | `workspace.Open` (same recovery path) | -| `binding.lsp = lsp.NewServer(...)` | `Runtime.LanguageServices()` | -| `stdlib.Resolve` + `SetStdlibRoot` | resolved once by the runtime, inherited by sessions | -| `WatchFiles` (own watcher) | `workspace.Watcher` → one mutation queue | -| `lsp.WatchGitHead` | `Runtime.startGitWatch` | -| `binding.awaitIndex`, `indexWaitLimit` | `workspace/status` with `waitReadyMs` | -| `binding.close` | close the connection; the daemon idles out | -| `mcp.Config{LSP, Store, ProjectRoot}` | `Runtime.LanguageServices()/Store/Root` accessors | - -### Two ways to attach - -Both shapes are reachable through the registries above, and they are not equally -cheap. Neither is wired up yet. The MCP Go SDK speaks JSON-RPC over an `io.ReadWriteCloser`, but its -`InMemoryTransport` keeps that field unexported and `newIOConn` is internal, so -there is no supported way to hand it a raw socket. - -**Control methods (recommended).** Keep the MCP protocol server in the frontend -process over stdio, exactly as it is today, and reach the workspace through -control calls. Register one method per tool backend — -`workspace/definition`, `workspace/rename`, `workspace/callHierarchy`, -`workspace/implementations`, `workspace/outline`, `workspace/moduleAPI`, -`workspace/search` — each a thin wrapper over the equivalent `internal/lsp/api.go` -call made against `mc.LSP()`. The frontend's per-root binding becomes -`daemon.Ensure(ctx, root)`; its index barrier becomes `client.WorkspaceStatus(ctx, -waitReadyMs)`; its watcher becomes `client.Watch(ctx, buffer, onChange)`; -its `close` becomes `client.Close()`. The cost is one local JSON round trip per -tool call, which is noise next to the model latency that triggered the call, and -it needs no transport work and no SDK coupling. Multi-root negotiation stays in -the frontend: one control connection per negotiated root. - -**Daemon-hosted frontend.** `RegisterFrontend("mcp", ...)` plus -`daemon.ProxyFrontend(ctx, root, "mcp", session, os.Stdin, os.Stdout)` would run -the MCP server inside the daemon, so tool calls never cross a socket and -`fc.LSP()` gives attached mode the editor's unsaved buffers. It requires -implementing the SDK's `Transport`/`Connection` pair over the socket — about the -same newline-delimited JSON framing the LSP stream already avoids by proxying -bytes — and that implementation tracks SDK internals. Worth revisiting only if -per-call latency ever becomes measurable, or if the SDK grows a constructor that -wraps an `io.ReadWriteCloser`. - -Rules for that work: - -- Attach per root. One daemon per negotiated workspace root, one control - connection each; the daemon itself does not become multi-root. -- Reach name-based operations through `Runtime.LanguageServices()`, or through the - attached editor session when the client explicitly named one. Add - protocol-neutral methods to the runtime, or register a control method, rather - than implementing queries in an adapter. -- Never open the store, start a watcher, or run a second LSP lifecycle in the - frontend process. -- Gate a cold workspace with `workspace/status` instead of a private barrier. -- Bump `ContractVersion` when a wire change breaks an older frontend, and treat - method names and payload shapes as the adapter contract. +`dexter mcp` keeps the MCP protocol server in the frontend process (over stdio, +or streamable HTTP with `--listen`) and reaches the workspace through one +registered control method, `mcp/tool`. It opens no store, starts no watcher, and +runs no LSP lifecycle of its own. + +- **Roots.** Multi-root negotiation stays in the frontend. Each MCP session's + root comes from MCP roots, resolved with the same project-root search as the + CLI, or from the command line. Sessions that resolve to the same root share + one control connection (`daemon.Ensure`), which is the lease that keeps the + daemon alive; the connection closes when the last such session ends. The + frontend keeps the spelling the client gave. When the daemon already serves + the workspace through another spelling, the frontend uses the daemon's root. +- **Tools.** `internal/mcp` registers `mcp/tool` at package initialization. A + call carries the tool name, its arguments, and `waitReadyMs`. The daemon runs + the tool body against `mc.LSP()` (the headless language service, since MCP + never names an editor session) and the runtime's store. One method instead + of one per tool keeps each tool's parameter type in one place, shared by the + input schema the agent sees and the body that decodes it. Definitions and + references use `LookupName` and `ReferenceNames`, the same name navigation as + the editor and the CLI. +- **Cold and degraded index.** A tool waits up to `waitReadyMs` (capped at + 30 s) for the initial reconciliation, then answers from what is indexed. An + answer from an index that is still building, or that has an active warning + or error condition, ends with a note that says so. The rename tool refuses + until the index is complete, because a rename from a partial index would + change some call sites and leave others with the old name. +- **Editor buffers.** Tools that read file text (outlines, definition and + module docs, reference lines) use the newest buffer that an attached editor + session holds open with unsaved changes: the editor changed it after its + last open or save (`didChange` after `didOpen`/`didSave`). A buffer with no + unsaved changes can be older than the disk (an editor that has not reloaded + the file after an agent wrote it), so the disk is used. When the file on + disk changed after the buffer's last change, the disk is used too, and the + answer warns that the editor's unsaved changes may conflict. The answer + names the files that came from unsaved buffers. The index positions refer + to the saved file, so a line is mapped into the buffer with a line diff + (Myers, line endings ignored; bounded, with a fallback to the lines that + both texts share at the start and at the end); a position on a changed line + shows the saved line, marked as such. Paths that the + agent gives must be inside the project root after symlinks are resolved, and + must name a regular file of at most 10 MB. +- **Limits and cancellation.** One frontend runs at most 32 tool calls at once + on a workspace connection, below the daemon's 64, and a call past the limit + waits for a slot. A call that the MCP client cancels sends `$/cancel`, so the + daemon stops waiting for the index for it. A canceled rename says that it may + have been applied. When the client's roots change to another project during + a call, the call ends with an error that says so. +- **HTTP.** `--listen` accepts only a loopback address unless + `--listen-unsafe` is given, because the server has no authentication. The + SDK refuses a non-loopback `Host` on a loopback connection (DNS rebinding), + cross-origin browser requests are refused, a request body over 4 MB gets + 413, and a session with no request for 30 minutes is closed, so a client + that went away does not keep its daemon alive. +- **Reconnects.** When the daemon goes away (an upgrade replaced it, or + `dexter stop --force`; a plain `dexter stop` is refused while MCP is + attached), the next tool call connects again, which starts a new daemon. + A read-only call that was in flight is sent once more; a rename is never + repeated. +- **Rename.** The rename runs on the headless language service and writes the + changed files on disk. It does not yet look at buffers that an editor attached + to the same daemon has open. The shared rule for frontends without an editor + (never write over unsaved editor work; refuse with an actionable error when an + affected file has unsaved changes) will replace the call in + `internal/mcp/rename.go` when it lands in `internal/lsp`. + +A daemon-hosted frontend (`RegisterFrontend("mcp", ...)`, with the MCP server +inside the daemon) would save the local round trip per tool call. It needs an +SDK `Transport` over the socket, which tracks SDK internals, and the round trip +is noise next to the model latency that caused the call. Revisit it only if the +per-call latency becomes measurable. + +Bump `ContractVersion` when a change to `mcp/tool` breaks an older frontend; the +method name and payload shapes are part of the adapter contract. ## Performance requirements @@ -363,8 +384,7 @@ behind the mutation coordinator. Benchmarks should cover warm definition/hover/completion latency, daemon startup, CLI lookup, large reference responses, event-to-index latency, concurrent reads -during a reindex, and total memory/CPU with LSP and MCP connected together (MCP -once it is migrated). +during a reindex, and total memory/CPU with LSP and MCP connected together. The target for the daemon hop is no more than 1 ms added p95 latency for hot LSP operations. `cmd/lspprobe` measures a real project over the wire; calling the diff --git a/go.mod b/go.mod index 3854e57..c3c0937 100644 --- a/go.mod +++ b/go.mod @@ -6,6 +6,7 @@ require ( github.com/fsnotify/fsevents v0.2.0 github.com/fsnotify/fsnotify v1.10.1 github.com/mattn/go-sqlite3 v1.14.38 + github.com/modelcontextprotocol/go-sdk v1.6.1 github.com/spf13/cobra v1.10.2 github.com/tree-sitter/go-tree-sitter v0.25.0 github.com/tree-sitter/tree-sitter-elixir v0.3.5 @@ -19,12 +20,15 @@ require ( replace github.com/tree-sitter/tree-sitter-elixir => github.com/elixir-lang/tree-sitter-elixir v0.3.5 require ( + github.com/google/jsonschema-go v0.4.3 // indirect github.com/inconshreveable/mousetrap v1.1.0 // indirect github.com/mattn/go-pointer v0.0.1 // indirect github.com/segmentio/asm v1.1.3 // indirect - github.com/segmentio/encoding v0.3.4 // indirect + github.com/segmentio/encoding v0.5.4 // indirect github.com/spf13/pflag v1.0.9 // indirect + github.com/yosida95/uritemplate/v3 v3.0.2 // indirect go.lsp.dev/pkg v0.0.0-20210717090340-384b27a52fb2 // indirect go.uber.org/atomic v1.9.0 // indirect go.uber.org/multierr v1.8.0 // indirect + golang.org/x/oauth2 v0.35.0 // indirect ) diff --git a/go.sum b/go.sum index 0959922..e16430a 100644 --- a/go.sum +++ b/go.sum @@ -10,9 +10,13 @@ github.com/fsnotify/fsevents v0.2.0 h1:BRlvlqjvNTfogHfeBOFvSC9N0Ddy+wzQCQukyoD7o github.com/fsnotify/fsevents v0.2.0/go.mod h1:B3eEk39i4hz8y1zaWS/wPrAP4O6wkIl7HQwKBr1qH/w= github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho= github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo= +github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY= +github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE= github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= -github.com/google/go-cmp v0.5.6 h1:BKbKCqvP6I+rmFHt06ZmyQtvB8xAkWdhFyr0ZUNZcxQ= -github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/jsonschema-go v0.4.3 h1:/DBOLZTfDow7pe2GmaJNhltueGTtDKICi8V8p+DQPd0= +github.com/google/jsonschema-go v0.4.3/go.mod h1:r5quNTdLOYEz95Ru18zA0ydNbBuYoo9tgaYcxEYhJVE= github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= @@ -22,6 +26,8 @@ github.com/mattn/go-pointer v0.0.1 h1:n+XhsuGeVO6MEAp7xyEukFINEa+Quek5psIR/ylA6o github.com/mattn/go-pointer v0.0.1/go.mod h1:2zXcozF6qYGgmsG+SeTZz3oAbFLdD3OWqnUbNvJZAlc= github.com/mattn/go-sqlite3 v1.14.38 h1:tDUzL85kMvOrvpCt8P64SbGgVFtJB11GPi2AdmITgb4= github.com/mattn/go-sqlite3 v1.14.38/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y= +github.com/modelcontextprotocol/go-sdk v1.6.1 h1:0zOSupjKUxPKSocPT1Wtago+mUHU2/uZ4xSOY0FGReU= +github.com/modelcontextprotocol/go-sdk v1.6.1/go.mod h1:kzm3kzFL1/+AziGOE0nUs3gvPoNxMCvkxokMkuFapXQ= github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= @@ -29,8 +35,8 @@ github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZN github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/segmentio/asm v1.1.3 h1:WM03sfUOENvvKexOLp+pCqgb/WDjsi7EK8gIsICtzhc= github.com/segmentio/asm v1.1.3/go.mod h1:Ld3L4ZXGNcSLRg4JBsZ3//1+f/TjYl0Mzen/DQy1EJg= -github.com/segmentio/encoding v0.3.4 h1:WM4IBnxH8B9TakiM2QD5LyNl9JSndh88QbHqVC+Pauc= -github.com/segmentio/encoding v0.3.4/go.mod h1:n0JeuIqEQrQoPDGsjo8UNd1iA0U8d8+oHAA4E3G3OxM= +github.com/segmentio/encoding v0.5.4 h1:OW1VRern8Nw6ITAtwSZ7Idrl3MXCFwXHPgqESYfvNt0= +github.com/segmentio/encoding v0.5.4/go.mod h1:HS1ZKa3kSN32ZHVZ7ZLPLXWvOVIiZtyJnO1gPH1sKt0= github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU= github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4= github.com/spf13/pflag v1.0.9 h1:9exaQaMOCwffKiiiYk6/BndUBv+iRViNW+4lEMi0PvY= @@ -66,6 +72,8 @@ github.com/tree-sitter/tree-sitter-ruby v0.23.1 h1:T/NKHUA+iVbHM440hFx+lzVOzS4dV github.com/tree-sitter/tree-sitter-ruby v0.23.1/go.mod h1:kUS4kCCQloFcdX6sdpr8p6r2rogbM6ZjTox5ZOQy8cA= github.com/tree-sitter/tree-sitter-rust v0.23.2 h1:6AtoooCW5GqNrRpfnvl0iUhxTAZEovEmLKDbyHlfw90= github.com/tree-sitter/tree-sitter-rust v0.23.2/go.mod h1:hfeGWic9BAfgTrc7Xf6FaOAguCFJRo3RBbs7QJ6D7MI= +github.com/yosida95/uritemplate/v3 v3.0.2 h1:Ed3Oyj9yrmi9087+NczuL5BwkIc4wvTb5zIM+UJPGz4= +github.com/yosida95/uritemplate/v3 v3.0.2/go.mod h1:ILOh0sOhIJR3+L/8afwt/kE++YT040gmv5BQTMR2HP4= github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= go.lsp.dev/jsonrpc2 v0.10.0 h1:Pr/YcXJoEOTMc/b6OTmcR1DPJ3mSWl/SWiU1Cct6VmI= go.lsp.dev/jsonrpc2 v0.10.0/go.mod h1:fmEzIdXPi/rf6d4uFcayi8HpFP1nBF99ERP1htC72Ac= @@ -94,6 +102,8 @@ golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= +golang.org/x/oauth2 v0.35.0 h1:Mv2mzuHuZuY2+bkyWXIHMfhNdJAdwW3FuWeCPYN5GVQ= +golang.org/x/oauth2 v0.35.0/go.mod h1:lzm5WQJQwKZ3nwavOZ3IS5Aulzxi68dUSgRHujetwEA= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -101,7 +111,6 @@ golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20211110154304-99a53858aa08/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k= golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= @@ -111,10 +120,11 @@ golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGm golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= +golang.org/x/tools v0.42.0 h1:uNgphsn75Tdz5Ji2q36v/nsFSfR/9BRFvqhGBaJGd5k= +golang.org/x/tools v0.42.0/go.mod h1:Ma6lCIwGZvHK6XtgbswSoWroEkhugApmsXyrUmBhfr0= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE= golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= diff --git a/integration_test.go b/integration_test.go index 970680d..00aedb4 100644 --- a/integration_test.go +++ b/integration_test.go @@ -1,11 +1,16 @@ package main import ( + "bufio" + "context" "os" "os/exec" "path/filepath" "strings" "testing" + "time" + + sdkmcp "github.com/modelcontextprotocol/go-sdk/mcp" "github.com/remoteoss/dexter/internal/daemon" "github.com/remoteoss/dexter/internal/lsptest" @@ -954,3 +959,253 @@ func TestIntegration_LegacyMigration(t *testing.T) { t.Errorf("expected lookup to work after migration, got: %s", out) } } + +// mcpConnect spawns `dexter mcp ` over stdio and returns a connected +// MCP client session. +func mcpConnect(t *testing.T, binary, root string) *sdkmcp.ClientSession { + t.Helper() + cmd := exec.Command(binary, "mcp", root) + cmd.Dir = root + cmd.Stderr = os.Stderr + // The daemon that the MCP session starts must not outlive the test by the + // default idle timeout. + cmd.Env = append(os.Environ(), "DEXTER_DAEMON_IDLE_TIMEOUT=1s", "PWD="+mustAbs(t, root)) + client := sdkmcp.NewClient(&sdkmcp.Implementation{Name: "integration-test", Version: "0.0.1"}, nil) + ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) + t.Cleanup(cancel) + session, err := client.Connect(ctx, &sdkmcp.CommandTransport{Command: cmd}, nil) + if err != nil { + t.Fatalf("connecting to dexter mcp: %v", err) + } + t.Cleanup(func() { _ = session.Close() }) + return session +} + +func mcpToolText(t *testing.T, res *sdkmcp.CallToolResult) string { + t.Helper() + var b strings.Builder + for _, c := range res.Content { + if tc, ok := c.(*sdkmcp.TextContent); ok { + b.WriteString(tc.Text) + } + } + return b.String() +} + +func TestIntegration_MCPStdio(t *testing.T) { + binary := buildDexter(t) + root := scaffoldProject(t) + runDexter(t, binary, root, "init", root) + + session := mcpConnect(t, binary, root) + ctx := context.Background() + + tools, err := session.ListTools(ctx, nil) + if err != nil { + t.Fatal(err) + } + names := map[string]bool{} + for _, tool := range tools.Tools { + names[tool.Name] = true + } + for _, want := range []string{"dexter_workspace", "dexter_search", "dexter_definition", "dexter_references", "dexter_module_api", "dexter_file_outline", "dexter_implementations", "dexter_call_hierarchy", "dexter_reindex", "dexter_rename_symbol"} { + if !names[want] { + t.Errorf("tool %s not advertised; got %v", want, names) + } + } + + res, err := session.CallTool(ctx, &sdkmcp.CallToolParams{Name: "dexter_workspace"}) + if err != nil { + t.Fatal(err) + } + if res.IsError { + t.Fatalf("dexter_workspace errored: %s", mcpToolText(t, res)) + } + out := mcpToolText(t, res) + for _, want := range []string{"Project root:", "mix.exs", "definitions"} { + if !strings.Contains(out, want) { + t.Errorf("workspace output missing %q:\n%s", want, out) + } + } + + res, err = session.CallTool(ctx, &sdkmcp.CallToolParams{Name: "dexter_definition", Arguments: map[string]any{"module": "MyApp.Repo", "function": "get"}}) + if err != nil { + t.Fatal(err) + } + out = mcpToolText(t, res) + if !strings.Contains(out, "lib/my_app/repo.ex") { + t.Errorf("definition output missing location:\n%s", out) + } +} + +func TestIntegration_MCPStdio_EmptyIndexBuildsOnStartup(t *testing.T) { + binary := buildDexter(t) + root := scaffoldProject(t) + // No `dexter init`: the daemon builds the index, and the tool call waits + // for that build. + + session := mcpConnect(t, binary, root) + res, err := session.CallTool(context.Background(), &sdkmcp.CallToolParams{Name: "dexter_search", Arguments: map[string]any{"query": "process_event"}}) + if err != nil { + t.Fatal(err) + } + out := mcpToolText(t, res) + if !strings.Contains(out, "MyApp.Handlers.Webhooks.process_event") { + t.Errorf("search after auto-index missing symbol:\n%s", out) + } +} + +func TestIntegration_MCPInstructions(t *testing.T) { + binary := buildDexter(t) + out := runDexter(t, binary, t.TempDir(), "mcp", "--instructions") + for _, want := range []string{"dexter_workspace", "dexter_reindex", "dexter_rename_symbol"} { + if !strings.Contains(out, want) { + t.Errorf("instructions missing %q", want) + } + } +} + +// TestIntegration_MCPSharesWorkspaceDaemon checks that `dexter mcp` is a +// frontend of the workspace daemon, like the CLI and the editor: it attaches to +// the one daemon of the workspace, sees changes through the daemon's watcher, +// and answers from the same index as `dexter lookup`. +func TestIntegration_MCPSharesWorkspaceDaemon(t *testing.T) { + binary := buildDexter(t) + root := scaffoldProject(t) + runDexter(t, binary, root, "init", root) + + session := mcpConnect(t, binary, root) + ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) + defer cancel() + res, err := session.CallTool(ctx, &sdkmcp.CallToolParams{Name: "dexter_definition", Arguments: map[string]any{"module": "MyApp.Repo", "function": "get"}}) + if err != nil { + t.Fatal(err) + } + if out := mcpToolText(t, res); !strings.Contains(out, "lib/my_app/repo.ex") { + t.Fatalf("definition output missing location:\n%s", out) + } + + client, err := daemon.Dial(ctx, root) + if err != nil { + t.Fatalf("no workspace daemon serves the MCP session: %v", err) + } + defer func() { _ = client.Close() }() + status, err := client.DaemonStatus(ctx) + if err != nil { + t.Fatal(err) + } + // The MCP control connection and this one. + if status.Clients != 2 { + t.Errorf("daemon has %d clients, want 2 (the MCP session and this test)", status.Clients) + } + + // Agent edits produce no editor notifications; the daemon's watcher sees + // them. + createdPath := filepath.Join(root, "lib", "my_app", "created_by_agent.ex") + if err := os.WriteFile(createdPath, []byte("defmodule MyApp.CreatedByAgent do\n def run, do: :ok\nend\n"), 0o644); err != nil { + t.Fatal(err) + } + deadline := time.Now().Add(10 * time.Second) + for { + res, err = session.CallTool(ctx, &sdkmcp.CallToolParams{Name: "dexter_definition", Arguments: map[string]any{"module": "MyApp.CreatedByAgent"}}) + if err != nil { + t.Fatal(err) + } + if strings.Contains(mcpToolText(t, res), "lib/my_app/created_by_agent.ex") { + break + } + if time.Now().After(deadline) { + t.Fatal("the daemon's watcher did not index an agent-created file") + } + time.Sleep(50 * time.Millisecond) + } + + // The CLI answers from the same daemon and the same index. + if out := runDexter(t, binary, root, "lookup", "MyApp.CreatedByAgent"); !strings.Contains(out, "created_by_agent.ex:1") { + t.Errorf("CLI lookup does not see the module the MCP session saw: %s", out) + } + after, err := client.DaemonStatus(ctx) + if err != nil { + t.Fatal(err) + } + if after.PID != status.PID { + t.Errorf("daemon pid changed from %d to %d: a second daemon served the workspace", status.PID, after.PID) + } +} + +// TestIntegration_MCPListenHTTP serves MCP over streamable HTTP and calls a +// tool, which the workspace daemon answers. +func TestIntegration_MCPListenHTTP(t *testing.T) { + binary := buildDexter(t) + root := scaffoldProject(t) + runDexter(t, binary, root, "init", root) + + cmd := exec.Command(binary, "mcp", "--listen=localhost:0", root) + cmd.Dir = root + cmd.Env = append(os.Environ(), "DEXTER_DAEMON_IDLE_TIMEOUT=1s", "PWD="+mustAbs(t, root)) + stderr, err := cmd.StderrPipe() + if err != nil { + t.Fatal(err) + } + if err := cmd.Start(); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { + _ = cmd.Process.Signal(os.Interrupt) + _, _ = cmd.Process.Wait() + }) + + addrCh := make(chan string, 1) + go func() { + scanner := bufio.NewScanner(stderr) + for scanner.Scan() { + if i := strings.Index(scanner.Text(), "MCP server listening on "); i >= 0 { + addrCh <- strings.TrimSpace(scanner.Text()[i+len("MCP server listening on "):]) + break + } + } + // Keep draining so the child never blocks on a full stderr pipe. + for scanner.Scan() { + } + }() + var addr string + select { + case addr = <-addrCh: + case <-time.After(30 * time.Second): + t.Fatal("timed out waiting for the MCP listen address on stderr") + } + + client := sdkmcp.NewClient(&sdkmcp.Implementation{Name: "integration-test", Version: "0.0.1"}, nil) + ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) + defer cancel() + session, err := client.Connect(ctx, &sdkmcp.StreamableClientTransport{Endpoint: "http://" + addr}, nil) + if err != nil { + t.Fatalf("connecting to the MCP HTTP server: %v", err) + } + defer func() { _ = session.Close() }() + + res, err := session.CallTool(ctx, &sdkmcp.CallToolParams{Name: "dexter_definition", Arguments: map[string]any{"module": "MyApp.Repo", "function": "get"}}) + if err != nil { + t.Fatal(err) + } + if out := mcpToolText(t, res); !strings.Contains(out, "lib/my_app/repo.ex") { + t.Errorf("definition over HTTP missing location:\n%s", out) + } +} + +// Regression: --listen accepted any address, and the server has no +// authentication, so a wildcard address exposed the code and the rename tool +// to the network. +func TestIntegration_MCPListenRefusesNonLoopback(t *testing.T) { + binary := buildDexter(t) + root := scaffoldProject(t) + cmd := exec.Command(binary, "mcp", "--listen=0.0.0.0:0", root) + cmd.Dir = root + out, err := cmd.CombinedOutput() + if err == nil { + t.Fatalf("dexter mcp --listen=0.0.0.0:0 started:\n%s", out) + } + if !strings.Contains(string(out), "refusing to listen") || !strings.Contains(string(out), "--listen-unsafe") { + t.Errorf("refusal does not explain itself:\n%s", out) + } +} diff --git a/internal/daemon/client.go b/internal/daemon/client.go index 50933b1..5460686 100644 --- a/internal/daemon/client.go +++ b/internal/daemon/client.go @@ -367,6 +367,13 @@ func (c *Client) Call(ctx context.Context, method string, params, result any) er c.pendingMu.Lock() delete(c.pending, id) c.pendingMu.Unlock() + // Tell the daemon, so the request stops waiting and frees its slot. + // Best effort: a connection that cannot take it is ending anyway. + if cancelParams, err := json.Marshal(CancelParams{ID: id}); err == nil { + c.writeMu.Lock() + _ = writeJSONLine(c.conn, request{Method: MethodCancel, Params: cancelParams}) + c.writeMu.Unlock() + } return ctx.Err() } } @@ -434,6 +441,10 @@ func (c *Client) WorkspaceStatus(ctx context.Context, waitReadyMs int) (Workspac return status, err } +// Done is closed when the connection ends, for example because the daemon +// exited. A long-lived frontend uses it to know that it must connect again. +func (c *Client) Done() <-chan struct{} { return c.readLoopDone } + // Close closes the control connection. func (c *Client) Close() error { c.closeOnce.Do(func() { diff --git a/internal/daemon/endpoint.go b/internal/daemon/endpoint.go index 7234094..8457f9a 100644 --- a/internal/daemon/endpoint.go +++ b/internal/daemon/endpoint.go @@ -18,7 +18,7 @@ import ( // running daemon, whose startup then rebuilds a populated index whose // IndexVersion differs. Routine changes that leave frontends and daemons // compatible do not bump it, so a running daemon is left alone. -const ContractVersion = 3 +const ContractVersion = 4 // maxSocketPath keeps a workspace socket inside sockaddr_un on every supported // platform (about 104 bytes on macOS, 108 on Linux), including the NUL. diff --git a/internal/daemon/registry.go b/internal/daemon/registry.go index 178e329..a6028ce 100644 --- a/internal/daemon/registry.go +++ b/internal/daemon/registry.go @@ -49,6 +49,8 @@ type Frontend interface { // MethodContext is what a registered control method may use. type MethodContext struct { + // Context is canceled when the client cancels this request (MethodCancel), + // when the connection ends, or when the daemon stops. Context context.Context Runtime *workspace.Runtime Session string @@ -129,7 +131,7 @@ func lookupMethod(name string) (MethodHandler, bool) { func isBuiltinMethod(name string) bool { switch name { case MethodStatus, MethodShutdown, MethodWorkspaceStatus, MethodLookup, - MethodReferences, MethodReindex, MethodWatch, MethodUnwatch: + MethodReferences, MethodReindex, MethodWatch, MethodUnwatch, MethodCancel: return true } return false diff --git a/internal/daemon/server.go b/internal/daemon/server.go index a32804d..58150fd 100644 --- a/internal/daemon/server.go +++ b/internal/daemon/server.go @@ -64,8 +64,19 @@ const ( MethodReindex = "workspace/reindex" MethodWatch = "workspace/watch" MethodUnwatch = "workspace/unwatch" + + // MethodCancel cancels one in-flight request of the same connection, like + // $/cancelRequest in LSP. It is sent with id 0 and gets no response; the + // canceled request still answers, usually with a context error, and the + // client has stopped waiting for it. + MethodCancel = "$/cancel" ) +// CancelParams names the request that MethodCancel cancels. +type CancelParams struct { + ID uint64 `json:"id"` +} + // Status describes the daemon serving a workspace. type Status struct { Root string `json:"root"` @@ -579,6 +590,11 @@ type conn struct { sem chan struct{} requests sync.WaitGroup + // inflight maps the id of each running request to the cancel func of its + // context, for MethodCancel. + inflightMu sync.Mutex + inflight map[uint64]context.CancelFunc + writeMu sync.Mutex subsMu sync.Mutex subs map[string]func() @@ -772,7 +788,7 @@ func (s lspStream) Close() error { return s.conn.Close() } // serveControl handles requests concurrently so a long reindex cannot block a // lookup, and serializes only the writes. func (s *server) serveControl(c *conn, reader *bufio.Reader, sessionID string) error { - mc := MethodContext{ + base := MethodContext{ Context: c.ctx, Runtime: s.runtime, Session: sessionID, @@ -785,6 +801,15 @@ func (s *server) serveControl(c *conn, reader *bufio.Reader, sessionID string) e if err := readJSONLine(reader, &req); err != nil { return err } + if req.Method == MethodCancel { + // Handled on the reader, outside the request limit, so a cancel + // gets through when every slot is taken. + var params CancelParams + if err := json.Unmarshal(req.Params, ¶ms); err == nil { + c.cancelRequest(params.ID) + } + continue + } select { case c.sem <- struct{}{}: case <-c.ctx.Done(): @@ -794,6 +819,12 @@ func (s *server) serveControl(c *conn, reader *bufio.Reader, sessionID string) e continue } c.requests.Add(1) + // Each request has its own context, so a cancel ends its waits and + // frees its slot before the connection ends. + mc := base + var cancel context.CancelFunc + mc.Context, cancel = context.WithCancel(c.ctx) + c.trackRequest(req.ID, cancel) go func(req request) { res := response{ID: req.ID} defer func() { @@ -808,6 +839,8 @@ func (s *server) serveControl(c *conn, reader *bufio.Reader, sessionID string) e log.Printf("Daemon control method %q: %v", req.Method, err) _ = c.write(response{ID: req.ID, Error: fmt.Sprintf("%s result is too large to send (%d bytes, limit %d); narrow the query", req.Method, len(res.Result), maxProtocolLine)}) } + c.untrackRequest(req.ID) + cancel() <-c.sem c.requests.Done() }() @@ -824,6 +857,30 @@ func (s *server) serveControl(c *conn, reader *bufio.Reader, sessionID string) e } } +func (c *conn) trackRequest(id uint64, cancel context.CancelFunc) { + c.inflightMu.Lock() + if c.inflight == nil { + c.inflight = make(map[uint64]context.CancelFunc) + } + c.inflight[id] = cancel + c.inflightMu.Unlock() +} + +func (c *conn) untrackRequest(id uint64) { + c.inflightMu.Lock() + delete(c.inflight, id) + c.inflightMu.Unlock() +} + +func (c *conn) cancelRequest(id uint64) { + c.inflightMu.Lock() + cancel := c.inflight[id] + c.inflightMu.Unlock() + if cancel != nil { + cancel() + } +} + func (s *server) handleRequest(c *conn, mc MethodContext, req request) (any, error) { switch req.Method { case MethodStatus: @@ -958,7 +1015,9 @@ func (s *server) handleRequest(c *conn, mc MethodContext, req request) (any, err // the only watchers, so a frontend uses this instead of watching the tree again. func (s *server) watch(c *conn, mc MethodContext, params WatchParams) (any, error) { changes, cancel := s.runtime.Subscribe(params.Buffer) - ctx, stop := context.WithCancel(mc.Context) + // The subscription outlives the request that made it: it ends with the + // connection or an unwatch. + ctx, stop := context.WithCancel(c.ctx) id := c.addSub(func() { stop() cancel() diff --git a/internal/daemon/server_test.go b/internal/daemon/server_test.go index 3b67eac..288334b 100644 --- a/internal/daemon/server_test.go +++ b/internal/daemon/server_test.go @@ -615,6 +615,10 @@ func init() { RegisterMethod("srvtest/panic", func(MethodContext, json.RawMessage) (any, error) { panic("method exploded on purpose") }) + RegisterMethod("srvtest/wait", func(mc MethodContext, _ json.RawMessage) (any, error) { + <-mc.Context.Done() + return nil, mc.Context.Err() + }) RegisterMethod("srvtest/block", func(mc MethodContext, _ json.RawMessage) (any, error) { blockingMethodMu.Lock() state := blockingMethod @@ -838,3 +842,83 @@ func TestDaemonExitsWhenItsSocketDisappears(t *testing.T) { t.Fatal("daemon kept running after its socket disappeared") } } + +// pipeControlClient serves one control connection over an in-memory pipe. +func pipeControlClient(t *testing.T) *Client { + t.Helper() + quietEnv(t) + root := t.TempDir() + s, endpoint := pipeServer(t, root) + serverConn, clientConn := net.Pipe() + go func() { _ = s.serveConn(serverConn) }() + reader := bufio.NewReader(clientConn) + if err := writeJSONLine(clientConn, hello{ + Contract: ContractVersion, + Kind: kindControl, + Root: endpoint.Root, + Identity: endpoint.Identity, + }); err != nil { + t.Fatal(err) + } + var response helloResponse + if err := readJSONLine(reader, &response); err != nil || !response.OK { + t.Fatalf("handshake = %+v, %v", response, err) + } + client := newClient(clientConn, reader) + t.Cleanup(func() { _ = client.Close() }) + return client +} + +// Regression: a call that the client canceled kept running in the daemon +// until the connection ended, so after 64 canceled calls every new call on the +// connection failed with "too many concurrent control requests". +func TestCanceledCallsFreeTheirSlots(t *testing.T) { + client := pipeControlClient(t) + for i := 0; i < 3*maxConcurrentRequests; i++ { + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Millisecond) + err := client.Call(ctx, "srvtest/wait", struct{}{}, nil) + cancel() + if !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("call %d: error = %v, want the deadline", i, err) + } + } + if _, err := client.DaemonStatus(context.Background()); err != nil { + t.Fatalf("connection unusable after canceled calls: %v", err) + } +} + +// lastContractWithoutCancel is the contract of the builds whose daemons do not +// know $/cancel. A frontend that sends it must not attach to such a daemon: +// its canceled requests would keep their slots. +const lastContractWithoutCancel = 3 + +// The contract was bumped with $/cancel, so the two builds refuse each other +// and the newer one replaces the older daemon (TestEnsureReplacesAnOlderDaemon) +// instead of sharing it. +func TestContractBumpedForRequestCancel(t *testing.T) { + if ContractVersion <= lastContractWithoutCancel { + t.Fatalf("ContractVersion = %d; $/cancel needs a contract above %d", ContractVersion, lastContractWithoutCancel) + } + quietEnv(t) + root := t.TempDir() + s, endpoint := pipeServer(t, root) + serverConn, clientConn := net.Pipe() + go func() { _ = s.serveConn(serverConn) }() + defer func() { _ = clientConn.Close() }() + reader := bufio.NewReader(clientConn) + if err := writeJSONLine(clientConn, hello{ + Contract: lastContractWithoutCancel, + Kind: kindControl, + Root: endpoint.Root, + Identity: endpoint.Identity, + }); err != nil { + t.Fatal(err) + } + var response helloResponse + if err := readJSONLine(reader, &response); err != nil { + t.Fatal(err) + } + if response.OK || !response.Incompatible || response.Exiting { + t.Fatalf("handshake from contract %d = %+v, want an incompatible refusal that keeps the daemon", lastContractWithoutCancel, response) + } +} diff --git a/internal/lsp/api.go b/internal/lsp/api.go new file mode 100644 index 0000000..5764bda --- /dev/null +++ b/internal/lsp/api.go @@ -0,0 +1,251 @@ +package lsp + +import ( + "context" + "fmt" + "sort" + "sync" + "time" + + "go.lsp.dev/uri" +) + +// This file is the exported, name-based surface of the LSP server for callers +// outside an editor session, such as the MCP tools that the workspace daemon +// runs. Everything here delegates to the same internals the LSP handlers use, +// so the results are the same for each frontend. + +// ReadFileText returns a file's current text, preferring an editor-owned buffer. +func (s *Server) ReadFileText(filePath string) (text string, open bool, ok bool) { + return s.readFileText(filePath) +} + +// FileLine returns one 1-based line, preferring an editor-owned buffer. +func (s *Server) FileLine(filePath string, lineNum int) (string, bool) { + return s.getFileLine(filePath, lineNum) +} + +// UnsavedBuffer returns the text of filePath when an editor holds it open in +// this session with changes that it has not saved, and the time of the last +// change. seq orders buffers across sessions: the higher one changed later. +func (s *Server) UnsavedBuffer(filePath string) (text string, seq uint64, changedAt time.Time, ok bool) { + return s.docs.UnsavedBuffer(string(uri.File(filePath))) +} + +// RenameSummary reports what a rename changed on disk. +type RenameSummary struct { + FilesChanged []string + FilesMoved map[string]string // old path → new path (conventional module renames) + // FilesFailed lists the files the rename could not change. They still use + // the old name. FailureReason is the first error. + FilesFailed []string + FailureReason string + + mu sync.Mutex + indexed []<-chan struct{} +} + +// changed records that the rename changed path. A nil summary ignores it, so +// the editor rename passes nil. +func (r *RenameSummary) changed(path string) { + if r == nil { + return + } + r.mu.Lock() + r.FilesChanged = append(r.FilesChanged, path) + r.mu.Unlock() +} + +// failed records that the rename could not change path. +func (r *RenameSummary) failed(path string, err error) { + if r == nil { + return + } + r.mu.Lock() + if len(r.FilesFailed) == 0 { + r.FailureReason = err.Error() + } + r.FilesFailed = append(r.FilesFailed, path) + r.mu.Unlock() +} + +// recordFailures copies the failures of one rename pass into the summary. +func (r *RenameSummary) recordFailures(f *renameFailures) { + if r == nil { + return + } + f.mu.Lock() + paths := append([]string(nil), f.paths...) + first := f.first + f.mu.Unlock() + if len(paths) == 0 { + return + } + r.mu.Lock() + if len(r.FilesFailed) == 0 { + r.FailureReason = first + } + r.FilesFailed = append(r.FilesFailed, paths...) + r.mu.Unlock() +} + +// recordModuleRename records the files and moves of a module rename. +func (r *RenameSummary) recordModuleRename(sitesByFile map[string][]moduleEditSite, movedFiles, clientRenames map[string]string, failures *renameFailures) { + if r == nil { + return + } + r.mu.Lock() + for path := range sitesByFile { + r.FilesChanged = append(r.FilesChanged, path) + } + if len(movedFiles)+len(clientRenames) > 0 && r.FilesMoved == nil { + r.FilesMoved = make(map[string]string, len(movedFiles)+len(clientRenames)) + } + for from, to := range movedFiles { + r.FilesMoved[from] = to + } + for from, to := range clientRenames { + r.FilesMoved[from] = to + } + r.mu.Unlock() + r.recordFailures(failures) +} + +// waitFor records an index update that the rename started. finish waits for +// it, so the caller sees an index that shows the rename. +func (r *RenameSummary) waitFor(indexed <-chan struct{}) { + if r == nil { + return + } + r.mu.Lock() + r.indexed = append(r.indexed, indexed) + r.mu.Unlock() +} + +// finish waits for the index updates, sorts the lists, removes duplicates, +// and removes the failed files from the changed files. +func (r *RenameSummary) finish() { + r.mu.Lock() + indexed := r.indexed + r.indexed = nil + r.mu.Unlock() + for _, ch := range indexed { + <-ch + } + + r.mu.Lock() + defer r.mu.Unlock() + r.FilesFailed = sortedUnique(r.FilesFailed) + failed := make(map[string]struct{}, len(r.FilesFailed)) + for _, path := range r.FilesFailed { + failed[path] = struct{}{} + } + all := sortedUnique(r.FilesChanged) + changed := make([]string, 0, len(all)) + for _, path := range all { + if _, ok := failed[path]; !ok { + changed = append(changed, path) + } + } + r.FilesChanged = changed +} + +func sortedUnique(paths []string) []string { + if len(paths) == 0 { + return paths + } + sort.Strings(paths) + out := paths[:1] + for _, path := range paths[1:] { + if path != out[len(out)-1] { + out = append(out, path) + } + } + return out +} + +// RenameFunction renames module.functionName to newName across the workspace +// with the same validation and the same machinery as the editor rename. It +// returns when the index shows the rename. +// +// The caller has no editor, so every file that this server does not hold open +// is written on disk. The workspace daemon calls it on its headless language +// service, which holds no editor buffers. +func (s *Server) RenameFunction(module, functionName, newName string) (*RenameSummary, error) { + return s.RenameFunctionContext(context.Background(), module, functionName, newName) +} + +// RenameFunctionContext is RenameFunction for a caller that can cancel. A +// rename canceled before it starts to write changes nothing; one canceled +// during its writes completes. +func (s *Server) RenameFunctionContext(ctx context.Context, module, functionName, newName string) (*RenameSummary, error) { + if !isValidFunctionName(newName) { + return nil, fmt.Errorf("invalid function name %q: must match [a-z_][a-z0-9_?!]*", newName) + } + defs, err := s.store.LookupFunction(module, functionName) + if err != nil { + return nil, err + } + if len(defs) == 0 { + return nil, fmt.Errorf("function %s.%s not found in the index", module, functionName) + } + if existing, err := s.store.LookupFunction(module, newName); err == nil && len(existing) > 0 { + return nil, fmt.Errorf("function %s.%s already exists", module, newName) + } + + summary := &RenameSummary{} + edit, err := s.renameFunctionEdits(ctx, module, functionName, newName, summary) + summary.finish() + if err != nil { + return nil, err + } + if err := requireNoBufferEdits(edit); err != nil { + return summary, err + } + return summary, nil +} + +// RenameModule renames oldModule (and its submodules) to newModule across the +// workspace, with the same validation and machinery as the editor rename, +// including the moves of files that follow the naming convention. It returns +// when the index shows the rename. +func (s *Server) RenameModule(oldModule, newModule string) (*RenameSummary, error) { + return s.RenameModuleContext(context.Background(), oldModule, newModule) +} + +// RenameModuleContext is RenameModule for a caller that can cancel, with the +// same rule as RenameFunctionContext. +func (s *Server) RenameModuleContext(ctx context.Context, oldModule, newModule string) (*RenameSummary, error) { + if !isValidModuleName(newModule) { + return nil, fmt.Errorf("invalid module name %q: must be CamelCase segments separated by dots", newModule) + } + defs, err := s.store.LookupModule(oldModule) + if err != nil { + return nil, err + } + if len(defs) == 0 { + return nil, fmt.Errorf("module %s not found in the index", oldModule) + } + + summary := &RenameSummary{} + edit, err := s.renameModuleEdits(ctx, oldModule, newModule, summary) + summary.finish() + if err != nil { + return nil, err + } + if err := requireNoBufferEdits(edit); err != nil { + return summary, err + } + return summary, nil +} + +// requireNoBufferEdits reports an error when the rename machinery left edits +// for buffers that the server holds open: a caller without an editor cannot +// apply them. The headless language service holds no buffers, so this is a +// guard, not an expected path. +func requireNoBufferEdits(edit *WorkspaceEdit) error { + if edit == nil || (len(edit.Changes) == 0 && len(edit.DocumentChanges) == 0) { + return nil + } + return fmt.Errorf("the rename has edits for files that are open in this language service; only an editor can apply them") +} diff --git a/internal/lsp/api_test.go b/internal/lsp/api_test.go new file mode 100644 index 0000000..2dea81a --- /dev/null +++ b/internal/lsp/api_test.go @@ -0,0 +1,254 @@ +package lsp + +import ( + "context" + "os" + "path/filepath" + "runtime" + "strings" + "testing" + "time" +) + +// A rename without an editor writes every changed file, reports the files, and +// returns only when the index shows the new name. +func TestRenameFunction_HeadlessWritesFilesAndIndex(t *testing.T) { + server, cleanup := setupTestServer(t) + defer cleanup() + + indexFile(t, server.store, server.projectRoot, "lib/accounts.ex", `defmodule MyApp.Accounts do + def fetch_user(id), do: id +end +`) + indexFile(t, server.store, server.projectRoot, "lib/caller.ex", `defmodule MyApp.Caller do + def go(id), do: MyApp.Accounts.fetch_user(id) +end +`) + callerPath := filepath.Join(server.projectRoot, "lib/caller.ex") + + summary, err := server.RenameFunction("MyApp.Accounts", "fetch_user", "get_user") + if err != nil { + t.Fatal(err) + } + + if len(summary.FilesChanged) != 2 || len(summary.FilesFailed) != 0 { + t.Errorf("FilesChanged = %v, FilesFailed = %v; want both files changed", summary.FilesChanged, summary.FilesFailed) + } + data, err := os.ReadFile(callerPath) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(data), "MyApp.Accounts.get_user(id)") { + t.Errorf("caller not written to disk:\n%s", data) + } + // No wait here: RenameFunction returns after the index update. + results, err := server.store.LookupFunction("MyApp.Accounts", "get_user") + if err != nil || len(results) == 0 { + t.Errorf("index not updated when the rename returned: %v, %v", results, err) + } +} + +// A file that the rename cannot write is reported, and is not counted as +// changed, so the caller can tell the user which files still use the old name. +func TestRenameFunction_ReportsFilesItCouldNotWrite(t *testing.T) { + if runtime.GOOS == "windows" || os.Geteuid() == 0 { + t.Skip("needs a file that the process cannot write") + } + server, cleanup := setupTestServer(t) + defer cleanup() + + indexFile(t, server.store, server.projectRoot, "lib/accounts.ex", `defmodule MyApp.Accounts do + def fetch_user(id), do: id +end +`) + indexFile(t, server.store, server.projectRoot, "lib/caller.ex", `defmodule MyApp.Caller do + def go(id), do: MyApp.Accounts.fetch_user(id) +end +`) + callerPath := filepath.Join(server.projectRoot, "lib/caller.ex") + if err := os.Chmod(callerPath, 0444); err != nil { + t.Fatal(err) + } + defer func() { _ = os.Chmod(callerPath, 0644) }() + + summary, err := server.RenameFunction("MyApp.Accounts", "fetch_user", "get_user") + if err != nil { + t.Fatal(err) + } + if len(summary.FilesFailed) != 1 || summary.FilesFailed[0] != callerPath || summary.FailureReason == "" { + t.Errorf("FilesFailed = %v (%q), want %s", summary.FilesFailed, summary.FailureReason, callerPath) + } + for _, path := range summary.FilesChanged { + if path == callerPath { + t.Errorf("FilesChanged includes the file that could not be written: %v", summary.FilesChanged) + } + } +} + +// Without an editor, nothing is open, so the server moves conventional files +// itself and the summary reports the move. +func TestRenameModule_HeadlessMovesFilesItself(t *testing.T) { + server, cleanup := setupTestServer(t) + defer cleanup() + + indexFile(t, server.store, server.projectRoot, "lib/accounts.ex", `defmodule MyApp.Accounts do + def list_users, do: [] +end +`) + oldPath := filepath.Join(server.projectRoot, "lib/accounts.ex") + newPath := filepath.Join(server.projectRoot, "lib/auth.ex") + + summary, err := server.RenameModule("MyApp.Accounts", "MyApp.Auth") + if err != nil { + t.Fatal(err) + } + + if _, err := os.Stat(oldPath); err == nil { + t.Error("expected accounts.ex to be gone") + } + data, err := os.ReadFile(newPath) + if err != nil { + t.Fatalf("expected auth.ex on disk: %v", err) + } + if !strings.Contains(string(data), "defmodule MyApp.Auth") { + t.Errorf("expected 'defmodule MyApp.Auth', got:\n%s", data) + } + if summary.FilesMoved[oldPath] != newPath { + t.Errorf("summary reports moves %v, want %s → %s", summary.FilesMoved, oldPath, newPath) + } + if results, err := server.store.LookupModule("MyApp.Auth"); err != nil || len(results) == 0 { + t.Errorf("index not updated when the rename returned: %v, %v", results, err) + } +} + +// A namespace-only module rename keeps the conventional file path, so the +// summary reports no move and the file is renamed in place. +func TestRenameModule_NamespaceOnlyKeepsPath(t *testing.T) { + server, cleanup := setupTestServer(t) + defer cleanup() + + path := filepath.Join(server.projectRoot, "lib", "user.ex") + indexFile(t, server.store, server.projectRoot, "lib/user.ex", `defmodule MyApp.Accounts.User do + def name(u), do: u.name +end +`) + + summary, err := server.RenameModule("MyApp.Accounts.User", "MyApp.Billing.User") + if err != nil { + t.Fatal(err) + } + if len(summary.FilesMoved) != 0 { + t.Errorf("namespace-only rename reported moves: %v", summary.FilesMoved) + } + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("file left its conventional path: %v", err) + } + if !strings.Contains(string(data), "defmodule MyApp.Billing.User") { + t.Errorf("module not renamed in place:\n%s", data) + } +} + +func TestRenameValidation(t *testing.T) { + server, cleanup := setupTestServer(t) + defer cleanup() + indexFile(t, server.store, server.projectRoot, "lib/accounts.ex", `defmodule MyApp.Accounts do + def fetch_user(id), do: id + def list_users, do: [] +end +`) + cases := []struct { + name string + run func() error + want string + }{ + {"invalid function name", func() error { _, err := server.RenameFunction("MyApp.Accounts", "fetch_user", "NotValid"); return err }, "invalid function name"}, + {"existing function", func() error { + _, err := server.RenameFunction("MyApp.Accounts", "fetch_user", "list_users") + return err + }, "already exists"}, + {"missing function", func() error { _, err := server.RenameFunction("MyApp.Accounts", "missing", "other"); return err }, "not found"}, + {"invalid module name", func() error { _, err := server.RenameModule("MyApp.Accounts", "my_app"); return err }, "invalid module name"}, + {"missing module", func() error { _, err := server.RenameModule("MyApp.Missing", "MyApp.Other"); return err }, "not found"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if err := tc.run(); err == nil || !strings.Contains(err.Error(), tc.want) { + t.Errorf("error = %v, want %q", err, tc.want) + } + }) + } +} + +// Regression: a rename that waited for the rename lock, or for the index to +// show the previous rename, ignored its context, so a rename that the client +// had canceled could still start to write. +func TestRenameContext_CanceledWhileWaitingChangesNothing(t *testing.T) { + server, cleanup := setupTestServer(t) + defer cleanup() + indexFile(t, server.store, server.projectRoot, "lib/accounts.ex", "defmodule MyApp.Accounts do\n def fetch_user(id), do: id\nend\n") + path := filepath.Join(server.projectRoot, "lib/accounts.ex") + + // Another rename holds the lock, and its index update has not finished. + pending := make(chan struct{}) + server.index.renameSerial.Lock() + server.index.renameIndexed = pending + + ctx, cancel := context.WithCancel(context.Background()) + errc := make(chan error, 1) + go func() { + _, err := server.RenameFunctionContext(ctx, "MyApp.Accounts", "fetch_user", "get_user") + errc <- err + }() + time.Sleep(20 * time.Millisecond) + cancel() + server.index.renameSerial.Unlock() // the other rename's writes end; its index update does not + + select { + case err := <-errc: + if err == nil || !strings.Contains(err.Error(), "canceled before it changed any file") { + t.Fatalf("error = %v, want a cancellation", err) + } + case <-time.After(5 * time.Second): + t.Fatal("the canceled rename kept waiting for the previous rename's index update") + } + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if strings.Contains(string(data), "get_user") { + t.Fatalf("a canceled rename wrote the file:\n%s", data) + } + close(pending) +} + +// A rename canceled while another rename held the lock does not write once +// it gets the lock. +func TestRenameContext_CanceledWhileLockedChangesNothing(t *testing.T) { + server, cleanup := setupTestServer(t) + defer cleanup() + indexFile(t, server.store, server.projectRoot, "lib/accounts.ex", "defmodule MyApp.Accounts do\n def fetch_user(id), do: id\nend\n") + path := filepath.Join(server.projectRoot, "lib/accounts.ex") + + server.index.renameSerial.Lock() + ctx, cancel := context.WithCancel(context.Background()) + errc := make(chan error, 1) + go func() { + _, err := server.RenameModuleContext(ctx, "MyApp.Accounts", "MyApp.Users") + errc <- err + }() + time.Sleep(20 * time.Millisecond) + cancel() + server.index.renameSerial.Unlock() + + if err := <-errc; err == nil || !strings.Contains(err.Error(), "canceled before it changed any file") { + t.Fatalf("error = %v, want a cancellation", err) + } + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if strings.Contains(string(data), "MyApp.Users") { + t.Fatalf("a canceled rename wrote the file:\n%s", data) + } +} diff --git a/internal/lsp/documents.go b/internal/lsp/documents.go index 8fec7ec..2a55cc4 100644 --- a/internal/lsp/documents.go +++ b/internal/lsp/documents.go @@ -5,6 +5,8 @@ import ( "os" "strings" "sync" + "sync/atomic" + "time" tree_sitter "github.com/tree-sitter/go-tree-sitter" tree_sitter_elixir "github.com/tree-sitter/tree-sitter-elixir/bindings/go" @@ -31,8 +33,20 @@ type cachedDoc struct { // LRU and evicted once the transient cap is reached. Editor-owned // entries (created via Set) are never transient and never evicted. transient bool + // seq orders editor-owned entries across every store in the process: + // the higher one was set later. A frontend without an editor uses it to + // pick the newest unsaved buffer when several editors hold one file. + seq uint64 + // dirty is true when the editor changed the buffer after its last open + // or save, at changedAt. A clean buffer can be older than the disk (an + // editor that has not reloaded yet), so only a dirty one is unsaved work. + dirty bool + changedAt time.Time } +// docSeq numbers editor-owned entries; see cachedDoc.seq. +var docSeq atomic.Uint64 + // refTree wraps a tree-sitter parse tree with refcounting so that // concurrent handlers walking the tree (RootNode, queries) aren't racing // with eviction or replacement, which would free the underlying C memory @@ -144,7 +158,32 @@ func (ds *DocumentStore) Set(uri string, text string) { } // Editor took ownership of this URI - drop any LRU tracking for it. ds.removeFromLRULocked(uri) - ds.docs[uri] = &cachedDoc{text: text} + ds.docs[uri] = &cachedDoc{text: text, seq: docSeq.Add(1)} +} + +// SetChanged is Set for an edit in the editor: the buffer has unsaved changes +// until MarkSaved. +func (ds *DocumentStore) SetChanged(uri string, text string) { + ds.mu.Lock() + defer ds.mu.Unlock() + if ds.closed { + return + } + if doc, ok := ds.docs[uri]; ok { + doc.tree.retireLocked() + } + ds.removeFromLRULocked(uri) + ds.docs[uri] = &cachedDoc{text: text, seq: docSeq.Add(1), dirty: true, changedAt: time.Now()} +} + +// MarkSaved records that the editor saved the buffer, so it has no unsaved +// changes. +func (ds *DocumentStore) MarkSaved(uri string) { + ds.mu.Lock() + defer ds.mu.Unlock() + if doc, ok := ds.docs[uri]; ok && !doc.transient { + doc.dirty = false + } } func (ds *DocumentStore) Close(uri string) { @@ -206,6 +245,19 @@ func (ds *DocumentStore) GetIfOpen(uri string) (string, bool) { return doc.text, true } +// UnsavedBuffer returns an editor-owned entry that has changes the editor has +// not saved, with its sequence number (see cachedDoc.seq) and the time of its +// last change. +func (ds *DocumentStore) UnsavedBuffer(uri string) (text string, seq uint64, changedAt time.Time, ok bool) { + ds.mu.RLock() + defer ds.mu.RUnlock() + doc, found := ds.docs[uri] + if !found || doc.transient || !doc.dirty { + return "", 0, time.Time{}, false + } + return doc.text, doc.seq, doc.changedAt, true +} + // GetOrLoad returns the text for the given URI, falling back to a disk // read if no editor has opened the document. Disk-loaded entries are // marked transient and tracked in an LRU; if the transient population diff --git a/internal/lsp/rename_test.go b/internal/lsp/rename_test.go index a5874ec..400ceed 100644 --- a/internal/lsp/rename_test.go +++ b/internal/lsp/rename_test.go @@ -3567,3 +3567,90 @@ func TestRename_Module_ConventionalPathUnchangedOpenFile(t *testing.T) { t.Errorf("expected 'defmodule MyApp.AbTest', got:\n%s", got) } } + +// Moving one member of a grouped alias to another namespace must not rewrite +// the shared prefix: the other members would then resolve to modules that do +// not exist. The moved member leaves the group and gets its own alias. +func TestRename_Module_GroupedAliasMemberMovedToOtherNamespace(t *testing.T) { + server, cleanup := setupTestServer(t) + defer cleanup() + + indexFile(t, server.store, server.projectRoot, "lib/shared_lib/worker.ex", `defmodule SharedLib.Worker do + def call, do: :ok +end +`) + indexFile(t, server.store, server.projectRoot, "lib/shared_lib/config.ex", `defmodule SharedLib.Config do + def get, do: :ok +end +`) + callerPath := filepath.Join(server.projectRoot, "lib", "runner.ex") + indexFile(t, server.store, server.projectRoot, "lib/runner.ex", `defmodule MyApp.Runner do + alias SharedLib.{Config, Worker} + + def run, do: {Config.get(), Worker.call()} +end +`) + + if _, err := server.RenameModule("SharedLib.Worker", "OtherLib.Worker"); err != nil { + t.Fatal(err) + } + + got, err := os.ReadFile(callerPath) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{" alias SharedLib.{Config}\n", " alias OtherLib.Worker\n"} { + if !strings.Contains(string(got), want) { + t.Errorf("expected %q, got:\n%s", want, got) + } + } + if strings.Contains(string(got), "OtherLib.{") { + t.Errorf("the shared prefix was rewritten, so Config now names OtherLib.Config:\n%s", got) + } +} + +// A grouped alias whose members continue on the next lines names the prefix +// only on the opening line, where the index records every member. Renaming +// the prefix module must rewrite that line. +func TestRename_Module_MultilineGroupedAlias(t *testing.T) { + server, cleanup := setupTestServer(t) + defer cleanup() + + indexFile(t, server.store, server.projectRoot, "lib/shared_lib.ex", `defmodule SharedLib do + def start, do: :ok +end +`) + indexFile(t, server.store, server.projectRoot, "lib/shared_lib/worker.ex", `defmodule SharedLib.Worker do + def call, do: :ok +end +`) + indexFile(t, server.store, server.projectRoot, "lib/shared_lib/config.ex", `defmodule SharedLib.Config do + def get, do: :ok +end +`) + callerPath := filepath.Join(server.projectRoot, "lib", "runner.ex") + indexFile(t, server.store, server.projectRoot, "lib/runner.ex", `defmodule MyApp.Runner do + alias SharedLib.{ + Config, + Worker + } + + def run, do: {Config.get(), Worker.call()} +end +`) + + if _, err := server.RenameModule("SharedLib", "CoreLib"); err != nil { + t.Fatal(err) + } + + got, err := os.ReadFile(callerPath) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(got), " alias CoreLib.{\n Config,\n Worker\n }") { + t.Errorf("expected the multi-line group to use CoreLib, got:\n%s", got) + } + if strings.Contains(string(got), "SharedLib") { + t.Errorf("SharedLib should be gone, got:\n%s", got) + } +} diff --git a/internal/lsp/server.go b/internal/lsp/server.go index cec6c75..b375fb5 100644 --- a/internal/lsp/server.go +++ b/internal/lsp/server.go @@ -97,6 +97,16 @@ type IndexCoordinator struct { stdlibMu sync.RWMutex stdlibRoot string + // renameSerial keeps renames one at a time across every session of the + // workspace, editor and headless alike. A rename reads the affected files + // and writes them back; two at once would each write over the other's + // edits. It is held from the first read of the affected files to the end + // of the writes, and no other request takes it. renameIndexed, guarded by + // it, closes when the index shows the last rename, so the next rename + // checks its new name against a current index. + renameSerial sync.Mutex + renameIndexed <-chan struct{} + // writes is held for writing by a cold full build and for reading by every // single-file write. The bulk path is insert-only and cannot overlap any // incremental mutation. @@ -120,6 +130,24 @@ type IndexCoordinator struct { cancelWork context.CancelFunc } +// lockRename takes renameSerial and waits until the index shows the last +// rename. It returns the unlock. When ctx ends first, it returns ctx's error +// and holds nothing, so a canceled rename does not start to write. +func (ic *IndexCoordinator) lockRename(ctx context.Context) (func(), error) { + ic.renameSerial.Lock() + if ic.renameIndexed != nil { + select { + case <-ic.renameIndexed: + case <-ctx.Done(): + } + } + if err := ctx.Err(); err != nil { + ic.renameSerial.Unlock() + return nil, fmt.Errorf("the rename was canceled before it changed any file: %w", err) + } + return ic.renameSerial.Unlock, nil +} + // CancelWork stops the reconciliation in flight and makes every later one // return at once. The workspace calls it first when it shuts down, so a pass // over a large change set cannot keep the process (and its workspace lock) @@ -1089,7 +1117,7 @@ func (s *Server) DidChange(ctx context.Context, params *protocol.DidChangeTextDo if len(params.ContentChanges) > 0 { // Full sync mode — last change contains the full text text := params.ContentChanges[len(params.ContentChanges)-1].Text - s.docs.Set(string(params.TextDocument.URI), text) + s.docs.SetChanged(string(params.TextDocument.URI), text) } return nil } @@ -1115,6 +1143,7 @@ func (s *Server) restartBeamForFormatterConfig(path string) { } func (s *Server) DidSave(ctx context.Context, params *protocol.DidSaveTextDocumentParams) error { + s.docs.MarkSaved(string(params.TextDocument.URI)) path := uriToPath(params.TextDocument.URI) if path == "" { return nil @@ -5737,7 +5766,7 @@ func (s *Server) RenameEdit(ctx context.Context, params *protocol.RenameParams) if existing, err := s.store.LookupFunction(fullModule, params.NewName); err == nil && len(existing) > 0 { return nil, fmt.Errorf("function %s.%s already exists", fullModule, params.NewName) } - return s.renameFunctionEdits(fullModule, functionName, params.NewName) + return s.renameFunctionEdits(ctx, fullModule, functionName, params.NewName, nil) } } else if moduleRef != "" { fullModule := resolveModule(moduleRef, aliases) @@ -5751,7 +5780,7 @@ func (s *Server) RenameEdit(ctx context.Context, params *protocol.RenameParams) if !isValidModuleName(newModule) { return nil, fmt.Errorf("invalid module name %q: must be CamelCase segments separated by dots", params.NewName) } - return s.renameModuleEdits(fullModule, newModule) + return s.renameModuleEdits(ctx, fullModule, newModule, nil) } } } @@ -5760,8 +5789,18 @@ func (s *Server) RenameEdit(ctx context.Context, params *protocol.RenameParams) } // renameFunctionEdits builds a WorkspaceEdit renaming all occurrences of -// module.functionName to newName across the codebase. -func (s *Server) renameFunctionEdits(module, functionName, newName string) (*WorkspaceEdit, error) { +// module.functionName to newName across the codebase. When report is not nil, +// it receives the files the rename changed and the files it could not change. +func (s *Server) renameFunctionEdits(ctx context.Context, module, functionName, newName string, report *RenameSummary) (*WorkspaceEdit, error) { + unlock, err := s.index.lockRename(ctx) + if err != nil { + return nil, err + } + defer unlock() + // The check before the lock can race another rename to the same name. + if existing, err := s.store.LookupFunction(module, newName); err == nil && len(existing) > 0 { + return nil, fmt.Errorf("function %s.%s already exists", module, newName) + } // Collect all (filePath, lineNumber) pairs — definitions + references type siteKey struct { filePath string @@ -5885,7 +5924,7 @@ func (s *Server) renameFunctionEdits(module, functionName, newName string) (*Wor } } - edit := s.buildTextEdits(sites, functionName, newName) + edit := s.buildTextEdits(sites, functionName, newName, report) // Update defdelegate lines that forward to this function: add or update // the `as:` option so the facade keeps working after the rename. @@ -5925,6 +5964,7 @@ func (s *Server) renameFunctionEdits(module, functionName, newName string) (*Wor } fileURI := protocol.DocumentURI(uri.File(del.FilePath)) + report.changed(del.FilePath) if open { if edit.Changes == nil { edit.Changes = make(map[protocol.DocumentURI][]protocol.TextEdit) @@ -5942,7 +5982,10 @@ func (s *Server) renameFunctionEdits(module, functionName, newName string) (*Wor newFileLines = append(newFileLines, fileLines[:spanStart]...) newFileLines = append(newFileLines, updatedSpan...) newFileLines = append(newFileLines, fileLines[spanEnd:]...) - _ = os.WriteFile(del.FilePath, []byte(strings.Join(newFileLines, "\n")), 0644) + if err := os.WriteFile(del.FilePath, []byte(strings.Join(newFileLines, "\n")), 0644); err != nil { + log.Printf("Rename: cannot write %s: %v", del.FilePath, err) + report.failed(del.FilePath, err) + } } } } @@ -5958,8 +6001,15 @@ func (s *Server) renameFunctionEdits(module, functionName, newName string) (*Wor // parallel goroutines. Only open buffers are included in the returned // WorkspaceEdit, keeping the response small and avoiding editor freezes. // Files following the naming convention are also renamed/moved: closed ones -// by the server, open ones by the client through rename operations. -func (s *Server) renameModuleEdits(oldModule, newModule string) (*WorkspaceEdit, error) { +// by the server, open ones by the client through rename operations. When +// report is not nil, it receives the files the rename changed or moved and the +// files it could not change. +func (s *Server) renameModuleEdits(ctx context.Context, oldModule, newModule string, report *RenameSummary) (*WorkspaceEdit, error) { + unlock, err := s.index.lockRename(ctx) + if err != nil { + return nil, err + } + defer unlock() mr := s.buildModuleRename(oldModule, newModule) // Check for collisions: verify that none of the target module names @@ -5975,8 +6025,10 @@ func (s *Server) renameModuleEdits(oldModule, newModule string) (*WorkspaceEdit, movedFiles, clientRenames := mr.moveConventionalFiles(fileCache) openChanges := mr.applyEdits(fileCache, movedFiles) - mr.reindex(fileCache, movedFiles, clientRenames) + indexed := mr.reindex(fileCache, movedFiles, clientRenames) s.reportRenameFailures(&mr.failures) + report.recordModuleRename(mr.sitesByFile, movedFiles, clientRenames, &mr.failures) + report.waitFor(indexed) if len(clientRenames) == 0 { return &WorkspaceEdit{Changes: openChanges}, nil @@ -6272,10 +6324,14 @@ func (mr *moduleRename) findModuleEdits(lineText string, token string) []moduleE // is indexed as its own reference — so the reference's full name never appears // on the line. // -// Which half moves depends on the rename: renaming the prefix rewrites the -// prefix, renaming a member rewrites that member inside the braces. Sites for -// the other members on the same line find nothing once the prefix is rewritten, -// so a group is only edited once. +// Which half moves depends on the rename. When the renamed module is the +// prefix or one of its ancestors, every member moves with it, so the prefix is +// rewritten; this is also correct for a group whose members continue on the +// next lines, where the index records every member on the opening line. Sites +// for the other members on the same line find nothing once the prefix is +// rewritten, so a group is only edited once. When only the member is renamed, +// the member is rewritten inside the braces, or, when it moves to another +// namespace, it leaves the group and gets its own alias. func (mr *moduleRename) findGroupedAliasEdits(lineText, token, newToken string) []moduleEditResult { dot := strings.LastIndexByte(token, '.') if dot <= 0 { @@ -6286,29 +6342,61 @@ func (mr *moduleRename) findGroupedAliasEdits(lineText, token, newToken string) if prefixCol < 0 { return nil } - memberCols := findAllTokenColumns(lineText[groupStart:groupEnd], member) - if len(memberCols) == 0 { - return nil - } newDot := strings.LastIndexByte(newToken, '.') if newDot <= 0 { // The member lost its namespace; a grouped alias cannot express that. return nil } - if newPrefix := newToken[:newDot]; newPrefix != prefix { + newPrefix, newMember := newToken[:newDot], newToken[newDot+1:] + if prefix == mr.oldModule || strings.HasPrefix(prefix, mr.oldModule+".") { + if newPrefix == prefix { + return nil + } return []moduleEditResult{{prefixCol, len(prefix), newPrefix}} } - newMember := newToken[newDot+1:] - if newMember == member { + memberCols := findAllTokenColumns(lineText[groupStart:groupEnd], member) + if len(memberCols) == 0 { return nil } - results := make([]moduleEditResult, 0, len(memberCols)) - for _, col := range memberCols { - results = append(results, moduleEditResult{groupStart + col, len(member), newMember}) + if newPrefix == prefix { + if newMember == member { + return nil + } + results := make([]moduleEditResult, 0, len(memberCols)) + for _, col := range memberCols { + results = append(results, moduleEditResult{groupStart + col, len(member), newMember}) + } + return results + } + return splitGroupedAlias(lineText, prefix, member, newToken, prefixCol, groupStart, groupEnd) +} + +// splitGroupedAlias moves one member out of `Prefix.{A, B}` to its own +// `alias New.A` line after the group, because only that member changes +// namespace. Rewriting the shared prefix instead would move the other members +// too. A group with no other member gets the new name in place. A group that +// continues on the next lines is left unchanged: the members are not on this +// line. +func splitGroupedAlias(lineText, prefix, member, newToken string, prefixCol, groupStart, groupEnd int) []moduleEditResult { + if groupEnd >= len(lineText) || lineText[groupEnd] != '}' { + return nil } - return results + var rest []string + for _, m := range strings.Split(lineText[groupStart:groupEnd], ",") { + if m = strings.TrimSpace(m); m != "" && m != member { + rest = append(rest, m) + } + } + span := groupEnd + 1 - prefixCol + if len(rest) == 0 { + return []moduleEditResult{{prefixCol, span, newToken}} + } + // The new line repeats what comes before the prefix: the indentation and + // the alias, require, or import keyword. + replacement := prefix + ".{" + strings.Join(rest, ", ") + "}\n" + lineText[:prefixCol] + newToken + return []moduleEditResult{{prefixCol, span, replacement}} } type moduleEditResult struct { @@ -6547,7 +6635,7 @@ func overlapsClaimed(claimed []moduleEditResult, e moduleEditResult) bool { // back from disk. clientRenames have not moved yet — the client applies them // when it receives the reply — so their new paths are indexed from the text // the edits produce. -func (mr *moduleRename) reindex(fileCache map[string]moduleFileInfo, movedFiles, clientRenames map[string]string) { +func (mr *moduleRename) reindex(fileCache map[string]moduleFileInfo, movedFiles, clientRenames map[string]string) <-chan struct{} { removePaths := make([]string, 0, len(movedFiles)+len(clientRenames)) for oldPath := range movedFiles { removePaths = append(removePaths, oldPath) @@ -6594,7 +6682,7 @@ func (mr *moduleRename) reindex(fileCache map[string]moduleFileInfo, movedFiles, } } - mr.server.reindexAfterRename(removePaths, reindexPaths, openReindexes) + return mr.server.reindexAfterRename(removePaths, reindexPaths, openReindexes) } type renameSite struct { @@ -6611,7 +6699,7 @@ type textReindex struct { // buildTextEdits creates a WorkspaceEdit replacing all whole-token occurrences // of oldToken with newToken. Open buffers are returned in the WorkspaceEdit; // closed files are written directly to disk in parallel goroutines. -func (s *Server) buildTextEdits(sites []renameSite, oldToken, newToken string) *WorkspaceEdit { +func (s *Server) buildTextEdits(sites []renameSite, oldToken, newToken string, report *RenameSummary) *WorkspaceEdit { // Group sites by file sitesByFile := make(map[string][]renameSite, len(sites)) for _, site := range sites { @@ -6728,8 +6816,14 @@ func (s *Server) buildTextEdits(sites []renameSite, oldToken, newToken string) * } wg.Wait() s.reportRenameFailures(&failures) + for fp := range sitesByFile { + if _, ok := fileCache[fp]; ok { + report.changed(fp) + } + } + report.recordFailures(&failures) - s.reindexAfterRename(nil, reindexPaths, openReindexes) + report.waitFor(s.reindexAfterRename(nil, reindexPaths, openReindexes)) return &WorkspaceEdit{Changes: openChanges} } @@ -6738,10 +6832,15 @@ func (s *Server) buildTextEdits(sites []renameSite, oldToken, newToken string) * // lock. Besides keeping each write out of a cold build, holding the lock across // removals and inserts prevents a build from observing an empty intermediate // state and starting its insert-only transaction in the middle of the rename. -func (s *Server) reindexAfterRename(removePaths, diskPaths []string, textPaths []textReindex) { +// +// The returned channel is closed when the index shows the rename. +func (s *Server) reindexAfterRename(removePaths, diskPaths []string, textPaths []textReindex) <-chan struct{} { + done := make(chan struct{}) + s.index.renameIndexed = done // the caller holds renameSerial s.index.backgroundWork.Add(1) go func() { defer s.index.backgroundWork.Done() + defer close(done) s.index.reindexing.Lock() defer s.index.reindexing.Unlock() @@ -6769,6 +6868,7 @@ func (s *Server) reindexAfterRename(removePaths, diskPaths []string, textPaths [ log.Printf("Rename: reindexed %d files", len(diskPaths)+len(textPaths)) // intentionally always logged — useful for user feedback } }() + return done } // isDepsFile returns true if filePath lives under the deps/ directory of some diff --git a/internal/lsp/server_test.go b/internal/lsp/server_test.go index c259be9..d70db75 100644 --- a/internal/lsp/server_test.go +++ b/internal/lsp/server_test.go @@ -6730,7 +6730,7 @@ end`) } }() - server.buildTextEdits([]renameSite{{filePath: path, line: 2}}, "old_name", "new_name") + server.buildTextEdits([]renameSite{{filePath: path, line: 2}}, "old_name", "new_name", nil) done := make(chan struct{}) go func() { server.index.backgroundWork.Wait() diff --git a/internal/mcp/call_hierarchy.go b/internal/mcp/call_hierarchy.go new file mode 100644 index 0000000..f15033c --- /dev/null +++ b/internal/mcp/call_hierarchy.go @@ -0,0 +1,88 @@ +package mcp + +import ( + "context" + "fmt" + "strings" + + "go.lsp.dev/protocol" +) + +type CallHierarchyParams struct { + Module string `json:"module" jsonschema:"fully-qualified module owning the function"` + Function string `json:"function" jsonschema:"function name without arity"` + Direction string `json:"direction,omitempty" jsonschema:"'incoming' (callers), 'outgoing' (callees), or 'both' (default)"` +} + +const maxCallsPerDirection = 50 + +func (h *Handler) callHierarchy(ctx context.Context, args CallHierarchyParams) (string, error) { + module := strings.TrimSpace(args.Module) + function := strings.TrimSpace(args.Function) + if module == "" || function == "" { + return "", fmt.Errorf("module and function must not be empty") + } + direction := strings.ToLower(strings.TrimSpace(args.Direction)) + switch direction { + case "": + direction = "both" + case "incoming", "outgoing", "both": + default: + return "", fmt.Errorf("direction must be 'incoming', 'outgoing', or 'both', got %q", args.Direction) + } + + // The LSP call-hierarchy handlers are name-based: they only read the + // module/function pair from Item.Data, so a synthetic item works. + item := protocol.CallHierarchyItem{ + Data: map[string]interface{}{"module": module, "function": function}, + } + + var b strings.Builder + fmt.Fprintf(&b, "Call hierarchy for %s.%s:\n", module, function) + found := false + + if direction == "incoming" || direction == "both" { + calls, err := h.lsp.IncomingCalls(ctx, &protocol.CallHierarchyIncomingCallsParams{Item: item}) + if err != nil { + return "", fmt.Errorf("incoming calls: %w", err) + } + fmt.Fprintf(&b, "\nIncoming (callers): %d\n", len(calls)) + for i, c := range calls { + if i == maxCallsPerDirection { + fmt.Fprintf(&b, " ... and %d more\n", len(calls)-maxCallsPerDirection) + break + } + lines := make([]string, 0, len(c.FromRanges)) + for _, r := range c.FromRanges { + lines = append(lines, fmt.Sprintf("%d", r.Start.Line+1)) + } + fmt.Fprintf(&b, " ← %s (%s:%d) calls at line %s\n", c.From.Name, h.relPath(uriToPath(c.From.URI)), c.From.Range.Start.Line+1, strings.Join(lines, ", ")) + } + found = found || len(calls) > 0 + } + + if direction == "outgoing" || direction == "both" { + calls, err := h.lsp.OutgoingCalls(ctx, &protocol.CallHierarchyOutgoingCallsParams{Item: item}) + if err != nil { + return "", fmt.Errorf("outgoing calls: %w", err) + } + fmt.Fprintf(&b, "\nOutgoing (callees): %d\n", len(calls)) + for i, c := range calls { + if i == maxCallsPerDirection { + fmt.Fprintf(&b, " ... and %d more\n", len(calls)-maxCallsPerDirection) + break + } + fmt.Fprintf(&b, " → %s (%s:%d)\n", c.To.Name, h.relPath(uriToPath(c.To.URI)), c.To.Range.Start.Line+1) + } + found = found || len(calls) > 0 + } + + if !found { + fmt.Fprintf(&b, "\nNo calls found. Check the module/function names (dexter_search can help), or call dexter_reindex if files changed recently.\n") + } + return b.String(), nil +} + +func uriToPath(u protocol.DocumentURI) string { + return u.Filename() +} diff --git a/internal/mcp/daemon.go b/internal/mcp/daemon.go new file mode 100644 index 0000000..9c3095a --- /dev/null +++ b/internal/mcp/daemon.go @@ -0,0 +1,239 @@ +package mcp + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "log" + "strings" + "sync" + "time" + + "github.com/remoteoss/dexter/internal/daemon" +) + +// MethodTool is the daemon control method that runs one MCP tool body. The +// frontend keeps the MCP protocol; the daemon, which owns the store, the +// watchers, and the language caches, answers the tool. +const MethodTool = "mcp/tool" + +// ToolParams is the payload of MethodTool. +type ToolParams struct { + Name string `json:"name"` + Arguments json.RawMessage `json:"arguments,omitempty"` + // WaitReadyMs is how long the call waits for the initial index before it + // answers from what is indexed, with a note that the index is still + // building. + WaitReadyMs int `json:"waitReadyMs,omitempty"` +} + +// ToolResult is the answer of MethodTool. +type ToolResult struct { + Text string `json:"text"` +} + +// indexWaitLimit caps how long a tool call waits for a workspace's initial +// index. The daemon caps its own index waits at the same value. A variable so +// tests can shrink it. +var indexWaitLimit = 30 * time.Second + +func init() { + daemon.RegisterMethod(MethodTool, serveTool) +} + +// serveTool runs in the workspace daemon. It answers from the headless +// language service, or from an editor session only when the connection +// explicitly named one. +func serveTool(mc daemon.MethodContext, raw json.RawMessage) (any, error) { + var params ToolParams + if err := json.Unmarshal(raw, ¶ms); err != nil { + return nil, err + } + h := NewHandler(mc.Runtime, mc.LSP()) + text, err := h.Call(mc.Context, params.Name, params.Arguments, time.Duration(params.WaitReadyMs)*time.Millisecond) + if err != nil { + return nil, err + } + return ToolResult{Text: text}, nil +} + +// Backend answers tool calls for one workspace root. +type Backend interface { + CallTool(ctx context.Context, name string, args json.RawMessage) (string, error) + Close() error +} + +// warmer is a Backend that can start its workspace before the first call, so +// that the index is already building when the agent asks. +type warmer interface { + warm(ctx context.Context) +} + +// DaemonBackend returns a Backend that forwards each tool call to the +// workspace daemon for root, starting the daemon when necessary. It holds one +// control connection, which keeps the daemon alive while the MCP session is +// open, and connects again when the daemon goes away. +func DaemonBackend(root string) Backend { + return newDaemonBackend(root, func(ctx context.Context, root string) (controlClient, error) { + client, err := daemon.Ensure(ctx, root) + if err != nil { + // Not a typed nil in an interface. + return nil, err + } + return client, nil + }) +} + +func newDaemonBackend(root string, ensure func(ctx context.Context, root string) (controlClient, error)) *daemonBackend { + return &daemonBackend{root: root, ensure: ensure, slots: make(chan struct{}, maxConcurrentToolCalls)} +} + +// controlClient is the part of *daemon.Client that a daemonBackend uses. +type controlClient interface { + Call(ctx context.Context, method string, params, result any) error + Done() <-chan struct{} + Close() error +} + +// maxConcurrentToolCalls bounds the tool calls that one workspace connection +// runs at the same time. It is below the daemon's limit of concurrent +// requests per connection, so the MCP frontend waits for a slot instead of +// getting refusals from the daemon. +const maxConcurrentToolCalls = 32 + +type daemonBackend struct { + mu sync.Mutex + root string + ensure func(ctx context.Context, root string) (controlClient, error) + client controlClient + closed bool + // closeReason is the error that calls get after Close. + closeReason error + + slots chan struct{} +} + +// retire closes the backend with the error that its calls in flight get. +func (b *daemonBackend) retire(reason error) error { + b.mu.Lock() + b.closeReason = reason + b.mu.Unlock() + return b.Close() +} + +func (b *daemonBackend) closedErr() error { + b.mu.Lock() + defer b.mu.Unlock() + if !b.closed { + return nil + } + if b.closeReason != nil { + return b.closeReason + } + return errClosed +} + +// connection returns the live control connection, connecting when there is +// none or the last one ended. +func (b *daemonBackend) connection(ctx context.Context) (controlClient, error) { + b.mu.Lock() + defer b.mu.Unlock() + if b.closed { + if b.closeReason != nil { + return nil, b.closeReason + } + return nil, errClosed + } + if b.client != nil { + select { + case <-b.client.Done(): + _ = b.client.Close() + b.client = nil + default: + return b.client, nil + } + } + client, err := b.ensure(ctx, b.root) + var mismatch *daemon.RootMismatchError + if errors.As(err, &mismatch) { + // A daemon already serves this workspace through another spelling of + // the root, for example the one an editor used. Share it: the agent + // gets paths in that spelling, which name the same files. + log.Printf("MCP: workspace %s is served as %s; using that root", b.root, mismatch.Daemon) + b.root = mismatch.Daemon + client, err = b.ensure(ctx, b.root) + } + if err != nil { + return nil, err + } + b.client = client + return client, nil +} + +func (b *daemonBackend) warm(ctx context.Context) { + if _, err := b.connection(ctx); err != nil { + log.Printf("MCP: cannot start the workspace daemon for %s: %v", b.root, err) + } +} + +func (b *daemonBackend) CallTool(ctx context.Context, name string, args json.RawMessage) (string, error) { + select { + case b.slots <- struct{}{}: + defer func() { <-b.slots }() + case <-ctx.Done(): + return "", ctx.Err() + } + params := ToolParams{Name: name, Arguments: args, WaitReadyMs: int(indexWaitLimit.Milliseconds())} + for attempt := 0; ; attempt++ { + client, err := b.connection(ctx) + if err != nil { + return "", err + } + var res ToolResult + err = client.Call(ctx, MethodTool, params, &res) + if err == nil { + return res.Text, nil + } + if strings.Contains(err.Error(), fmt.Sprintf("unknown daemon method %q", MethodTool)) { + return "", fmt.Errorf("the dexter daemon for %s was started by a build without MCP tools; run `dexter stop --force` in the project, then retry", b.root) + } + if name == renameToolName && ctx.Err() != nil { + // A client that canceled may not read the answer, so the log + // keeps it too. + log.Printf("MCP: a rename was canceled while it ran in %s; it may have been applied, so check git status", b.root) + return "", fmt.Errorf("the rename was canceled while it ran, so it may have been applied; check git status (%w)", ctx.Err()) + } + if reason := b.closedErr(); reason != nil { + // The frontend closed this connection during the call, for + // example because the client's roots changed. + if name == renameToolName { + return "", fmt.Errorf("%w; the rename may have been applied before the connection closed, so check git status", reason) + } + return "", reason + } + select { + case <-client.Done(): + // The daemon went away, for example because a newer build + // replaced it. Connect again once, but never repeat a rename: it + // may have been applied before the connection ended. + if attempt == 0 && name != renameToolName { + continue + } + default: + } + return "", err + } +} + +func (b *daemonBackend) Close() error { + b.mu.Lock() + defer b.mu.Unlock() + b.closed = true + if b.client == nil { + return nil + } + err := b.client.Close() + b.client = nil + return err +} diff --git a/internal/mcp/daemon_test.go b/internal/mcp/daemon_test.go new file mode 100644 index 0000000..efe02c2 --- /dev/null +++ b/internal/mcp/daemon_test.go @@ -0,0 +1,215 @@ +package mcp + +import ( + "context" + "errors" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/modelcontextprotocol/go-sdk/mcp" +) + +// blockingClient is a daemon connection whose calls run until the caller +// cancels them or the connection closes, like a tool call that waits for a +// cold index or a rename that is writing files. +type blockingClient struct { + done chan struct{} + closeOnce sync.Once + inflight atomic.Int32 + peak atomic.Int32 +} + +func newBlockingClient() *blockingClient { return &blockingClient{done: make(chan struct{})} } + +func (c *blockingClient) Call(ctx context.Context, method string, params, result any) error { + n := c.inflight.Add(1) + defer c.inflight.Add(-1) + for { + p := c.peak.Load() + if n <= p || c.peak.CompareAndSwap(p, n) { + break + } + } + select { + case <-ctx.Done(): + return ctx.Err() + case <-c.done: + return errors.New("daemon connection closed") + } +} + +func (c *blockingClient) Done() <-chan struct{} { return c.done } + +func (c *blockingClient) Close() error { + c.closeOnce.Do(func() { close(c.done) }) + return nil +} + +func backendOver(client *blockingClient) *daemonBackend { + return newDaemonBackend("/project", func(context.Context, string) (controlClient, error) { return client, nil }) +} + +// Regression: a rename that the client canceled reported a bare context +// error, although the daemon can have written some files already. +func TestDaemonBackend_CanceledRenameSaysItMayBeApplied(t *testing.T) { + b := backendOver(newBlockingClient()) + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Millisecond) + defer cancel() + _, err := b.CallTool(ctx, renameToolName, nil) + if err == nil || !strings.Contains(err.Error(), "may have been applied; check git status") { + t.Fatalf("error = %v, want a note that the rename may have been applied", err) + } + + // Other tools keep the plain context error. + ctx2, cancel2 := context.WithTimeout(context.Background(), 10*time.Millisecond) + defer cancel2() + if _, err := b.CallTool(ctx2, "dexter_search", nil); !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("error = %v, want the deadline", err) + } +} + +// Regression: when the client's roots changed during a call, the frontend +// closed the old workspace connection, and the call said "the MCP server is +// shutting down". +func TestDaemonBackend_RootsChangeDuringCall(t *testing.T) { + for _, tool := range []string{"dexter_search", renameToolName} { + b := backendOver(newBlockingClient()) + errc := make(chan error, 1) + go func() { + _, err := b.CallTool(context.Background(), tool, nil) + errc <- err + }() + time.Sleep(10 * time.Millisecond) + if err := b.retire(errRootsChanged); err != nil { + t.Fatal(err) + } + var err error + select { + case err = <-errc: + case <-time.After(5 * time.Second): + t.Fatal("call did not end") + } + if err == nil || !strings.Contains(err.Error(), "roots changed") || strings.Contains(err.Error(), "shutting down") { + t.Fatalf("%s: error = %v, want a roots-changed error", tool, err) + } + if tool == renameToolName && !strings.Contains(err.Error(), "may have been applied") { + t.Fatalf("rename error = %v, want a note that it may have been applied", err) + } + } +} + +// One MCP frontend must not take every request slot of the daemon +// connection: calls past the frontend's limit wait for a slot. +func TestDaemonBackend_LimitsConcurrentCalls(t *testing.T) { + client := newBlockingClient() + b := backendOver(client) + ctx, cancel := context.WithCancel(context.Background()) + var wg sync.WaitGroup + for i := 0; i < 100; i++ { + wg.Add(1) + go func() { + defer wg.Done() + _, _ = b.CallTool(ctx, "dexter_search", nil) + }() + } + deadline := time.Now().Add(5 * time.Second) + for client.inflight.Load() < maxConcurrentToolCalls && time.Now().Before(deadline) { + time.Sleep(5 * time.Millisecond) + } + time.Sleep(50 * time.Millisecond) + cancel() + wg.Wait() + if peak := client.peak.Load(); peak != maxConcurrentToolCalls { + t.Fatalf("peak concurrent daemon calls = %d, want %d", peak, maxConcurrentToolCalls) + } + if maxConcurrentToolCalls >= 64 { + t.Fatalf("frontend limit %d is not below the daemon's 64", maxConcurrentToolCalls) + } +} + +// The frontend retires the old workspace connection with errRootsChanged, so +// a call still running on it says why it ended. +func TestFrontend_RootsChangeEndsCallWithRootsChangedError(t *testing.T) { + var mu sync.Mutex + clients := map[string]*blockingClient{} + cfg := Config{Root: t.TempDir(), Connect: func(root string) Backend { + mu.Lock() + defer mu.Unlock() + c := newBlockingClient() + clients[root] = c + return backendOver(c) + }} + e := setupNegotiationWith(t, cfg) + e.f.cfg.Connect = cfg.Connect // setupNegotiationWith installs fake backends + rootA, uriA := projectDir(t) + _, uriB := projectDir(t) + cs, client := e.connect(nil, uriA) + t.Cleanup(func() { + // Calls still waiting on a fake daemon end before the session closes. + mu.Lock() + defer mu.Unlock() + for _, c := range clients { + _ = c.Close() + } + }) + + first := make(chan string, 1) + go func() { + out, _ := toolText(t, cs, "dexter_search", map[string]any{"query": "x"}) + first <- out + }() + eventually(t, "the first call to reach the daemon", func() bool { + mu.Lock() + defer mu.Unlock() + c := clients[rootA] + return c != nil && c.inflight.Load() == 1 + }) + + client.RemoveRoots(uriA) + client.AddRoots(&mcp.Root{URI: uriB}) + go func() { + // The next call negotiates again and moves the session to rootB. + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + _, _ = cs.CallTool(ctx, &mcp.CallToolParams{Name: "dexter_search", Arguments: map[string]any{"query": "x"}}) + }() + select { + case out := <-first: + wantContains(t, out, "roots changed") + wantNotContains(t, out, "shutting down") + case <-time.After(10 * time.Second): + t.Fatal("the first call did not end") + } +} + +// errorClient is a daemon connection whose calls fail with one error. +type errorClient struct { + err error + done chan struct{} +} + +func (c *errorClient) Call(context.Context, string, any, any) error { return c.err } +func (c *errorClient) Done() <-chan struct{} { return c.done } +func (c *errorClient) Close() error { return nil } + +// Regression: recovery hints said to run a plain dexter stop, which a daemon +// refuses while this MCP session is attached. +func TestRecoveryHintsSayForce(t *testing.T) { + b := newDaemonBackend("/project", func(context.Context, string) (controlClient, error) { + return &errorClient{err: errors.New(`unknown daemon method "mcp/tool"`), done: make(chan struct{})}, nil + }) + _, err := b.CallTool(context.Background(), "dexter_search", nil) + if err == nil || !strings.Contains(err.Error(), "dexter stop --force") { + t.Errorf("old-daemon error = %v, want the --force hint", err) + } + + e := setupProject(t) + if err := e.store.SetIndexVersion(1); err != nil { + t.Fatal(err) + } + out := e.callTool("dexter_workspace", nil) + wantContains(t, out, "dexter stop --force") +} diff --git a/internal/mcp/definition.go b/internal/mcp/definition.go new file mode 100644 index 0000000..3e34774 --- /dev/null +++ b/internal/mcp/definition.go @@ -0,0 +1,168 @@ +package mcp + +import ( + "context" + "fmt" + "strings" + + "github.com/remoteoss/dexter/internal/lsp" +) + +type DefinitionParams struct { + Module string `json:"module" jsonschema:"fully-qualified module name, e.g. MyApp.Accounts (aliases are not resolved)"` + Function string `json:"function,omitempty" jsonschema:"function/macro/type name without arity; omit to look up the module itself"` +} + +func (h *Handler) definition(ctx context.Context, args DefinitionParams) (string, error) { + module := strings.TrimSpace(args.Module) + if module == "" { + return "", fmt.Errorf("module must not be empty") + } + function := strings.TrimSpace(args.Function) + + if function == "" { + return h.moduleDefinition(module) + } + + // The same name navigation as go-to-definition and `dexter lookup`: use + // chains, generated functions, and the lines that declared them. Direct + // definitions come first because they show whether this is a defdelegate + // facade. + direct, err := h.lsp.LookupName(module, function, lsp.NameLookupOptions{}) + if err != nil { + return "", fmt.Errorf("looking up function: %w", err) + } + + var b strings.Builder + if len(direct) == 0 { + // No direct definition. The function can still resolve through a + // defdelegate chain. + resolved, err := h.lsp.LookupName(module, function, lsp.NameLookupOptions{FollowDelegates: true}) + if err != nil { + return "", fmt.Errorf("looking up function: %w", err) + } + if len(resolved) == 0 { + return fmt.Sprintf("%s.%s is not in the index. It can be generated by a macro that the index cannot see, or misspelled. Try dexter_search or dexter_module_api %s.", module, function, module), nil + } + for i, r := range resolved { + if i == maxDefinitionClauses { + writeMoreClauses(&b, len(resolved)-i) + break + } + h.writeDefinition(&b, module, function, r) + } + return b.String(), nil + } + + for i, r := range direct { + if i == maxDefinitionClauses { + writeMoreClauses(&b, len(direct)-i) + break + } + h.writeDefinition(&b, module, function, r) + if r.Kind != "defdelegate" { + continue + } + targetModule, targetFunction, ok := h.delegateTargetAt(module, function, r) + if !ok { + continue + } + targets, err := h.lsp.LookupName(module, function, lsp.NameLookupOptions{FollowDelegates: true}) + if err != nil || len(targets) == 0 { + continue + } + fmt.Fprintf(&b, "\nDelegates to %s.%s:\n", targetModule, targetFunction) + written := 0 + for j, t := range targets { + if t.FilePath == r.FilePath && t.Line == r.Line { + continue + } + if written == maxDefinitionClauses { + writeMoreClauses(&b, len(targets)-j) + break + } + h.writeDefinition(&b, targetModule, targetFunction, t) + written++ + } + } + return b.String(), nil +} + +// maxDefinitionClauses caps the clauses that one definition answer shows. A +// generated function can have thousands of clauses, and each one shown costs +// its docs and its head line. +const maxDefinitionClauses = 20 + +func writeMoreClauses(b *strings.Builder, n int) { + fmt.Fprintf(b, "\n… and %d more clause(s) not shown.\n", n) +} + +// delegateTargetAt names the function that the defdelegate at r forwards to. +func (h *Handler) delegateTargetAt(module, function string, r lsp.NameLocation) (targetModule, targetFunction string, ok bool) { + defs, err := h.store.LookupFunction(module, function) + if err != nil { + return "", "", false + } + for _, d := range defs { + if d.FilePath != r.FilePath || d.Line != r.Line || d.DelegateTo == "" { + continue + } + targetFunction = function + if d.DelegateAs != "" { + targetFunction = d.DelegateAs + } + return d.DelegateTo, targetFunction, true + } + return "", "", false +} + +func (h *Handler) moduleDefinition(module string) (string, error) { + results, err := h.lsp.LookupName(module, "", lsp.NameLookupOptions{}) + if err != nil { + return "", fmt.Errorf("looking up module: %w", err) + } + if len(results) == 0 { + return fmt.Sprintf("Module %s is not in the index. Use dexter_search to find the right name, or dexter_reindex if it was just created.", module), nil + } + + var b strings.Builder + for _, r := range results { + a, ok := h.at(r.FilePath, r.Line) + if !ok { + fmt.Fprintf(&b, "%s %s - %s:%d\n", moduleKindLabel(r.Kind), module, h.relPath(r.FilePath), r.Line) + continue + } + fmt.Fprintf(&b, "%s %s - %s:%d%s\n", moduleKindLabel(r.Kind), module, h.relPath(r.FilePath), a.line, a.label()) + if r.Kind != "defimpl" { + if doc := a.tokenized().ExtractModuledoc(a.line - 1); doc != "" { + fmt.Fprintf(&b, "\n%s\n", strings.TrimRight(doc, "\n")) + } + } + } + return b.String(), nil +} + +// writeDefinition renders one definition with location, @spec/@doc, and the +// definition head line. +func (h *Handler) writeDefinition(b *strings.Builder, module, function string, r lsp.NameLocation) { + kind := r.Kind + if kind == "" { + kind = "def" + } + a, ok := h.at(r.FilePath, r.Line) + if !ok { + fmt.Fprintf(b, "%s (%s) - %s:%d\n", symbolName(module, function, r.Arity), kind, h.relPath(r.FilePath), r.Line) + return + } + fmt.Fprintf(b, "%s (%s) - %s:%d%s\n", symbolName(module, function, r.Arity), kind, h.relPath(r.FilePath), a.line, a.label()) + doc, spec := a.tokenized().ExtractDocAbove(a.line - 1) + if spec != "" { + fmt.Fprintf(b, "%s\n", spec) + } + if head, ok := a.lineText(); ok { + fmt.Fprintf(b, "%s\n", strings.TrimRight(head, " \t")) + } + if doc != "" { + fmt.Fprintf(b, "\n%s\n", strings.TrimRight(doc, "\n")) + } +} diff --git a/internal/mcp/file_outline.go b/internal/mcp/file_outline.go new file mode 100644 index 0000000..5f5eae7 --- /dev/null +++ b/internal/mcp/file_outline.go @@ -0,0 +1,118 @@ +package mcp + +import ( + "context" + "errors" + "fmt" + "sort" + "strings" + + "github.com/remoteoss/dexter/internal/parser" +) + +type FileOutlineParams struct { + File string `json:"file" jsonschema:"path to a .ex/.exs file, absolute or relative to the project root"` +} + +func (h *Handler) fileOutline(ctx context.Context, args FileOutlineParams) (string, error) { + if strings.TrimSpace(args.File) == "" { + return "", fmt.Errorf("file must not be empty") + } + path, err := h.userPath(args.File) + if errors.Is(err, errNotFound) { + return fmt.Sprintf("File not found: %s", h.relPath(path)), nil + } + if err != nil { + return "", err + } + text, err := h.readSource(path) + if errors.Is(err, errNotFound) { + return fmt.Sprintf("File not found: %s", h.relPath(path)), nil + } + if err != nil { + return "", err + } + + // Parse fresh source so the outline is correct when either the index is + // stale or an attached editor has unsaved changes. + defs, _, err := parser.ParseText(path, text) + if err != nil { + return "", fmt.Errorf("parsing %s: %w", h.relPath(path), err) + } + if len(defs) == 0 { + return fmt.Sprintf("%s defines no modules or functions.", h.relPath(path)), nil + } + + // Split into module declarations (in line order) and their members. + type moduleEntry struct { + def parser.Definition + members []parser.Definition + } + var modules []*moduleEntry + byName := make(map[string]*moduleEntry) + var orphans []parser.Definition + + sorted := make([]parser.Definition, len(defs)) + copy(sorted, defs) + sort.SliceStable(sorted, func(i, j int) bool { return sorted[i].Line < sorted[j].Line }) + + for _, d := range sorted { + if d.Function == "" { + e := &moduleEntry{def: d} + modules = append(modules, e) + byName[d.Module] = e + } + } + for _, d := range sorted { + if d.Function == "" { + continue + } + if e, ok := byName[d.Module]; ok { + e.members = append(e.members, d) + } else { + orphans = append(orphans, d) + } + } + + var b strings.Builder + fmt.Fprintf(&b, "%s\n", h.relPath(path)) + for _, e := range modules { + fmt.Fprintf(&b, "\n%s %s (line %d)\n", moduleKindLabel(e.def.Kind), e.def.Module, e.def.Line) + for _, m := range e.members { + b.WriteString(" " + memberLine(m) + "\n") + } + } + for _, m := range orphans { + b.WriteString(memberLine(m) + "\n") + } + return b.String(), nil +} + +func moduleKindLabel(kind string) string { + switch kind { + case "module": + return "defmodule" + default: // defprotocol, defimpl + return kind + } +} + +func memberLine(d parser.Definition) string { + label := d.Kind + switch d.Kind { + case "type", "opaque", "callback", "macrocallback": + label = "@" + d.Kind + } + line := fmt.Sprintf("%4d: %s %s/%d", d.Line, label, d.Function, d.Arity) + if d.Params != "" { + line += fmt.Sprintf(" (%s)", d.Params) + } + if d.DelegateTo != "" { + target := d.DelegateTo + if d.DelegateAs != "" { + target += "." + d.DelegateAs + } + line += " → " + target + } + return line +} diff --git a/internal/mcp/frontend.go b/internal/mcp/frontend.go new file mode 100644 index 0000000..0fa5df9 --- /dev/null +++ b/internal/mcp/frontend.go @@ -0,0 +1,284 @@ +package mcp + +import ( + "context" + "encoding/json" + "errors" + "log" + "sync" + + "github.com/modelcontextprotocol/go-sdk/mcp" + + "github.com/remoteoss/dexter/internal/store" + "github.com/remoteoss/dexter/internal/version" +) + +var errClosed = errors.New("the MCP server is shutting down") + +// errRootsChanged is the error of a call whose workspace connection closed +// because the client's roots changed to another project. +var errRootsChanged = errors.New("the MCP client's workspace roots changed during the call, so the call ended; retry it") + +// retirer is a Backend that can close with the error its calls in flight get. +type retirer interface { + retire(reason error) error +} + +// Config configures a Frontend. +type Config struct { + // Root is the workspace root for sessions that give no usable MCP root. + // With Fixed it is the root of every session. + Root string + Fixed bool + // FallbackErr, when set, refuses sessions that give no usable MCP root + // instead of using Root. + FallbackErr error + + // ResolveRoot finds the project root for a directory that a client gives + // as an MCP root. It must resolve the way the CLI and the editor do, so + // that every frontend attaches to the same daemon. Nil uses the store's + // marker search. + ResolveRoot func(dir string) (string, error) + + // Connect returns the Backend for a resolved root. Nil connects to the + // shared workspace daemon. Tests replace it. + Connect func(root string) Backend +} + +// Frontend is the MCP side of `dexter mcp`. It owns no workspace state: each +// session's root is negotiated through MCP roots (or fixed), and every tool +// call goes to the Backend of that root, normally the shared workspace daemon. +// Sessions whose roots resolve to the same project share one Backend. +type Frontend struct { + cfg Config + + mu sync.Mutex + closed bool + conns map[string]*workspaceConn + sessions map[*mcp.ServerSession]*sessionState +} + +type workspaceConn struct { + backend Backend + users int +} + +type sessionState struct { + root string + dirty bool // the client's roots changed; resolve again on the next call +} + +// NewFrontend returns a Frontend for cfg. +func NewFrontend(cfg Config) *Frontend { + if cfg.Connect == nil { + cfg.Connect = DaemonBackend + } + return &Frontend{ + cfg: cfg, + conns: make(map[string]*workspaceConn), + sessions: make(map[*mcp.ServerSession]*sessionState), + } +} + +// backendFor returns the Backend of the session's workspace, negotiating the +// root first when the session is new or its roots changed. +func (f *Frontend) backendFor(ctx context.Context, ss *mcp.ServerSession) (Backend, error) { + f.mu.Lock() + if f.closed { + f.mu.Unlock() + return nil, errClosed + } + if st, ok := f.sessions[ss]; ok && !st.dirty { + b := f.conns[st.root].backend + f.mu.Unlock() + return b, nil + } + f.mu.Unlock() + + // Outside the lock: ListRoots waits on the client. + root, source, err := f.rootFor(ctx, ss) + if err != nil { + return nil, err + } + + f.mu.Lock() + if f.closed { + f.mu.Unlock() + return nil, errClosed + } + st, bound := f.sessions[ss] + if bound && st.root == root { + st.dirty = false + b := f.conns[root].backend + f.mu.Unlock() + return b, nil + } + var orphan Backend + if bound { + orphan = f.releaseLocked(st.root) + } + c, exists := f.conns[root] + if !exists { + c = &workspaceConn{backend: f.cfg.Connect(root)} + f.conns[root] = c + } + c.users++ + f.sessions[ss] = &sessionState{root: root} + f.mu.Unlock() + + log.Printf("MCP session workspace: %s (%s)", root, source) + if !bound { + go func() { + _ = ss.Wait() + f.detach(ss) + }() + } + if orphan != nil { + if r, ok := orphan.(retirer); ok { + if err := r.retire(errRootsChanged); err != nil { + log.Printf("MCP: closing workspace connection: %v", err) + } + } else { + closeBackend(orphan) + } + } + // Warm only a project: a root that the user gave explicitly but that is + // not one starts its daemon at the first tool call, not before. + if w, ok := c.backend.(warmer); ok && !exists && store.NonProjectRootError(root) == nil { + go w.warm(context.WithoutCancel(ctx)) + } + return c.backend, nil +} + +// rootFor resolves the workspace root of a session. +func (f *Frontend) rootFor(ctx context.Context, ss *mcp.ServerSession) (root, source string, err error) { + if f.cfg.Fixed { + return f.cfg.Root, "command line", nil + } + root, ok, err := negotiatedRoot(ctx, ss, f.resolve) + if err != nil { + return "", "", err + } + if !ok { + if f.cfg.FallbackErr != nil { + return "", "", f.cfg.FallbackErr + } + return f.cfg.Root, "fallback", nil + } + return root, "client roots", nil +} + +func (f *Frontend) resolve(dir string) (string, error) { + if f.cfg.ResolveRoot != nil { + return f.cfg.ResolveRoot(dir) + } + return defaultResolveRoot(dir) +} + +// releaseLocked drops one user of root and returns its Backend when no session +// uses it any more. The caller closes it outside the lock. +func (f *Frontend) releaseLocked(root string) Backend { + c, ok := f.conns[root] + if !ok { + return nil + } + c.users-- + if c.users > 0 { + return nil + } + delete(f.conns, root) + return c.backend +} + +// detach drops a closed session, and closes its workspace connection when no +// other session uses it. +func (f *Frontend) detach(ss *mcp.ServerSession) { + f.mu.Lock() + var orphan Backend + if st, ok := f.sessions[ss]; ok { + delete(f.sessions, ss) + orphan = f.releaseLocked(st.root) + } + f.mu.Unlock() + if orphan != nil { + closeBackend(orphan) + } +} + +func closeBackend(b Backend) { + if err := b.Close(); err != nil { + log.Printf("MCP: closing workspace connection: %v", err) + } +} + +// onInitialized starts the new session's workspace, so the index is already +// building at the first tool call. Failures show at that first call, which +// negotiates again with its own context. +func (f *Frontend) onInitialized(ctx context.Context, req *mcp.InitializedRequest) { + _, _ = f.backendFor(ctx, req.Session) +} + +// onRootsChanged marks the session for a new negotiation. It happens at the +// session's next tool call, whose request context reaches the client on every +// transport. Roots that still resolve to the same project keep the workspace. +func (f *Frontend) onRootsChanged(_ context.Context, req *mcp.RootsListChangedRequest) { + f.mu.Lock() + if st, ok := f.sessions[req.Session]; ok { + st.dirty = true + } + f.mu.Unlock() +} + +// Close closes every workspace connection. The daemons stay up for their +// other clients, and stop on their idle timeout. +func (f *Frontend) Close() { + f.mu.Lock() + if f.closed { + f.mu.Unlock() + return + } + f.closed = true + conns := f.conns + f.conns = nil + f.sessions = nil + f.mu.Unlock() + for _, c := range conns { + closeBackend(c.backend) + } +} + +// addTool declares one tool on srv. Its handler sends the typed arguments to +// the Backend of the session's workspace. +func addTool[In any](srv *mcp.Server, f *Frontend, t mcp.Tool) { + mcp.AddTool(srv, &t, func(ctx context.Context, req *mcp.CallToolRequest, args In) (*mcp.CallToolResult, any, error) { + b, err := f.backendFor(ctx, req.Session) + if err != nil { + return nil, nil, err + } + raw, err := json.Marshal(args) + if err != nil { + return nil, nil, err + } + text, err := b.CallTool(ctx, t.Name, raw) + if err != nil { + return nil, nil, err + } + return &mcp.CallToolResult{Content: []mcp.Content{&mcp.TextContent{Text: text}}}, nil, nil + }) +} + +// NewServer returns an MCP protocol server with all dexter tools declared. +func NewServer(f *Frontend) *mcp.Server { + opts := &mcp.ServerOptions{Instructions: Instructions, InitializedHandler: f.onInitialized} + if !f.cfg.Fixed { + opts.RootsListChangedHandler = f.onRootsChanged + } + srv := mcp.NewServer( + &mcp.Implementation{Name: "dexter", Title: "Dexter Elixir language tools", Version: version.Version}, + opts, + ) + for _, spec := range toolSpecs() { + spec.register(srv, f, spec.tool) + } + return srv +} diff --git a/internal/mcp/http_test.go b/internal/mcp/http_test.go new file mode 100644 index 0000000..31158ea --- /dev/null +++ b/internal/mcp/http_test.go @@ -0,0 +1,124 @@ +package mcp + +import ( + "bytes" + "context" + "io" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/modelcontextprotocol/go-sdk/mcp" +) + +func TestCheckListenAddr(t *testing.T) { + for addr, ok := range map[string]bool{ + "localhost:0": true, + "127.0.0.1:8080": true, + "127.1.2.3:8080": true, + "[::1]:8080": true, + ":8080": false, + "0.0.0.0:8080": false, + "[::]:8080": false, + "192.168.1.20:8080": false, + "example.com:8080": false, + "localhost": false, // no port + } { + err := CheckListenAddr(addr, false) + if (err == nil) != ok { + t.Errorf("CheckListenAddr(%q) = %v, want ok=%v", addr, err, ok) + } + } + if err := CheckListenAddr("0.0.0.0:8080", true); err != nil { + t.Errorf("an explicit unsafe listen was refused: %v", err) + } +} + +const initializeBody = `{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"t","version":"1"}}}` + +func postMCP(t *testing.T, url string, body io.Reader, header map[string]string) *http.Response { + t.Helper() + req, err := http.NewRequest(http.MethodPost, url, body) + if err != nil { + t.Fatal(err) + } + req.Header.Set("Content-Type", "application/json") + req.Header.Set("Accept", "application/json, text/event-stream") + for k, v := range header { + req.Header.Set(k, v) + } + res, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = res.Body.Close() }) + return res +} + +func httpTestServer(t *testing.T) (*httptest.Server, *negotiationEnv) { + t.Helper() + e := setupNegotiationWith(t, Config{Root: t.TempDir(), Fixed: true}) + srv := httptest.NewServer(HTTPHandler(e.f)) + t.Cleanup(srv.Close) + return srv, e +} + +// Regression: a request body had no size limit; a 300 MB POST made the server +// hold gigabytes. +func TestHTTPHandler_CapsBodySize(t *testing.T) { + srv, _ := httpTestServer(t) + if res := postMCP(t, srv.URL, strings.NewReader(initializeBody), nil); res.StatusCode != http.StatusOK { + t.Fatalf("a small initialize got status %d", res.StatusCode) + } + // Valid JSON, padded past the limit. + padded := initializeBody + strings.Repeat(" ", maxHTTPBodyBytes) + if res := postMCP(t, srv.URL, bytes.NewReader([]byte(padded)), nil); res.StatusCode != http.StatusRequestEntityTooLarge { + t.Fatalf("a %d-byte body got status %d, want 413", len(padded), res.StatusCode) + } + // A chunked body has no Content-Length; its limit shows while it is read. + chunked := io.MultiReader(strings.NewReader(padded)) + if res := postMCP(t, srv.URL, chunked, nil); res.StatusCode != http.StatusRequestEntityTooLarge { + t.Fatalf("a chunked %d-byte body got status %d, want 413", len(padded), res.StatusCode) + } + if res := postMCP(t, srv.URL, io.MultiReader(strings.NewReader(initializeBody)), nil); res.StatusCode != http.StatusOK { + t.Fatalf("a small chunked initialize got status %d", res.StatusCode) + } +} + +// A web page in the user's browser must not drive the tools. +func TestHTTPHandler_RefusesCrossOriginRequests(t *testing.T) { + srv, _ := httpTestServer(t) + for _, header := range []map[string]string{ + {"Sec-Fetch-Site": "cross-site"}, + {"Origin": "http://attacker.example"}, + } { + if res := postMCP(t, srv.URL, strings.NewReader(initializeBody), header); res.StatusCode != http.StatusForbidden { + t.Errorf("cross-origin request %v got status %d, want 403", header, res.StatusCode) + } + } +} + +// Regression: HTTP sessions never timed out, so a client that went away +// without closing its session kept its workspace connection, and the daemon, +// alive forever. +func TestHTTPHandler_IdleSessionReleasesWorkspace(t *testing.T) { + old := httpSessionTimeout + httpSessionTimeout = 200 * time.Millisecond + t.Cleanup(func() { httpSessionTimeout = old }) + srv, e := httpTestServer(t) + + client := mcp.NewClient(&mcp.Implementation{Name: "t", Version: "1"}, nil) + cs, err := client.Connect(context.Background(), &mcp.StreamableClientTransport{Endpoint: srv.URL, DisableStandaloneSSE: true}, nil) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = cs.Close() }) + mustTool(t, cs, "dexter_search", map[string]any{"query": "x"}) + if n := e.connCount(); n != 1 { + t.Fatalf("%d workspace connections, want 1", n) + } + // The client stays silent; the session times out and lets go. + eventually(t, "the idle session to release its workspace connection", func() bool { return e.connCount() == 0 }) +} diff --git a/internal/mcp/implementations.go b/internal/mcp/implementations.go new file mode 100644 index 0000000..33fce0b --- /dev/null +++ b/internal/mcp/implementations.go @@ -0,0 +1,161 @@ +package mcp + +import ( + "context" + "fmt" + "strings" +) + +type ImplementationsParams struct { + Module string `json:"module" jsonschema:"behaviour or protocol module, fully qualified"` + Function string `json:"function,omitempty" jsonschema:"callback or protocol function name; when set, locate its definition in each implementor"` +} + +func (h *Handler) implementations(ctx context.Context, args ImplementationsParams) (string, error) { + module := strings.TrimSpace(args.Module) + if module == "" { + return "", fmt.Errorf("module must not be empty") + } + + modResults, err := h.store.LookupModule(module) + if err != nil { + return "", fmt.Errorf("looking up module: %w", err) + } + + // Protocol: implementations are the defimpl rows indexed under the protocol name. + isProtocol := false + var impls, decls []int + for i, r := range modResults { + switch r.Kind { + case "defprotocol": + isProtocol = true + decls = append(decls, i) + case "defimpl": + impls = append(impls, i) + } + } + if isProtocol { + var b strings.Builder + fmt.Fprintf(&b, "%s is a protocol (defprotocol at %s:%d).\n", module, h.relPath(modResults[decls[0]].FilePath), modResults[decls[0]].Line) + if function := strings.TrimSpace(args.Function); function != "" { + defs, err := h.store.LookupFunction(module, function) + if err != nil { + return "", fmt.Errorf("looking up protocol function: %w", err) + } + + // Functions in defprotocol and defimpl blocks share the protocol's + // module name in the index. Attribute each definition to the nearest + // preceding declaration in its file so a declaration and one or more + // implementations can safely coexist in the same file. + scopeKind := func(filePath string, line int) string { + kind, scopeLine := "", -1 + for _, r := range modResults { + if r.FilePath == filePath && r.Line <= line && r.Line > scopeLine { + kind, scopeLine = r.Kind, r.Line + } + } + return kind + } + arities := make(map[int]bool) + for _, d := range defs { + if scopeKind(d.FilePath, d.Line) == "defprotocol" { + arities[d.Arity] = true + } + } + if len(arities) == 0 { + return fmt.Sprintf("%s does not define a protocol function named %s. List its functions with dexter_module_api.", module, function), nil + } + + fmt.Fprintf(&b, "\nImplementations of protocol function %s.%s:\n", module, function) + found := 0 + for _, d := range defs { + if !arities[d.Arity] || scopeKind(d.FilePath, d.Line) != "defimpl" { + continue + } + fmt.Fprintf(&b, " %s - %s:%d\n", symbolName(module, function, d.Arity), h.relPath(d.FilePath), d.Line) + found++ + } + if found == 0 { + fmt.Fprintf(&b, " (no indexed defimpl defines %s)\n", function) + } + return b.String(), nil + } + if len(impls) == 0 { + fmt.Fprintf(&b, "No defimpl implementations found in the index.\n") + return b.String(), nil + } + fmt.Fprintf(&b, "\nImplementations (%d):\n", len(impls)) + for _, i := range impls { + r := modResults[i] + fmt.Fprintf(&b, " %s:%d\n", h.relPath(r.FilePath), r.Line) + } + fmt.Fprintf(&b, "\nNote: the defimpl target type is on the cited line (defimpl %s, for: Type).\n", module) + return b.String(), nil + } + + // Behaviour: modules that declare @behaviour or `use` this module. + implementors, err := h.store.LookupBehaviourImplementors(module) + if err != nil { + return "", fmt.Errorf("looking up implementors: %w", err) + } + if len(implementors) == 0 { + if len(modResults) == 0 { + return fmt.Sprintf("Module %s is not in the index. Use dexter_search to find the right name.", module), nil + } + return fmt.Sprintf("No modules declare @behaviour %s (or use it) in the index.", module), nil + } + + var b strings.Builder + + if args.Function != "" { + // Locate the callback's implementation in each implementor. + function := strings.TrimSpace(args.Function) + cbs, err := h.store.LookupCallbackDef(module, function) + if err != nil { + return "", fmt.Errorf("looking up callback: %w", err) + } + if len(cbs) == 0 { + return fmt.Sprintf("%s does not define a @callback named %s. List its callbacks with dexter_module_api.", module, function), nil + } + fmt.Fprintf(&b, "Implementations of callback %s.%s:\n", module, function) + arities := make(map[int]bool, len(cbs)) + for _, cb := range cbs { + arities[cb.Arity] = true + } + found := 0 + for _, impl := range implementors { + defs, err := h.store.LookupFunction(impl.Module, function) + if err != nil { + continue + } + for _, d := range defs { + if !arities[d.Arity] { + continue + } + fmt.Fprintf(&b, " %s - %s:%d\n", symbolName(impl.Module, function, d.Arity), h.relPath(d.FilePath), d.Line) + found++ + } + } + if found == 0 { + fmt.Fprintf(&b, " (none of the %d implementor(s) define %s; they may rely on a default implementation injected via use)\n", len(implementors), function) + } + return b.String(), nil + } + + fmt.Fprintf(&b, "Modules implementing behaviour %s (%d):\n", module, len(implementors)) + const maxImpls = 50 + for i, impl := range implementors { + if i == maxImpls { + fmt.Fprintf(&b, " ... and %d more\n", len(implementors)-maxImpls) + break + } + fmt.Fprintf(&b, " %s - %s\n", impl.Module, h.relPath(impl.FilePath)) + } + if cbs, err := h.store.ListModuleCallbacks(module); err == nil && len(cbs) > 0 { + fmt.Fprintf(&b, "\nCallbacks defined by %s:\n", module) + for _, cb := range cbs { + fmt.Fprintf(&b, " @%s %s/%d\n", cb.Kind, cb.Function, cb.Arity) + } + } + return b.String(), nil +} diff --git a/internal/mcp/instructions.md b/internal/mcp/instructions.md new file mode 100644 index 0000000..027c520 --- /dev/null +++ b/internal/mcp/instructions.md @@ -0,0 +1,36 @@ +# Dexter: Elixir code intelligence + +Dexter indexes every module, function, and call site in this Elixir workspace +by parsing source directly (no compilation needed). Use these tools instead of +grep or reading whole files whenever you navigate or ask questions about +Elixir code: they resolve aliases, imports, defdelegate chains, use-chain +injection, and the Elixir stdlib, which text search cannot. + +Which tool for which question: + +- Locate a symbol by name fragment: `dexter_search` +- Where or what is Module.function: `dexter_definition` +- Understand a module before reading its source: `dexter_module_api` +- Who calls or uses something: `dexter_references` or `dexter_call_hierarchy` +- Implementations of a behaviour or protocol: `dexter_implementations` +- What a specific file defines: `dexter_file_outline` +- Project layout and index freshness: `dexter_workspace` +- Rename a module or function everywhere: `dexter_rename_symbol` (writes the + changes; review with `git diff`) + +The index updates automatically: the dexter daemon, which the editor shares, +watches file changes and git branch switches. If a lookup ever seems stale, +`dexter_reindex` forces an immediate incremental update. An answer that ends +with a note about the index (still building, being rebuilt, files that could +not be indexed) can be incomplete; retry after the note goes away. + +Elixir specifics: modules are not tied to files (use `dexter_file_outline` for +a file, `dexter_definition` for a module); pass fully-qualified module names, +not aliases; function names take no arity. Functions that a `__using__` +quote block injects resolve through the use chain, but code generated in other +ways (other macros, `unquote` names) can be missing from the index, so an empty +lookup can mean macro-generated code. + +Answers read files the way the user sees them: a file open in an attached +editor with unsaved changes is read from that buffer, unless the file on disk +changed later, and the answer says so. diff --git a/internal/mcp/linemap.go b/internal/mcp/linemap.go new file mode 100644 index 0000000..e061b4f --- /dev/null +++ b/internal/mcp/linemap.go @@ -0,0 +1,125 @@ +package mcp + +import "strings" + +// Bounds of the line diff. On a 10,000-line file, a few edits cost about +// 0.3 ms and 900 edits about 2 ms and 5 MB; outside the bounds lineMap falls +// back to the lines that both texts share at the start and at the end. +const ( + maxDiffLines = 50000 // lines of both texts after the shared start and end + maxDiffEdits = 1000 // inserted plus deleted lines +) + +// lineMap maps each 1-based line of the saved file (where the index puts +// definitions and references) to the same line in an editor buffer, or to 0 +// when the buffer changed or deleted that line. A line counts as equal +// whatever its line ending, so a buffer that differs from the disk only in +// CRLF against LF changes no line. +type lineMap struct { + to []int32 // to[i] is the buffer line of saved line i; index 0 is unused +} + +func newLineMap(disk, text string) lineMap { + a, b := splitLines(disk), splitLines(text) + m := lineMap{to: make([]int32, len(a)+1)} + prefix := 0 + for prefix < len(a) && prefix < len(b) && a[prefix] == b[prefix] { + m.to[prefix+1] = int32(prefix + 1) + prefix++ + } + suffix := 0 + for suffix < len(a)-prefix && suffix < len(b)-prefix && a[len(a)-1-suffix] == b[len(b)-1-suffix] { + m.to[len(a)-suffix] = int32(len(b) - suffix) + suffix++ + } + midA, midB := a[prefix:len(a)-suffix], b[prefix:len(b)-suffix] + if len(midA) == 0 || len(midB) == 0 || len(midA)+len(midB) > maxDiffLines { + return m + } + matchLines(midA, midB, func(i, j int) { + m.to[prefix+i+1] = int32(prefix + j + 1) + }) + return m +} + +// locate returns the buffer line of saved line n, or false when the buffer +// changed it. +func (m lineMap) locate(n int) (int, bool) { + if n < 1 || n >= len(m.to) || m.to[n] == 0 { + return n, false + } + return int(m.to[n]), true +} + +// splitLines splits text into lines without their line endings. +func splitLines(text string) []string { + lines := strings.Split(text, "\n") + for i, l := range lines { + lines[i] = strings.TrimSuffix(l, "\r") + } + return lines +} + +// matchLines reports the pairs of equal lines of a shortest edit script from a +// to b (Myers, "An O(ND) Difference Algorithm"), in order. It reports nothing +// when the script needs more than maxDiffEdits edits. +func matchLines(a, b []string, match func(i, j int)) { + n, m := len(a), len(b) + maxD := n + m + if maxD > maxDiffEdits { + maxD = maxDiffEdits + } + off := maxD + 1 + v := make([]int32, 2*maxD+3) + // trace[d] is v[off-d-1 : off+d+2] before step d, for the backtrack. + var trace [][]int32 + for d := 0; d <= maxD; d++ { + trace = append(trace, append([]int32(nil), v[off-d-1:off+d+2]...)) + for k := -d; k <= d; k += 2 { + var x int + if k == -d || (k != d && v[off+k-1] < v[off+k+1]) { + x = int(v[off+k+1]) + } else { + x = int(v[off+k-1]) + 1 + } + y := x - k + for x < n && y < m && a[x] == b[y] { + x++ + y++ + } + v[off+k] = int32(x) + if x >= n && y >= m { + backtrack(trace, n, m, match) + return + } + } + } +} + +func backtrack(trace [][]int32, n, m int, match func(i, j int)) { + type pair struct{ i, j int } + var pairs []pair + x, y := n, m + for d := len(trace) - 1; d >= 0; d-- { + v := trace[d] // v[k] is at index k+d+1 + at := func(k int) int { return int(v[k+d+1]) } + k := x - y + var prevK int + if k == -d || (k != d && at(k-1) < at(k+1)) { + prevK = k + 1 + } else { + prevK = k - 1 + } + prevX := at(prevK) + prevY := prevX - prevK + for x > prevX && y > prevY { + x-- + y-- + pairs = append(pairs, pair{x, y}) + } + x, y = prevX, prevY + } + for i := len(pairs) - 1; i >= 0; i-- { + match(pairs[i].i, pairs[i].j) + } +} diff --git a/internal/mcp/mcp_test.go b/internal/mcp/mcp_test.go new file mode 100644 index 0000000..81679f0 --- /dev/null +++ b/internal/mcp/mcp_test.go @@ -0,0 +1,225 @@ +package mcp + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/modelcontextprotocol/go-sdk/mcp" + + "github.com/remoteoss/dexter/internal/lsp" + "github.com/remoteoss/dexter/internal/store" + "github.com/remoteoss/dexter/internal/workspace" +) + +// testEnv is a full in-memory MCP round trip: client session <-> Frontend +// with all tools declared <-> the tool bodies over a real workspace runtime in +// a temp dir. Going through the SDK session exercises schema inference and +// argument validation; the local backend calls the same Handler.Call that the +// daemon's control method calls. +type testEnv struct { + t *testing.T + rt *workspace.Runtime + store *store.Store + lsp *lsp.Server + root string + session *mcp.ClientSession +} + +// localBackend runs tool bodies in-process, as the daemon does. +type localBackend struct { + h *Handler + wait time.Duration +} + +func (b *localBackend) CallTool(ctx context.Context, name string, args json.RawMessage) (string, error) { + return b.h.Call(ctx, name, args, b.wait) +} + +func (b *localBackend) Close() error { return nil } + +// openTestRuntime opens a workspace runtime over root without native +// watching, so tests decide when the index changes. +func openTestRuntime(t *testing.T, root string, opts workspace.Options) *workspace.Runtime { + t.Helper() + // Keep stdlib and version-manager detection from finding a real Elixir + // install and indexing it. + t.Setenv("DEXTER_ELIXIR_LIB_ROOT", t.TempDir()) + t.Setenv("PATH", t.TempDir()) + t.Setenv("SHELL", "/bin/false") + opts.NoWatch = true + rt, err := workspace.OpenWithOptions(root, opts) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = rt.Close() }) + return rt +} + +func setupTestEnv(t *testing.T) *testEnv { + t.Helper() + root := t.TempDir() + rt := openTestRuntime(t, root, workspace.Options{}) + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + if err := rt.WaitReady(ctx); err != nil { + t.Fatal(err) + } + h := NewHandler(rt, rt.LanguageServices()) + session := connectFrontend(t, NewFrontend(Config{Root: root, Fixed: true, Connect: func(string) Backend { + return &localBackend{h: h, wait: time.Second} + }})) + return &testEnv{t: t, rt: rt, store: rt.Store(), lsp: rt.LanguageServices(), root: root, session: session} +} + +// connectFrontend connects an in-memory MCP client to a server over f. +func connectFrontend(t *testing.T, f *Frontend, rootURIs ...string) *mcp.ClientSession { + t.Helper() + t.Cleanup(f.Close) + ctx := context.Background() + serverTransport, clientTransport := mcp.NewInMemoryTransports() + serverSession, err := NewServer(f).Connect(ctx, serverTransport, nil) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = serverSession.Close() }) + + client := mcp.NewClient(&mcp.Implementation{Name: "test-client", Version: "0.0.1"}, nil) + for _, u := range rootURIs { + client.AddRoots(&mcp.Root{URI: u}) + } + session, err := client.Connect(ctx, clientTransport, nil) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = session.Close() }) + return session +} + +// indexFile writes an Elixir source file under the project root and waits +// until the workspace has indexed it. +func (e *testEnv) indexFile(relPath, content string) string { + e.t.Helper() + path := filepath.Join(e.root, relPath) + if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil { + e.t.Fatal(err) + } + if err := os.WriteFile(path, []byte(content), 0644); err != nil { + e.t.Fatal(err) + } + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + if err := e.rt.ReindexPath(ctx, path); err != nil { + e.t.Fatal(err) + } + return path +} + +func (e *testEnv) callTool(name string, args map[string]any) string { + e.t.Helper() + res, err := e.session.CallTool(context.Background(), &mcp.CallToolParams{Name: name, Arguments: args}) + if err != nil { + e.t.Fatalf("CallTool(%s): %v", name, err) + } + if res.IsError { + e.t.Fatalf("CallTool(%s) returned tool error: %s", name, resultText(res)) + } + return resultText(res) +} + +func (e *testEnv) callToolExpectError(name string, args map[string]any) string { + e.t.Helper() + res, err := e.session.CallTool(context.Background(), &mcp.CallToolParams{Name: name, Arguments: args}) + if err != nil { + return err.Error() + } + if !res.IsError { + e.t.Fatalf("CallTool(%s) succeeded, want error; got: %s", name, resultText(res)) + } + return resultText(res) +} + +func resultText(res *mcp.CallToolResult) string { + var b strings.Builder + for _, c := range res.Content { + if tc, ok := c.(*mcp.TextContent); ok { + b.WriteString(tc.Text) + } + } + return b.String() +} + +func wantContains(t *testing.T, got string, wants ...string) { + t.Helper() + for _, w := range wants { + if !strings.Contains(got, w) { + t.Errorf("output missing %q.\nFull output:\n%s", w, got) + } + } +} + +func wantNotContains(t *testing.T, got string, unwanted ...string) { + t.Helper() + for _, w := range unwanted { + if strings.Contains(got, w) { + t.Errorf("output unexpectedly contains %q.\nFull output:\n%s", w, got) + } + } +} + +func TestListTools(t *testing.T) { + e := setupTestEnv(t) + res, err := e.session.ListTools(context.Background(), nil) + if err != nil { + t.Fatal(err) + } + want := []string{ + "dexter_call_hierarchy", + "dexter_definition", + "dexter_file_outline", + "dexter_implementations", + "dexter_module_api", + "dexter_references", + "dexter_reindex", + "dexter_rename_symbol", + "dexter_search", + "dexter_workspace", + } + if len(res.Tools) != len(want) { + t.Errorf("registered %d tools, want %d", len(res.Tools), len(want)) + } + var got []string + for _, tool := range res.Tools { + got = append(got, tool.Name) + } + for _, w := range want { + found := false + for _, g := range got { + if g == w { + found = true + } + } + if !found { + t.Errorf("tool %s not registered; got %v", w, got) + } + } + + for _, tool := range res.Tools { + a := tool.Annotations + if a == nil { + t.Errorf("tool %s has no annotations", tool.Name) + continue + } + if a.OpenWorldHint == nil || *a.OpenWorldHint { + t.Errorf("tool %s not marked closed-world", tool.Name) + } + wantReadOnly := tool.Name != "dexter_reindex" && tool.Name != "dexter_rename_symbol" + if a.ReadOnlyHint != wantReadOnly { + t.Errorf("tool %s ReadOnlyHint = %v, want %v", tool.Name, a.ReadOnlyHint, wantReadOnly) + } + } +} diff --git a/internal/mcp/module_api.go b/internal/mcp/module_api.go new file mode 100644 index 0000000..89cfcc2 --- /dev/null +++ b/internal/mcp/module_api.go @@ -0,0 +1,184 @@ +package mcp + +import ( + "context" + "fmt" + "strings" + + "github.com/remoteoss/dexter/internal/store" +) + +type ModuleAPIParams struct { + Module string `json:"module" jsonschema:"fully-qualified module name, e.g. MyApp.Accounts (aliases are not resolved)"` + IncludePrivate bool `json:"include_private,omitempty" jsonschema:"also list defp/defmacrop definitions (default false)"` +} + +func (h *Handler) moduleAPI(ctx context.Context, args ModuleAPIParams) (string, error) { + module := strings.TrimSpace(args.Module) + if module == "" { + return "", fmt.Errorf("module must not be empty") + } + + modResults, err := h.store.LookupModule(module) + if err != nil { + return "", fmt.Errorf("looking up module: %w", err) + } + var moduleDef *store.LookupResult + implCount := 0 + isProtocol := false + for i := range modResults { + switch modResults[i].Kind { + case "defimpl": + implCount++ + case "defprotocol": + isProtocol = true + moduleDef = &modResults[i] + case "module": + if moduleDef == nil { + moduleDef = &modResults[i] + } + } + } + if moduleDef == nil { + return fmt.Sprintf("Module %s is not in the index. Use dexter_search to find the right name, or dexter_reindex if the module was just created.", module), nil + } + + var b strings.Builder + kind := "module" + if isProtocol { + kind = "protocol" + } + fmt.Fprintf(&b, "%s %s - %s:%d\n", kind, module, h.relPath(moduleDef.FilePath), moduleDef.Line) + if isProtocol && implCount > 0 { + fmt.Fprintf(&b, "%d defimpl implementation(s); list them with dexter_implementations.\n", implCount) + } + + if moduledoc := h.extractModuledoc(moduleDef.FilePath, moduleDef.Line); moduledoc != "" { + fmt.Fprintf(&b, "\n%s\n", strings.TrimRight(moduledoc, "\n")) + } + + funcs, err := h.store.ListModuleFunctions(module, !args.IncludePrivate) + if err != nil { + return "", fmt.Errorf("listing functions: %w", err) + } + callbacks, err := h.store.ListModuleCallbacks(module) + if err != nil { + return "", fmt.Errorf("listing callbacks: %w", err) + } + + // Bucket by section, preserving store order (name, arity). + sections := map[string][]store.CompletionResult{} + for _, f := range funcs { + sections[sectionFor(f.Kind)] = append(sections[sectionFor(f.Kind)], f) + } + + docs := h.newDocExtractor() + writeSection := func(title string, entries []store.CompletionResult) { + if len(entries) == 0 { + return + } + fmt.Fprintf(&b, "\n%s:\n", title) + for _, e := range entries { + sig := fmt.Sprintf("%s/%d", e.Function, e.Arity) + if e.Params != "" { + sig = fmt.Sprintf("%s(%s)", e.Function, e.Params) + } + line := fmt.Sprintf(" %s [%s:%d]", sig, h.relPath(e.FilePath), e.Line) + if e.Kind == "defdelegate" { + if target := h.delegateTarget(module, e.Function, e.Arity); target != "" { + line += " → " + target + } + } + if doc := docs.docFor(e.FilePath, e.Line); doc != "" { + line += "\n " + doc + } + b.WriteString(line + "\n") + } + } + + writeSection("Functions", sections["functions"]) + writeSection("Macros", sections["macros"]) + writeSection("Guards", sections["guards"]) + writeSection("Delegates", sections["delegates"]) + writeSection("Types", sections["types"]) + writeSection("Private functions", sections["private"]) + writeSection("Callbacks (this module is a behaviour)", callbacks) + + if subs, err := h.store.ListSubmodules(module); err == nil && len(subs) > 0 { + fmt.Fprintf(&b, "\nSubmodules (%d):\n", len(subs)) + const maxSubs = 20 + for i, s := range subs { + if i == maxSubs { + fmt.Fprintf(&b, " ... and %d more\n", len(subs)-maxSubs) + break + } + fmt.Fprintf(&b, " %s\n", s) + } + } + + if len(funcs) == 0 && len(callbacks) == 0 { + fmt.Fprintf(&b, "\nNo functions indexed for this module.\n") + } + return b.String(), nil +} + +func sectionFor(kind string) string { + switch kind { + case "defmacro": + return "macros" + case "defguard": + return "guards" + case "defdelegate": + return "delegates" + case "type", "opaque": + return "types" + case "defp", "defmacrop", "defguardp": + return "private" + default: + return "functions" + } +} + +// delegateTarget renders "Target.function" for a defdelegate entry. +func (h *Handler) delegateTarget(module, function string, arity int) string { + results, err := h.store.LookupFunction(module, function) + if err != nil { + return "" + } + for _, r := range results { + if r.Kind == "defdelegate" && r.Arity == arity && r.DelegateTo != "" { + target := r.DelegateTo + "." + function + if r.DelegateAs != "" { + target = r.DelegateTo + "." + r.DelegateAs + } + return target + } + } + return "" +} + +func (h *Handler) extractModuledoc(filePath string, defLine int) string { + a, ok := h.at(filePath, defLine) + if !ok { + return "" + } + return a.tokenized().ExtractModuledoc(a.line - 1) +} + +// docExtractor extracts @doc summaries, tokenizing each source file at most once. +type docExtractor struct { + h *Handler +} + +func (h *Handler) newDocExtractor() *docExtractor { + return &docExtractor{h: h} +} + +func (d *docExtractor) docFor(filePath string, defLine int) string { + a, ok := d.h.at(filePath, defLine) + if !ok { + return "" + } + doc, _ := a.tokenized().ExtractDocAbove(a.line - 1) + return firstDocLine(doc) +} diff --git a/internal/mcp/negotiation_test.go b/internal/mcp/negotiation_test.go new file mode 100644 index 0000000..3c333a9 --- /dev/null +++ b/internal/mcp/negotiation_test.go @@ -0,0 +1,499 @@ +package mcp + +import ( + "context" + "encoding/json" + "errors" + "net/url" + "os" + "path/filepath" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/modelcontextprotocol/go-sdk/mcp" +) + +// fakeBackend stands in for a workspace daemon connection. It answers every +// tool call with the root it was opened for. +type fakeBackend struct { + root string + mu sync.Mutex + closed bool +} + +func (b *fakeBackend) CallTool(ctx context.Context, name string, args json.RawMessage) (string, error) { + return "root=" + b.root, nil +} + +func (b *fakeBackend) Close() error { + b.mu.Lock() + defer b.mu.Unlock() + b.closed = true + return nil +} + +func (b *fakeBackend) isClosed() bool { + b.mu.Lock() + defer b.mu.Unlock() + return b.closed +} + +// negotiationEnv is a negotiating Frontend over fake backends, plus helpers to +// connect clients that advertise chosen roots. +type negotiationEnv struct { + t *testing.T + f *Frontend + fallback string + + mu sync.Mutex + backends []*fakeBackend +} + +func setupNegotiation(t *testing.T) *negotiationEnv { + t.Helper() + return setupNegotiationWith(t, Config{Root: t.TempDir()}) +} + +func setupNegotiationWith(t *testing.T, cfg Config) *negotiationEnv { + t.Helper() + e := &negotiationEnv{t: t, fallback: cfg.Root} + cfg.Connect = func(root string) Backend { + b := &fakeBackend{root: root} + e.mu.Lock() + e.backends = append(e.backends, b) + e.mu.Unlock() + return b + } + e.f = NewFrontend(cfg) + t.Cleanup(e.f.Close) + return e +} + +func (e *negotiationEnv) backendsFor(root string) []*fakeBackend { + e.mu.Lock() + defer e.mu.Unlock() + var out []*fakeBackend + for _, b := range e.backends { + if b.root == root { + out = append(out, b) + } + } + return out +} + +func (e *negotiationEnv) connCount() int { + e.f.mu.Lock() + defer e.f.mu.Unlock() + return len(e.f.conns) +} + +// connect wires a new client session to the negotiating server. Roots are +// added before connecting so they are visible from the first roots/list. +func (e *negotiationEnv) connect(opts *mcp.ClientOptions, rootURIs ...string) (*mcp.ClientSession, *mcp.Client) { + e.t.Helper() + ctx := context.Background() + serverTransport, clientTransport := mcp.NewInMemoryTransports() + ss, err := NewServer(e.f).Connect(ctx, serverTransport, nil) + if err != nil { + e.t.Fatal(err) + } + e.t.Cleanup(func() { _ = ss.Close() }) + + client := mcp.NewClient(&mcp.Implementation{Name: "test-client", Version: "0.0.1"}, opts) + for _, u := range rootURIs { + client.AddRoots(&mcp.Root{URI: u}) + } + cs, err := client.Connect(ctx, clientTransport, nil) + if err != nil { + e.t.Fatal(err) + } + e.t.Cleanup(func() { _ = cs.Close() }) + return cs, client +} + +// projectDir creates a repository directory and returns its path and file URI. +func projectDir(t *testing.T) (string, string) { + t.Helper() + dir := t.TempDir() + if err := os.MkdirAll(filepath.Join(dir, ".git"), 0755); err != nil { + t.Fatal(err) + } + return dir, fileURI(dir) +} + +func writeSource(t *testing.T, dir, rel, content string) { + t.Helper() + path := filepath.Join(dir, rel) + if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(content), 0644); err != nil { + t.Fatal(err) + } +} + +func fileURI(path string) string { + return (&url.URL{Scheme: "file", Path: filepath.ToSlash(path)}).String() +} + +func toolText(t *testing.T, cs *mcp.ClientSession, name string, args map[string]any) (string, bool) { + t.Helper() + res, err := cs.CallTool(context.Background(), &mcp.CallToolParams{Name: name, Arguments: args}) + if err != nil { + return err.Error(), false + } + return resultText(res), !res.IsError +} + +func mustTool(t *testing.T, cs *mcp.ClientSession, name string, args map[string]any) string { + t.Helper() + out, ok := toolText(t, cs, name, args) + if !ok { + t.Fatalf("CallTool(%s) failed: %s", name, out) + } + return out +} + +func eventually(t *testing.T, what string, cond func() bool) { + t.Helper() + deadline := time.Now().Add(10 * time.Second) + for time.Now().Before(deadline) { + if cond() { + return + } + time.Sleep(20 * time.Millisecond) + } + t.Fatalf("timed out waiting for %s", what) +} + +func TestFileURIToPath(t *testing.T) { + cases := []struct { + uri string + want string // "" means an error is expected + }{ + {"file:///a/b", "/a/b"}, + {"file:///a/b/", "/a/b"}, // trailing slash must not key a second workspace + {"file:///a/./b/../b", "/a/b"}, + {"file://localhost/a/b", "/a/b"}, + {"file:///a/my%20project", "/a/my project"}, + {"file://otherhost/a", ""}, + {"file://a", ""}, // host form, no path + {"file:relative", ""}, + } + for _, tc := range cases { + got, err := fileURIToPath(tc.uri) + if tc.want == "" { + if err == nil { + t.Errorf("fileURIToPath(%q) = %q, want error", tc.uri, got) + } + continue + } + if err != nil || got != tc.want { + t.Errorf("fileURIToPath(%q) = %q, %v; want %q", tc.uri, got, err, tc.want) + } + } +} + +func TestNegotiation_BindsClientRoot(t *testing.T) { + e := setupNegotiation(t) + root, uri := projectDir(t) + cs, _ := e.connect(nil, uri) + + wantContains(t, mustTool(t, cs, "dexter_search", map[string]any{"query": "x"}), "root="+root) + if len(e.backendsFor(e.fallback)) != 0 { + t.Error("connected to the fallback root despite a negotiated root") + } +} + +func TestNegotiation_FallsBackWithoutUsableRoots(t *testing.T) { + cases := []struct { + name string + opts *mcp.ClientOptions + uris []string + }{ + // A default go-sdk client advertises roots with an empty list; this is + // what most clients look like, not an edge case. + {name: "empty roots list", opts: nil}, + {name: "roots capability off", opts: &mcp.ClientOptions{Capabilities: &mcp.ClientCapabilities{}}}, + {name: "non-file roots only", opts: nil, uris: []string{"https://example.com/project"}}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + e := setupNegotiation(t) + cs, _ := e.connect(tc.opts, tc.uris...) + wantContains(t, mustTool(t, cs, "dexter_search", map[string]any{"query": "x"}), "root="+e.fallback) + }) + } +} + +// A launch directory that is not a project is not indexed: a session without +// roots gets the reason, and a session with roots still works. +func TestNegotiation_FallbackRefused(t *testing.T) { + e := setupNegotiationWith(t, Config{Root: t.TempDir(), FallbackErr: errors.New("refusing to use the launch directory")}) + noRoots, _ := e.connect(&mcp.ClientOptions{Capabilities: &mcp.ClientCapabilities{}}) + out, ok := toolText(t, noRoots, "dexter_search", map[string]any{"query": "x"}) + if ok || !strings.Contains(out, "refusing to use the launch directory") { + t.Fatalf("session without roots was not refused: %s", out) + } + if len(e.backendsFor(e.fallback)) != 0 { + t.Error("connected to a refused fallback root") + } + + root, uri := projectDir(t) + withRoots, _ := e.connect(nil, uri) + wantContains(t, mustTool(t, withRoots, "dexter_search", map[string]any{"query": "x"}), "root="+root) +} + +// A root inside a repository resolves upward to the repository: an existing +// index or .git wins, and a nested mix.exs does not stop the walk. +func TestNegotiation_ResolvesRootUpward(t *testing.T) { + e := setupNegotiation(t) + repo, _ := projectDir(t) + writeSource(t, repo, "apps/web/mix.exs", "defmodule Web.MixProject do\nend\n") + cs, _ := e.connect(nil, fileURI(filepath.Join(repo, "apps", "web"))) + + wantContains(t, mustTool(t, cs, "dexter_search", map[string]any{"query": "x"}), "root="+repo) +} + +// The resolver that cmd passes decides the root, so that the MCP frontend +// attaches to the same daemon as the CLI and the editor. +func TestNegotiation_UsesConfiguredResolver(t *testing.T) { + resolved := t.TempDir() + var asked string + e := setupNegotiationWith(t, Config{Root: t.TempDir(), ResolveRoot: func(dir string) (string, error) { + asked = dir + return resolved, nil + }}) + dir, uri := projectDir(t) + cs, _ := e.connect(nil, uri) + + wantContains(t, mustTool(t, cs, "dexter_search", map[string]any{"query": "x"}), "root="+resolved) + if asked != dir { + t.Errorf("resolver got %q, want the client root %q", asked, dir) + } +} + +func TestNegotiation_BadRootIsRetryable(t *testing.T) { + e := setupNegotiation(t) + badURI := "file:///nonexistent/dexter-negotiation-test" + cs, client := e.connect(nil, badURI) + + out, ok := toolText(t, cs, "dexter_search", map[string]any{"query": "x"}) + if ok { + t.Fatalf("tool call succeeded against a nonexistent root: %s", out) + } + if !strings.Contains(out, "not a directory") { + t.Errorf("error does not name the problem: %s", out) + } + + // The failure is not cached: with the roots fixed, the same session works. + root, goodURI := projectDir(t) + client.RemoveRoots(badURI) + client.AddRoots(&mcp.Root{URI: goodURI}) + eventually(t, "session to bind the corrected root", func() bool { + out, ok := toolText(t, cs, "dexter_search", map[string]any{"query": "x"}) + return ok && strings.Contains(out, "root="+root) + }) +} + +// Sessions with different roots use their own workspaces; sessions with the +// same root share one connection. +func TestNegotiation_MultipleRoots(t *testing.T) { + e := setupNegotiation(t) + rootA, uriA := projectDir(t) + rootB, uriB := projectDir(t) + csA, _ := e.connect(nil, uriA) + csB, _ := e.connect(nil, uriB) + csA2, _ := e.connect(nil, uriA) + + wantContains(t, mustTool(t, csA, "dexter_search", map[string]any{"query": "x"}), "root="+rootA) + wantContains(t, mustTool(t, csB, "dexter_search", map[string]any{"query": "x"}), "root="+rootB) + wantContains(t, mustTool(t, csA2, "dexter_search", map[string]any{"query": "x"}), "root="+rootA) + + if n := e.connCount(); n != 2 { + t.Errorf("3 sessions over 2 roots hold %d workspace connections, want 2", n) + } + if n := len(e.backendsFor(rootA)); n != 1 { + t.Errorf("2 sessions on one root opened %d connections, want 1", n) + } +} + +func TestNegotiation_RootsChangedSwapsWorkspace(t *testing.T) { + e := setupNegotiation(t) + rootA, uriA := projectDir(t) + rootB, uriB := projectDir(t) + cs, client := e.connect(nil, uriA) + + wantContains(t, mustTool(t, cs, "dexter_search", map[string]any{"query": "x"}), "root="+rootA) + + client.RemoveRoots(uriA) + client.AddRoots(&mcp.Root{URI: uriB}) + eventually(t, "session to move to the new root", func() bool { + out, ok := toolText(t, cs, "dexter_search", map[string]any{"query": "x"}) + return ok && strings.Contains(out, "root="+rootB) + }) + + // The old workspace connection is closed: nothing else uses it. + for _, b := range e.backendsFor(rootA) { + if !b.isClosed() { + t.Error("old workspace connection still open after the swap") + } + } + if n := e.connCount(); n != 1 { + t.Errorf("%d workspace connections after the swap, want 1", n) + } +} + +// A roots change that resolves to the same project keeps the connection. +func TestNegotiation_SameRootChangeIsNoop(t *testing.T) { + e := setupNegotiation(t) + root, uri := projectDir(t) + cs, client := e.connect(nil, uri) + mustTool(t, cs, "dexter_search", map[string]any{"query": "x"}) + + // Same project, different advertised directory: the subdirectory + // resolves upward through .git. + subdir := filepath.Join(root, "lib") + if err := os.MkdirAll(subdir, 0755); err != nil { + t.Fatal(err) + } + client.AddRoots(&mcp.Root{URI: fileURI(subdir)}) + eventually(t, "roots change notification to arrive", func() bool { + e.f.mu.Lock() + defer e.f.mu.Unlock() + for _, st := range e.f.sessions { + if st.dirty { + return true + } + } + return false + }) + wantContains(t, mustTool(t, cs, "dexter_search", map[string]any{"query": "x"}), "root="+root) // negotiates again + + backends := e.backendsFor(root) + if len(backends) != 1 || backends[0].isClosed() { + t.Errorf("workspace connection was replaced for a change that resolves to the same root") + } +} + +// A workspace root with characters that URI-encode (spaces) binds correctly. +func TestNegotiation_RootWithSpaces(t *testing.T) { + e := setupNegotiation(t) + root := filepath.Join(t.TempDir(), "my project") + if err := os.MkdirAll(filepath.Join(root, ".git"), 0755); err != nil { + t.Fatal(err) + } + uri := fileURI(root) + if !strings.Contains(uri, "%20") { + t.Fatalf("test URI %q does not exercise percent-encoding", uri) + } + cs, _ := e.connect(nil, uri) + wantContains(t, mustTool(t, cs, "dexter_search", map[string]any{"query": "x"}), "root="+root) +} + +// A session disconnecting releases its workspace connection. +func TestNegotiation_SessionCloseReleasesWorkspace(t *testing.T) { + e := setupNegotiation(t) + root, uri := projectDir(t) + cs, _ := e.connect(nil, uri) + mustTool(t, cs, "dexter_search", map[string]any{"query": "x"}) + + if err := cs.Close(); err != nil { + t.Fatal(err) + } + eventually(t, "workspace connection to be released", func() bool { + backends := e.backendsFor(root) + return len(backends) == 1 && backends[0].isClosed() && e.connCount() == 0 + }) +} + +// A fixed root (`dexter mcp `) ignores the client's roots. +func TestFixedRootIgnoresClientRoots(t *testing.T) { + fixed := t.TempDir() + e := setupNegotiationWith(t, Config{Root: fixed, Fixed: true}) + _, uri := projectDir(t) + cs, _ := e.connect(nil, uri) + wantContains(t, mustTool(t, cs, "dexter_search", map[string]any{"query": "x"}), "root="+fixed) +} + +// A Windows file URI carries the drive letter after a leading slash. Without +// removing that slash, `C:/project` is not an absolute path on Windows and the +// negotiation fails. Other systems keep the path as it is. +func TestURIPathDriveLetter(t *testing.T) { + cases := []struct { + path, goos, want string + }{ + {"/C:/project", "windows", "C:/project"}, + {"/c:/my project", "windows", "c:/my project"}, + {"/project", "windows", "/project"}, + {"/C:/project", "linux", "/C:/project"}, + {"/C:/project", "darwin", "/C:/project"}, + } + for _, tc := range cases { + if got := uriPath(tc.path, tc.goos); got != tc.want { + t.Errorf("uriPath(%q, %s) = %q, want %q", tc.path, tc.goos, got, tc.want) + } + } +} + +// An unusable root before a usable one must not fail the session: the client +// can list a stale or deleted directory first. +func TestNegotiation_SkipsUnusableRoot(t *testing.T) { + e := setupNegotiation(t) + root, uri := projectDir(t) + file := filepath.Join(t.TempDir(), "not-a-dir.txt") + if err := os.WriteFile(file, nil, 0o644); err != nil { + t.Fatal(err) + } + cs, _ := e.connect(nil, "file:///nonexistent/dexter-negotiation-test", fileURI(file), uri) + wantContains(t, mustTool(t, cs, "dexter_search", map[string]any{"query": "x"}), "root="+root) +} + +// warmCounter is a fake backend that counts warm calls. +type warmCounter struct { + fakeBackend + warms *atomic.Int32 +} + +func (w *warmCounter) warm(context.Context) { w.warms.Add(1) } + +// Regression: a session's workspace was warmed (its daemon started and +// indexing) even when the root was not an Elixir project. +func TestFrontend_WarmsOnlyProjects(t *testing.T) { + for _, tc := range []struct { + name string + project bool + }{{"project", true}, {"plain directory", false}} { + t.Run(tc.name, func(t *testing.T) { + root := t.TempDir() + if tc.project { + writeSource(t, root, "mix.exs", "defmodule App.MixProject do\nend\n") + } + var warms atomic.Int32 + f := NewFrontend(Config{Root: root, Fixed: true, Connect: func(root string) Backend { + return &warmCounter{fakeBackend: fakeBackend{root: root}, warms: &warms} + }}) + cs := connectFrontend(t, f) + mustTool(t, cs, "dexter_search", map[string]any{"query": "x"}) + time.Sleep(50 * time.Millisecond) // warm runs in a goroutine + if got := warms.Load() > 0; got != tc.project { + t.Errorf("warmed = %v, want %v", got, tc.project) + } + }) + } +} + +// A client root that is not a project is unusable: a later project root wins. +func TestNegotiation_SkipsNonProjectRoot(t *testing.T) { + e := setupNegotiation(t) + plain := t.TempDir() + project, projectURI := projectDir(t) + cs, _ := e.connect(nil, fileURI(plain), projectURI) + wantContains(t, mustTool(t, cs, "dexter_search", map[string]any{"query": "x"}), "root="+project) + if got := e.backendsFor(plain); len(got) != 0 { + t.Errorf("a backend was opened for the non-project root %s", plain) + } +} diff --git a/internal/mcp/references.go b/internal/mcp/references.go new file mode 100644 index 0000000..458eea9 --- /dev/null +++ b/internal/mcp/references.go @@ -0,0 +1,77 @@ +package mcp + +import ( + "context" + "fmt" + "sort" + "strings" + + "github.com/remoteoss/dexter/internal/lsp" +) + +type ReferencesParams struct { + Module string `json:"module" jsonschema:"fully-qualified module name, e.g. MyApp.Accounts (aliases are not resolved)"` + Function string `json:"function,omitempty" jsonschema:"function name; omit to list references to the module itself (aliases, imports, uses, qualified calls)"` +} + +const maxReferenceLines = 100 + +func (h *Handler) references(ctx context.Context, args ReferencesParams) (string, error) { + module := strings.TrimSpace(args.Module) + if module == "" { + return "", fmt.Errorf("module must not be empty") + } + function := strings.TrimSpace(args.Function) + + // The same reference search as find-references and `dexter references`: + // use chains, aliases injected by __using__, bare calls in the defining + // module, and calls through defdelegate facades. + refs, err := h.lsp.ReferenceNames(module, function, lsp.NameReferenceOptions{ + FollowDelegates: true, + ExcludeStdlib: true, + }) + if err != nil { + return "", fmt.Errorf("finding references: %w", err) + } + + target := module + if function != "" { + target = module + "." + function + } + if len(refs) == 0 { + return fmt.Sprintf("No references to %s found in the index. If files changed recently, call dexter_reindex first.", target), nil + } + sort.Slice(refs, func(i, j int) bool { + if refs[i].FilePath != refs[j].FilePath { + return refs[i].FilePath < refs[j].FilePath + } + return refs[i].Line < refs[j].Line + }) + + var b strings.Builder + fmt.Fprintf(&b, "%d reference(s) to %s:\n", len(refs), target) + + written := 0 + var lastFile string + truncated := 0 + for _, r := range refs { + if written >= maxReferenceLines { + truncated++ + continue + } + if r.FilePath != lastFile { + fmt.Fprintf(&b, "\n%s\n", h.relPath(r.FilePath)) + lastFile = r.FilePath + } + srcLine, line, label, ok := h.sourceLine(r.FilePath, r.Line) + if !ok { + line = r.Line + } + fmt.Fprintf(&b, " %d: %s%s\n", line, strings.TrimSpace(srcLine), label) + written++ + } + if truncated > 0 { + fmt.Fprintf(&b, "\n... and %d more reference(s) not shown. Narrow the search (e.g. pass a function name) to see the rest.\n", truncated) + } + return b.String(), nil +} diff --git a/internal/mcp/reindex.go b/internal/mcp/reindex.go new file mode 100644 index 0000000..477b4b9 --- /dev/null +++ b/internal/mcp/reindex.go @@ -0,0 +1,20 @@ +package mcp + +import ( + "context" + "fmt" + "time" +) + +type ReindexParams struct{} + +func (h *Handler) reindex(ctx context.Context, args ReindexParams) (string, error) { + // The workspace daemon owns the index: this is the same barrier as + // `dexter reindex`. Every change that the daemon accepted before the call + // is in the index when it returns. + start := time.Now() + if err := h.rt.Reindex(ctx); err != nil { + return "", fmt.Errorf("reindexing: %w", err) + } + return fmt.Sprintf("Reindexed the workspace in %s. The index is up to date.", time.Since(start).Round(time.Millisecond)), nil +} diff --git a/internal/mcp/rename.go b/internal/mcp/rename.go new file mode 100644 index 0000000..9a83c99 --- /dev/null +++ b/internal/mcp/rename.go @@ -0,0 +1,82 @@ +package mcp + +import ( + "context" + "fmt" + "sort" + "strings" + + "github.com/remoteoss/dexter/internal/lsp" +) + +type RenameParams struct { + Module string `json:"module" jsonschema:"module being renamed, or the module owning the function"` + Function string `json:"function,omitempty" jsonschema:"if set, rename this function; otherwise rename the module itself (and its submodules)"` + NewName string `json:"new_name" jsonschema:"new function name (e.g. get_user), or new fully-qualified module name (e.g. MyApp.Clients)"` +} + +func (h *Handler) rename(ctx context.Context, args RenameParams) (string, error) { + module := strings.TrimSpace(args.Module) + function := strings.TrimSpace(args.Function) + newName := strings.TrimSpace(args.NewName) + if module == "" || newName == "" { + return "", fmt.Errorf("module and new_name must not be empty") + } + if err := h.renameAllowed(); err != nil { + return "", err + } + + target := fmt.Sprintf("%s to %s", module, newName) + if function != "" { + target = fmt.Sprintf("%s.%s to %s", module, function, newName) + } + // The last point where a canceled rename has changed nothing. + if err := ctx.Err(); err != nil { + return "", fmt.Errorf("the rename was canceled before it changed any file: %w", err) + } + summary, err := renameSymbol(ctx, h.lsp, module, function, newName) + if err != nil { + return "", err + } + + var b strings.Builder + fmt.Fprintf(&b, "Renamed %s across %d file(s). The index is updated.\n", target, len(summary.FilesChanged)) + if len(summary.FilesFailed) > 0 { + fmt.Fprintf(&b, "\nWARNING: the rename could not change %d file(s) (%s). These files still use the old name; fix them by hand or revert the rename:\n", len(summary.FilesFailed), summary.FailureReason) + for _, fp := range summary.FilesFailed { + fmt.Fprintf(&b, " %s\n", h.relPath(fp)) + } + } + if len(summary.FilesMoved) > 0 { + fmt.Fprintf(&b, "\nFiles moved to follow the naming convention:\n") + from := make([]string, 0, len(summary.FilesMoved)) + for path := range summary.FilesMoved { + from = append(from, path) + } + sort.Strings(from) + for _, path := range from { + fmt.Fprintf(&b, " %s → %s\n", h.relPath(path), h.relPath(summary.FilesMoved[path])) + } + } + fmt.Fprintf(&b, "\nChanged files:\n") + for _, fp := range summary.FilesChanged { + fmt.Fprintf(&b, " %s\n", h.relPath(fp)) + } + fmt.Fprintf(&b, "\nReview with git diff; revert with git checkout.\n") + return b.String(), nil +} + +// renameSymbol is the only place where the MCP rename calls the rename +// machinery in internal/lsp. +// +// TODO: The workspace daemon is shared with editors. When the shared rule for +// renames from a frontend without an editor lands in internal/lsp (do not +// write over unsaved editor buffers: write closed files and clean open +// files, and refuse with an actionable error when an affected file has unsaved +// changes in an editor), call its headless entry point here instead. +func renameSymbol(ctx context.Context, server *lsp.Server, module, function, newName string) (*lsp.RenameSummary, error) { + if function != "" { + return server.RenameFunctionContext(ctx, module, function, newName) + } + return server.RenameModuleContext(ctx, module, newName) +} diff --git a/internal/mcp/rename_test.go b/internal/mcp/rename_test.go new file mode 100644 index 0000000..b0d3507 --- /dev/null +++ b/internal/mcp/rename_test.go @@ -0,0 +1,207 @@ +package mcp + +import ( + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "sync" + "testing" + "time" +) + +func readFile(t *testing.T, root, rel string) string { + t.Helper() + b, err := os.ReadFile(filepath.Join(root, rel)) + if err != nil { + t.Fatal(err) + } + return string(b) +} + +func TestRenameTool_Function(t *testing.T) { + e := setupProject(t) + out := e.callTool("dexter_rename_symbol", map[string]any{ + "module": "MyApp.Accounts", "function": "fetch_user", "new_name": "get_user", + }) + wantContains(t, out, + "Renamed MyApp.Accounts.fetch_user to get_user", + "lib/my_app/accounts.ex", + "lib/my_app/worker.ex", + "git diff", + ) + + accounts := readFile(t, e.root, "lib/my_app/accounts.ex") + wantContains(t, accounts, "def get_user(id)", "@spec get_user(integer())") + wantNotContains(t, accounts, "fetch_user") + + worker := readFile(t, e.root, "lib/my_app/worker.ex") + wantContains(t, worker, "MyApp.Accounts.get_user(1)", "MyApp.Accounts.get_user(2)") + + // The rename reindexes what it wrote: lookups resolve the new name only. + wantContains(t, e.callTool("dexter_definition", map[string]any{"module": "MyApp.Accounts", "function": "get_user"}), "get_user/1 (def)") + wantContains(t, e.callTool("dexter_definition", map[string]any{"module": "MyApp.Accounts", "function": "fetch_user"}), "not in the index") +} + +func TestRenameTool_Module_MovesFiles(t *testing.T) { + e := setupProject(t) + out := e.callTool("dexter_rename_symbol", map[string]any{ + "module": "MyApp.Accounts", "new_name": "MyApp.Users", + }) + wantContains(t, out, + "Renamed MyApp.Accounts to MyApp.Users", + "Files moved to follow the naming convention:", + "lib/my_app/accounts.ex → lib/my_app/users.ex", + "lib/my_app/accounts/creator.ex → lib/my_app/users/creator.ex", + ) + + if _, err := os.Stat(filepath.Join(e.root, "lib/my_app/accounts.ex")); !os.IsNotExist(err) { + t.Error("old module file still exists after rename") + } + wantContains(t, readFile(t, e.root, "lib/my_app/users.ex"), "defmodule MyApp.Users do") + wantContains(t, readFile(t, e.root, "lib/my_app/users/creator.ex"), "defmodule MyApp.Users.Creator do") + wantContains(t, readFile(t, e.root, "lib/my_app/worker.ex"), "MyApp.Users.fetch_user(1)") + + wantContains(t, e.callTool("dexter_definition", map[string]any{"module": "MyApp.Users"}), "defmodule MyApp.Users") +} + +func TestRenameTool_Errors(t *testing.T) { + e := setupProject(t) + + errText := e.callToolExpectError("dexter_rename_symbol", map[string]any{ + "module": "MyApp.Accounts", "function": "fetch_user", "new_name": "NotValid", + }) + wantContains(t, errText, "invalid function name") + + errText = e.callToolExpectError("dexter_rename_symbol", map[string]any{ + "module": "MyApp.Accounts", "function": "fetch_user", "new_name": "list_users", + }) + wantContains(t, errText, "already exists") + + errText = e.callToolExpectError("dexter_rename_symbol", map[string]any{ + "module": "MyApp.Missing", "new_name": "MyApp.New", + }) + wantContains(t, errText, "not found") + + // Failed renames must not touch disk. + if s := readFile(t, e.root, "lib/my_app/accounts.ex"); !strings.Contains(s, "def fetch_user(id)") { + t.Error("failed rename modified files") + } +} + +// Regression: the rename once applied editor positions (UTF-16 columns) as +// byte offsets on disk, so a non-ASCII character left of the name moved the +// edit and corrupted the line. +func TestRenameTool_NonASCIIBeforeName(t *testing.T) { + e := setupProject(t) + e.indexFile("lib/my_app/greeter.ex", `defmodule MyApp.Greeter do + def greet(id), do: {"héllo wörld ✓", MyApp.Accounts.fetch_user(id)} +end +`) + e.callTool("dexter_rename_symbol", map[string]any{ + "module": "MyApp.Accounts", "function": "fetch_user", "new_name": "get_user", + }) + wantContains(t, readFile(t, e.root, "lib/my_app/greeter.ex"), + ` def greet(id), do: {"héllo wörld ✓", MyApp.Accounts.get_user(id)}`) +} + +// Regression: two renames at the same time each read the affected files and +// wrote them back, so the later write dropped the other rename's edits while +// both reported success. +func TestRenameTool_ConcurrentRenamesKeepBothEdits(t *testing.T) { + e := setupProject(t) + for i := 0; i < 40; i++ { + e.indexFile(fmt.Sprintf("lib/my_app/caller_%d.ex", i), fmt.Sprintf(`defmodule MyApp.Caller%d do + def run do + MyApp.Accounts.fetch_user(1) + MyApp.Accounts.list_users([]) + end +end +`, i)) + } + h := NewHandler(e.rt, e.lsp) + pairs := [][2][2]string{ + {{"fetch_user", "get_user"}, {"list_users", "all_users"}}, + {{"get_user", "fetch_user"}, {"all_users", "list_users"}}, + } + for round := 0; round < 6; round++ { + renames := pairs[round%2] + var wg sync.WaitGroup + errs := make([]error, 2) + for i, r := range renames { + wg.Add(1) + go func() { + defer wg.Done() + args, _ := json.Marshal(map[string]any{"module": "MyApp.Accounts", "function": r[0], "new_name": r[1]}) + _, errs[i] = h.Call(context.Background(), renameToolName, args, 0) + }() + } + wg.Wait() + for _, err := range errs { + if err != nil { + t.Fatalf("round %d: rename failed: %v", round, err) + } + } + for i := 0; i < 40; i++ { + text := readFile(t, e.root, fmt.Sprintf("lib/my_app/caller_%d.ex", i)) + for _, r := range renames { + if !strings.Contains(text, "MyApp.Accounts."+r[1]+"(") || strings.Contains(text, "MyApp.Accounts."+r[0]+"(") { + t.Fatalf("round %d: caller_%d.ex lost the rename %s → %s:\n%s", round, i, r[0], r[1], text) + } + } + } + } +} + +// Regression: a call canceled while it waited for the index still ran, so a +// rename could start its writes after the client was told that it may have +// been applied. +func TestRenameTool_CanceledCallChangesNothing(t *testing.T) { + e := setupProject(t) + h := NewHandler(e.rt, e.lsp) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + args, _ := json.Marshal(map[string]any{"module": "MyApp.Accounts", "function": "fetch_user", "new_name": "get_user"}) + if _, err := h.Call(ctx, renameToolName, args, time.Second); err == nil { + t.Fatal("a canceled rename succeeded") + } + wantContains(t, readFile(t, e.root, "lib/my_app/worker.ex"), "MyApp.Accounts.fetch_user(1)") + wantNotContains(t, readFile(t, e.root, "lib/my_app/accounts.ex"), "get_user") +} + +// Regression: the check that the new name is free ran before the rename +// lock, so two renames to the same new name could both pass it and both +// write. +func TestRenameTool_ConcurrentRenamesToOneNameOnlyOneWins(t *testing.T) { + e := setupProject(t) + h := NewHandler(e.rt, e.lsp) + current := [2]string{"fetch_user", "list_users"} + for round := 0; round < 8; round++ { + target := fmt.Sprintf("taken_%d", round) + var wg sync.WaitGroup + errs := make([]error, 2) + for i := range current { + wg.Add(1) + go func() { + defer wg.Done() + args, _ := json.Marshal(map[string]any{"module": "MyApp.Accounts", "function": current[i], "new_name": target}) + _, errs[i] = h.Call(context.Background(), renameToolName, args, 0) + }() + } + wg.Wait() + wins := 0 + for i, err := range errs { + if err == nil { + wins++ + current[i] = target + } else if !strings.Contains(err.Error(), "already exists") { + t.Fatalf("round %d: unexpected error: %v", round, err) + } + } + if wins != 1 { + t.Fatalf("round %d: %d renames to %s succeeded, want 1:\n%s", round, wins, target, readFile(t, e.root, "lib/my_app/accounts.ex")) + } + } +} diff --git a/internal/mcp/roots.go b/internal/mcp/roots.go new file mode 100644 index 0000000..2f93b18 --- /dev/null +++ b/internal/mcp/roots.go @@ -0,0 +1,109 @@ +package mcp + +import ( + "context" + "fmt" + "net/url" + "os" + "path/filepath" + "runtime" + "strings" + + "github.com/modelcontextprotocol/go-sdk/mcp" + + "github.com/remoteoss/dexter/internal/store" +) + +// fileURIToPath converts a file:// URI to a clean absolute filesystem path, so +// that spellings such as a trailing slash name the same root. +func fileURIToPath(raw string) (string, error) { + u, err := url.Parse(raw) + if err != nil { + return "", fmt.Errorf("invalid root URI %q: %w", raw, err) + } + if u.Host != "" && u.Host != "localhost" { + return "", fmt.Errorf("root URI %q names a remote host", raw) + } + path := filepath.Clean(filepath.FromSlash(uriPath(u.Path, runtime.GOOS))) + if !filepath.IsAbs(path) { + return "", fmt.Errorf("root URI %q has no absolute path", raw) + } + return path, nil +} + +// uriPath returns the filesystem form of a file URI's path. On Windows, +// file:///C:/project carries the drive letter after a leading slash. +func uriPath(p, goos string) string { + if goos == "windows" && len(p) >= 3 && p[0] == '/' && p[2] == ':' && isDriveLetter(p[1]) { + return p[1:] + } + return p +} + +func isDriveLetter(c byte) bool { + return ('a' <= c && c <= 'z') || ('A' <= c && c <= 'Z') +} + +// negotiatedRoot resolves a session's workspace root from the MCP roots the +// client advertises. ok is false when the client offers no usable root (no +// roots capability, an empty list, or no file:// root): callers fall back to +// the launch-directory root. A transport failure, or file:// roots that are +// all unusable, is an error the caller should surface and retry, not cache. +// +// A usable root goes through resolve, which finds the project root the same +// way for every frontend. The spelling the client used is kept: the daemon +// indexes paths in the spelling of the frontend that started it, and refuses +// other spellings of the same directory. +func negotiatedRoot(ctx context.Context, ss *mcp.ServerSession, resolve func(string) (string, error)) (root string, ok bool, err error) { + params := ss.InitializeParams() + if params == nil || params.Capabilities == nil || params.Capabilities.RootsV2 == nil { + return "", false, nil + } + res, err := ss.ListRoots(ctx, nil) + if err != nil { + return "", false, fmt.Errorf("listing client roots: %w", err) + } + // The first usable file:// root wins. An unusable one (a stale or deleted + // directory, a file) is skipped, so it cannot hide a usable root after + // it; when no file:// root is usable, the first error is reported. + var firstErr error + for _, r := range res.Roots { + if !strings.HasPrefix(r.URI, "file:") { + continue + } + root, err := usableRoot(r.URI, resolve) + if err == nil { + return root, true, nil + } + if firstErr == nil { + firstErr = err + } + } + if firstErr != nil { + return "", false, firstErr + } + return "", false, nil +} + +func usableRoot(uri string, resolve func(string) (string, error)) (string, error) { + path, err := fileURIToPath(uri) + if err != nil { + return "", err + } + info, err := os.Stat(path) + if err != nil || !info.IsDir() { + return "", fmt.Errorf("client root %q is not a directory", path) + } + return resolve(path) +} + +// defaultResolveRoot finds the project root above dir with the store's marker +// search (an existing index, then a repository), and refuses a directory that +// is not a project. +func defaultResolveRoot(dir string) (string, error) { + root := store.FindProjectRoot(dir) + if err := store.NonProjectRootError(root); err != nil { + return "", err + } + return root, nil +} diff --git a/internal/mcp/search.go b/internal/mcp/search.go new file mode 100644 index 0000000..c051d1b --- /dev/null +++ b/internal/mcp/search.go @@ -0,0 +1,37 @@ +package mcp + +import ( + "context" + "fmt" + "strings" +) + +type SearchParams struct { + Query string `json:"query" jsonschema:"fuzzy symbol query, e.g. 'Accounts.fetch' or 'fetch_user'"` + IncludeStdlib bool `json:"include_stdlib,omitempty" jsonschema:"also match Elixir stdlib symbols (default false)"` +} + +func (h *Handler) search(ctx context.Context, args SearchParams) (string, error) { + query := strings.TrimSpace(args.Query) + if query == "" { + return "", fmt.Errorf("query must not be empty") + } + + var exclude []string + if stdlibRoot := h.lsp.StdlibRoot(); !args.IncludeStdlib && stdlibRoot != "" { + exclude = append(exclude, stdlibRoot) + } + results, err := h.store.SearchSymbols(query, exclude...) + if err != nil { + return "", fmt.Errorf("searching symbols: %w", err) + } + if len(results) == 0 { + return fmt.Sprintf("No symbols matched %q. Try a shorter or less specific query; matching is fuzzy on module and function names.", query), nil + } + + var b strings.Builder + for _, r := range results { + fmt.Fprintf(&b, "%s (%s) - %s:%d\n", symbolName(r.Module, r.Function, r.Arity), r.Kind, h.relPath(r.FilePath), r.Line) + } + return b.String(), nil +} diff --git a/internal/mcp/serve.go b/internal/mcp/serve.go new file mode 100644 index 0000000..0b148b5 --- /dev/null +++ b/internal/mcp/serve.go @@ -0,0 +1,94 @@ +package mcp + +import ( + "bytes" + "context" + "errors" + "fmt" + "io" + "net" + "net/http" + "time" + + "github.com/modelcontextprotocol/go-sdk/mcp" +) + +// RunStdio serves MCP over stdin/stdout until ctx is canceled or the client +// disconnects. +func RunStdio(ctx context.Context, f *Frontend) error { + return NewServer(f).Run(ctx, &mcp.StdioTransport{}) +} + +// httpSessionTimeout closes an HTTP session that sends no request for this +// long. A client that goes away without closing its session would otherwise +// keep its workspace connection, and with it the daemon, alive forever. A +// variable so tests can shrink it. +var httpSessionTimeout = 30 * time.Minute + +// maxHTTPBodyBytes caps one HTTP request body. MCP requests from an agent are +// small; the cap keeps one request from filling the server's memory. +const maxHTTPBodyBytes = 4 << 20 + +// HTTPHandler returns a streamable-HTTP handler serving MCP. Each session +// gets its own protocol server; they all share the Frontend. +// +// The SDK refuses a request that reaches a loopback address with a Host that +// is not loopback (DNS rebinding). Cross-origin browser requests are refused +// too, so a web page cannot drive the tools, and bodies are capped. +func HTTPHandler(f *Frontend) http.Handler { + h := mcp.NewStreamableHTTPHandler(func(*http.Request) *mcp.Server { return NewServer(f) }, &mcp.StreamableHTTPOptions{ + SessionTimeout: httpSessionTimeout, + }) + return limitBody(http.NewCrossOriginProtection().Handler(h), maxHTTPBodyBytes) +} + +// limitBody reads a request body of at most limit bytes before next sees the +// request, and answers a longer one with 413. The SDK turns a read error into +// 400, so the limit cannot be left to a reader that next drains. +func limitBody(next http.Handler, limit int64) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.ContentLength > limit { + http.Error(w, "request body too large", http.StatusRequestEntityTooLarge) + return + } + if r.Body != nil && r.Body != http.NoBody { + body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, limit)) + if err != nil { + var tooLarge *http.MaxBytesError + if errors.As(err, &tooLarge) { + http.Error(w, "request body too large", http.StatusRequestEntityTooLarge) + return + } + http.Error(w, "failed to read the request body", http.StatusBadRequest) + return + } + r.Body = io.NopCloser(bytes.NewReader(body)) + } + next.ServeHTTP(w, r) + }) +} + +// CheckListenAddr refuses an HTTP listen address that other machines can +// reach, because the server has no authentication: anyone who reaches it can +// read the code and rename symbols. allowRemote is the explicit way through. +func CheckListenAddr(addr string, allowRemote bool) error { + if allowRemote { + return nil + } + host, _, err := net.SplitHostPort(addr) + if err != nil { + return fmt.Errorf("invalid listen address %q: %w", addr, err) + } + if isLoopbackHost(host) { + return nil + } + return fmt.Errorf("refusing to listen on %q: the MCP HTTP server has no authentication, so it listens only on a loopback address (for example localhost:8080 or 127.0.0.1:8080). Pass --listen-unsafe to listen on %q anyway", addr, addr) +} + +func isLoopbackHost(host string) bool { + if host == "localhost" { + return true + } + ip := net.ParseIP(host) + return ip != nil && ip.IsLoopback() +} diff --git a/internal/mcp/source.go b/internal/mcp/source.go new file mode 100644 index 0000000..6353c6a --- /dev/null +++ b/internal/mcp/source.go @@ -0,0 +1,347 @@ +package mcp + +import ( + "errors" + "fmt" + "io" + "io/fs" + "os" + "path/filepath" + "sort" + "strings" + "sync" + "time" + + "github.com/remoteoss/dexter/internal/lsp" +) + +// maxSourceBytes caps the size of one file that a tool reads. Elixir source +// files are far smaller; the cap keeps a wrong path from filling the daemon's +// memory. +const maxSourceBytes = 10 << 20 + +// sourceCache holds the file text that one tool call read, so that each file +// is read and tokenized at most once per call, and records which files came +// from unsaved editor buffers. +type sourceCache struct { + mu sync.Mutex + views map[string]*sourceView // nil for a file that cannot be read + unsaved map[string]struct{} + // conflicts are files read from disk that also have older unsaved + // changes in an editor. + conflicts map[string]struct{} +} + +// errNotFound reports a file that does not exist. +var errNotFound = errors.New("file not found") + +// userPath resolves a file path that the agent gave. A relative path is +// resolved against the project root. The path, with every symlink resolved, +// must be inside the project root, also with its symlinks resolved. The +// result is spelled under the root as the daemon spells it, so that it names +// the same file as the index and the editors. +func (h *Handler) userPath(p string) (string, error) { + candidate := p + if !filepath.IsAbs(candidate) { + candidate = filepath.Join(h.projectRoot, candidate) + } + candidate = filepath.Clean(candidate) + root, err := filepath.EvalSymlinks(h.projectRoot) + if err != nil { + return "", fmt.Errorf("resolving the project root: %w", err) + } + real, err := filepath.EvalSymlinks(candidate) + if errors.Is(err, fs.ErrNotExist) { + // A file that is not on disk can still be open in an editor. Only a + // path inside the root as spelled can name one. + if rel, ok := inside(h.projectRoot, candidate); ok { + path := filepath.Join(h.projectRoot, rel) + if _, open := h.rt.UnsavedBuffer(path); open { + return path, nil + } + return path, errNotFound + } + return "", outsideRootError(p, h.projectRoot) + } + if err != nil { + return "", err + } + rel, ok := inside(root, real) + if !ok { + return "", outsideRootError(p, h.projectRoot) + } + return filepath.Join(h.projectRoot, rel), nil +} + +func outsideRootError(p, root string) error { + return fmt.Errorf("%s is outside the project root %s; dexter reads only files inside the project", p, root) +} + +// inside returns path relative to root when it is root or under it. +func inside(root, path string) (string, bool) { + rel, err := filepath.Rel(root, path) + if err != nil || filepath.IsAbs(rel) || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) { + return "", false + } + return rel, true +} + +// readRegularFile reads a regular file of at most maxSourceBytes. Devices, +// FIFOs, sockets, and directories are refused before they are opened, so a +// read can neither block nor run without end. +func readRegularFile(path string) (string, time.Time, error) { + info, err := os.Stat(path) + if err != nil { + if errors.Is(err, fs.ErrNotExist) { + return "", time.Time{}, errNotFound + } + return "", time.Time{}, err + } + if !info.Mode().IsRegular() { + return "", time.Time{}, fmt.Errorf("%s is not a regular file", path) + } + if info.Size() > maxSourceBytes { + return "", time.Time{}, fmt.Errorf("%s is %d bytes, more than the %d MB limit for one file", path, info.Size(), maxSourceBytes>>20) + } + f, err := os.Open(path) + if err != nil { + return "", time.Time{}, err + } + defer func() { _ = f.Close() }() + // The path can change between the stat and the open. + opened, err := f.Stat() + if err != nil || !os.SameFile(info, opened) { + return "", time.Time{}, fmt.Errorf("%s changed while it was read; retry", path) + } + data, err := io.ReadAll(io.LimitReader(f, maxSourceBytes+1)) + if err != nil { + return "", time.Time{}, err + } + if len(data) > maxSourceBytes { + return "", time.Time{}, fmt.Errorf("%s is more than the %d MB limit for one file", path, maxSourceBytes>>20) + } + return string(data), opened.ModTime(), nil +} + +// sourceView is one file as a tool call reads it. When an attached editor +// holds the file open with changes that are not saved, text is that buffer +// and disk is the saved file, which the index positions refer to. +type sourceView struct { + text string + unsaved bool + disk string + hasDisk bool + // lines maps the saved file's lines, which the index positions refer + // to, into the buffer. + lines lineMap + + tf, diskTF *lsp.TokenizedFile +} + +func newSourceView(text string, unsaved bool, disk string, hasDisk bool) *sourceView { + v := &sourceView{text: text, unsaved: unsaved, disk: disk, hasDisk: hasDisk} + if unsaved && hasDisk { + v.lines = newLineMap(disk, text) + } + return v +} + +// locate maps a 1-based line of the saved file, where the index puts a +// definition or a reference, to the same line in the text. inText is false +// when the buffer changed that line. +func (v *sourceView) locate(indexLine int) (line int, inText bool) { + if !v.unsaved || !v.hasDisk { + return indexLine, true + } + return v.lines.locate(indexLine) +} + +func (v *sourceView) tokenized(inText bool) *lsp.TokenizedFile { + if inText { + if v.tf == nil { + v.tf = lsp.NewTokenizedFile(v.text) + } + return v.tf + } + if v.diskTF == nil { + v.diskTF = lsp.NewTokenizedFile(v.disk) + } + return v.diskTF +} + +// sourceAt is one index position in a file as the user sees it. +type sourceAt struct { + v *sourceView + line int // the 1-based line to show + // changed is true when an unsaved buffer changed this line: the text and + // the line are the saved file's. + changed bool +} + +func (a sourceAt) lineText() (string, bool) { + if a.changed { + return nthLine(a.v.disk, a.line) + } + return nthLine(a.v.text, a.line) +} + +func (a sourceAt) tokenized() *lsp.TokenizedFile { return a.v.tokenized(!a.changed) } + +// label is the suffix of a location whose text did not come from the buffer +// the user sees. +func (a sourceAt) label() string { + if a.changed { + return " (saved text; this part is changed in an unsaved editor buffer)" + } + return "" +} + +// readSource returns the text of path as the user sees it: the newest buffer +// that an attached editor holds open when it differs from the disk, otherwise +// the file on disk. Text from an unsaved buffer is recorded, so the answer +// says so. +func (h *Handler) readSource(path string) (string, error) { + v, err := h.view(path) + if err != nil { + return "", err + } + return v.text, nil +} + +// at finds the index position path:indexLine in the text the user sees. +func (h *Handler) at(path string, indexLine int) (sourceAt, bool) { + v, err := h.view(path) + if err != nil { + return sourceAt{}, false + } + line, inText := v.locate(indexLine) + return sourceAt{v: v, line: line, changed: !inText}, true +} + +// view reads path once per tool call. +func (h *Handler) view(path string) (*sourceView, error) { + c := &h.sources + c.mu.Lock() + if cached, ok := c.views[path]; ok { + c.mu.Unlock() + if cached == nil { + return nil, errNotFound + } + return cached, nil + } + c.mu.Unlock() + + v, conflict, err := h.loadSource(path) + + c.mu.Lock() + defer c.mu.Unlock() + if c.views == nil { + c.views = make(map[string]*sourceView) + } + c.views[path] = v + if v != nil && v.unsaved { + if c.unsaved == nil { + c.unsaved = make(map[string]struct{}) + } + c.unsaved[path] = struct{}{} + } + if conflict { + if c.conflicts == nil { + c.conflicts = make(map[string]struct{}) + } + c.conflicts[path] = struct{}{} + } + return v, err +} + +// loadSource reads path as the user sees it. A buffer that an editor holds +// open is used only when it has changes that the editor has not saved and the +// file on disk did not change after them. When both changed, the disk wins, +// because the index follows the disk, and conflict is true. +func (h *Handler) loadSource(path string) (v *sourceView, conflict bool, err error) { + disk, mtime, diskErr := readRegularFile(path) + if buffer, open := h.rt.UnsavedBuffer(path); open { + switch { + case diskErr == nil && disk == buffer.Text: + return newSourceView(disk, false, "", false), false, nil + case diskErr == nil && mtime.After(buffer.ChangedAt): + return newSourceView(disk, false, "", false), true, nil + default: + return newSourceView(buffer.Text, true, disk, diskErr == nil), false, nil + } + } + if diskErr != nil { + return nil, false, diskErr + } + return newSourceView(disk, false, "", false), false, nil +} + +// sourceLine returns the text of one index position. A file that no editor +// holds open is scanned only up to the line, so a reference list does not +// read whole files. +func (h *Handler) sourceLine(path string, indexLine int) (text string, line int, label string, ok bool) { + c := &h.sources + c.mu.Lock() + _, cached := c.views[path] + c.mu.Unlock() + if !cached { + if _, open := h.rt.UnsavedBuffer(path); !open { + text, ok := h.lsp.FileLine(path, indexLine) + return text, indexLine, "", ok + } + } + a, ok := h.at(path, indexLine) + if !ok { + return "", 0, "", false + } + text, ok = a.lineText() + return text, a.line, a.label(), ok +} + +// nthLine returns the 1-based line n of text. +func nthLine(text string, n int) (string, bool) { + if n < 1 { + return "", false + } + for i := 1; ; i++ { + end := strings.IndexByte(text, '\n') + if i == n { + if end < 0 { + return strings.TrimSuffix(text, "\r"), true + } + return strings.TrimSuffix(text[:end], "\r"), true + } + if end < 0 { + return "", false + } + text = text[end+1:] + } +} + +// unsavedNote names the files whose text came from unsaved editor buffers in +// this call, and the files that changed on disk after an editor's unsaved +// changes to them, or is empty when there are none. +func (h *Handler) unsavedNote() string { + c := &h.sources + c.mu.Lock() + unsaved := h.relPaths(c.unsaved) + conflicts := h.relPaths(c.conflicts) + c.mu.Unlock() + var notes []string + if len(unsaved) > 0 { + notes = append(notes, "Note: read from unsaved editor buffers (the files on disk differ; line numbers are the buffer's): "+strings.Join(unsaved, ", ")) + } + if len(conflicts) > 0 { + notes = append(notes, "Note: read from disk, but an editor also has unsaved changes to these files, made before the files on disk changed; the two may conflict: "+strings.Join(conflicts, ", ")) + } + return strings.Join(notes, "\n") +} + +func (h *Handler) relPaths(set map[string]struct{}) []string { + out := make([]string, 0, len(set)) + for p := range set { + out = append(out, h.relPath(p)) + } + sort.Strings(out) + return out +} diff --git a/internal/mcp/source_test.go b/internal/mcp/source_test.go new file mode 100644 index 0000000..5789a02 --- /dev/null +++ b/internal/mcp/source_test.go @@ -0,0 +1,395 @@ +package mcp + +import ( + "context" + "fmt" + "math/rand/v2" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "go.lsp.dev/protocol" + "go.lsp.dev/uri" + + "github.com/remoteoss/dexter/internal/lsp" +) + +// Regression: dexter_file_outline read any path the agent gave, so an +// absolute path or one with .. could read every file the user can read, and +// /dev/zero filled the daemon's memory. +func TestFileOutline_RefusesPathsOutsideRoot(t *testing.T) { + e := setupProject(t) + outside := filepath.Join(t.TempDir(), "secret.ex") + if err := os.WriteFile(outside, []byte("defmodule Secret do\n def key, do: 1\nend\n"), 0644); err != nil { + t.Fatal(err) + } + if err := os.Symlink(outside, filepath.Join(e.root, "lib", "link.ex")); err != nil { + t.Fatal(err) + } + if err := os.Symlink("/dev/zero", filepath.Join(e.root, "lib", "zero.ex")); err != nil { + t.Fatal(err) + } + rel, err := filepath.Rel(e.root, outside) + if err != nil { + t.Fatal(err) + } + for _, p := range []string{outside, rel, "lib/link.ex", "/dev/zero", "lib/zero.ex", "../" + filepath.Base(e.root) + "x/a.ex"} { + errText := e.callToolExpectError("dexter_file_outline", map[string]any{"file": p}) + wantContains(t, errText, "outside the project root") + wantNotContains(t, errText, "Secret") + } + + // A symlink that stays inside the root is still fine. + if err := os.Symlink(filepath.Join(e.root, "lib/my_app/worker.ex"), filepath.Join(e.root, "lib", "alias.ex")); err != nil { + t.Fatal(err) + } + wantContains(t, e.callTool("dexter_file_outline", map[string]any{"file": "lib/alias.ex"}), "defmodule MyApp.Worker") +} + +func TestFileOutline_RefusesLargeFilesAndDirectories(t *testing.T) { + e := setupProject(t) + big := filepath.Join(e.root, "lib", "big.ex") + if err := os.WriteFile(big, make([]byte, maxSourceBytes+1), 0644); err != nil { + t.Fatal(err) + } + wantContains(t, e.callToolExpectError("dexter_file_outline", map[string]any{"file": "lib/big.ex"}), "MB limit") + wantContains(t, e.callToolExpectError("dexter_file_outline", map[string]any{"file": "lib"}), "not a regular file") + wantContains(t, e.callTool("dexter_file_outline", map[string]any{"file": "lib/missing.ex"}), "File not found: lib/missing.ex") +} + +// Regression: a function with thousands of clauses gave an answer of hundreds +// of kilobytes, and each clause read and tokenized the file again. +func TestDefinitionTool_CapsClauses(t *testing.T) { + e := setupProject(t) + var src strings.Builder + src.WriteString("defmodule MyApp.Big do\n") + for i := 0; i < 300; i++ { + fmt.Fprintf(&src, " def code(%d), do: %d\n", i, i) + } + src.WriteString("end\n") + e.indexFile("lib/my_app/big.ex", src.String()) + + out := e.callTool("dexter_definition", map[string]any{"module": "MyApp.Big", "function": "code"}) + if got := strings.Count(out, "MyApp.Big.code/1 (def)"); got != 20 { + t.Errorf("answer shows %d clauses, want 20:\n%s", got, out) + } + wantContains(t, out, "… and 280 more clause(s) not shown.") +} + +// MCP answers from what the user sees: a buffer that an attached editor holds +// open, with changes not yet saved, wins over the disk, and the answer says so. +// The index positions refer to the saved file, so they are mapped into the +// buffer. +func TestTools_SeeUnsavedEditorBuffers(t *testing.T) { + e := setupProject(t) + _, session, release := e.rt.AttachLSPSession() + t.Cleanup(release) + + change := func(rel, text string) { + t.Helper() + path := filepath.Join(e.root, rel) + if err := session.DidChange(context.Background(), &protocol.DidChangeTextDocumentParams{ + TextDocument: protocol.VersionedTextDocumentIdentifier{TextDocumentIdentifier: protocol.TextDocumentIdentifier{URI: protocol.DocumentURI(uri.File(path))}}, + ContentChanges: []protocol.TextDocumentContentChangeEvent{{Text: text}}, + }); err != nil { + t.Fatal(err) + } + } + + // A new module at the end of the buffer shows in the outline. + change("lib/my_app/worker.ex", workerSource+"\ndefmodule MyApp.Draft do\n def draft_only(x), do: x\nend\n") + out := e.callTool("dexter_file_outline", map[string]any{"file": "lib/my_app/worker.ex"}) + wantContains(t, out, "defmodule MyApp.Draft", "draft_only/1", "unsaved editor buffers", "lib/my_app/worker.ex") + + // Two lines inserted at the top move every reference down by two. + change("lib/my_app/worker.ex", "# draft\n# notes\n"+workerSource) + refs := e.callTool("dexter_references", map[string]any{"module": "MyApp.Accounts", "function": "fetch_user"}) + wantContains(t, refs, "5: MyApp.Accounts.fetch_user(1)", "10: MyApp.Accounts.fetch_user(2)", "unsaved editor buffers") + + // A reference on a line that the buffer changed shows the saved line, + // marked as such. + change("lib/my_app/worker.ex", strings.Replace(workerSource, "MyApp.Accounts.fetch_user(1)", "MyApp.Accounts.fetch_user(id)", 1)) + refs = e.callTool("dexter_references", map[string]any{"module": "MyApp.Accounts", "function": "fetch_user"}) + wantContains(t, refs, "3: MyApp.Accounts.fetch_user(1) (saved text; this part is changed in an unsaved editor buffer)", "8: MyApp.Accounts.fetch_user(2)\n") + + // A definition below an inserted function: the head, @spec, and @doc come + // from the buffer at the moved line. + change("lib/my_app/accounts.ex", strings.Replace(accountsSource, " @doc \"\"\"\n Fetches", " def helper, do: :ok\n\n @doc \"\"\"\n Fetches a user, unsaved.\n Fetches", 1)) + def := e.callTool("dexter_definition", map[string]any{"module": "MyApp.Accounts", "function": "fetch_user"}) + wantContains(t, def, "lib/my_app/accounts.ex:13", "@spec fetch_user(integer())", "def fetch_user(id) do", "Fetches a user, unsaved.", "unsaved editor buffers") + + // A buffer that matches the disk is not unsaved. + change("lib/my_app/worker.ex", workerSource) + change("lib/my_app/accounts.ex", accountsSource) + out = e.callTool("dexter_file_outline", map[string]any{"file": "lib/my_app/worker.ex"}) + wantNotContains(t, out, "MyApp.Draft", "unsaved") + def = e.callTool("dexter_definition", map[string]any{"module": "MyApp.Accounts", "function": "fetch_user"}) + wantContains(t, def, "lib/my_app/accounts.ex:10", "def fetch_user(id) do") + wantNotContains(t, def, "unsaved") +} + +func TestSourceViewLocate(t *testing.T) { + disk := "a\nb\nc\nd\ne" + v := newSourceView("a\nX\nY\nc\nd\ne", true, disk, true) // b replaced by X, Y + for line, want := range map[int][2]int{1: {1, 1}, 2: {2, 0}, 3: {4, 1}, 5: {6, 1}} { + got, in := v.locate(line) + if in != (want[1] == 1) || (in && got != want[0]) { + t.Errorf("locate(%d) = %d, %v; want %v", line, got, in, want) + } + } +} + +// editorSession is an attached editor session that a test drives. +type editorSession struct { + t *testing.T + root string + srv *lsp.Server +} + +func attachEditor(t *testing.T, e *testEnv) *editorSession { + t.Helper() + _, session, release := e.rt.AttachLSPSession() + t.Cleanup(release) + return &editorSession{t: t, root: e.root, srv: session} +} + +func (s *editorSession) uri(rel string) protocol.DocumentURI { + return protocol.DocumentURI(uri.File(filepath.Join(s.root, rel))) +} + +func (s *editorSession) open(rel, text string) { + s.t.Helper() + if err := s.srv.DidOpen(context.Background(), &protocol.DidOpenTextDocumentParams{ + TextDocument: protocol.TextDocumentItem{URI: s.uri(rel), LanguageID: "elixir", Version: 1, Text: text}, + }); err != nil { + s.t.Fatal(err) + } +} + +func (s *editorSession) change(rel, text string) { + s.t.Helper() + if err := s.srv.DidChange(context.Background(), &protocol.DidChangeTextDocumentParams{ + TextDocument: protocol.VersionedTextDocumentIdentifier{TextDocumentIdentifier: protocol.TextDocumentIdentifier{URI: s.uri(rel)}}, + ContentChanges: []protocol.TextDocumentContentChangeEvent{{Text: text}}, + }); err != nil { + s.t.Fatal(err) + } +} + +func (s *editorSession) save(rel string) { + s.t.Helper() + if err := s.srv.DidSave(context.Background(), &protocol.DidSaveTextDocumentParams{ + TextDocument: protocol.TextDocumentIdentifier{URI: s.uri(rel)}, + }); err != nil { + s.t.Fatal(err) + } +} + +// writeLater writes a file with a modification time after every editor change +// so far, as an agent's write that follows them does. +func writeLater(t *testing.T, path, text string) { + t.Helper() + time.Sleep(20 * time.Millisecond) + if err := os.WriteFile(path, []byte(text), 0644); err != nil { + t.Fatal(err) + } +} + +const agentWorkerSource = "# agent\n# notes\n" + workerSource + "\ndefmodule MyApp.AgentAdded do\n def added(x), do: x\nend\n" + +// Regression: an editor buffer that the user never changed, but that is older +// than the disk (the editor has not reloaded the file after the agent wrote +// it), hid the agent's edits and was reported as unsaved work. +func TestTools_CleanStaleBufferReadsDisk(t *testing.T) { + e := setupProject(t) + ed := attachEditor(t, e) + ed.open("lib/my_app/worker.ex", workerSource) + path := filepath.Join(e.root, "lib/my_app/worker.ex") + writeLater(t, path, agentWorkerSource) + + out := e.callTool("dexter_file_outline", map[string]any{"file": "lib/my_app/worker.ex"}) + wantContains(t, out, "defmodule MyApp.AgentAdded") + wantNotContains(t, out, "unsaved") +} + +// A buffer with unsaved changes, made after the disk last changed, is what the +// user sees, so it wins. +func TestTools_DirtyBufferWins(t *testing.T) { + e := setupProject(t) + ed := attachEditor(t, e) + ed.open("lib/my_app/worker.ex", workerSource) + ed.change("lib/my_app/worker.ex", workerSource+"\ndefmodule MyApp.Typed do\nend\n") + out := e.callTool("dexter_file_outline", map[string]any{"file": "lib/my_app/worker.ex"}) + wantContains(t, out, "defmodule MyApp.Typed", "read from unsaved editor buffers") +} + +// When both the buffer and the disk changed, and the disk changed later, the +// disk wins (the index follows it), and the answer warns about the editor's +// older unsaved changes. +func TestTools_DirtyBufferOlderThanDiskReadsDiskWithNote(t *testing.T) { + e := setupProject(t) + ed := attachEditor(t, e) + ed.open("lib/my_app/worker.ex", workerSource) + ed.change("lib/my_app/worker.ex", workerSource+"\ndefmodule MyApp.Typed do\nend\n") + writeLater(t, filepath.Join(e.root, "lib/my_app/worker.ex"), agentWorkerSource) + + out := e.callTool("dexter_file_outline", map[string]any{"file": "lib/my_app/worker.ex"}) + wantContains(t, out, "defmodule MyApp.AgentAdded", "an editor also has unsaved changes", "may conflict") + wantNotContains(t, out, "MyApp.Typed", "read from unsaved editor buffers") +} + +// A save ends the buffer's unsaved changes: a later write by the agent is +// read from disk with no warning. +func TestTools_SaveClearsUnsavedState(t *testing.T) { + e := setupProject(t) + ed := attachEditor(t, e) + path := filepath.Join(e.root, "lib/my_app/worker.ex") + ed.open("lib/my_app/worker.ex", workerSource) + edited := workerSource + "\ndefmodule MyApp.Typed do\nend\n" + ed.change("lib/my_app/worker.ex", edited) + if err := os.WriteFile(path, []byte(edited), 0644); err != nil { + t.Fatal(err) + } + ed.save("lib/my_app/worker.ex") + writeLater(t, path, agentWorkerSource) + + out := e.callTool("dexter_file_outline", map[string]any{"file": "lib/my_app/worker.ex"}) + wantContains(t, out, "defmodule MyApp.AgentAdded") + wantNotContains(t, out, "unsaved") +} + +// Regression: lines were mapped through the lines that both texts share at +// the start and at the end, so an edit at the top and one at the bottom +// marked every line between them as changed, and a buffer with CRLF line +// endings marked every line. +func TestLineMap(t *testing.T) { + disk := "a\nb\nc\nd\ne\nf" + m := newLineMap(disk, "TOP\na\nb\nc\nd\ne\nBOTTOM") // f changed, line added at top + for line, want := range map[int]int{1: 2, 2: 3, 3: 4, 4: 5, 5: 6} { + if got, ok := m.locate(line); !ok || got != want { + t.Errorf("locate(%d) = %d, %v; want %d", line, got, ok, want) + } + } + if _, ok := m.locate(6); ok { + t.Error("the changed last line is mapped") + } + + crlf := newLineMap(disk, strings.ReplaceAll(disk, "\n", "\r\n")) + for line := 1; line <= 6; line++ { + if got, ok := crlf.locate(line); !ok || got != line { + t.Errorf("CRLF: locate(%d) = %d, %v; want %d", line, got, ok, line) + } + } + + // A deleted line in the middle, and a changed one. + mid := newLineMap(disk, "a\nc\nD\ne\nf") + for line, want := range map[int]int{1: 1, 3: 2, 5: 4, 6: 5} { + if got, ok := mid.locate(line); !ok || got != want { + t.Errorf("mid: locate(%d) = %d, %v; want %d", line, got, ok, want) + } + } + for _, line := range []int{2, 4} { + if _, ok := mid.locate(line); ok { + t.Errorf("mid: changed line %d is mapped", line) + } + } +} + +// The diff is exact on random edits: every mapped pair is equal, mapped lines +// keep their order, and no more lines are left unmapped than a shortest edit +// script deletes. +func TestLineMapRandomEdits(t *testing.T) { + rng := rand.New(rand.NewPCG(1, 2)) + for round := 0; round < 200; round++ { + var a []string + for i := 0; i < 1+rng.IntN(60); i++ { + a = append(a, fmt.Sprintf("l%d", rng.IntN(8))) + } + var b []string + for _, l := range a { + switch rng.IntN(6) { + case 0: // delete + case 1: + b = append(b, "new", l) + case 2: + b = append(b, "changed") + default: + b = append(b, l) + } + } + m := newLineMap(strings.Join(a, "\n"), strings.Join(b, "\n")) + last := 0 + mapped := 0 + for i := 1; i <= len(a); i++ { + j, ok := m.locate(i) + if !ok { + continue + } + mapped++ + if j <= last || a[i-1] != b[j-1] { + t.Fatalf("round %d: line %d mapped to %d (last %d): %q vs %q", round, i, j, last, a[i-1], b[j-1]) + } + last = j + } + if want := lcsLen(a, b); mapped != want { + t.Fatalf("round %d: %d lines mapped, want the LCS length %d", round, mapped, want) + } + } +} + +func lcsLen(a, b []string) int { + dp := make([][]int, len(a)+1) + for i := range dp { + dp[i] = make([]int, len(b)+1) + } + for i := 1; i <= len(a); i++ { + for j := 1; j <= len(b); j++ { + if a[i-1] == b[j-1] { + dp[i][j] = dp[i-1][j-1] + 1 + } else { + dp[i][j] = max(dp[i-1][j], dp[i][j-1]) + } + } + } + return dp[len(a)][len(b)] +} + +// BenchmarkLineMap10k measures a 10,000-line file with edits at the top, in +// the middle, and at the bottom. +func BenchmarkLineMap10k(b *testing.B) { + var lines []string + for i := 0; i < 10000; i++ { + lines = append(lines, fmt.Sprintf(" def f%d(x), do: x + %d", i, i)) + } + disk := strings.Join(lines, "\n") + edited := append([]string{"# top"}, lines...) + edited[5000] = " # changed" + edited = append(edited[:7000], append([]string{" def added(x), do: x"}, edited[7000:]...)...) + edited[len(edited)-1] = "# bottom" + text := strings.Join(edited, "\n") + b.ResetTimer() + for i := 0; i < b.N; i++ { + newLineMap(disk, text) + } +} + +// BenchmarkLineMap10kManyEdits is a 10,000-line file with edits near the +// bound (about 900 inserted and deleted lines). +func BenchmarkLineMap10kManyEdits(b *testing.B) { + var lines []string + for i := 0; i < 10000; i++ { + lines = append(lines, fmt.Sprintf(" def f%d(x), do: x + %d", i, i)) + } + disk := strings.Join(lines, "\n") + edited := append([]string(nil), lines...) + for i := 0; i < len(edited); i += 22 { + edited[i] = " # changed" + } + text := strings.Join(edited, "\n") + b.ResetTimer() + for i := 0; i < b.N; i++ { + newLineMap(disk, text) + } +} diff --git a/internal/mcp/source_unix_test.go b/internal/mcp/source_unix_test.go new file mode 100644 index 0000000..18049dc --- /dev/null +++ b/internal/mcp/source_unix_test.go @@ -0,0 +1,18 @@ +//go:build unix + +package mcp + +import ( + "path/filepath" + "syscall" + "testing" +) + +// Regression: a FIFO blocked the read forever. +func TestFileOutline_RefusesFIFO(t *testing.T) { + e := setupProject(t) + if err := syscall.Mkfifo(filepath.Join(e.root, "lib", "pipe.ex"), 0644); err != nil { + t.Fatal(err) + } + wantContains(t, e.callToolExpectError("dexter_file_outline", map[string]any{"file": "lib/pipe.ex"}), "not a regular file") +} diff --git a/internal/mcp/tools.go b/internal/mcp/tools.go new file mode 100644 index 0000000..3172c60 --- /dev/null +++ b/internal/mcp/tools.go @@ -0,0 +1,262 @@ +// Package mcp implements dexter's Model Context Protocol frontend. It exposes +// the index as a set of coarse, agent-oriented tools (modeled on gopls mcp), +// addressed by module/function name rather than file positions because Elixir +// modules are not tied to files. +// +// The package has two halves. The frontend (frontend.go) speaks MCP to the +// agent and owns nothing of the workspace: for each workspace root it connects +// to the shared workspace daemon, like the editor and the CLI. The tool bodies +// (this file and one file per tool) run inside the daemon, as the control +// method MethodTool, against the daemon's store and headless language service. +package mcp + +import ( + "context" + _ "embed" + "encoding/json" + "errors" + "fmt" + "path/filepath" + "strings" + "time" + + "github.com/modelcontextprotocol/go-sdk/mcp" + + "github.com/remoteoss/dexter/internal/lsp" + "github.com/remoteoss/dexter/internal/notify" + "github.com/remoteoss/dexter/internal/store" + "github.com/remoteoss/dexter/internal/workspace" +) + +// Instructions is the agent-facing usage guide, offered to MCP clients via the +// server's instructions field and printable with `dexter mcp --instructions`. +// +//go:embed instructions.md +var Instructions string + +// Handler runs tool bodies against one workspace. In production it lives in +// the workspace daemon; tests build it over an in-process runtime. +type Handler struct { + rt *workspace.Runtime + lsp *lsp.Server + store *store.Store + projectRoot string + + // sources caches the files that one tool call reads. Call gives each call + // its own Handler, so the cache never outlives the call. + sources sourceCache +} + +// NewHandler returns a Handler over the runtime's store and the given language +// service. +func NewHandler(rt *workspace.Runtime, server *lsp.Server) *Handler { + return &Handler{rt: rt, lsp: server, store: rt.Store(), projectRoot: rt.Root()} +} + +const renameToolName = "dexter_rename_symbol" + +// toolSpec is one tool: its MCP declaration for the frontend and its body for +// the daemon. The two halves share the parameter type, so the input schema the +// agent sees and the arguments the body decodes cannot drift apart. +type toolSpec struct { + tool mcp.Tool + register func(srv *mcp.Server, f *Frontend, t mcp.Tool) + run func(h *Handler, ctx context.Context, raw json.RawMessage) (string, error) + // ownStatus is true for a tool whose answer already describes the index + // state, so Call adds no notes to it. + ownStatus bool +} + +func newToolSpec[In any](t mcp.Tool, body func(*Handler, context.Context, In) (string, error)) toolSpec { + return toolSpec{ + tool: t, + register: func(srv *mcp.Server, f *Frontend, t mcp.Tool) { addTool[In](srv, f, t) }, + run: func(h *Handler, ctx context.Context, raw json.RawMessage) (string, error) { + var in In + if len(raw) > 0 && string(raw) != "null" { + if err := json.Unmarshal(raw, &in); err != nil { + return "", fmt.Errorf("invalid arguments for %s: %w", t.Name, err) + } + } + return body(h, ctx, in) + }, + } +} + +// toolSpecs lists every tool in the order that tools/list shows them. +func toolSpecs() []toolSpec { + // The pointer hints distinguish explicit false from unset; clients must + // treat unset pessimistically (destructive, open world). + readOnly := &mcp.ToolAnnotations{ReadOnlyHint: true, OpenWorldHint: new(bool)} + + workspaceSpec := newToolSpec(mcp.Tool{ + Name: "dexter_workspace", + Annotations: readOnly, + Description: "Overview of the Elixir workspace: Mix projects, index size and state, stdlib status, and active warnings. Call once at the start of Elixir work.", + }, (*Handler).workspace) + workspaceSpec.ownStatus = true + + return []toolSpec{ + workspaceSpec, + newToolSpec(mcp.Tool{ + Name: "dexter_search", + Annotations: readOnly, + Description: "Locate Elixir modules and functions by fuzzy name match. More precise than grep for finding symbols: results are exact definitions with file:line.", + }, (*Handler).search), + newToolSpec(mcp.Tool{ + Name: "dexter_definition", + Annotations: readOnly, + Description: "Definition of an Elixir module or function by name: location, @doc/@spec, and source snippet, following defdelegate to the real implementation. Use instead of grep or reading files to answer where or what a symbol is.", + }, (*Handler).definition), + newToolSpec(mcp.Tool{ + Name: "dexter_references", + Annotations: readOnly, + Description: "All call sites of an Elixir module or function, resolved through aliases, imports, and use-chain injection that grep cannot see. Use for any 'who calls or uses X' question.", + }, (*Handler).references), + newToolSpec(mcp.Tool{ + Name: "dexter_module_api", + Annotations: readOnly, + Description: "A module's public API in one call: moduledoc, functions with signatures and doc summaries, macros, delegates, types, callbacks, and submodules. Use before reading a module's source.", + }, (*Handler).moduleAPI), + newToolSpec(mcp.Tool{ + Name: "dexter_file_outline", + Annotations: readOnly, + Description: "Everything an Elixir file defines: modules, functions, macros, and types with line numbers. Use instead of reading a file to map its contents; one Elixir file can define many modules.", + }, (*Handler).fileOutline), + newToolSpec(mcp.Tool{ + Name: "dexter_implementations", + Annotations: readOnly, + Description: "Implementations of an Elixir behaviour (@behaviour/use) or protocol (defimpl), optionally locating one callback in each implementor. Grep cannot resolve these relationships.", + }, (*Handler).implementations), + newToolSpec(mcp.Tool{ + Name: "dexter_call_hierarchy", + Annotations: readOnly, + Description: "Incoming callers and outgoing callees of an Elixir function, with file:line locations. Use to trace execution paths without reading files.", + }, (*Handler).callHierarchy), + newToolSpec(mcp.Tool{ + Name: "dexter_reindex", + Annotations: &mcp.ToolAnnotations{DestructiveHint: new(bool), IdempotentHint: true, OpenWorldHint: new(bool)}, + Description: "Force an immediate incremental reindex. The index already updates automatically as files change; use this only when a lookup seems stale. It writes only dexter's own index database.", + }, (*Handler).reindex), + newToolSpec(mcp.Tool{ + Name: renameToolName, + Description: "Rename an Elixir module or function across the whole workspace, exactly like an editor rename: writes the changes to disk, moves files that follow the naming convention, and updates the index. Reports every file changed; review with git diff.", + Annotations: &mcp.ToolAnnotations{OpenWorldHint: new(bool)}, + }, (*Handler).rename), + } +} + +// Call runs one tool. It first waits up to wait for the workspace's initial +// index, and then adds a note when the answer comes from an index that is +// still building or that is degraded, so the agent does not take an +// incomplete answer as complete. +func (h *Handler) Call(ctx context.Context, name string, args json.RawMessage, wait time.Duration) (string, error) { + var spec *toolSpec + for _, s := range toolSpecs() { + if s.tool.Name == name { + spec = &s + break + } + } + if spec == nil { + return "", fmt.Errorf("unknown tool %q", name) + } + if wait > 0 && !h.rt.IsReady() { + waitCtx, cancel := context.WithTimeout(ctx, wait) + _ = h.rt.WaitReady(waitCtx) + cancel() + } + + // A call canceled during the wait does not run: a rename must not start + // its writes after the client was told that it may have been applied. + if err := ctx.Err(); err != nil { + return "", err + } + call := &Handler{rt: h.rt, lsp: h.lsp, store: h.store, projectRoot: h.projectRoot} + text, err := spec.run(call, ctx, args) + var noteList []string + if !spec.ownStatus { + if n := h.indexNotes(); n != "" { + noteList = append(noteList, n) + } + } + if n := call.unsavedNote(); n != "" { + noteList = append(noteList, n) + } + notes := strings.Join(noteList, "\n") + if err != nil { + if notes != "" { + return "", fmt.Errorf("%w\n%s", err, notes) + } + return "", err + } + if notes != "" { + text = strings.TrimRight(text, "\n") + "\n\n" + notes + } + return text, nil +} + +// indexNotes describes an index that is still building, or that has a warning +// or an error, the same conditions that an editor shows and that `dexter +// lookup` prints. It is empty for a ready index with no such condition. +func (h *Handler) indexNotes() string { + var notes []string + if !h.rt.IsReady() { + notes = append(notes, "Note: the workspace index is still building, so this answer can be incomplete. Retry shortly for complete results.") + } + for _, c := range h.rt.IndexConditions() { + if c.Severity == notify.Info { + continue + } + notes = append(notes, fmt.Sprintf("Note (%s): %s", c.Severity, strings.TrimPrefix(c.Message, "Dexter: "))) + } + return strings.Join(notes, "\n") +} + +// renameAllowed refuses a rename while the index is incomplete: the rename +// finds its sites in the index, so it would change some call sites and leave +// others with the old name. +func (h *Handler) renameAllowed() error { + if !h.rt.IsReady() { + return errors.New("the workspace index is still building, so a rename now could miss call sites. Nothing was changed. Retry when dexter_workspace shows the index as ready") + } + for _, c := range h.rt.IndexConditions() { + switch c.Key { + case lsp.CondIndexBuild, lsp.CondIndexRebuild, lsp.CondIndexUnavailable, lsp.CondIndexFallback: + return fmt.Errorf("the index is not complete, so a rename now could miss call sites. Nothing was changed. %s", strings.TrimPrefix(c.Message, "Dexter: ")) + } + } + return nil +} + +// relPath renders p relative to the project root when it is inside it. +func (h *Handler) relPath(p string) string { + if rel, err := filepath.Rel(h.projectRoot, p); err == nil && !strings.HasPrefix(rel, "..") { + return rel + } + return p +} + +// symbolName renders Module.function/arity (or just the module name). +func symbolName(module, function string, arity int) string { + if function == "" { + return module + } + return fmt.Sprintf("%s.%s/%d", module, function, arity) +} + +// firstDocLine returns the first non-empty line of a doc string, truncated. +func firstDocLine(doc string) string { + for _, line := range strings.Split(doc, "\n") { + line = strings.TrimSpace(line) + if line == "" { + continue + } + const max = 120 + if len(line) > max { + return line[:max-3] + "..." + } + return line + } + return "" +} diff --git a/internal/mcp/tools_test.go b/internal/mcp/tools_test.go new file mode 100644 index 0000000..e6d9be8 --- /dev/null +++ b/internal/mcp/tools_test.go @@ -0,0 +1,421 @@ +package mcp + +import ( + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/remoteoss/dexter/internal/lsp" + "github.com/remoteoss/dexter/internal/notify" + "github.com/remoteoss/dexter/internal/workspace" +) + +const accountsSource = `defmodule MyApp.Accounts do + @moduledoc """ + The accounts context. + """ + + @doc """ + Fetches a user by id. + """ + @spec fetch_user(integer()) :: {:ok, map()} | {:error, :not_found} + def fetch_user(id) do + {:ok, %{id: id}} + end + + def list_users(opts) do + opts + end + + defp validate(id), do: id + + defdelegate create_user(attrs), to: MyApp.Accounts.Creator, as: :create + + @type user_id :: integer() +end +` + +const creatorSource = `defmodule MyApp.Accounts.Creator do + def create(attrs) do + attrs + end +end +` + +const workerSource = `defmodule MyApp.Worker do + def run do + MyApp.Accounts.fetch_user(1) + end + + def run_all do + MyApp.Accounts.list_users([]) + MyApp.Accounts.fetch_user(2) + end +end +` + +func setupProject(t *testing.T) *testEnv { + t.Helper() + e := setupTestEnv(t) + e.indexFile("mix.exs", "defmodule MyApp.MixProject do\nend\n") + e.indexFile("lib/my_app/accounts.ex", accountsSource) + e.indexFile("lib/my_app/accounts/creator.ex", creatorSource) + e.indexFile("lib/my_app/worker.ex", workerSource) + return e +} + +func TestWorkspaceTool(t *testing.T) { + e := setupProject(t) + out := e.callTool("dexter_workspace", nil) + wantContains(t, out, + "Project root: "+e.root, + "mix.exs", + "definitions", + "references", + ) +} + +func TestSearchTool(t *testing.T) { + e := setupProject(t) + + out := e.callTool("dexter_search", map[string]any{"query": "fetch_user"}) + wantContains(t, out, "MyApp.Accounts.fetch_user/1", "lib/my_app/accounts.ex") + + out = e.callTool("dexter_search", map[string]any{"query": "zzz_nothing_matches"}) + wantContains(t, out, "No symbols matched") + + errText := e.callToolExpectError("dexter_search", map[string]any{"query": " "}) + wantContains(t, errText, "query must not be empty") +} + +func TestDefinitionTool_Function(t *testing.T) { + e := setupProject(t) + out := e.callTool("dexter_definition", map[string]any{"module": "MyApp.Accounts", "function": "fetch_user"}) + wantContains(t, out, + "MyApp.Accounts.fetch_user/1 (def)", + "lib/my_app/accounts.ex:10", + "@spec fetch_user(integer())", + "Fetches a user by id.", + "def fetch_user(id) do", + ) +} + +func TestDefinitionTool_FollowsDelegate(t *testing.T) { + e := setupProject(t) + out := e.callTool("dexter_definition", map[string]any{"module": "MyApp.Accounts", "function": "create_user"}) + wantContains(t, out, + "(defdelegate)", + "Delegates to MyApp.Accounts.Creator.create", + "lib/my_app/accounts/creator.ex", + ) +} + +func TestDefinitionTool_Module(t *testing.T) { + e := setupProject(t) + out := e.callTool("dexter_definition", map[string]any{"module": "MyApp.Accounts"}) + wantContains(t, out, + "defmodule MyApp.Accounts - lib/my_app/accounts.ex:1", + "The accounts context.", + ) +} + +func TestDefinitionTool_NotFound(t *testing.T) { + e := setupProject(t) + out := e.callTool("dexter_definition", map[string]any{"module": "MyApp.Missing"}) + wantContains(t, out, "not in the index") +} + +func TestReferencesTool(t *testing.T) { + e := setupProject(t) + out := e.callTool("dexter_references", map[string]any{"module": "MyApp.Accounts", "function": "fetch_user"}) + wantContains(t, out, + "reference(s) to MyApp.Accounts.fetch_user", + "lib/my_app/worker.ex", + "MyApp.Accounts.fetch_user(1)", + "MyApp.Accounts.fetch_user(2)", + ) +} + +func TestReferencesTool_DelegateFacade(t *testing.T) { + e := setupProject(t) + // Calls to the facade MyApp.Accounts.create_user should count as + // references to the delegate target Creator.create. + e.indexFile("lib/my_app/caller.ex", `defmodule MyApp.Caller do + def go(attrs) do + MyApp.Accounts.create_user(attrs) + end +end +`) + out := e.callTool("dexter_references", map[string]any{"module": "MyApp.Accounts.Creator", "function": "create"}) + wantContains(t, out, "lib/my_app/caller.ex") +} + +func TestReferencesTool_Truncation(t *testing.T) { + e := setupProject(t) + var b strings.Builder + b.WriteString("defmodule MyApp.Spammy do\n def go do\n") + for i := 0; i < maxReferenceLines+20; i++ { + fmt.Fprintf(&b, " MyApp.Accounts.list_users(%d)\n", i) + } + b.WriteString(" end\nend\n") + e.indexFile("lib/my_app/spammy.ex", b.String()) + + out := e.callTool("dexter_references", map[string]any{"module": "MyApp.Accounts", "function": "list_users"}) + wantContains(t, out, "more reference(s) not shown") +} + +func TestModuleAPITool(t *testing.T) { + e := setupProject(t) + out := e.callTool("dexter_module_api", map[string]any{"module": "MyApp.Accounts"}) + wantContains(t, out, + "module MyApp.Accounts - lib/my_app/accounts.ex:1", + "The accounts context.", + "Functions:", + "fetch_user(id)", + "Fetches a user by id.", + "Delegates:", + "create_user(attrs)", + "→ MyApp.Accounts.Creator.create", + "Types:", + "user_id/0", + "Submodules", + "Creator", + ) + wantNotContains(t, out, "validate") + + out = e.callTool("dexter_module_api", map[string]any{"module": "MyApp.Accounts", "include_private": true}) + wantContains(t, out, "validate") +} + +func TestFileOutlineTool(t *testing.T) { + e := setupProject(t) + out := e.callTool("dexter_file_outline", map[string]any{"file": "lib/my_app/accounts.ex"}) + wantContains(t, out, + "defmodule MyApp.Accounts (line 1)", + "def fetch_user/1", + "defp validate/1", + "defdelegate create_user/1", + "→ MyApp.Accounts.Creator.create", + "@type user_id/0", + ) + + out = e.callTool("dexter_file_outline", map[string]any{"file": "lib/nope.ex"}) + wantContains(t, out, "File not found") +} + +func TestFileOutlineTool_NestedModules(t *testing.T) { + e := setupProject(t) + e.indexFile("lib/my_app/outer.ex", `defmodule MyApp.Outer do + def outer_fun, do: :ok + + defmodule Inner do + def inner_fun, do: :ok + end +end +`) + out := e.callTool("dexter_file_outline", map[string]any{"file": "lib/my_app/outer.ex"}) + wantContains(t, out, + "defmodule MyApp.Outer (line 1)", + "defmodule MyApp.Outer.Inner (line 4)", + "def inner_fun/0", + ) +} + +// The outline parses the file on disk, so it is correct before the index has +// seen the change. +func TestFileOutlineTool_ReadsDiskNotIndex(t *testing.T) { + e := setupProject(t) + path := filepath.Join(e.root, "lib/my_app/accounts.ex") + changed := `defmodule MyApp.Accounts do + def not_yet_indexed, do: :ok +end +` + if err := os.WriteFile(path, []byte(changed), 0644); err != nil { + t.Fatal(err) + } + + out := e.callTool("dexter_file_outline", map[string]any{"file": "lib/my_app/accounts.ex"}) + wantContains(t, out, "def not_yet_indexed/0") + wantNotContains(t, out, "def fetch_user/1") +} + +func TestImplementationsTool_Behaviour(t *testing.T) { + e := setupProject(t) + e.indexFile("lib/my_app/notifier.ex", `defmodule MyApp.Notifier do + @callback deliver(map()) :: :ok | {:error, term()} +end +`) + e.indexFile("lib/my_app/email_notifier.ex", `defmodule MyApp.EmailNotifier do + @behaviour MyApp.Notifier + + @impl true + def deliver(msg) do + :ok + end +end +`) + out := e.callTool("dexter_implementations", map[string]any{"module": "MyApp.Notifier"}) + wantContains(t, out, + "Modules implementing behaviour MyApp.Notifier", + "MyApp.EmailNotifier", + "@callback deliver/1", + ) + + out = e.callTool("dexter_implementations", map[string]any{"module": "MyApp.Notifier", "function": "deliver"}) + wantContains(t, out, + "Implementations of callback MyApp.Notifier.deliver", + "MyApp.EmailNotifier.deliver/1", + "lib/my_app/email_notifier.ex", + ) +} + +func TestImplementationsTool_Protocol(t *testing.T) { + e := setupProject(t) + e.indexFile("lib/my_app/size.ex", `defprotocol MyApp.Size do + def size(data) +end +`) + e.indexFile("lib/my_app/size_impls.ex", `defimpl MyApp.Size, for: BitString do + def size(binary), do: byte_size(binary) +end + +defimpl MyApp.Size, for: Map do + def size(map), do: map_size(map) +end +`) + out := e.callTool("dexter_implementations", map[string]any{"module": "MyApp.Size"}) + wantContains(t, out, + "MyApp.Size is a protocol", + "Implementations (2)", + "lib/my_app/size_impls.ex:1", + "lib/my_app/size_impls.ex:5", + ) + + out = e.callTool("dexter_implementations", map[string]any{"module": "MyApp.Size", "function": "size"}) + wantContains(t, out, + "Implementations of protocol function MyApp.Size.size", + "lib/my_app/size_impls.ex:2", + "lib/my_app/size_impls.ex:6", + ) + wantNotContains(t, out, "lib/my_app/size_impls.ex:1", "lib/my_app/size_impls.ex:5") +} + +func TestCallHierarchyTool(t *testing.T) { + e := setupProject(t) + out := e.callTool("dexter_call_hierarchy", map[string]any{"module": "MyApp.Accounts", "function": "fetch_user"}) + wantContains(t, out, + "Call hierarchy for MyApp.Accounts.fetch_user", + "Incoming (callers)", + "MyApp.Worker.run/0", + "lib/my_app/worker.ex", + ) + + out = e.callTool("dexter_call_hierarchy", map[string]any{"module": "MyApp.Worker", "function": "run", "direction": "outgoing"}) + wantContains(t, out, "Outgoing (callees)", "MyApp.Accounts.fetch_user") + wantNotContains(t, out, "Incoming") + + errText := e.callToolExpectError("dexter_call_hierarchy", map[string]any{"module": "MyApp.Worker", "function": "run", "direction": "sideways"}) + wantContains(t, errText, "direction must be") +} + +func TestReindexTool(t *testing.T) { + e := setupProject(t) + + // Write a new file WITHOUT indexing it: the tool must pick it up. + path := filepath.Join(e.root, "lib/my_app/fresh.ex") + if err := os.WriteFile(path, []byte("defmodule MyApp.Fresh do\n def new_fun, do: :ok\nend\n"), 0644); err != nil { + t.Fatal(err) + } + + out := e.callTool("dexter_reindex", nil) + wantContains(t, out, "Reindexed the workspace", "up to date") + + out = e.callTool("dexter_search", map[string]any{"query": "new_fun"}) + wantContains(t, out, "MyApp.Fresh.new_fun/0") +} + +// A tool call while the initial index is still building answers from what is +// indexed and says so, instead of waiting without end or failing. +func TestToolNotesIndexStillBuilding(t *testing.T) { + root := t.TempDir() + release := make(chan struct{}) + rt := openTestRuntime(t, root, workspace.Options{BeforeInitialReconcile: func() { <-release }}) + // Registered after the runtime, so it runs before the runtime closes: a + // close waits for the held reconciliation. + t.Cleanup(func() { + select { + case <-release: + default: + close(release) + } + }) + h := NewHandler(rt, rt.LanguageServices()) + + out, err := h.Call(context.Background(), "dexter_search", json.RawMessage(`{"query":"x"}`), 10*time.Millisecond) + if err != nil { + t.Fatal(err) + } + wantContains(t, out, "still building") + + // The rename finds its sites in the index, so it refuses until the index + // is complete. + _, err = h.Call(context.Background(), "dexter_rename_symbol", json.RawMessage(`{"module":"MyApp.A","new_name":"MyApp.B"}`), 10*time.Millisecond) + if err == nil || !strings.Contains(err.Error(), "still building") { + t.Fatalf("rename during the initial build: %v, want a refusal", err) + } + + close(release) + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + if err := rt.WaitReady(ctx); err != nil { + t.Fatal(err) + } + out, err = h.Call(context.Background(), "dexter_search", json.RawMessage(`{"query":"x"}`), 0) + if err != nil { + t.Fatal(err) + } + wantNotContains(t, out, "still building") +} + +// A degraded index condition, the same one an editor shows, is added to the +// tool answer so the agent does not take an incomplete answer as complete. +func TestToolNotesDegradedIndex(t *testing.T) { + e := setupProject(t) + e.rt.Reporter().Set(lsp.CondIndexFiles, notify.Warning, "Dexter: 2 files could not be indexed: a.ex, b.ex (permission denied).") + out := e.callTool("dexter_search", map[string]any{"query": "fetch_user"}) + wantContains(t, out, "MyApp.Accounts.fetch_user/1", "Note (warning): 2 files could not be indexed") + + // The workspace tool lists every active condition itself. + out = e.callTool("dexter_workspace", nil) + wantContains(t, out, "Workspace conditions:", "warning: 2 files could not be indexed") +} + +// Regression: a call written through an alias that a `__using__` block +// injects is indexed under the short name. The editor's find-references found +// it, and dexter_references must find it too. +func TestReferencesTool_ViaUseInjectedAlias(t *testing.T) { + e := setupProject(t) + e.indexFile("lib/my_app/repo.ex", `defmodule MyApp.Repo do + defmacro __using__(_) do + quote do + alias MyApp.Repo + end + end + + def all(q), do: q +end +`) + e.indexFile("lib/my_app/users.ex", `defmodule MyApp.Users do + use MyApp.Repo + + def list, do: Repo.all(:users) +end +`) + out := e.callTool("dexter_references", map[string]any{"module": "MyApp.Repo", "function": "all"}) + wantContains(t, out, "lib/my_app/users.ex", "4: def list, do: Repo.all(:users)") +} diff --git a/internal/mcp/workspace.go b/internal/mcp/workspace.go new file mode 100644 index 0000000..bc4a7f9 --- /dev/null +++ b/internal/mcp/workspace.go @@ -0,0 +1,94 @@ +package mcp + +import ( + "context" + "fmt" + "os" + "path/filepath" + "sort" + "strings" + + "github.com/remoteoss/dexter/internal/version" +) + +type WorkspaceParams struct{} + +func (h *Handler) workspace(ctx context.Context, args WorkspaceParams) (string, error) { + var b strings.Builder + fmt.Fprintf(&b, "Dexter %s\n", version.Version) + fmt.Fprintf(&b, "Project root: %s\n", h.projectRoot) + + if projects := findMixProjects(h.projectRoot); len(projects) > 0 { + fmt.Fprintf(&b, "\nMix projects:\n") + for _, p := range projects { + fmt.Fprintf(&b, " %s\n", p) + } + } else { + fmt.Fprintf(&b, "\nNo mix.exs found at the project root. The index may cover a plain directory of Elixir files.\n") + } + + st, err := h.rt.IndexStatus() + if err != nil { + return "", fmt.Errorf("reading index status: %w", err) + } + if st.StdlibRoot != "" { + fmt.Fprintf(&b, "\nElixir stdlib: %s (indexed; stdlib symbols resolve in lookups)\n", st.StdlibRoot) + } else { + fmt.Fprintf(&b, "\nElixir stdlib: not detected. Set DEXTER_ELIXIR_LIB_ROOT to enable stdlib lookups.\n") + } + + fmt.Fprintf(&b, "\nIndex: %d files, %d definitions, %d references\n", st.Files, st.Definitions, st.References) + if st.Ready { + fmt.Fprintf(&b, "Index state: ready\n") + } else { + fmt.Fprintf(&b, "Index state: still building; answers can be incomplete until it is ready\n") + } + if st.IndexVersion != st.ExpectedIndexVersion && st.Ready { + fmt.Fprintf(&b, "WARNING: index version %d does not match this binary (%d). Run `dexter stop --force` in the project so the next call starts a current daemon (a plain stop is refused while this MCP session is attached).\n", st.IndexVersion, st.ExpectedIndexVersion) + } + if st.Watching { + fmt.Fprintf(&b, "\nThe index updates automatically as files change and on git branch switches; dexter_reindex forces an immediate update.\n") + } else { + fmt.Fprintf(&b, "\nFile watching is not active, so the index updates only on git branch switches, editor saves, and dexter_reindex.\n") + } + + if conditions := h.rt.Reporter().Conditions(); len(conditions) > 0 { + fmt.Fprintf(&b, "\nWorkspace conditions:\n") + for _, c := range conditions { + fmt.Fprintf(&b, " %s: %s\n", c.Severity, strings.TrimPrefix(c.Message, "Dexter: ")) + } + } + return b.String(), nil +} + +// findMixProjects lists mix.exs locations relative to root: the root itself, +// umbrella apps under apps/, and direct children with their own mix.exs. +// The scan is deliberately shallow; no full tree walk. +func findMixProjects(root string) []string { + var projects []string + if _, err := os.Stat(filepath.Join(root, "mix.exs")); err == nil { + projects = append(projects, "mix.exs") + } + for _, pattern := range []string{"apps/*/mix.exs", "*/mix.exs"} { + matches, _ := filepath.Glob(filepath.Join(root, pattern)) + for _, m := range matches { + if rel, err := filepath.Rel(root, m); err == nil && rel != "mix.exs" { + projects = append(projects, rel) + } + } + } + sort.Strings(projects) + return dedupeStrings(projects) +} + +func dedupeStrings(in []string) []string { + out := in[:0] + var prev string + for i, s := range in { + if i == 0 || s != prev { + out = append(out, s) + } + prev = s + } + return out +} diff --git a/internal/store/project.go b/internal/store/project.go index a02aaf4..16f8dca 100644 --- a/internal/store/project.go +++ b/internal/store/project.go @@ -1,6 +1,7 @@ package store import ( + "fmt" "os" "path/filepath" ) @@ -47,3 +48,20 @@ func gitMarker(path string) bool { info, err := os.Stat(path) return err == nil && (info.IsDir() || info.Mode().IsRegular()) } + +// NonProjectRootError explains why dir is not a workspace to index, or returns +// nil when it is one. Every frontend refuses such a directory unless the user +// insists, because indexing it (the home directory, a mistyped path) would +// read every file under it. +func NonProjectRootError(dir string) error { + if IsHomeDir(dir) { + if HasIndex(dir) { + return nil + } + return fmt.Errorf("refusing to use %s as a workspace: it is your home directory, not a project", dir) + } + if LooksLikeProject(dir) { + return nil + } + return fmt.Errorf("refusing to use %s as a workspace: no mix.exs, .git, or Dexter database found, so it does not look like an Elixir project", dir) +} diff --git a/internal/store/store.go b/internal/store/store.go index 9ccd0fc..ff6c71b 100644 --- a/internal/store/store.go +++ b/internal/store/store.go @@ -1282,6 +1282,29 @@ func (s *Store) ListModuleFunctions(module string, publicOnly bool) ([]Completio return results, rows.Err() } +// ListModuleCallbacks returns the @callback and @macrocallback definitions of +// the given behaviour module (these are excluded from ListModuleFunctions). +func (s *Store) ListModuleCallbacks(module string) ([]CompletionResult, error) { + rows, err := s.db.Query( + "SELECT d.module, d.function, d.arity, d.kind, f.path, d.line, d.params FROM definitions d JOIN files f ON f.id = d.file_id WHERE d.module = ? AND d.kind IN ('callback', 'macrocallback') GROUP BY d.function, d.arity ORDER BY d.function, d.arity LIMIT 100", + module, + ) + if err != nil { + return nil, err + } + defer func() { _ = rows.Close() }() + + var results []CompletionResult + for rows.Next() { + var r CompletionResult + if err := rows.Scan(&r.Module, &r.Function, &r.Arity, &r.Kind, &r.FilePath, &r.Line, &r.Params); err != nil { + return nil, err + } + results = append(results, r) + } + return results, rows.Err() +} + // IndexStats summarizes the size of the index. type IndexStats struct { Files int diff --git a/internal/store/store_test.go b/internal/store/store_test.go index 69f2f26..6d2e3d0 100644 --- a/internal/store/store_test.go +++ b/internal/store/store_test.go @@ -1868,6 +1868,73 @@ func TestSetBulkPragmas_AppliesWhenExclusive(t *testing.T) { } } +func TestListModuleCallbacks(t *testing.T) { + s, dir := setupTestStore(t) + defer func() { _ = s.Close() }() + + path := writeElixirFile(t, dir, "lib/notifier.ex", `defmodule MyApp.Notifier do + @callback deliver(map()) :: :ok | {:error, term()} + @callback name() :: String.t() + @macrocallback render(term()) :: Macro.t() + def dispatch(msg), do: msg +end +`) + defs, _, err := parser.ParseFile(path) + if err != nil { + t.Fatal(err) + } + if err := s.IndexFile(path, defs); err != nil { + t.Fatal(err) + } + + results, err := s.ListModuleCallbacks("MyApp.Notifier") + if err != nil { + t.Fatal(err) + } + if len(results) != 3 { + t.Fatalf("callbacks = %d, want 3: %+v", len(results), results) + } + kinds := make(map[string]string, len(results)) + for _, result := range results { + kinds[result.Function] = result.Kind + } + if kinds["deliver"] != "callback" || kinds["render"] != "macrocallback" { + t.Errorf("callback kinds = %v", kinds) + } +} + +func TestStats(t *testing.T) { + s, dir := setupTestStore(t) + defer func() { _ = s.Close() }() + + stats, err := s.Stats() + if err != nil { + t.Fatal(err) + } + if stats.Files != 0 || stats.Definitions != 0 || stats.References != 0 { + t.Errorf("empty store stats = %+v, want zeros", stats) + } + + path := writeElixirFile(t, dir, "lib/worker.ex", `defmodule SharedLib.Worker do + def run, do: MyApp.Accounts.fetch_user(1) +end +`) + defs, refs, err := parser.ParseFile(path) + if err != nil { + t.Fatal(err) + } + if err := s.IndexFileWithRefs(path, defs, refs); err != nil { + t.Fatal(err) + } + stats, err = s.Stats() + if err != nil { + t.Fatal(err) + } + if stats.Files != 1 || stats.Definitions < 2 || stats.References < 1 { + t.Errorf("populated store stats = %+v", stats) + } +} + // ModuleFunctionKeys must return the complete set. Callers diff it against // another source of truth, so the 100-row cap ListModuleFunctions applies for // completion would silently make every function past the cap look unindexed. diff --git a/internal/workspace/runtime.go b/internal/workspace/runtime.go index c083bdf..fd8c1f2 100644 --- a/internal/workspace/runtime.go +++ b/internal/workspace/runtime.go @@ -222,6 +222,40 @@ func (r *Runtime) Session(id string) (*lsp.Server, bool) { return s.server, true } +// EditorBuffer is a buffer that an attached editor holds open with changes it +// has not saved. +type EditorBuffer struct { + Text string + ChangedAt time.Time // the time of the last change in the editor +} + +// UnsavedBuffer returns the newest buffer for path that an attached editor +// holds open with unsaved changes, or false when there is none. A frontend +// without an editor, such as MCP, uses it to read what the user is editing +// instead of the disk. A buffer the editor has not changed since it opened or +// saved it is not returned: it can be older than the disk. +func (r *Runtime) UnsavedBuffer(path string) (EditorBuffer, bool) { + r.sessMu.Lock() + if len(r.sessions) == 0 { + r.sessMu.Unlock() + return EditorBuffer{}, false + } + servers := make([]*lsp.Server, 0, len(r.sessions)) + for _, s := range r.sessions { + servers = append(servers, s.server) + } + r.sessMu.Unlock() + var out EditorBuffer + var newest uint64 + found := false + for _, srv := range servers { + if t, seq, at, ok := srv.UnsavedBuffer(path); ok && (!found || seq > newest) { + out, newest, found = EditorBuffer{Text: t, ChangedAt: at}, seq, true + } + } + return out, found +} + // SessionInfo describes one attached editor session for status output. type SessionInfo struct { ID string