diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ea2de7f..a691067 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -73,12 +73,12 @@ jobs: cd internal/lsp/testdata/monorepo/apps/app_with_styler && mix deps.get && mix deps.compile && cd - cd internal/lsp/testdata/monorepo/apps/app_with_ecto_migration && mix deps.get && mix deps.compile && cd - cd internal/lsp/testdata/monorepo/apps/app_basic && mix deps.get && cd - - # Every formatter test and the generated-BEAM completion test need mix on - # PATH and skip themselves without it, so this job is the only place they - # run — the test/test-race jobs have no Elixir. Match the formatter family - # and explicitly include the self-contained BEAM regression fixture. + # Every formatter test and the generated-BEAM completion and definition + # tests need mix on PATH and skip themselves without it, so this job is the + # only place they run — the test/test-race jobs have no Elixir. Match the + # formatter family and explicitly include the self-contained BEAM fixtures. - name: Run integration tests - run: go test ./internal/lsp/ -run 'TestFormatter|TestDidSave_Formatter|TestCompletion_GeneratedFunctionsFromConsumerBEAM' -v -timeout 600s + run: go test ./internal/lsp/ -run 'TestFormatter|TestDidSave_Formatter|TestCompletion_GeneratedFunctionsFromConsumerBEAM|TestDefinition_GeneratedFunctionsFromCompiler' -v -timeout 600s lint: runs-on: ubuntu-latest diff --git a/CHANGELOG.md b/CHANGELOG.md index 34072b9..b69b5a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,8 @@ ### Added +- **Go-to-definition reaches the line that declared a generated function** — a function a macro generated used to resolve to the top of its module. Dexter now reads the line from the compiled module's debug info, which is standard compiler output, so no framework is special-cased. A generator that expands each function at the line of the call that declared it, or stamps it with `@file {file, line}`, sends definition, call hierarchy, the references declaration, and `dexter lookup` (including `--strict`) to that line. The line is used only when it was compiled from the file being opened; a BEAM older than the source still gives its line, because Dexter cannot compile the project and the last compile's line is closer than the module line; only edits to the declaring file move it, and the next compile makes it exact. A function whose only recorded line is the module line, such as one a `@before_compile` hook made, goes to the call in its module that declares it by name. When several calls spell the name, as an Ash action and the code interface that runs it do, the macro whose calls name the most of the module's generated functions wins, and a tie keeps the module line. A function with a clause per DSL call goes to every clause. A line past the end of the file is never returned. A module compiled without debug info falls back to its Docs chunk annotation, which is often, but not always, the same line. A generated module with no source of its own, such as one `Module.create` made or a Spark DSL entity, goes to the file it was compiled from, rebased onto the project when it was built elsewhere, and so does go-to-definition on its name. A module a macro made with `defmodule` and a name it computed records no line of its own, so its name goes to its first function's line. A bare call to a generated function of an imported module now resolves. Ash code interfaces go to their `define` line with released Ash, and from the recorded line with an Ash release that includes [ash-project/ash#2971](https://github.com/ash-project/ash/pull/2971) ([#108](https://github.com/remoteoss/dexter/issues/108)) + - **`--root`/`-C` names the workspace on every command** — `dexter init --root ~/project`, `dexter lookup --root ~/project MyApp.Repo`, and `dexter lsp --root ~/project` all run as if they had been started in that directory, so an agent, a script, or an editor wrapper can index or query a project from anywhere. Relative paths, a `reindex` target included, resolve from the named root - **Watching and runtime locations recover without periodic reindexes** — a directory the kernel refuses to watch (an inotify watch limit on a large tree) is tracked instead of disabling the whole native watcher. Dexter reconciles once when coverage is lost, retries only failed registrations, and reconciles once when coverage returns; it does not run recurring full-tree passes that cause CPU spikes. Runtime files use the environment-independent `/tmp/dexter-` directory so a GUI editor and a shell cannot derive different ownership locks. Elixir and mix detection also searches the standard mise, asdf, and Homebrew locations and falls back to a login shell, so an editor that starts the daemon with a stripped PATH no longer disables stdlib indexing or formatting @@ -26,6 +28,10 @@ ### Fixed +- **A def inside a macro's `quote` is no longer indexed as a function of the macro's module** — `defmacro route(...) do quote do def handle(...) end end` made the index say that the DSL module defines `handle/2`, which it does not. A consumer that imports the DSL then resolved `Consumer.handle` into the macro's body. Such a def is now skipped, so the call goes to the line in the consumer that declared it, from the compiled BEAM. What `__using__` injects, and a quote in a helper function, are still indexed as before. The index is rebuilt once after the upgrade + +- **Compressed BEAM files are read** — a module compiled with the `compressed` option, as some Erlang dependencies are, is a gzip stream around the BEAM container. Dexter rejected it as an invalid BEAM, so its exports were missing from completion and generated-function navigation. It is now decompressed, with the same size limit as an uncompressed file + - **Git worktrees nested inside a project were indexed as part of it** — a linked worktree checked out below the project root (Claude Code's `.claude/worktrees/`, or any gitignored worktree folder) is a full copy of the repository, so every go-to-definition returned one result per checkout. The indexer, the incremental sweep, the native watcher and single-file updates now skip any directory that is a linked worktree, including worktrees of bare repositories; submodules and Mix git dependencies stay indexed. Adding, changing, moving or removing such a worktree, including its `mix.exs` and `mix.lock`, no longer reindexes the enclosing project, and an existing index drops the worktree's files at the next start. A worktree that git still records stays skipped after its `.git` file is gone, as during `git worktree remove`, until `git worktree prune`. Starting Dexter inside such a worktree no longer climbs to the enclosing checkout's `.dexter/dexter.db` either: the worktree is its own project - **Branch switches were missed when the project root is a linked worktree or a submodule** — the HEAD poller read `/.git/HEAD`, but in such a checkout `.git` is a file that names the git directory. The poller now follows that file, so a branch switch reconciles the index even when the native file watcher is unavailable - **Find references through an injected alias was slow and memory-hungry on large projects** — a module such as `MyApp.Repo`, aliased by a `__using__` block that most of the project uses, made every references query read and tokenize each file that used the injector: about 20,000 files and 2.4 GB of allocations per query on a large monorepo. Only files that contain a candidate reference are read now, which brought that query from 1–2 s to under 0.3 s and its allocations to about 70 MB diff --git a/docs/architecture.md b/docs/architecture.md index e503e73..780b385 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -7,7 +7,7 @@ Dexter is a fast Elixir LSP server. It indexes module and function definitions f - `cmd/main.go` — CLI entrypoint: `init`, `reindex`, `lookup`, `references`, `lsp`, `version`, and the hidden `daemon` subcommand. `lsp`, `lookup`, `references`, and `reindex` are frontends to the workspace daemon: `lsp` is a raw stdio proxy onto it. Only `init` still opens the store itself, under the same ownership lock the daemon takes, because it needs the workspace to itself. - `internal/indexer/` — the cold build: walk and stat on all cores, parse on all cores, then one bulk transaction with the indexes dropped. `dexter init` and the LSP server (when it finds an empty index) both call `FullBuild`. `Options.InProcess` marks the server, which shares the database with live readers and so cannot use the connection-wide bulk pragmas. - `internal/parser/` — Elixir parser backed by a hand-rolled tokenizer (`tokenizer.go`). The tokenizer produces a flat token stream (handling heredocs, sigils, strings, comments as opaque tokens; the code inside a `#{}` interpolation is tokenized as well, into the separate `TokenResult.Interp` stream — see below) and `parser_tokenized.go` walks it to extract defmodule, def, defp, defmacro, defdelegate, defguard, defprotocol, defimpl, @type, @callback, alias, import, use, and Module.function references. Handles module nesting, alias resolution for defdelegate targets, and multi-line expressions natively via bracket depth tracking. -- `internal/beam/` — bounded, bounds-checked readers for the BEAM container, export table, Elixir Docs chunk, and persisted module attributes. These readers extract generated callables and DSL-provider metadata without starting the Erlang VM. +- `internal/beam/` — bounded, bounds-checked readers for the BEAM container, export table, Elixir Docs chunk, Elixir debug info, and persisted module attributes. These readers extract generated callables, their definition lines, and DSL-provider metadata without starting the Erlang VM. - `internal/store/` — SQLite layer. Tables: `files` (path + mtime), `definitions` (module, function, kind, line, file_path, delegate_to, delegate_as), `refs` (module, function, line, file_path, kind). - `internal/lsp/` — LSP server. `server.go` handles all LSP methods. `elixir.go` contains pure functions for cursor expression extraction, alias/import/use extraction (tokenizer-based), and use-chain parsing. `rename.go` has rename helpers. `hover.go` has hover formatting. `documents.go` is an in-memory open-buffer store. - `internal/workspace/` — the protocol-independent owner of one workspace: the store, the shared `lsp.IndexCoordinator`, the headless language-service instance, stdlib discovery, the native and Git watchers, and the single mutation queue every index change enters. `runtime.go` is the lifecycle and the queue; `watch.go` defines the recursive watcher abstraction, with an FSEvents backend on macOS and an fsnotify backend elsewhere or as fallback. @@ -137,6 +137,26 @@ The block walk is passed as a thunk and runs only after the compiled consumer re Generated-symbol resolution is shared by completion, hover, definition, signature help, references, and call-hierarchy preparation. Definition and call hierarchy cannot point at a source definition for a source-less provider, so they walk the provider's lexical module parents and use the closest module that has an indexed source location. Completion resolve and signature help read the same lazily cached BEAM documentation used by hover. +### Definition lines for generated functions + +A generated function has no source definition, but the compiled module's `Dbgi` chunk still records where each def came from. `beam.ReadDefinitionLines` walks `{:debug_info_v1, :elixir_erl, {:elixir_v1, map, specs}}` for the `file`, `relative_file`, and `definitions` keys, and steps over clause bodies without allocating. Clause ASTs can nest very deep, so the ETF reader steps over a term with a count of the terms still to skip instead of recursion: nesting costs no stack, and a count larger than the bytes left is rejected as corrupt. + +Each def carries two locations. Its `:line` is the line in the module's own file that was being compiled when the def was produced: the macro call, or wherever a before-compile hook ran. A `file: {path, line}` entry, left by `@file` or `quote location: :keep`, is preferred when `path` is the module's own source, because that is where the code asked for the function. When `path` is another file it is the generator's implementation, and `:line` is used instead. A `file:` line at or before the module's own line is not used either: it is a `location: :keep` quote in a macro defined above the module in the same file. Elixir 1.17 and earlier record the module's line under the map's `line` key instead of `anno`, and both are read. Ash's code interfaces take the second shape: each generated def is expanded at its `define` line, so `:line` is that line and `file` still names `code_interface.ex`, which keeps stacktraces unchanged. No framework is recognized by name. + +The parser does not index a def inside a `quote` in a macro other than `__using__` (`defmacro route ... do quote do def handle ... end end`) as a function of the macro's module: it is code the macro generates in its caller, and the caller's BEAM records where it was declared. Without that, a consumer that imports the DSL resolved `Consumer.handle` through the import into the macro's body before the BEAM was asked. What `__using__` injects and a quote in a helper function stay indexed, because the use-chain features above resolve them. + +`generatedDefinitionResultsFor` is the single entry point that definition, call-hierarchy preparation, and the references declaration share; qualified definition, the references declaration, and `dexter lookup` reach it through `LookupName`. It starts from `generatedDefinitionResults` and moves to the recorded line only when that line belongs to the file being opened: the debug info must have been compiled from that file (the same absolute path, or else the most path components shared from the end, which covers a moved, copied, or symlinked project; when a module has rows in two files, as two umbrella apps can, the most specific match wins, and an equal match trusts neither), and the line must fall after the module's own line. Any other case keeps the module result. As for generated names, source-vs-BEAM mtime is not a gate: Dexter cannot compile the project, so a stale BEAM is the usual state while editing, and the last compile's line is closer than the module line. + +`beam.ReadDefinitionLines` also reads the module's own line (the map's `anno`, a line or `{line, column}`) and the line of each clause when a def's clauses were made at different lines (`DebugInfo.Clauses`: a DSL that adds a clause per call, such as `route :get, "/a"`, gets a location per clause). + +Lines are checked against the current text, which is the buffer open in the editor if there is one and the file on disk otherwise (`currentSource`). A line at or before the module's line carries no information and is dropped; both lines come from the same compile (the module's line is Dbgi's `anno` of the BEAM that holds the module, which is the parent's for a module a macro nested in it, and the index's only when no BEAM records one), so the comparison holds even when the file has changed since. A line past the end of the current text is dropped too, because a generator can give a def any line (`quote line: 99`) and an edit can remove lines. When nothing is left, which is the case for a def a `@before_compile` hook made at the module line (and for released Ash's code interfaces), the owning module's body is searched for the declaring call: a call, outside heredocs, whose first argument is the function's name as an atom, also trying the name without a trailing `!` or `?`. That is the generic shape of a macro call that declares a name; an atom in a keyword, a typespec, a later argument, or a module attribute does not count, and neither does a call in a sibling module of the same file (found with the tokenizer). When one line matches, it is used. When several do, as an Ash action (`update :publish`) and the code interface that runs it (`define :publish`) do, the macro that made the function is taken to be the one whose calls name the most of the module's generated functions (`define` names every interface; an action names only itself), and a tie gives no answer. The search is not used for a function whose clauses were made at several lines: those came from calls that do not spell its name (`get "/a"` makes a `match/2` clause), so a call that does, such as `plug :match`, is not where they were declared. + +A stale BEAM's lines are not corrected. Only edits to the declaring file move them, by the lines added or removed above, and those files (DSL modules, schemas, routers, dependencies) change less often than their callers. Correcting them meant guessing at text the BEAM was compiled from, and every signal tried either gave up often or could send the editor to a line that looked meaningful and was not. The line from the last compile is near, and the next compile makes it exact. + +A module compiled without debug info still has its Docs chunk, whose per-entry anno (`beam.Function.Line`) is often the same `:line`, but not always: a generator can give the def one line and its docs another, as Ash's code interfaces do. It is used when there is no Dbgi, and the `CInf` chunk's `:source` (`beam.ReadSourcePath`) decides which file it belongs to. The `CInf` source also covers a generated module with no source row of its own, such as a Spark entity module: its recorded file is usually the generator's own file. A recorded path inside the project is used as it is. Otherwise it is rebased: first each tail of the recorded path under the project root, longest first, which covers a moved or copied checkout and an umbrella's `apps//lib`, then the Mix layouts for the application named below the last `lib/` (`lib/`, `deps//lib/`, or `deps/`). The application comes from the recorded path, not the BEAM's, because Spark creates Ash's entity modules in ash's ebin from spark's source. A recorded path outside the project is the last choice: it is right for a `path:` dependency, but a `_build` copied from another worktree records that worktree, which usually still exists. If no file is found, the lexical parent stays the answer. `CInf` records the path as codepoints, which are decoded to UTF-8. A precise line is the function's own definition, so `LookupName` returns it even for a strict (`ExactModule`) lookup; without one, the existing module results stand. `LookupName` on the name of a module that has no source row returns the file from `CInf` at the module's recorded line, or at its first function's line when the module line is zero, as it is for a module a macro made with `defmodule unquote(name)` and a name it computed, so go-to-definition on a module that `Module.create` made works too. A bare call that no source resolves also checks the BEAMs of the file's imported modules for a generated function of that name. + +The debug info and compile info are read on the first definition request that needs them and memoized on the module's generated-function cache entry, so the BEAM stamp invalidates them with everything else. A module compiled without debug info, or an Erlang module, is memoized as empty. + For references, the parser records bare injected calls under the direct `use` module because the generated provider is unavailable while source is indexed. At lookup time, generated-symbol resolution queries those injector rows and validates every candidate against the compiled provider active at that candidate's block path. This keeps same-named macros from another DSL or another section out of the result. Statement-level injected calls with arguments are indexed even when they omit both parentheses and a `do` block, as in `authorize_if always()`. ## References — injector scan diff --git a/internal/beam/compile_info.go b/internal/beam/compile_info.go new file mode 100644 index 0000000..ab19e28 --- /dev/null +++ b/internal/beam/compile_info.go @@ -0,0 +1,159 @@ +package beam + +import ( + "errors" + "unicode/utf8" +) + +// ReadSourcePath returns the source file recorded in a BEAM's compile info +// (the CInf chunk). +// +// The chunk is a short keyword list — `[version: ..., options: [...], source: +// ...]` — so this is a handful of terms, far cheaper than the Docs chunk it is +// usually read alongside. +// +// The path is the one the artifact was compiled in, which is not necessarily +// where the artifact lives now: a `_build` copied between checkouts, or any +// Docker/CI build, records a directory that no longer exists. Callers must treat +// the result as a hint and verify it before returning it to an editor. +// +// This matters for modules that have no source file of their own. Spark creates +// entity modules such as `Ash.Resource.Dsl.CodeInterface.Define` with an explicit +// location, so the module's recorded source is the framework file that generated +// it, and the source index has no row to consult. +func ReadSourcePath(path string) (string, bool) { + raw, err := readChunk(path, "CInf") + if err != nil { + return "", false + } + return parseCompileSource(raw) +} + +// parseCompileSource reads `{version, options, source}` pairs and returns the +// source entry. Any other shape is skipped rather than failing the chunk: a +// module without usable compile info must still yield its generated functions. +func parseCompileSource(raw []byte) (string, bool) { + if len(raw) < 1 || raw[0] != etfVersion { + return "", false + } + r := &etfReader{buf: raw[1:]} + + count, hasTail, err := r.enterList() + if err != nil { + return "", false + } + + for i := int64(0); i < count; i++ { + arity, err := r.enterTuple() + if err != nil { + return "", false + } + if arity != 2 { + if err := r.skipTerms(int64(arity)); err != nil { + return "", false + } + continue + } + + key, err := r.readAtom() + if err != nil { + return "", false + } + if key != "source" { + if err := r.skip(); err != nil { + return "", false + } + continue + } + + value, err := r.readStringTerm() + if err != nil || value == "" { + return "", false + } + return value, true + } + + // enterList reports a tail for every LIST_EXT, not only improper ones, so the + // terminator still has to be consumed when the loop runs to completion. + if hasTail { + if err := r.skip(); err != nil { + return "", false + } + } + return "", false +} + +// readStringTerm reads a path-shaped term: a binary, a string, or a charlist of +// codepoints, returned as UTF-8. The compiler records :source as a charlist, +// stored as a compact string term when every codepoint is below 256 and as a +// list otherwise; other producers of the same chunk use a binary. +func (r *etfReader) readStringTerm() (string, error) { + tag, err := r.peekTag() + if err != nil { + return "", err + } + + switch tag { + case tagNil: + _, err := r.u8() + return "", err + + case tagBinary: + return r.readBinary() + + case tagString: + if _, err := r.u8(); err != nil { + return "", err + } + n, err := r.u16() + if err != nil { + return "", err + } + if err := r.need(n); err != nil { + return "", err + } + // Each byte is one codepoint below 256, not a UTF-8 byte: a path with + // `é` in it stores 0xE9. + out := make([]byte, 0, n) + for _, b := range r.buf[r.pos : r.pos+n] { + out = utf8.AppendRune(out, rune(b)) + } + r.pos += n + return string(out), nil + + case tagList: + count, hasTail, err := r.enterList() + if err != nil { + return "", err + } + // The chunk size already bounds count; this only keeps a corrupt length + // from preallocating an absurd buffer. + if count > maxSourcePathLen { + return "", errors.New("charlist too long") + } + out := make([]byte, 0, count) + for i := int64(0); i < count; i++ { + n, err := r.readInt() + if err != nil { + return "", err + } + if n < 0 || n > utf8.MaxRune || !utf8.ValidRune(rune(n)) { + return "", errors.New("charlist element out of range") + } + out = utf8.AppendRune(out, rune(n)) + } + if hasTail { + if err := r.skip(); err != nil { + return "", err + } + } + return string(out), nil + + default: + return "", errors.New("unexpected source path encoding") + } +} + +// maxSourcePathLen bounds a charlist path. Real paths are far shorter; the limit +// exists so a corrupt length cannot drive a large allocation. +const maxSourcePathLen = 4096 diff --git a/internal/beam/compile_info_test.go b/internal/beam/compile_info_test.go new file mode 100644 index 0000000..5903a69 --- /dev/null +++ b/internal/beam/compile_info_test.go @@ -0,0 +1,145 @@ +package beam + +import ( + "path/filepath" + "testing" +) + +// compileInfoTerm builds a CInf payload: a keyword list shaped like the one the +// compiler writes, with the source entry encoded in the requested form. +func compileInfoTerm(t *testing.T, source func(*etfTestWriter, string)) []byte { + t.Helper() + // parseCompileSource takes the chunk payload, which begins with the ETF + // version byte, exactly as writeTestBEAM lays it out. + var w etfTestWriter + w.version() + + w.listHeader(2) + + w.smallTuple(2) + w.atom("version") + w.string("10.0.4") + + w.smallTuple(2) + w.atom("source") + source(&w, "/build/agent/deps/shared_lib/lib/shared_lib/worker.ex") + + w.nil() + return w.buf +} + +func TestParseCompileSource(t *testing.T) { + cases := []struct { + name string + source func(*etfTestWriter, string) + }{ + { + // What the compiler writes: a printable charlist stored as a string. + name: "string term", + source: func(w *etfTestWriter, path string) { + w.string(path) + }, + }, + { + name: "binary term", + source: func(w *etfTestWriter, path string) { + w.binary(path) + }, + }, + { + name: "charlist term", + source: func(w *etfTestWriter, path string) { + w.listHeader(len(path)) + for i := 0; i < len(path); i++ { + w.smallInt(int(path[i])) + } + w.nil() + }, + }, + } + + const want = "/build/agent/deps/shared_lib/lib/shared_lib/worker.ex" + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, ok := parseCompileSource(compileInfoTerm(t, tc.source)) + if !ok { + t.Fatalf("parseCompileSource reported no source") + } + if got != want { + t.Errorf("source = %q, want %q", got, want) + } + }) + } +} + +func TestParseCompileSourceMissing(t *testing.T) { + // A compile info list without a source entry, and payloads that are not a + // list at all, must all report "no source" rather than inventing one. + var noSource etfTestWriter + noSource.smallTuple(0) + + var notAList etfTestWriter + notAList.smallInt(1) + + cases := map[string][]byte{ + "empty": nil, + "no entry": noSource.buf, + "not a list": notAList.buf, + } + for name, raw := range cases { + if got, ok := parseCompileSource(raw); ok { + t.Errorf("%s: got %q, want no source", name, got) + } + } +} + +func TestReadSourcePath(t *testing.T) { + path := filepath.Join(t.TempDir(), "Elixir.LibFixture.beam") + writeTestBEAMOpts(t, path, testBEAMOptions{ + atomNames: defaultTestAtoms, + cinf: compileInfoTerm(t, func(w *etfTestWriter, path string) { + w.string(path) + }), + }) + + got, ok := ReadSourcePath(path) + if !ok || got != "/build/agent/deps/shared_lib/lib/shared_lib/worker.ex" { + t.Errorf("ReadSourcePath = %q, %v; want the recorded source", got, ok) + } + + missing := filepath.Join(t.TempDir(), "Elixir.NoCompileInfo.beam") + writeTestBEAM(t, missing, buildDocsTerm()) + if got, ok := ReadSourcePath(missing); ok { + t.Errorf("ReadSourcePath without CInf = %q, want no source", got) + } +} + +// Erlang records :source as codepoints. A path with `é` is stored as a string +// term whose byte is 0xE9, and one with `日本` as a list, because 26085 does not +// fit in a byte. Both must come back as UTF-8. +func TestParseCompileSourceUnicode(t *testing.T) { + const want = "/home/zoë/日本/lib/app.ex" + codepoints := []rune(want) + list := compileInfoTerm(t, func(w *etfTestWriter, _ string) { + w.listHeader(len(codepoints)) + for _, c := range codepoints { + w.smallInt(int(c)) + } + w.nil() + }) + if got, ok := parseCompileSource(list); !ok || got != want { + t.Errorf("list: source = %q, %v; want %q", got, ok, want) + } + + const latin = "/home/zoë/lib/app.ex" + short := compileInfoTerm(t, func(w *etfTestWriter, _ string) { + var bytes []byte + for _, c := range latin { + bytes = append(bytes, byte(c)) + } + w.string(string(bytes)) + }) + if got, ok := parseCompileSource(short); !ok || got != latin { + t.Errorf("string term: source = %q, %v; want %q", got, ok, latin) + } +} diff --git a/internal/beam/debug_info.go b/internal/beam/debug_info.go new file mode 100644 index 0000000..27b4435 --- /dev/null +++ b/internal/beam/debug_info.go @@ -0,0 +1,431 @@ +package beam + +import ( + "errors" + "fmt" + "slices" +) + +// maxPreallocatedSites bounds the definitions slice allocated up front. +const maxPreallocatedSites = 1024 + +// FunctionKey names one callable by name and arity. +type FunctionKey struct { + Name string + Arity int +} + +// DebugInfo is what ReadDefinitionLines extracts from an Elixir Dbgi chunk. +type DebugInfo struct { + // File is the absolute source path the module was compiled from, and + // RelativeFile the same path relative to the compiler's working directory. + // Either may be empty if the chunk omits it. + File string + RelativeFile string + + // ModuleLine is the line of the module itself in File: its defmodule, or + // wherever Module.create was called. It is the map's anno, or its line + // before Elixir 1.18. Zero if the chunk has none. + ModuleLine int + + // Lines maps each public function and macro to its line in File. Entries + // without a usable line are absent. + Lines map[FunctionKey]int + + // Clauses holds the line of each clause for a public definition whose + // clauses were made at more than one line, in clause order: a DSL that + // adds one clause per call, such as `route :get, "/a"`. Definitions with + // one line are absent; Lines has it. + Clauses map[FunctionKey][]int +} + +// ReadDefinitionLines reads the line of every public definition from an Elixir +// BEAM's debug info, for functions that have no source definition to index +// because a macro generated them. +// +// A generated def carries two locations. Its :line is the line in the module's +// own file that was being compiled when the def was produced: the macro call, +// or wherever a before-compile hook ran. A `file: {path, line}` entry, left by +// `@file` or by `quote location: :keep`, names a more specific place. When that +// path is the module's own source and the line is after the module's own, it +// is where the code asked for the function. Otherwise it is the generator's own +// implementation, in another file or in a macro defined above the module, which +// is not what a reader of the module is looking for, so :line is used instead. +// +// Everything here is standard compiler output. No framework is recognized by +// name: a generator that wants its functions to navigate to the line that +// declared them either expands each def at that line, as Ash's code interfaces +// do, or stamps it with `@file {file, line}`. +func ReadDefinitionLines(path string) (DebugInfo, error) { + chunk, err := readChunk(path, "Dbgi") + if err != nil { + return DebugInfo{}, err + } + inflated, err := inflateDocsTerm(chunk) + if err != nil { + return DebugInfo{}, fmt.Errorf("decode Dbgi chunk: %w", err) + } + return parseDebugInfo(inflated) +} + +var errNoElixirDebugInfo = errors.New("no Elixir debug info") + +// definitionSite is one definition's raw locations, kept until the module's +// source path is known: small maps put definitions ahead of file and +// relative_file, so they cannot be resolved while walking. +type definitionSite struct { + key FunctionKey + line int + keepFile string + keepLine int + clauseLines []int +} + +// parseDebugInfo walks {:debug_info_v1, :elixir_erl, {:elixir_v1, map, specs}}. +// Only three keys of the map are read; clause bodies, which are nearly all of +// the chunk, are stepped over without allocating. +func parseDebugInfo(buf []byte) (info DebugInfo, err error) { + // Bounds-checked throughout, but as with Docs a reader mistake must not take + // down the server over a corrupt build artifact. + defer func() { + if recovered := recover(); recovered != nil { + info = DebugInfo{} + err = fmt.Errorf("invalid Dbgi term: %v", recovered) + } + }() + + r := &etfReader{buf: buf} + if arity, err := r.enterTuple(); err != nil { + return DebugInfo{}, err + } else if arity != 3 { + return DebugInfo{}, fmt.Errorf("debug_info tuple arity %d", arity) + } + if version, err := r.readAtom(); err != nil { + return DebugInfo{}, err + } else if version != "debug_info_v1" { + return DebugInfo{}, fmt.Errorf("unsupported debug info version %q", version) + } + if backend, err := r.readAtom(); err != nil { + return DebugInfo{}, err + } else if backend != "elixir_erl" { + return DebugInfo{}, fmt.Errorf("%w: backend %q", errNoElixirDebugInfo, backend) + } + // Compiling with `debug_info: false` leaves :none here. + if tag, err := r.peekTag(); err != nil { + return DebugInfo{}, err + } else if tag != tagSmallTuple && tag != tagLargeTuple { + return DebugInfo{}, errNoElixirDebugInfo + } + if arity, err := r.enterTuple(); err != nil { + return DebugInfo{}, err + } else if arity != 3 { + return DebugInfo{}, fmt.Errorf("elixir debug info tuple arity %d", arity) + } + if format, err := r.readAtom(); err != nil { + return DebugInfo{}, err + } else if format != "elixir_v1" { + return DebugInfo{}, fmt.Errorf("unsupported Elixir debug info %q", format) + } + + pairs, err := r.enterMap() + if err != nil { + return DebugInfo{}, err + } + var sites []definitionSite + var legacyModuleLine int + for i := int64(0); i < pairs; i++ { + var key string + if tag, err := r.peekTag(); err != nil { + return DebugInfo{}, err + } else if isAtomTag(tag) { + if key, err = r.readAtom(); err != nil { + return DebugInfo{}, err + } + } else if err := r.skip(); err != nil { + return DebugInfo{}, err + } + switch key { + case "definitions": + if sites, err = readDefinitionSites(r); err != nil { + return DebugInfo{}, err + } + case "file": + if info.File, err = readOptionalBinary(r); err != nil { + return DebugInfo{}, err + } + case "relative_file": + if info.RelativeFile, err = readOptionalBinary(r); err != nil { + return DebugInfo{}, err + } + case "anno": + if info.ModuleLine, err = readAnnoLine(r); err != nil { + return DebugInfo{}, err + } + case "line": + // Elixir 1.17 and earlier record the module's line here, as a + // bare line, and have no anno. + if legacyModuleLine, err = readAnnoLine(r); err != nil { + return DebugInfo{}, err + } + default: + if err := r.skip(); err != nil { + return DebugInfo{}, err + } + } + } + // The specs are not needed, but consuming them proves the term is whole. + if err := r.skip(); err != nil { + return DebugInfo{}, err + } + + if info.ModuleLine == 0 { + info.ModuleLine = legacyModuleLine + } + + info.Lines = make(map[FunctionKey]int, len(sites)) + for _, site := range sites { + line := site.line + // A location at or before the module's own line cannot be where the + // module asked for the function. It is a `location: :keep` quote in a + // macro defined above the module in the same file, so :line, the + // call, is used. + stamped := site.keepFile != "" && site.keepLine > 0 && + (site.keepFile == info.RelativeFile || site.keepFile == info.File) && + (info.ModuleLine == 0 || site.keepLine > info.ModuleLine) + if stamped { + line = site.keepLine + } + if line > 0 { + info.Lines[site.key] = line + } + // A stamped def names its own line; its clauses' lines are the + // generator's, so only an unstamped def's clauses are kept. + if !stamped && len(site.clauseLines) > 1 { + if info.Clauses == nil { + info.Clauses = make(map[FunctionKey][]int) + } + info.Clauses[site.key] = site.clauseLines + } + } + return info, nil +} + +// readDefinitionSites reads the definitions list, whose entries are +// {{name, arity}, kind, meta, clauses}. Private definitions are skipped: +// generated functions are found through the export table, so only public ones +// can be asked about. +func readDefinitionSites(r *etfReader) ([]definitionSite, error) { + count, hasTail, err := r.enterList() + if err != nil { + return nil, err + } + // A corrupt count is bounded only by the chunk size, which can still ask + // for gigabytes of sites before the walk fails. Real modules have far + // fewer definitions, so the slice grows past this if it must. + sites := make([]definitionSite, 0, min(count, maxPreallocatedSites)) + for i := int64(0); i < count; i++ { + arity, err := r.enterTuple() + if err != nil { + return nil, err + } + if arity != 4 { + if err := r.skipTerms(int64(arity)); err != nil { + return nil, err + } + continue + } + if keyArity, err := r.enterTuple(); err != nil { + return nil, err + } else if keyArity != 2 { + return nil, fmt.Errorf("definition key arity %d", keyArity) + } + name, err := r.readAtom() + if err != nil { + return nil, err + } + functionArity, err := r.readInt() + if err != nil { + return nil, err + } + kind, err := r.readAtom() + if err != nil { + return nil, err + } + site := definitionSite{key: FunctionKey{Name: name, Arity: functionArity}} + if err := readDefinitionMeta(r, &site); err != nil { + return nil, err + } + if site.clauseLines, err = readClauseLines(r); err != nil { + return nil, err + } + if kind == "def" || kind == "defmacro" { + sites = append(sites, site) + } + } + if hasTail { + if err := r.skip(); err != nil { + return nil, err + } + } + return sites, nil +} + +// readClauseLines reads the distinct :line of each {meta, args, guards, body} +// clause, in order, and steps over everything else without allocating. A value +// that is not a list is skipped. +func readClauseLines(r *etfReader) ([]int, error) { + if tag, err := r.peekTag(); err != nil { + return nil, err + } else if tag != tagList && tag != tagNil { + return nil, r.skip() + } + count, hasTail, err := r.enterList() + if err != nil { + return nil, err + } + var lines []int + for i := int64(0); i < count; i++ { + if tag, err := r.peekTag(); err != nil { + return nil, err + } else if tag != tagSmallTuple { + if err := r.skip(); err != nil { + return nil, err + } + continue + } + arity, err := r.enterTuple() + if err != nil { + return nil, err + } + if arity != 4 { + if err := r.skipTerms(int64(arity)); err != nil { + return nil, err + } + continue + } + var clause definitionSite + if err := readDefinitionMeta(r, &clause); err != nil { + return nil, err + } + if err := r.skipTerms(3); err != nil { // args, guards, body + return nil, err + } + if clause.line > 0 && !slices.Contains(lines, clause.line) { + lines = append(lines, clause.line) + } + } + if hasTail { + if err := r.skip(); err != nil { + return nil, err + } + } + return lines, nil +} + +// readDefinitionMeta reads :line and a `file: {path, line}` entry from a +// definition's keyword metadata. The first of each wins, as Keyword.get would. +func readDefinitionMeta(r *etfReader, site *definitionSite) error { + count, hasTail, err := r.enterList() + if err != nil { + return err + } + seenLine, seenFile := false, false + for i := int64(0); i < count; i++ { + if tag, err := r.peekTag(); err != nil { + return err + } else if tag != tagSmallTuple { + if err := r.skip(); err != nil { + return err + } + continue + } + arity, err := r.enterTuple() + if err != nil { + return err + } + if arity != 2 { + if err := r.skipTerms(int64(arity)); err != nil { + return err + } + continue + } + var key string + if tag, err := r.peekTag(); err != nil { + return err + } else if isAtomTag(tag) { + if key, err = r.readAtom(); err != nil { + return err + } + } else if err := r.skip(); err != nil { + return err + } + switch { + case key == "line" && !seenLine: + seenLine = true + if site.line, err = readOptionalInt(r); err != nil { + return err + } + case key == "file" && !seenFile: + seenFile = true + if site.keepFile, site.keepLine, err = readFileLocation(r); err != nil { + return err + } + default: + if err := r.skip(); err != nil { + return err + } + } + } + if hasTail { + return r.skip() + } + return nil +} + +// readFileLocation reads a {path, line} tuple. Any other shape is skipped and +// reported as no location. +func readFileLocation(r *etfReader) (string, int, error) { + if tag, err := r.peekTag(); err != nil { + return "", 0, err + } else if tag != tagSmallTuple { + return "", 0, r.skip() + } + arity, err := r.enterTuple() + if err != nil { + return "", 0, err + } + if arity != 2 { + return "", 0, r.skipTerms(int64(arity)) + } + file, err := readOptionalBinary(r) + if err != nil { + return "", 0, err + } + line, err := readOptionalInt(r) + if err != nil { + return "", 0, err + } + return file, line, nil +} + +// readOptionalBinary reads a binary, or skips a term of any other type and +// returns "". +func readOptionalBinary(r *etfReader) (string, error) { + if tag, err := r.peekTag(); err != nil { + return "", err + } else if tag != tagBinary { + return "", r.skip() + } + return r.readBinary() +} + +// readOptionalInt reads a small or 32-bit integer, or skips a term of any other +// type and returns 0. +func readOptionalInt(r *etfReader) (int, error) { + if tag, err := r.peekTag(); err != nil { + return 0, err + } else if tag != tagSmallInteger && tag != tagInteger { + return 0, r.skip() + } + return r.readInt() +} diff --git a/internal/beam/debug_info_test.go b/internal/beam/debug_info_test.go new file mode 100644 index 0000000..0643d3b --- /dev/null +++ b/internal/beam/debug_info_test.go @@ -0,0 +1,504 @@ +package beam + +import ( + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +// testDefinition is one entry of the definitions list in a synthetic Dbgi chunk. +type testDefinition struct { + name string + arity int + kind string // def, defp, defmacro, defmacrop + line int // the :line meta; zero omits it + + // keepFile/keepLine write a `file: {path, line}` meta entry, which is what + // both `@file {path, line}` and `quote location: :keep` leave on a def. + keepFile string + keepLine int +} + +// buildDebugInfoTerm encodes {:debug_info_v1, :elixir_erl, {:elixir_v1, map, specs}} +// the way the Elixir compiler writes a Dbgi chunk. Each definition carries a +// clause body so the reader has to step over real AST shapes to reach the next. +func buildDebugInfoTerm(file, relativeFile string, definitions ...testDefinition) []byte { + return buildDebugInfoTermWithModuleLine(file, relativeFile, "", 0, definitions...) +} + +// buildDebugInfoTermWithModuleLine also writes the module's own line under +// moduleKey: "anno" as Elixir 1.18 and later do, or "line" as earlier +// versions do. An empty key writes none. +func buildDebugInfoTermWithModuleLine(file, relativeFile, moduleKey string, moduleLine int, definitions ...testDefinition) []byte { + var w etfTestWriter + w.smallTuple(3) + w.atom("debug_info_v1") + w.atom("elixir_erl") + w.smallTuple(3) + w.atom("elixir_v1") + + // Keys in the order ERTS sorts a small map's atom keys, which puts + // definitions before file and relative_file: the reader must not depend on + // having seen the file first. + if moduleKey != "" { + w.mapHeader(5) + } else { + w.mapHeader(4) + } + w.atom("attributes") + w.nil() + w.atom("definitions") + w.listHeader(len(definitions)) + for _, definition := range definitions { + w.smallTuple(4) + w.smallTuple(2) + w.atom(definition.name) + w.smallInt(definition.arity) + w.atom(definition.kind) + + pairs := 1 // generated: true + if definition.line > 0 { + pairs++ + } + if definition.keepFile != "" { + pairs++ + } + w.listHeader(pairs) + if definition.line > 0 { + w.smallTuple(2) + w.atom("line") + w.smallInt(definition.line) + } + if definition.keepFile != "" { + w.smallTuple(2) + w.atom("file") + w.smallTuple(2) + w.binary(definition.keepFile) + w.smallInt(definition.keepLine) + } + w.smallTuple(2) + w.atom("generated") + w.atom("true") + w.nil() + + // One clause, {meta, args, guards, body}, with a body shaped like a + // remote call: {{:., [], [Mod, :fun]}, [line: 1], [arg]}. + w.listHeader(1) + w.smallTuple(4) + w.nil() + w.listHeader(1) + w.smallTuple(3) + w.atom("value") + w.nil() + w.atom("nil") + w.nil() + w.nil() + w.smallTuple(3) + w.smallTuple(3) + w.atom(".") + w.nil() + w.listHeader(2) + w.atom("Elixir.Enum") + w.atom("map") + w.nil() + w.nil() + w.listHeader(1) + w.binary("a string literal in the body") + w.nil() + w.nil() + } + w.nil() + w.atom("file") + w.binary(file) + w.atom("relative_file") + w.binary(relativeFile) + if moduleKey != "" { + w.atom(moduleKey) + w.smallInt(moduleLine) + } + + w.nil() // specs + return w.buf +} + +func writeDebugInfoBEAM(t *testing.T, path string, dbgi []byte) { + t.Helper() + writeTestBEAMOpts(t, path, testBEAMOptions{ + atomNames: defaultTestAtoms, + exports: defaultTestExports, + dbgi: dbgi, + }) +} + +// A generator that stamps `@file {path, line}` on the functions it defines +// points each one at the source line that asked for it, such as an Ash +// `define :get_by_slug` inside a domain. When that path is the module's own +// source, that line is the definition. +func TestReadDefinitionLinesPrefersOwnFileLocation(t *testing.T) { + beamPath := filepath.Join(t.TempDir(), "Elixir.MyApp.Accounts.beam") + writeDebugInfoBEAM(t, beamPath, buildDebugInfoTerm( + "/src/my_app/lib/my_app/accounts.ex", "lib/my_app/accounts.ex", + testDefinition{name: "get_user!", arity: 1, kind: "def", line: 1, keepFile: "lib/my_app/accounts.ex", keepLine: 7}, + testDefinition{name: "get_user!", arity: 2, kind: "def", line: 1, keepFile: "lib/my_app/accounts.ex", keepLine: 7}, + )) + + info, err := ReadDefinitionLines(beamPath) + if err != nil { + t.Fatal(err) + } + if info.File != "/src/my_app/lib/my_app/accounts.ex" || info.RelativeFile != "lib/my_app/accounts.ex" { + t.Fatalf("source = %q / %q", info.File, info.RelativeFile) + } + for _, arity := range []int{1, 2} { + if got := info.Lines[FunctionKey{Name: "get_user!", Arity: arity}]; got != 7 { + t.Errorf("get_user!/%d line = %d, want 7", arity, got) + } + } +} + +// Elixir 1.17 and earlier record the module's line under :line, and later +// versions under :anno. +func TestReadDefinitionLinesModuleLineKeys(t *testing.T) { + for _, key := range []string{"anno", "line"} { + t.Run(key, func(t *testing.T) { + beamPath := filepath.Join(t.TempDir(), "Elixir.MyApp.Accounts.beam") + writeDebugInfoBEAM(t, beamPath, buildDebugInfoTermWithModuleLine( + "/src/my_app/lib/my_app/accounts.ex", "lib/my_app/accounts.ex", key, 3, + testDefinition{name: "get_user", arity: 1, kind: "def", line: 5}, + )) + info, err := ReadDefinitionLines(beamPath) + if err != nil { + t.Fatal(err) + } + if info.ModuleLine != 3 { + t.Errorf("ModuleLine = %d, want 3", info.ModuleLine) + } + }) + } +} + +// A location in the module's own file at or before the module's line is a +// `location: :keep` quote in a macro defined above the module, not a stamp +// that names where the module asked for the function, so :line is used. +func TestReadDefinitionLinesOwnFileLocationBeforeModule(t *testing.T) { + beamPath := filepath.Join(t.TempDir(), "Elixir.MyApp.Accounts.beam") + writeDebugInfoBEAM(t, beamPath, buildDebugInfoTermWithModuleLine( + "/src/my_app/lib/my_app/accounts.ex", "lib/my_app/accounts.ex", "anno", 9, + testDefinition{name: "kept", arity: 0, kind: "def", line: 12, keepFile: "lib/my_app/accounts.ex", keepLine: 4}, + testDefinition{name: "stamped", arity: 0, kind: "def", line: 9, keepFile: "lib/my_app/accounts.ex", keepLine: 14}, + )) + info, err := ReadDefinitionLines(beamPath) + if err != nil { + t.Fatal(err) + } + for name, want := range map[string]int{"kept": 12, "stamped": 14} { + if got := info.Lines[FunctionKey{Name: name, Arity: 0}]; got != want { + t.Errorf("%s/0 line = %d, want %d", name, got, want) + } + } +} + +// `quote location: :keep` inside a library records the library's own file. +// That line belongs to the generator, not to the module, so the module-side +// :line is used instead. +func TestReadDefinitionLinesIgnoresForeignFileLocation(t *testing.T) { + beamPath := filepath.Join(t.TempDir(), "Elixir.MyApp.Accounts.beam") + writeDebugInfoBEAM(t, beamPath, buildDebugInfoTerm( + "/src/my_app/lib/my_app/accounts.ex", "lib/my_app/accounts.ex", + testDefinition{name: "list_users", arity: 0, kind: "def", line: 12, keepFile: "deps/shared_lib/lib/shared_lib/interface.ex", keepLine: 1112}, + testDefinition{name: "count_users", arity: 0, kind: "def", line: 1, keepFile: "deps/shared_lib/lib/shared_lib/interface.ex", keepLine: 1112}, + )) + + info, err := ReadDefinitionLines(beamPath) + if err != nil { + t.Fatal(err) + } + if got := info.Lines[FunctionKey{Name: "list_users", Arity: 0}]; got != 12 { + t.Errorf("list_users/0 line = %d, want the module-side line 12", got) + } + if got := info.Lines[FunctionKey{Name: "count_users", Arity: 0}]; got != 1 { + t.Errorf("count_users/0 line = %d, want 1", got) + } +} + +func TestReadDefinitionLinesPlainLineAndMacros(t *testing.T) { + beamPath := filepath.Join(t.TempDir(), "Elixir.MyApp.Worker.beam") + writeDebugInfoBEAM(t, beamPath, buildDebugInfoTerm( + "/src/my_app/lib/my_app/worker.ex", "lib/my_app/worker.ex", + testDefinition{name: "new", arity: 1, kind: "def", line: 2}, + testDefinition{name: "build", arity: 1, kind: "defmacro", line: 9}, + testDefinition{name: "helper", arity: 0, kind: "defp", line: 4}, + testDefinition{name: "no_line", arity: 0, kind: "def"}, + )) + + info, err := ReadDefinitionLines(beamPath) + if err != nil { + t.Fatal(err) + } + want := map[FunctionKey]int{ + {Name: "new", Arity: 1}: 2, + {Name: "build", Arity: 1}: 9, + } + if len(info.Lines) != len(want) { + t.Fatalf("lines = %v, want only public definitions with a line %v", info.Lines, want) + } + for key, line := range want { + if info.Lines[key] != line { + t.Errorf("%v line = %d, want %d", key, info.Lines[key], line) + } + } +} + +func TestReadDefinitionLinesRejectsMissingOrForeignDebugInfo(t *testing.T) { + dir := t.TempDir() + + noChunk := filepath.Join(dir, "Elixir.NoChunk.beam") + writeTestBEAM(t, noChunk, buildDocsTerm()) + if _, err := ReadDefinitionLines(noChunk); err == nil { + t.Error("expected an error for a BEAM without a Dbgi chunk") + } + + // `debug_info: false` still writes the chunk, with :none as the payload. + var none etfTestWriter + none.smallTuple(3) + none.atom("debug_info_v1") + none.atom("elixir_erl") + none.atom("none") + stripped := filepath.Join(dir, "Elixir.Stripped.beam") + writeDebugInfoBEAM(t, stripped, none.buf) + if _, err := ReadDefinitionLines(stripped); err == nil { + t.Error("expected an error for stripped debug info") + } + + // An Erlang module's abstract code is a different backend entirely. + var erlang etfTestWriter + erlang.smallTuple(3) + erlang.atom("debug_info_v1") + erlang.atom("erl_abstract_code") + erlang.smallTuple(2) + erlang.atom("none") + erlang.nil() + erlangBeam := filepath.Join(dir, "erlang_mod.beam") + writeDebugInfoBEAM(t, erlangBeam, erlang.buf) + if _, err := ReadDefinitionLines(erlangBeam); err == nil { + t.Error("expected an error for a non-Elixir debug info backend") + } +} + +// A corrupt build artifact must fail the read, never the server. +func TestReadDefinitionLinesTruncated(t *testing.T) { + full := buildDebugInfoTerm("/src/a.ex", "a.ex", + testDefinition{name: "run", arity: 0, kind: "def", line: 3, keepFile: "a.ex", keepLine: 5}) + dir := t.TempDir() + for cut := 1; cut < len(full); cut += 7 { + path := filepath.Join(dir, "Elixir.Cut.beam") + writeDebugInfoBEAM(t, path, full[:cut]) + if _, err := ReadDefinitionLines(path); err == nil { + t.Fatalf("expected an error when the term is cut at %d of %d bytes", cut, len(full)) + } + } +} + +// Clause bodies are arbitrary AST and nest far deeper than a Docs chunk. A long +// pipeline or nested case is stepped over without recursion, so it costs no +// stack and does not lose the module. +func TestReadDefinitionLinesDeepClauseBody(t *testing.T) { + var w etfTestWriter + w.smallTuple(3) + w.atom("debug_info_v1") + w.atom("elixir_erl") + w.smallTuple(3) + w.atom("elixir_v1") + w.mapHeader(2) + w.atom("definitions") + w.listHeader(2) + for i, name := range []string{"deep", "after_deep"} { + w.smallTuple(4) + w.smallTuple(2) + w.atom(name) + w.smallInt(0) + w.atom("def") + w.listHeader(1) + w.smallTuple(2) + w.atom("line") + w.smallInt(10 + i) + w.nil() + // 100,000 nested {:|>, [], [left, ...]} levels: 200,000 ETF levels. + const depth = 100_000 + for range depth { + w.smallTuple(3) + w.atom("|>") + w.nil() + w.listHeader(1) + } + w.atom("nil") + for range depth { + w.nil() + } + } + w.nil() + w.atom("relative_file") + w.binary("lib/deep.ex") + w.nil() + + path := filepath.Join(t.TempDir(), "Elixir.Deep.beam") + writeDebugInfoBEAM(t, path, w.buf) + info, err := ReadDefinitionLines(path) + if err != nil { + t.Fatal(err) + } + if info.Lines[FunctionKey{Name: "after_deep", Arity: 0}] != 11 { + t.Fatalf("definition after a deep body = %v, want line 11", info.Lines) + } +} + +// The synthetic fixtures encode what the compiler is believed to write. This +// compiles real modules so a change in Elixir's debug info layout fails here +// instead of silently sending every definition back to the module line. +func TestReadDefinitionLinesFromElixirCompiler(t *testing.T) { + elixirc, err := exec.LookPath("elixirc") + if err != nil { + t.Skip("elixirc not installed") + } + dir := t.TempDir() + write := func(name, source string) { + t.Helper() + if err := os.WriteFile(filepath.Join(dir, name), []byte(source), 0o644); err != nil { + t.Fatal(err) + } + } + // SharedLib.Interface generates functions three ways: with `@file` pointing + // at the line of the call that asked for it (as a DSL records it), from a + // `location: :keep` quote, and plainly. + write("interface.ex", `defmodule SharedLib.Interface do + defmacro define(name) do + line = __CALLER__.line + + quote bind_quoted: [name: name, line: line], location: :keep do + @file {__ENV__.file, line} + def unquote(name)(), do: :stamped + end + end + + defmacro define_kept(name) do + quote bind_quoted: [name: name], location: :keep do + def unquote(name)(), do: :kept + end + end + + defmacro define_plain(name) do + quote do + def unquote(name)(), do: :plain + end + end +end +`) + write("accounts.ex", `defmodule MyApp.Accounts do + require SharedLib.Interface + + SharedLib.Interface.define(:stamped) + + SharedLib.Interface.define_kept(:kept) + + SharedLib.Interface.define_plain(:plain) +end +`) + cmd := exec.Command(elixirc, "-o", dir, "interface.ex", "accounts.ex") + cmd.Dir = dir + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("elixirc: %v\n%s", err, out) + } + + info, err := ReadDefinitionLines(filepath.Join(dir, "Elixir.MyApp.Accounts.beam")) + if err != nil { + t.Fatal(err) + } + if !strings.HasSuffix(filepath.ToSlash(info.File), "/accounts.ex") { + t.Errorf("File = %q, want the compiled source", info.File) + } + want := map[string]int{"stamped": 4, "kept": 6, "plain": 8} + for name, line := range want { + if got := info.Lines[FunctionKey{Name: name, Arity: 0}]; got != line { + t.Errorf("%s/0 line = %d, want %d (all: %v)", name, got, line, info.Lines) + } + } + + // The Docs anno is the fallback for a module without debug info, and the + // compile info says which file it is in. + source, ok := ReadSourcePath(filepath.Join(dir, "Elixir.MyApp.Accounts.beam")) + if !ok || !strings.HasSuffix(filepath.ToSlash(source), "/accounts.ex") { + t.Errorf("ReadSourcePath = %q, %v; want the compiled source", source, ok) + } + documented, err := ReadDocumentedFunctions(filepath.Join(dir, "Elixir.MyApp.Accounts.beam")) + if err != nil { + t.Fatal(err) + } + for _, function := range documented { + if line, ok := want[function.Name]; ok && function.Line != line { + t.Errorf("%s/0 Docs line = %d, want %d", function.Name, function.Line, line) + } + } + if len(documented) != len(want) { + t.Errorf("documented = %+v, want %d functions", documented, len(want)) + } +} + +// A macro defined above its caller in the same file, with `location: :keep`, +// records that file and the line inside the macro. The call is the answer. +func TestReadDefinitionLinesSameFileKeepFromElixirCompiler(t *testing.T) { + elixirc, err := exec.LookPath("elixirc") + if err != nil { + t.Skip("elixirc not installed") + } + dir := t.TempDir() + source := `defmodule Same.Dsl do + defmacro kept(name) do + quote location: :keep do + def unquote(name)(), do: :kept + end + end +end + +defmodule Same.User do + require Same.Dsl + + Same.Dsl.kept(:one) +end +` + if err := os.WriteFile(filepath.Join(dir, "same.ex"), []byte(source), 0o644); err != nil { + t.Fatal(err) + } + cmd := exec.Command(elixirc, "-o", dir, "same.ex") + cmd.Dir = dir + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("elixirc: %v\n%s", err, out) + } + info, err := ReadDefinitionLines(filepath.Join(dir, "Elixir.Same.User.beam")) + if err != nil { + t.Fatal(err) + } + if info.ModuleLine != 9 { + t.Errorf("ModuleLine = %d, want 9", info.ModuleLine) + } + if got := info.Lines[FunctionKey{Name: "one", Arity: 0}]; got != 12 { + t.Errorf("one/0 line = %d, want 12, the call (all: %v)", got, info.Lines) + } +} + +// A corrupt definitions count must fail the parse, not allocate for it first. +func TestParseDebugInfoHugeDefinitionCount(t *testing.T) { + var w etfTestWriter + w.smallTuple(3) + w.atom("debug_info_v1") + w.atom("elixir_erl") + w.smallTuple(3) + w.atom("elixir_v1") + w.mapHeader(1) + w.atom("definitions") + w.listHeader(1 << 30) + if _, err := parseDebugInfo(w.buf); err == nil { + t.Fatal("parseDebugInfo accepted a truncated definitions list") + } +} diff --git a/internal/beam/docs.go b/internal/beam/docs.go index f2e2df4..d4b042a 100644 --- a/internal/beam/docs.go +++ b/internal/beam/docs.go @@ -4,6 +4,7 @@ package beam import ( "bytes" + "compress/gzip" "compress/zlib" "encoding/binary" "errors" @@ -29,6 +30,18 @@ type Function struct { Kind string Hidden bool + // Line is the source annotation the compiler recorded for this callable, as + // emitted in the Docs chunk. It is zero when the callable came from the export + // table alone, which has no annotation. + // + // The annotation names the site the code was created at, not the module it + // ended up in: a symbol created while expanding `use X` is attributed to the + // `use` line, while one created by a transformer or a `@before_compile` hook is + // attributed to the module line. Combined with the owning module's source file + // (see ReadSourcePath) that is a usable definition target for code that exists + // only in compiled form. + Line int + // DocOffset and DocLen locate the documentation prose inside the inflated // Docs chunk, so a hover can read one function's docs without decoding the // chunk again. They are zero when the function has no prose. zlib output is @@ -125,11 +138,26 @@ func readChunks(path string, wanted ...string) (map[string][]byte, error) { if _, err := io.ReadFull(f, header[:]); err != nil { return nil, err } + // A module compiled with the `compressed` option, as some Erlang + // dependencies are, is a gzip stream around the usual container. + var r io.ReaderAt = f + size := info.Size() + if header[0] == 0x1f && header[1] == 0x8b { + data, err := gunzipBEAM(f) + if err != nil { + return nil, err + } + if len(data) < 12 { + return nil, fmt.Errorf("invalid BEAM size %d", len(data)) + } + r, size = bytes.NewReader(data), int64(len(data)) + copy(header[:], data) + } if string(header[:4]) != "FOR1" || string(header[8:]) != "BEAM" { return nil, errors.New("invalid BEAM header") } declared := int64(binary.BigEndian.Uint32(header[4:8])) + 8 - if declared > info.Size() { + if declared > size { return nil, errors.New("truncated BEAM container") } @@ -141,7 +169,7 @@ func readChunks(path string, wanted ...string) (map[string][]byte, error) { for offset := int64(12); offset+8 <= declared && len(found) < len(pending); { var chunkHeader [8]byte - if _, err := f.ReadAt(chunkHeader[:], offset); err != nil { + if _, err := r.ReadAt(chunkHeader[:], offset); err != nil { return nil, err } length := int64(binary.BigEndian.Uint32(chunkHeader[4:8])) @@ -157,7 +185,7 @@ func readChunks(path string, wanted ...string) (map[string][]byte, error) { return nil, fmt.Errorf("%s chunk too large: %d", name, length) } data := make([]byte, length) - if _, err := f.ReadAt(data, dataOffset); err != nil { + if _, err := r.ReadAt(data, dataOffset); err != nil { return nil, err } found[name] = data @@ -167,6 +195,28 @@ func readChunks(path string, wanted ...string) (map[string][]byte, error) { return found, nil } +// gunzipBEAM decompresses a compressed BEAM from its start. The output is +// bounded like an uncompressed file, so a small file cannot expand into an +// unbounded allocation. +func gunzipBEAM(f *os.File) ([]byte, error) { + if _, err := f.Seek(0, io.SeekStart); err != nil { + return nil, err + } + zr, err := gzip.NewReader(f) + if err != nil { + return nil, fmt.Errorf("decompress BEAM: %w", err) + } + defer func() { _ = zr.Close() }() + data, err := io.ReadAll(io.LimitReader(zr, maxBEAMSize+1)) + if err != nil { + return nil, fmt.Errorf("decompress BEAM: %w", err) + } + if len(data) > maxBEAMSize { + return nil, errors.New("decompressed BEAM too large") + } + return data, nil +} + func readChunk(path, wanted string) ([]byte, error) { chunks, err := readChunks(path, wanted) if err != nil { @@ -456,7 +506,8 @@ func parseDocsEntry(r *etfReader) ([]Function, bool, error) { return nil, false, err } - if err := r.skip(); err != nil { // anno + line, err := readAnnoLine(r) + if err != nil { return nil, false, err } params, err := readSignatureParams(r) @@ -502,6 +553,7 @@ func parseDocsEntry(r *etfReader) ([]Function, bool, error) { Params: strings.Join(callParams, ","), Kind: definitionKind, Hidden: hidden, + Line: line, DocOffset: docOffset, DocLen: docLen, }) @@ -509,6 +561,44 @@ func parseDocsEntry(r *etfReader) ([]Function, bool, error) { return out, true, nil } +// readAnnoLine reads the per-entry source annotation. Elixir writes it as a bare +// line number, but the field is an erl_anno and may hold other shapes: a +// {line, column} tuple gives its line, and anything else, such as `none`, is +// consumed and reported as line 0 rather than failing the whole entry. A dropped annotation must never cost the +// module its generated functions. +func readAnnoLine(r *etfReader) (int, error) { + // rest counts the elements after each tuple's first one, which are stepped + // over once the line is found. A loop rather than recursion, so a corrupt + // chunk of nested tuples costs no stack. + rest := int64(0) + for { + tag, err := r.peekTag() + if err != nil { + return 0, err + } + switch tag { + case tagSmallInteger, tagInteger: + line, err := r.readInt() + if err != nil { + return 0, err + } + return line, r.skipTerms(rest) + case tagSmallTuple: + // {line, column}, as erl_anno writes a location with a column. + arity, err := r.enterTuple() + if err != nil { + return 0, err + } + if arity == 0 { + return 0, r.skipTerms(rest) + } + rest += int64(arity) - 1 + continue + } + return 0, r.skipTerms(rest + 1) + } +} + // readSignatureParams consumes the signature list and parses the first signature // into argument names. Signatures are short, so unlike documentation prose they // are worth copying. diff --git a/internal/beam/docs_test.go b/internal/beam/docs_test.go index 9233d92..d01abfd 100644 --- a/internal/beam/docs_test.go +++ b/internal/beam/docs_test.go @@ -2,6 +2,7 @@ package beam import ( "bytes" + "compress/gzip" "compress/zlib" "encoding/binary" "os" @@ -135,20 +136,35 @@ func TestParseDocsMalformed(t *testing.T) { } } -// Deeply nested terms must be rejected rather than followed into a stack -// overflow, which a corrupt length field could otherwise cause. -func TestParseDocsRejectsDeepNesting(t *testing.T) { +// Stepping over a deeply nested term costs no stack, and a nested anno is read +// the same way: a million levels, which a corrupt chunk could hold, must not +// overflow the stack. +func TestParseDocsDeepNestingCostsNoStack(t *testing.T) { var w etfTestWriter w.smallTuple(7) w.atom("docs_v1") - // The anno field is skipped, so nesting it past the limit exercises the - // depth guard inside skip rather than the header checks. - for range maxETFDepth + 10 { + for range 1_000_000 { w.smallTuple(1) } w.nil() + // The rest of docs_v1 is missing, so the parse fails, but only after it + // stepped over the nested anno. if _, err := parseDocs(w.buf); err == nil { - t.Error("expected deeply nested input to be rejected") + t.Error("expected the incomplete term to be rejected") + } + + var anno etfTestWriter + for range 1_000_000 { + anno.smallTuple(2) + } + anno.smallInt(7) + for range 1_000_000 { + anno.smallInt(1) + } + r := &etfReader{buf: anno.buf} + line, err := readAnnoLine(r) + if err != nil || line != 7 || r.remaining() != 0 { + t.Errorf("readAnnoLine = %d, %v with %d bytes left; want 7 and the whole term read", line, err, r.remaining()) } } @@ -197,6 +213,59 @@ func TestReadExports(t *testing.T) { } } +// A module compiled with the `compressed` option is a gzip stream around the +// container. Its exports read the same, and a compressed file that does not +// hold a container is still rejected. +func TestReadExportsCompressedBEAM(t *testing.T) { + dir := t.TempDir() + plain := filepath.Join(dir, "plain.beam") + writeTestBEAM(t, plain, buildDocsTerm()) + data, err := os.ReadFile(plain) + if err != nil { + t.Fatal(err) + } + gzipped := func(data []byte) []byte { + var buf bytes.Buffer + zw := gzip.NewWriter(&buf) + if _, err := zw.Write(data); err != nil { + t.Fatal(err) + } + if err := zw.Close(); err != nil { + t.Fatal(err) + } + return buf.Bytes() + } + + compressed := filepath.Join(dir, "Elixir.Example.beam") + if err := os.WriteFile(compressed, gzipped(data), 0o644); err != nil { + t.Fatal(err) + } + want, err := ReadExports(plain) + if err != nil { + t.Fatal(err) + } + got, err := ReadExports(compressed) + if err != nil { + t.Fatal(err) + } + if len(got) != len(want) { + t.Fatalf("compressed exports = %#v, want %#v", got, want) + } + for i := range want { + if got[i] != want[i] { + t.Fatalf("compressed export %d = %#v, want %#v", i, got[i], want[i]) + } + } + + notBEAM := filepath.Join(dir, "not.beam") + if err := os.WriteFile(notBEAM, gzipped([]byte("this is not a BEAM container")), 0o644); err != nil { + t.Fatal(err) + } + if _, err := ReadExports(notBEAM); err == nil { + t.Fatal("ReadExports accepted a compressed file that holds no BEAM container") + } +} + // OTP 28 widened atom names past 255 bytes and re-encoded the AtU8 chunk: the // count is stored negated and every length becomes a tagged varint instead of a // raw byte. The chunk kept its name, so parsers must branch on the sign bit. @@ -374,6 +443,8 @@ type testBEAMOptions struct { exports [][3]uint32 docs []byte attrs []byte + dbgi []byte // written compressed, as the compiler writes it + cinf []byte // written as given } func writeTestBEAM(t *testing.T, path string, docs []byte) { @@ -389,25 +460,11 @@ func writeTestBEAMOpts(t *testing.T, path string, opts testBEAMOptions) { t.Helper() var size [4]byte - var raw bytes.Buffer - raw.WriteByte(etfVersion) + var raw []byte if len(opts.docs) > 0 { - raw.WriteByte(tagCompressed) - var uncompressed bytes.Buffer - uncompressed.Write(opts.docs) - binary.BigEndian.PutUint32(size[:], uint32(uncompressed.Len())) - raw.Write(size[:]) - var zw bytes.Buffer - compressor := zlib.NewWriter(&zw) - if _, err := compressor.Write(uncompressed.Bytes()); err != nil { - t.Fatal(err) - } - if err := compressor.Close(); err != nil { - t.Fatal(err) - } - raw.Write(zw.Bytes()) + raw = compressedTestTerm(t, opts.docs) } else { - raw.WriteByte(tagNil) + raw = []byte{etfVersion, tagNil} } var atoms bytes.Buffer @@ -443,7 +500,13 @@ func writeTestBEAMOpts(t *testing.T, path string, opts testBEAMOptions) { if len(opts.attrs) > 0 { writeTestChunk(&chunks, "Attr", opts.attrs) } - writeTestChunk(&chunks, "Docs", raw.Bytes()) + writeTestChunk(&chunks, "Docs", raw) + if len(opts.dbgi) > 0 { + writeTestChunk(&chunks, "Dbgi", compressedTestTerm(t, opts.dbgi)) + } + if len(opts.cinf) > 0 { + writeTestChunk(&chunks, "CInf", opts.cinf) + } var file bytes.Buffer file.WriteString("FOR1") @@ -456,6 +519,25 @@ func writeTestBEAMOpts(t *testing.T, path string, opts testBEAMOptions) { } } +// compressedTestTerm wraps an encoded term as a versioned COMPRESSED_TERM, the +// form Elixir writes its Docs and Dbgi chunks in. +func compressedTestTerm(t *testing.T, term []byte) []byte { + t.Helper() + var size [4]byte + raw := []byte{etfVersion, tagCompressed} + binary.BigEndian.PutUint32(size[:], uint32(len(term))) + raw = append(raw, size[:]...) + var zw bytes.Buffer + compressor := zlib.NewWriter(&zw) + if _, err := compressor.Write(term); err != nil { + t.Fatal(err) + } + if err := compressor.Close(); err != nil { + t.Fatal(err) + } + return append(raw, zw.Bytes()...) +} + func writeTestChunk(chunks *bytes.Buffer, name string, data []byte) { chunks.WriteString(name) var size [4]byte @@ -488,6 +570,10 @@ type docEntry struct { doc string // "none", "hidden", or the documentation prose defaults int + // anno is the source annotation the compiler records for the entry: the line + // the code was created at. Zero means the usual module line. + anno int + // extraMetadata writes additional pairs into the entry's metadata map, the // way a framework's `@doc spark_opts` lands alongside `defaults`. extraMetadata func(*etfTestWriter) @@ -523,7 +609,11 @@ func buildDocsTerm(entries ...docEntry) []byte { w.atom(entry.name) w.smallInt(entry.arity) - w.smallInt(1) // anno + anno := entry.anno + if anno == 0 { + anno = 1 + } + w.smallInt(anno) w.listHeader(1) w.binary(entry.signature) @@ -599,6 +689,14 @@ func (w *etfTestWriter) binary(text string) { w.buf = append(w.buf, text...) } +// string writes a STRING_EXT term, the compact encoding Erlang uses for the +// printable charlists it records in compile info. +func (w *etfTestWriter) string(text string) { + w.byte(tagString) + w.be16(uint16(len(text))) + w.buf = append(w.buf, text...) +} + func (w *etfTestWriter) smallInt(value int) { if value >= 0 && value < 256 { w.byte(tagSmallInteger) @@ -665,3 +763,76 @@ func TestParseDocsMetadataWithCapturedFunction(t *testing.T) { t.Errorf("entry after the captured function = %+v, want list/0", got[1]) } } + +// The compile-time annotation is what makes a generated callable navigable: it +// names the line the code was created at, which for a macro-emitted symbol is a +// specific site rather than the module the symbol ended up in. +func TestReadDocumentedFunctionsRecordsSourceLine(t *testing.T) { + docs := buildDocsTerm( + docEntry{kind: "macro", name: "code_interface", arity: 1, signature: "code_interface(body)", doc: "none", anno: 1811}, + docEntry{kind: "function", name: "generated_at_module_line", arity: 0, signature: "generated_at_module_line()", doc: "none", anno: 1}, + ) + path := filepath.Join(t.TempDir(), "Elixir.LibFixture.beam") + writeTestBEAM(t, path, docs) + + functions, err := ReadDocumentedFunctions(path) + if err != nil { + t.Fatalf("ReadDocumentedFunctions: %v", err) + } + + got := make(map[string]int, len(functions)) + for _, f := range functions { + got[f.Name] = f.Line + } + for name, want := range map[string]int{ + "code_interface": 1811, + "generated_at_module_line": 1, + } { + if got[name] != want { + t.Errorf("%s: Line = %d, want %d", name, got[name], want) + } + } +} + +// An annotation the reader does not understand must cost only the line, never +// the entry: dropping the whole Docs chunk would take every generated function +// with it. +func TestReadDocumentedFunctionsToleratesNonIntegerAnno(t *testing.T) { + // writeTestBEAM prepends the ETF version byte, so the payload starts at the + // term itself — same as buildDocsTerm. + var w etfTestWriter + w.smallTuple(7) + w.atom("docs_v1") + w.smallInt(1) + w.atom("elixir") + w.binary("text/markdown") + w.atom("none") + w.mapHeader(0) + w.listHeader(1) + w.smallTuple(5) + w.smallTuple(3) + w.atom("function") + w.atom("kept") + w.smallInt(0) + w.atom("none") // anno as an atom instead of a line + w.listHeader(1) + w.binary("kept()") + w.nil() + w.atom("none") + w.mapHeader(0) + w.nil() + + path := filepath.Join(t.TempDir(), "Elixir.LibFixture.beam") + writeTestBEAM(t, path, w.buf) + + functions, err := ReadDocumentedFunctions(path) + if err != nil { + t.Fatalf("ReadDocumentedFunctions: %v", err) + } + if len(functions) != 1 || functions[0].Name != "kept" { + t.Fatalf("functions = %+v, want the single kept/0 entry", functions) + } + if functions[0].Line != 0 { + t.Errorf("Line = %d, want 0 for an unreadable anno", functions[0].Line) + } +} diff --git a/internal/beam/etf.go b/internal/beam/etf.go index 58e5de2..5db9899 100644 --- a/internal/beam/etf.go +++ b/internal/beam/etf.go @@ -40,15 +40,10 @@ func isAtomTag(tag byte) bool { return tag == tagAtom || tag == tagAtomUTF8 || tag == tagSmallAtom || tag == tagSmallAtomUTF8 } -// maxETFDepth bounds recursion in skip. A Docs chunk nests at most five levels -// deep, so a corrupt length that implied more is rejected rather than followed. -const maxETFDepth = 64 - var ( errUnsupportedTag = errors.New("unsupported ETF tag") errTruncated = errors.New("truncated ETF term") errBadCount = errors.New("ETF count exceeds remaining bytes") - errTooDeep = errors.New("ETF nesting too deep") ) // etfReader walks an ETF term in place. @@ -63,9 +58,8 @@ var ( // Every read is bounds-checked, so malformed input yields an error rather than a // panic or an out-of-range slice. type etfReader struct { - buf []byte - pos int - depth int + buf []byte + pos int } func (r *etfReader) remaining() int { return len(r.buf) - r.pos } @@ -298,129 +292,119 @@ func (r *etfReader) binarySpan() (start, length int, err error) { // skip advances past one term of any shape without allocating for it. func (r *etfReader) skip() error { - if r.depth >= maxETFDepth { - return errTooDeep + return r.skipTerms(1) +} + +// skipTerms steps over count terms. It does not recurse: a tuple, list or map +// adds its elements to the terms still to step over, so a deeply nested term, +// as a clause body in a Dbgi chunk can be, costs no stack. Every term takes at +// least one byte, so more terms to step over than bytes left is corrupt input, +// and is rejected before it can drive the loop. +func (r *etfReader) skipTerms(count int64) error { + pending := int64(0) + add := func(n int64) error { + if n < 0 { + return errBadCount + } + pending += n + if pending > int64(r.remaining()) { + return errBadCount + } + return nil } - tag, err := r.u8() - if err != nil { + if err := add(count); err != nil { return err } - switch tag { - case tagNil: - return nil - case tagSmallInteger: - return r.skipBytes(1) - case tagInteger: - return r.skipBytes(4) - case tagNewFloat: - return r.skipBytes(8) - case tagFloat: - return r.skipBytes(31) - case tagAtom, tagAtomUTF8, tagString: - n, err := r.u16() - if err != nil { - return err - } - return r.skipBytes(int64(n)) - case tagSmallAtom, tagSmallAtomUTF8: - n, err := r.u8() - if err != nil { - return err - } - return r.skipBytes(int64(n)) - case tagBinary: - n, err := r.u32() - if err != nil { - return err - } - return r.skipBytes(n) - case tagBitBinary: - n, err := r.u32() - if err != nil { - return err - } - if err := r.skipBytes(1); err != nil { - return err - } - return r.skipBytes(n) - case tagSmallBig: - n, err := r.u8() - if err != nil { - return err - } - return r.skipBytes(int64(n) + 1) - case tagLargeBig: - n, err := r.u32() - if err != nil { - return err - } - return r.skipBytes(n + 1) - case tagSmallTuple: - n, err := r.u8() + for pending > 0 { + // Counted off before its header is read, so that its elements are + // compared with the bytes left after it, not with it as well. + pending-- + tag, err := r.u8() if err != nil { return err } - return r.skipTerms(int64(n)) - case tagLargeTuple: - n, err := r.u32() - if err != nil { - return err + switch tag { + case tagNil: + case tagSmallInteger: + err = r.skipBytes(1) + case tagInteger: + err = r.skipBytes(4) + case tagNewFloat: + err = r.skipBytes(8) + case tagFloat: + err = r.skipBytes(31) + case tagAtom, tagAtomUTF8, tagString: + var n int + if n, err = r.u16(); err == nil { + err = r.skipBytes(int64(n)) + } + case tagSmallAtom, tagSmallAtomUTF8: + var n byte + if n, err = r.u8(); err == nil { + err = r.skipBytes(int64(n)) + } + case tagBinary: + var n int64 + if n, err = r.u32(); err == nil { + err = r.skipBytes(n) + } + case tagBitBinary: + var n int64 + if n, err = r.u32(); err == nil { + err = r.skipBytes(n + 1) + } + case tagSmallBig: + var n byte + if n, err = r.u8(); err == nil { + err = r.skipBytes(int64(n) + 1) + } + case tagLargeBig: + var n int64 + if n, err = r.u32(); err == nil { + err = r.skipBytes(n + 1) + } + case tagSmallTuple: + var n byte + if n, err = r.u8(); err == nil { + err = add(int64(n)) + } + case tagLargeTuple: + var n int64 + if n, err = r.u32(); err == nil { + err = add(n) + } + case tagList: + // The elements, then the tail. + var n int64 + if n, err = r.u32(); err == nil { + err = add(n + 1) + } + case tagMap: + var n int64 + if n, err = r.u32(); err == nil { + err = add(n * 2) + } + case tagNewFun: + // Size counts the whole term including the four size bytes themselves. + var n int64 + if n, err = r.u32(); err == nil { + err = r.skipBytes(n - 4) + } + case tagExport: + // An external fun, `fun M:F/A`. Unlike NEW_FUN_EXT, whose Arity is a + // bare byte, this one encodes the arity as an integer term, so + // skipping a fixed byte would misalign every term that follows it. + if _, err = r.readAtom(); err == nil { + if _, err = r.readAtom(); err == nil { + err = add(1) + } + } + default: + err = fmt.Errorf("%w: %d", errUnsupportedTag, tag) } - return r.skipTerms(n) - case tagList: - n, err := r.u32() if err != nil { return err } - if err := r.checkCount(n, 1); err != nil { - return err - } - if err := r.skipTerms(n); err != nil { - return err - } - return r.skip() // the tail - case tagMap: - n, err := r.u32() - if err != nil { - return err - } - if err := r.checkCount(n, 2); err != nil { - return err - } - return r.skipTerms(n * 2) - case tagNewFun: - // Size counts the whole term including the four size bytes themselves. - n, err := r.u32() - if err != nil { - return err - } - return r.skipBytes(n - 4) - case tagExport: - // An external fun, `fun M:F/A`. Unlike NEW_FUN_EXT, whose Arity is a bare - // byte, this one encodes the arity as an integer term, so skipping a fixed - // byte would misalign every term that follows it. - if _, err := r.readAtom(); err != nil { - return err - } - if _, err := r.readAtom(); err != nil { - return err - } - return r.skip() - default: - return fmt.Errorf("%w: %d", errUnsupportedTag, tag) - } -} - -func (r *etfReader) skipTerms(count int64) error { - if err := r.checkCount(count, 1); err != nil { - return err - } - r.depth++ - defer func() { r.depth-- }() - for i := int64(0); i < count; i++ { - if err := r.skip(); err != nil { - return err - } } return nil } diff --git a/internal/beam/etf_test.go b/internal/beam/etf_test.go index d38de00..05359a1 100644 --- a/internal/beam/etf_test.go +++ b/internal/beam/etf_test.go @@ -45,3 +45,53 @@ func TestSkipExportExtKeepsAlignment(t *testing.T) { }) } } + +// A term that ends exactly at the end of the input is stepped over whole, and +// every strict prefix of it is rejected: the count of terms still to step over +// is compared with the bytes after each header, not with the header as well. +func TestSkipTermsThatEndTheInput(t *testing.T) { + cases := map[string]func(w *etfTestWriter){ + "tuple of an empty list": func(w *etfTestWriter) { + w.smallTuple(1) + w.nil() + }, + "empty map": func(w *etfTestWriter) { + w.mapHeader(0) + }, + "list of empty lists": func(w *etfTestWriter) { + w.listHeader(2) + w.nil() + w.nil() + w.nil() + }, + "map of empty lists": func(w *etfTestWriter) { + w.mapHeader(1) + w.nil() + w.nil() + }, + "nested tuples": func(w *etfTestWriter) { + w.smallTuple(2) + w.smallTuple(1) + w.nil() + w.listHeader(1) + w.smallTuple(0) + w.nil() + }, + } + for name, build := range cases { + t.Run(name, func(t *testing.T) { + var w etfTestWriter + build(&w) + r := &etfReader{buf: w.buf} + if err := r.skip(); err != nil || r.remaining() != 0 { + t.Fatalf("skip = %v with %d bytes left, want the whole term", err, r.remaining()) + } + for i := range len(w.buf) { + r := &etfReader{buf: w.buf[:i]} + if err := r.skip(); err == nil { + t.Errorf("skip accepted a truncation at %d of %d bytes", i, len(w.buf)) + } + } + }) + } +} diff --git a/internal/daemon/endpoint.go b/internal/daemon/endpoint.go index 9c550bd..f5bdb68 100644 --- a/internal/daemon/endpoint.go +++ b/internal/daemon/endpoint.go @@ -18,7 +18,7 @@ import ( // running daemon, whose startup then rebuilds a populated index whose // IndexVersion differs. Routine changes that leave frontends and daemons // compatible do not bump it, so a running daemon is left alone. -const ContractVersion = 2 +const ContractVersion = 3 // maxSocketPath keeps a workspace socket inside sockaddr_un on every supported // platform (about 104 bytes on macOS, 108 on Linux), including the NUL. diff --git a/internal/lsp/beam_path_test.go b/internal/lsp/beam_path_test.go index ad35ce0..57886f3 100644 --- a/internal/lsp/beam_path_test.go +++ b/internal/lsp/beam_path_test.go @@ -157,17 +157,33 @@ type testAttr struct { // makes a compiled module record facts its source does not state, which is the // only way a framework's macro provider can be discovered. func minimalBeamWithAttrs(attrs []testAttr, docs string, exports ...beamExport) []byte { - return buildTestBeam(attrs, docs, exports...) + return buildTestBeam(testBeamChunks{attrs: attrs, docs: docs}, exports...) } // minimalBeamWithDocs is minimalBeam plus a Docs chunk whose inflated payload is // docs. ReadDocBody inflates that payload and slices it, so a test can put one // function's prose at a known offset without encoding a real docs_v1 term. func minimalBeamWithDocs(docs string, exports ...beamExport) []byte { - return buildTestBeam(nil, docs, exports...) + return buildTestBeam(testBeamChunks{docs: docs}, exports...) } -func buildTestBeam(attrs []testAttr, docs string, exports ...beamExport) []byte { +// minimalBeamWithDbgi adds a Dbgi chunk holding the encoded debug info term, +// which is where a compiled module records the line of each definition. +func minimalBeamWithDbgi(dbgi []byte, exports ...beamExport) []byte { + return buildTestBeam(testBeamChunks{dbgi: dbgi}, exports...) +} + +// testBeamChunks are the optional chunks of a synthetic BEAM. +type testBeamChunks struct { + attrs []testAttr + docs string + dbgi []byte + // source is written as the CInf chunk's :source, the file the module was + // compiled from. + source string +} + +func buildTestBeam(chunks testBeamChunks, exports ...beamExport) []byte { names := make([]string, 0, len(exports)+1) names = append(names, "Elixir.Minimal") for _, export := range exports { @@ -189,21 +205,28 @@ func buildTestBeam(attrs []testAttr, docs string, exports ...beamExport) []byte writeBE32(&table, uint32(index+1)) // label } - var chunks bytes.Buffer - writeBeamChunk(&chunks, "AtU8", atoms.Bytes()) - writeBeamChunk(&chunks, "ExpT", table.Bytes()) - if docs != "" { - writeBeamChunk(&chunks, "Docs", docsChunk(docs)) + var body bytes.Buffer + writeBeamChunk(&body, "AtU8", atoms.Bytes()) + writeBeamChunk(&body, "ExpT", table.Bytes()) + if chunks.docs != "" { + writeBeamChunk(&body, "Docs", docsChunk(chunks.docs)) } - if len(attrs) > 0 { - writeBeamChunk(&chunks, "Attr", attrChunk(attrs)) + if len(chunks.attrs) > 0 { + writeBeamChunk(&body, "Attr", attrChunk(chunks.attrs)) + } + if len(chunks.dbgi) > 0 { + // Uncompressed, like docsChunk; the reader accepts both forms. + writeBeamChunk(&body, "Dbgi", append([]byte{131}, chunks.dbgi...)) + } + if chunks.source != "" { + writeBeamChunk(&body, "CInf", compileInfoChunk(chunks.source)) } var file bytes.Buffer file.WriteString("FOR1") - writeBE32(&file, uint32(chunks.Len()+4)) + writeBE32(&file, uint32(body.Len()+4)) file.WriteString("BEAM") - file.Write(chunks.Bytes()) + file.Write(body.Bytes()) return file.Bytes() } @@ -215,6 +238,19 @@ func docsChunk(docs string) []byte { return append(chunk, docs...) } +// compileInfoChunk encodes the CInf chunk ReadSourcePath reads, with :source +// as the compact string term the compiler writes for a charlist. +func compileInfoChunk(source string) []byte { + out := []byte{131} // ETF version byte + out = append(out, etfListHeader(1)...) + out = append(out, 104, 2) // small tuple of arity 2 + out = append(out, etfAtom("source")...) + out = append(out, 107) // STRING_EXT + out = binary.BigEndian.AppendUint16(out, uint16(len(source))) + out = append(out, source...) + return append(out, 106) // NIL_EXT +} + // attrChunk encodes the Attr chunk ReadModuleAttributes reads: an ETF list of // {name, [values]} pairs, with the values as atoms because that is how the // compiler persists module names. diff --git a/internal/lsp/generated_completion.go b/internal/lsp/generated_completion.go index af48e8c..d89140b 100644 --- a/internal/lsp/generated_completion.go +++ b/internal/lsp/generated_completion.go @@ -43,6 +43,13 @@ type generatedFunctionCacheEntry struct { providers []string providersResolved bool + // definitionSources holds what the BEAM records about where its functions + // were defined, read on the first definition request that needs it. + // Memoized on the entry for the same reason as providers: the BEAM stamp + // invalidates both. + definitionSources generatedDefinitionSources + definitionSourcesResolved bool + // docs memoizes rendered documentation prose for functions a hover has asked // about, so sweeping the mouse does not re-inflate the Docs chunk each time. // It is a pointer because get hands out a copy of the entry: the copy has to @@ -692,11 +699,12 @@ func generatedFunctionsNamed(functions []beam.Function, name string) []beam.Func // generatedDefinitionResults returns the closest source-backed module for a // generated provider. Generated nested modules have no source row, so walking // their lexical parents yields a stable artifact-level destination without -// knowing which framework created them. -func (s *Server) generatedDefinitionResults(module string) []store.LookupResult { +// knowing which framework created them. owner is the module those results +// belong to: module itself, or the lexical parent that was found. +func (s *Server) generatedDefinitionResults(module string) (results []store.LookupResult, owner string) { for candidate := module; candidate != ""; { if results, err := s.store.LookupModule(candidate); err == nil && len(results) > 0 { - return results + return results, candidate } dot := strings.LastIndexByte(candidate, '.') if dot < 0 { @@ -704,7 +712,7 @@ func (s *Server) generatedDefinitionResults(module string) []store.LookupResult } candidate = candidate[:dot] } - return nil + return nil, "" } // filterGeneratedProviderReferences removes the conservative false positives diff --git a/internal/lsp/generated_definition.go b/internal/lsp/generated_definition.go new file mode 100644 index 0000000..d8c9cbf --- /dev/null +++ b/internal/lsp/generated_definition.go @@ -0,0 +1,643 @@ +package lsp + +import ( + "os" + "path/filepath" + "slices" + "strings" + "time" + + "github.com/remoteoss/dexter/internal/beam" + "github.com/remoteoss/dexter/internal/parser" + "github.com/remoteoss/dexter/internal/store" +) + +// generatedDefinitionSources is what a compiled module records about where its +// functions were defined. +type generatedDefinitionSources struct { + // debugInfo holds the definition lines from the Dbgi chunk. It is empty for + // a module compiled without debug info, and for Erlang modules. + debugInfo beam.DebugInfo + + // compileSource is the source path from the CInf chunk: the file the module + // was compiled from, as it was spelled on the machine that compiled it. + compileSource string +} + +// generatedDefinitionSourcesFor returns a compiled module's definition sources. +// They are read once per BEAM, and a failed read is memoized as empty: a module +// compiled without debug info will not grow it until rebuilt. +func (s *Server) generatedDefinitionSourcesFor(module, beamPath string) (generatedDefinitionSources, bool) { + // Resolving first validates the entry and yields the BEAM path. + _ = s.generatedFunctionsFor(module, beamPath) + + entry, ok := s.generatedCache.get(module) + if !ok || entry.beamPath == "" { + return generatedDefinitionSources{}, false + } + if !entry.definitionSourcesResolved { + started := s.debugNow() + info, err := beam.ReadDefinitionLines(entry.beamPath) + source, _ := beam.ReadSourcePath(entry.beamPath) + if !started.IsZero() { + s.debugf("Generated BEAM definition sources: module=%s definitions=%d error=%v source=%q total=%s", module, len(info.Lines), err, source, time.Since(started).Round(time.Microsecond)) + } + entry.definitionSources = generatedDefinitionSources{debugInfo: info, compileSource: source} + entry.definitionSourcesResolved = true + // Another request may have loaded a newer BEAM while this one read the + // chunks. Its entry must not be replaced with this older one. + if current, ok := s.generatedCache.get(module); ok && current.beamPath == entry.beamPath && current.beamStamp == entry.beamStamp { + s.generatedCache.put(module, entry) + } + } + return entry.definitionSources, true +} + +// generatedDefinitionResultsFor is where navigation sends a generated function. +// It starts from generatedDefinitionResults, the closest source-backed module, +// and moves to the function's own line when the compiled module records one in +// that module's source: for a function a DSL declared, that is the line that +// declared it. precise reports whether it did. +// +// A module with no source row of its own, such as a Spark entity module, may +// still record a line in the file it was compiled from, which is usually a +// framework file under deps. That file is tried after the indexed ones. +// +// The BEAM may be older than the source. Dexter cannot compile the project, +// and the line from the last compile is still the best one it has: an edit +// moves it by the lines added or removed above it, and the next compile makes +// it exact again. Only the declaring file's own edits move it; edits to the +// callers, which are most edits, do not. +func (s *Server) generatedDefinitionResultsFor(module, beamPath string, functions []beam.Function) (results []store.LookupResult, precise bool) { + results, owner := s.generatedDefinitionResults(module) + if len(functions) == 0 { + return results, false + } + sources, ok := s.generatedDefinitionSourcesFor(module, beamPath) + if !ok { + return results, false + } + if result, found, ambiguous := sources.bestDescribedResult(results); found { + result.Line = s.compiledModuleLine(module, owner, result.Line, sources) + generated := s.generatedFunctionsFor(module, beamPath) + if lines := recordedLinesIn(module, owner, result, s.currentSourceOf(result.FilePath), sources, functions, generated); len(lines) > 0 { + return lines, true + } + return results, false + } else if ambiguous { + return results, false + } + if owner != module { + if source := s.compiledSourceFile(sources.compileSource); source != "" { + if lines := recordedLinesIn(module, "", store.LookupResult{FilePath: source, Line: 1}, s.currentSourceOf(source), sources, functions, nil); len(lines) > 0 { + return lines, true + } + } + } + return results, false +} + +// compiledModuleLine returns owner's line as the BEAM that holds it recorded +// it, so that it compares with the recorded function lines, which are from a +// compile too: the index has the line in the current text, which an edit since +// the compile has moved. The BEAM is module's own when owner is module, and +// the parent's when module is a generated module nested in it. The index's +// line is the fallback when neither BEAM records one. +func (s *Server) compiledModuleLine(module, owner string, indexed int, sources generatedDefinitionSources) int { + if owner == module { + if line := sources.debugInfo.ModuleLine; line > 0 { + return line + } + return indexed + } + if ownerSources, ok := s.generatedDefinitionSourcesFor(owner, ""); ok && ownerSources.debugInfo.ModuleLine > 0 { + return ownerSources.debugInfo.ModuleLine + } + return indexed +} + +// generatedModuleLocation is where a module that exists only as a BEAM is +// defined: the file it was compiled from, at the line the compiler recorded +// for the module, such as a Module.create call. It is empty when the BEAM does +// not record a line, or the line is not in the file. +func (s *Server) generatedModuleLocation(module string) []store.LookupResult { + if len(s.generatedFunctionsForModule(module)) == 0 { + return nil + } + sources, ok := s.generatedDefinitionSourcesFor(module, "") + if !ok { + return nil + } + line := recordedModuleLine(sources.debugInfo) + if line < 1 { + return nil + } + source := s.compiledSourceFile(sources.compileSource) + if source == "" { + return nil + } + if src := s.currentSourceOf(source); src == nil || line > len(src.lines) { + return nil + } + return []store.LookupResult{{Module: module, FilePath: source, Line: line, Kind: "module"}} +} + +// recordedModuleLine is the module's recorded line or, when the compiler +// recorded none, its first function's. A macro that expands +// `defmodule unquote(name)` with a name it computed leaves the module line at +// zero, but each def in the body still has the line of the call that made it. +func recordedModuleLine(info beam.DebugInfo) int { + if info.ModuleLine > 0 { + return info.ModuleLine + } + first := 0 + for _, line := range info.Lines { + if line > 0 && (first == 0 || line < first) { + first = line + } + } + return first +} + +// recordedLinesIn returns a result for each distinct line in result's file +// that the compiled module records for functions, after the module's own line. +// +// result.Line is the module's line from the same compile as the function lines +// (see compiledModuleLine), so the comparison holds even when the file has +// changed since. A line past the end of the current +// text is dropped, because a generator can give a def any line +// (`quote line: 99`) and an edit can remove lines: an editor is never sent to +// a line the file does not have. +// +// When nothing is left, which is the case for a def a @before_compile hook +// made at the module line, the module's body is searched for the call that +// declared the function: a call whose first argument is its name as an atom +// (`later :deferred`). See declaration for how one of several is chosen; +// generated is every function the module's BEAM adds to its source. +func recordedLinesIn(module, owner string, result store.LookupResult, src *currentSource, sources generatedDefinitionSources, functions, generated []beam.Function) []store.LookupResult { + if src == nil { + return nil + } + recorded := generatedFunctionLines(module, result.FilePath, sources, functions) + // Clauses made at several lines come from calls that do not spell the + // function's name (`get "/a"` makes a `match/2` clause), so a call that + // does, such as `plug :match`, is not where they were declared. + severalClauses := len(recorded) > len(functions) + lines := slices.DeleteFunc(recorded, func(r store.LookupResult) bool { + return r.Line <= result.Line || r.Line > len(src.lines) + }) + if len(lines) > 0 || owner == "" || severalClauses { + return lines + } + if line, ok := src.declaration(owner, functions, generated); ok { + return []store.LookupResult{{Module: module, FilePath: result.FilePath, Line: line, Kind: functions[0].Kind, Arity: functions[0].Arity}} + } + return nil +} + +// generatedFunctionLines returns a result for each distinct line that the +// compiled module records for functions in file: one per clause when the +// clauses were made at different lines, as a DSL that adds a clause per call +// does. +// +// The Dbgi line is preferred because it also honors `@file`. The Docs chunk +// annotation is the fallback, because a module compiled without debug info +// still has it. It is often the line the def was expanded at, but not always: +// a generator can give the def one line and its docs another. +func generatedFunctionLines(module, file string, sources generatedDefinitionSources, functions []beam.Function) []store.LookupResult { + useDebugInfo := len(sources.debugInfo.Lines) > 0 + var lines []store.LookupResult + for _, function := range functions { + key := beam.FunctionKey{Name: function.Name, Arity: function.Arity} + recorded := []int{function.Line} + if useDebugInfo { + recorded = []int{sources.debugInfo.Lines[key]} + if clauses := sources.debugInfo.Clauses[key]; len(clauses) > 0 { + recorded = clauses + } + } + for _, line := range recorded { + if line <= 0 || slices.ContainsFunc(lines, func(r store.LookupResult) bool { return r.Line == line }) { + continue + } + lines = append(lines, store.LookupResult{ + Module: module, + FilePath: file, + Line: line, + Kind: function.Kind, + Arity: function.Arity, + }) + } + } + return lines +} + +// currentSource is a file's text as it is now: in the open buffer if there is +// one, otherwise on disk. +type currentSource struct { + text string + // lines is text split into lines, with the lines inside heredocs blanked. + lines []string + owners []string +} + +// currentSourceOf returns path's current text, or nil if it cannot be read. +// Only a buffer open in the editor counts: a file that another request cached +// may be older than the file on disk. +func (s *Server) currentSourceOf(path string) *currentSource { + text, ok := s.docs.GetIfOpen(string(pathToURI(path))) + if !ok { + data, err := os.ReadFile(path) + if err != nil { + return nil + } + text = string(data) + } + return ¤tSource{text: text, lines: blankHeredocs(strings.Split(text, "\n"))} +} + +// ownerAt returns the innermost module whose body holds the 1-based line. +func (src *currentSource) ownerAt(line int) string { + if src.owners == nil { + src.owners = moduleOwnersByLine([]byte(src.text), len(src.lines)) + } + if line < 1 || line >= len(src.owners) { + return "" + } + return src.owners[line] +} + +// declaration returns the line in owner's body of the call that declared one +// of functions by name. A sibling module in the same file that declares the +// same name does not count. +// +// When one call spells the name, it is the answer. When several do, as +// `update :publish` and `define :publish` can, the macro that generated the +// function is taken to be the one whose calls name the most of the module's +// generated functions: `define` names every function it made, while an action +// names only its own. A tie gives no answer. +func (src *currentSource) declaration(owner string, functions, generated []beam.Function) (int, bool) { + names := declarationNames(functions[0].Name) + type match struct { + line int + callee string + } + var matches []match + for line := 1; line <= len(src.lines); line++ { + callee, atom, ok := declaringCall(src.lines[line-1]) + if !ok || !slices.Contains(names, atom) || src.ownerAt(line) != owner { + continue + } + matches = append(matches, match{line: line, callee: callee}) + } + switch len(matches) { + case 0: + return 0, false + case 1: + return matches[0].line, true + } + + generatedNames := make(map[string]bool) + for _, function := range generated { + for _, name := range declarationNames(function.Name) { + generatedNames[name] = true + } + } + named := make(map[string]map[string]bool) + for _, m := range matches { + named[m.callee] = make(map[string]bool) + } + for line := 1; line <= len(src.lines); line++ { + callee, atom, ok := declaringCall(src.lines[line-1]) + if !ok || named[callee] == nil || !generatedNames[atom] || src.ownerAt(line) != owner { + continue + } + named[callee][atom] = true + } + best, bestScore, tied := 0, -1, false + for _, m := range matches { + switch score := len(named[m.callee]); { + case score > bestScore: + best, bestScore, tied = m.line, score, false + case score == bestScore: + tied = true + } + } + return best, !tied +} + +// declarationNames is the names a declaring call can spell for function: the +// name itself, and without a trailing `!` or `?`, as `define :get` makes +// `get!` and `flag :active` makes `active?`. +func declarationNames(function string) []string { + names := []string{function} + if base := strings.TrimRight(function, "!?"); base != function && base != "" { + names = append(names, base) + } + return names +} + +// blankHeredocs replaces the lines inside heredocs with empty lines, so that +// an example in a @moduledoc or @doc, such as `define :foo`, is not taken for +// a declaration. The line that opens a heredoc keeps its text before the +// delimiter, as `define :foo, description: """` still declares foo. Line +// numbers do not change. +func blankHeredocs(lines []string) []string { + out := make([]string, len(lines)) + var open string + for i, line := range lines { + inside := open != "" + opened := -1 + for _, delimiter := range []string{`"""`, "'''"} { + if open != "" && open != delimiter { + continue + } + if strings.Count(line, delimiter)%2 == 1 { + if open == "" { + open = delimiter + opened = strings.LastIndex(line, delimiter) + } else { + open = "" + } + } + } + switch { + case !inside && open == "": + out[i] = line + case !inside && opened >= 0: + out[i] = line[:opened] + } + } + return out +} + +// lineDeclaresAtom reports whether text is a call whose first argument is one +// of names as an atom (see declaringCall). +func lineDeclaresAtom(text string, names []string) bool { + _, atom, ok := declaringCall(text) + return ok && slices.Contains(names, atom) +} + +// declaringCall returns the callee and the first argument of text when text +// is a call whose first argument is an atom: `define :list_rooms`, +// `field :email, :string`, or `Lib.define(:name)`. That is the shape of a +// macro call that declares a name. An atom anywhere else, such as a keyword +// value or a typespec, is not. +func declaringCall(text string) (callee, atom string, ok bool) { + rest := strings.TrimLeft(text, " \t") + end := 0 + for end < len(rest) && (isIdentifierByte(rest[end]) || rest[end] == '.') { + end++ + } + // A module attribute such as `@tag :slow` sets a value; it declares nothing. + if end == 0 || rest[0] == ':' || rest[0] == '@' || (rest[0] >= '0' && rest[0] <= '9') { + return "", "", false + } + args := strings.TrimLeft(rest[end:], " \t") + if len(args) == len(rest[end:]) { + // No space after the callee: only a parenthesized call qualifies. + if !strings.HasPrefix(args, "(") { + return "", "", false + } + } + args = strings.TrimLeft(strings.TrimPrefix(args, "("), " \t") + if !strings.HasPrefix(args, ":") { + return "", "", false + } + // Bytes from 0x80 up are parts of a UTF-8 identifier such as `:café`. + atomEnd := 1 + for atomEnd < len(args) && (isIdentifierByte(args[atomEnd]) || args[atomEnd] >= 0x80) { + atomEnd++ + } + if atomEnd == 1 { + return "", "", false + } + return rest[:end], args[1:atomEnd], true +} + +func isIdentifierByte(b byte) bool { + return b == '_' || b == '!' || b == '?' || b == '@' || + (b >= 'a' && b <= 'z') || (b >= 'A' && b <= 'Z') || (b >= '0' && b <= '9') +} + +// bestDescribedResult returns the result whose file the recorded lines belong +// to. Usually there is one row, but a module name defined in two files (two +// umbrella apps, say) has two, and a moved project matches neither exactly. +// The row that matches the recorded path most specifically wins. If two match +// equally, ambiguous is set, because a line from one file means nothing in the +// other. +func (sources generatedDefinitionSources) bestDescribedResult(results []store.LookupResult) (best store.LookupResult, found, ambiguous bool) { + bestScore := 0 + for _, result := range results { + score := sources.describe(result.FilePath) + switch { + case score == 0: + case score > bestScore: + best, bestScore, ambiguous = result, score, false + case score == bestScore && result.FilePath != best.FilePath: + ambiguous = true + } + } + if ambiguous { + return store.LookupResult{}, false, true + } + return best, bestScore > 0, false +} + +// describe scores how specifically the recorded source names path: zero when +// it does not, highest for the same absolute path. When the module has debug +// info, its file names are the test; otherwise the compile info's. +func (sources generatedDefinitionSources) describe(path string) int { + if len(sources.debugInfo.Lines) > 0 { + return recordedPathMatch(sources.debugInfo.File, sources.debugInfo.RelativeFile, path) + } + return recordedPathMatch(sources.compileSource, "", path) +} + +// exactPathMatch is the score for a recorded absolute path equal to the path. +const exactPathMatch = 1 << 20 + +// recordedPathMatch scores how specifically a recorded source path names path. +// The same absolute path is best. Otherwise the two are compared from their +// ends, which covers a project that has moved, was copied, or is reached +// through a symlink since it was compiled: each shared trailing component +// counts, and at least a directory and the file name must be shared. relative, +// the path relative to the compiler's working directory, counts its own +// components when it is a suffix of path. +func recordedPathMatch(recorded, relative, path string) int { + if recorded != "" && recorded == path { + return exactPathMatch + } + score := 0 + if recorded != "" { + if shared := sharedTrailingComponents(recorded, path); shared >= 2 { + score = shared + } + } + if relative != "" && !filepath.IsAbs(relative) { + slash := filepath.ToSlash(relative) + if strings.HasSuffix(filepath.ToSlash(path), "/"+slash) { + score = max(score, strings.Count(slash, "/")+1) + } + } + return score +} + +// sharedTrailingComponents counts the path components a and b share at their +// ends. +func sharedTrailingComponents(a, b string) int { + as := strings.Split(filepath.ToSlash(a), "/") + bs := strings.Split(filepath.ToSlash(b), "/") + shared := 0 + for shared < len(as) && shared < len(bs) { + component := as[len(as)-1-shared] + if component == "" || component != bs[len(bs)-1-shared] { + break + } + shared++ + } + return shared +} + +// moduleOwnersByLine returns, for each 1-based line of source, the innermost +// module whose body holds it, found with the tokenizer so that strings, +// heredocs and comments do not count. Index 0 is unused. +func moduleOwnersByLine(source []byte, lineCount int) []string { + owners := make([]string, lineCount+1) + tokens := parser.Tokenize(source) + type frame struct { + name string + depth int + } + var stack []frame + depth := 0 + current := func() string { + if len(stack) == 0 { + return "" + } + return stack[len(stack)-1].name + } + filled := 0 + fill := func(through int) { + for filled < through && filled < lineCount { + filled++ + owners[filled] = current() + } + } + for i := 0; i < len(tokens); i++ { + token := tokens[i] + // A line belongs to the module that is open at its first token. + fill(token.Line) + switch token.Kind { + case parser.TokDo, parser.TokFn: + parser.TrackBlockDepth(token.Kind, &depth) + case parser.TokEnd: + before := depth + parser.TrackBlockDepth(token.Kind, &depth) + if len(stack) > 0 && stack[len(stack)-1].depth == before { + stack = stack[:len(stack)-1] + } + case parser.TokDefmodule, parser.TokDefprotocol, parser.TokDefimpl: + name, next, hasDo := tokParseModuleDef(source, tokens, i+1, current()) + if name == "" { + continue + } + if hasDo { + depth++ + stack = append(stack, frame{name: name, depth: depth}) + } + i = next - 1 + } + } + fill(lineCount) + return owners +} + +// compiledSourceFile maps a source path from compile info onto this checkout. +// +// The path was recorded wherever the artifact was built, which for a dependency +// is often a different directory: a `_build` copied between checkouts, a Docker +// build, a vendored artifact. When it does not exist here, only the tail below +// the application's own lib directory is stable. +// +// The application is read from the recorded path, not from the BEAM path. A +// generated module can live in one application while its source lives in +// another: Spark creates Ash's entity modules, so the artifact sits in ash's +// ebin while the file that generated it is under spark. +// +// A recorded path outside the project is tried last. It is right for a `path:` +// dependency, but a `_build` copied from another checkout or worktree records +// that checkout, which usually still exists: returning it would open the file +// in the wrong tree. +func (s *Server) compiledSourceFile(recorded string) string { + if recorded == "" { + return "" + } + if pathWithin(recorded, s.projectRoot) && regularFileExists(recorded) { + return recorded + } + for _, candidate := range sourceRebaseCandidates(recorded, s.projectRoot) { + if regularFileExists(candidate) { + return candidate + } + } + if regularFileExists(recorded) { + return recorded + } + return "" +} + +// pathWithin reports whether path is root or below it. +func pathWithin(path, root string) bool { + if root == "" { + return false + } + rel, err := filepath.Rel(root, path) + return err == nil && rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator)) +} + +// sourceRebaseCandidates lists the plausible on-disk locations for a recorded +// source path, most specific first. +// +// First come the recorded path's own tails under the project root, longest +// first: a checkout that moved or was copied keeps its layout below the root, +// whether that is `lib/`, an umbrella's `apps//lib/`, or `deps/`. A tail +// needs at least one directory, so a bare file name never matches a file at +// the root. +// +// Then every Mix layout for the application named below the last `lib/`, +// because an artifact built elsewhere does not record which one applies here: +// `lib//...` is the application's own repository, `deps//lib//...` +// is a dependency, and `deps//...` is the older vendored shape. +func sourceRebaseCandidates(recorded, projectRoot string) []string { + if projectRoot == "" { + return nil + } + slash := filepath.ToSlash(recorded) + var candidates []string + parts := strings.Split(strings.TrimPrefix(slash, "/"), "/") + for i := 0; i+2 <= len(parts); i++ { + candidates = append(candidates, filepath.Join(projectRoot, filepath.FromSlash(strings.Join(parts[i:], "/")))) + } + + if idx := strings.LastIndex(slash, "/lib/"); idx >= 0 { + appTail := strings.SplitN(slash[idx+len("/lib/"):], "/", 2) + if len(appTail) == 2 && appTail[0] != "" && appTail[1] != "" { + app, tail := appTail[0], filepath.FromSlash(appTail[1]) + candidates = append(candidates, + filepath.Join(projectRoot, "lib", app, tail), + filepath.Join(projectRoot, "deps", app, "lib", app, tail), + filepath.Join(projectRoot, "deps", app, tail), + ) + } + } + return slices.Compact(candidates) +} + +// regularFileExists reports whether path names a readable regular file. +// Anything else is treated as absent, so an editor is never sent to a location +// it cannot open. +func regularFileExists(path string) bool { + info, err := os.Stat(path) + return err == nil && info.Mode().IsRegular() +} diff --git a/internal/lsp/generated_definition_integration_test.go b/internal/lsp/generated_definition_integration_test.go new file mode 100644 index 0000000..b47cda3 --- /dev/null +++ b/internal/lsp/generated_definition_integration_test.go @@ -0,0 +1,405 @@ +package lsp + +import ( + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "testing" + + "go.lsp.dev/uri" +) + +// The fixture compiles DSLs of every shape navigation has to handle, with the +// real compiler, so a change in what Elixir records fails here. It needs mix, +// so it runs in the integration job. + +const compiledDslSource = `defmodule Weird.Dsl do + defmacro outer(name), do: quote(do: Weird.Dsl.inner(unquote(name))) + defmacro inner(name), do: quote(do: def(unquote(name)(), do: :inner)) + + defmacro flag(name) do + fun = :"#{name}?" + quote do: def(unquote(fun)(map), do: Map.get(map, unquote(name), false)) + end + + defmacro later(name), do: quote(do: @later_names(unquote(name))) + + defmacro __before_compile__(env) do + for name <- Module.get_attribute(env.module, :later_names) do + quote do: def(unquote(name)(), do: :later) + end + end + + defmacro __using__(_) do + quote do + import Weird.Dsl + Module.register_attribute(__MODULE__, :later_names, accumulate: true) + @before_compile Weird.Dsl + end + end + + defmacro pinned(name), do: quote(line: 99, do: def(unquote(name)(), do: :pinned)) + + defmacro route(verb, path) do + quote do: def(match(unquote(verb), unquote(path)), do: {unquote(verb), unquote(path)}) + end + + defmacro plug(_name), do: nil + defmacro action(_name), do: nil + + defmacro on(event) do + quote do + def handle_event(unquote(event)), do: unquote(event) + end + end + + defmacro helper(name) do + quote bind_quoted: [name: name] do + defmodule Module.concat(__MODULE__, Macro.camelize(Atom.to_string(name))) do + def run, do: :helper + end + end + end + + # The module name is computed here, so the compiler records no line for + # the module itself. + defmacro named_helper(name) do + module = Module.concat(__CALLER__.module, Macro.camelize(Atom.to_string(name))) + + quote do + defmodule unquote(module) do + def run, do: :named + end + end + end + + defmacro keep_route(name, verb) do + quote bind_quoted: [name: name, verb: verb], location: :keep do + def unquote(name)(unquote(verb)), do: unquote(verb) + end + end + + defmacro with_default(name), do: quote(do: def(unquote(name)(value \\ 1), do: value)) + + defmacro elsewhere(name) do + quote bind_quoted: [name: name] do + Module.create( + Module.concat(Weird.Elsewhere, Macro.camelize(Atom.to_string(name))), + quote(do: def(run, do: :elsewhere)), + __ENV__ + ) + end + end + + defmacro delegate(name, to), do: quote(do: defdelegate(unquote(name)(value), to: unquote(to))) + + defmacro many(prefix, count) do + for i <- 1..count do + fun = :"#{prefix}_#{i}" + quote do: def(unquote(fun)(), do: unquote(i)) + end + end + + defmacro kept_field(name) do + quote bind_quoted: [name: name], location: :keep do + def unquote(name)(), do: unquote(name) + end + end + + defmacro stamped_field(name) do + line = __CALLER__.line + + quote bind_quoted: [name: name, line: line] do + @file {__ENV__.file, line} + def unquote(name)(), do: unquote(name) + end + end +end +` + +const compiledUserSource = `defmodule Weird.User do + use Weird.Dsl + + outer :two_level + + flag :active + + later :deferred + + action :submit + later :submit + + on :click + on :hover + + pinned :pinned_fun + + plug :match + + route :get, "/a" + route :post, "/b" + + keep_route :handle, :get + keep_route :handle, :post + + with_default :defaulted + + elsewhere :remote + + delegate :double, Weird.Target + + many :gen, 2000 + + kept_field :kept + + stamped_field :stamped + + helper :audit + + named_helper :report + + for name <- [:loop_a, :loop_b] do + def unquote(name)(), do: unquote(name) + end + + defmodule Nested do + use Weird.Dsl + flag :nested + end +end +` + +const compiledCallerSource = `defmodule Weird.Caller do + import Weird.User, only: [active?: 1] + + def run do + Weird.User.two_level() + Weird.User.active?(%{}) + Weird.User.deferred() + Weird.User.submit() + Weird.User.handle_event(:click) + Weird.User.pinned_fun() + Weird.User.match(:get, "/a") + Weird.User.handle(:get) + Weird.User.defaulted(2) + Weird.Elsewhere.Remote.run() + Weird.User.double(2) + Weird.User.gen_1500() + Weird.User.kept() + Weird.User.stamped() + Weird.User.loop_b() + Weird.User.Audit.run() + Weird.User.Report.run() + Weird.User.Nested.nested?(%{}) + active?(%{}) + f = &Weird.User.two_level/0 + f.() + end +end +` + +const ( + compiledDslRel = "lib/weird/dsl.ex" + compiledUserRel = "lib/weird/user.ex" + compiledCallerRel = "lib/weird/caller.ex" +) + +// compiledFixture is the compiled DSL project and the server that indexed it. +type compiledFixture struct { + server *Server + t *testing.T +} + +func newCompiledFixture(t *testing.T, elixircOptions string) *compiledFixture { + t.Helper() + if _, err := exec.LookPath("mix"); err != nil { + t.Skip("mix not available") + } + server, cleanup := setupTestServer(t) + t.Cleanup(cleanup) + + mixExs := `defmodule Weird.MixProject do + use Mix.Project + + def project, do: [app: :weird, version: "0.1.0", elixir: "~> 1.18", elixirc_options: [` + elixircOptions + `]] +end +` + if err := os.WriteFile(filepath.Join(server.projectRoot, "mix.exs"), []byte(mixExs), 0o644); err != nil { + t.Fatal(err) + } + for rel, source := range map[string]string{ + compiledDslRel: compiledDslSource, + compiledUserRel: compiledUserSource, + compiledCallerRel: compiledCallerSource, + "lib/weird/target.ex": `defmodule Weird.Target do + def double(value), do: value * 2 +end +`, + } { + indexFile(t, server.store, server.projectRoot, rel, source) + } + cmd := exec.Command("mix", "compile") + cmd.Dir = server.projectRoot + if output, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("compile the DSL fixture: %v\n%s", err, output) + } + return &compiledFixture{server: server, t: t} +} + +// saveLater saves and indexes an edit made after the compile. +func (f *compiledFixture) saveLater(rel, source string) { + f.t.Helper() + indexFile(f.t, f.server.store, f.server.projectRoot, rel, source) +} + +// lineOf returns the 1-based line of the nth occurrence of needle in source. +func lineOf(t *testing.T, source, needle string, nth int) int { + t.Helper() + for i, line := range strings.Split(source, "\n") { + if strings.Contains(line, needle) { + if nth--; nth == 0 { + return i + 1 + } + } + } + t.Fatalf("%q not found in source", needle) + return 0 +} + +// definitionOf returns the definition lines for the nth occurrence of needle in +// the caller, with the cursor inside the name the needle starts with. +func (f *compiledFixture) definitionOf(needle string, nth int) []string { + f.t.Helper() + line := lineOf(f.t, compiledCallerSource, needle, nth) + text := strings.Split(compiledCallerSource, "\n")[line-1] + col := strings.Index(text, needle) + 1 + locations := generatedDefinitionAt(f.t, f.server, compiledCallerRel, compiledCallerSource, line-1, col) + var got []string + for _, location := range locations { + rel, _ := filepath.Rel(f.server.projectRoot, uriToPath(location.URI)) + got = append(got, rel+":"+strconv.Itoa(int(location.Range.Start.Line)+1)) + } + return got +} + +func at(rel, source, needle string, t *testing.T) string { + t.Helper() + return rel + ":" + strconv.Itoa(lineOf(t, source, needle, 1)) +} + +func expectDefinition(t *testing.T, got []string, want ...string) { + t.Helper() + if strings.Join(got, " ") != strings.Join(want, " ") { + t.Errorf("definition = %v, want %v", got, want) + } +} + +func TestDefinition_GeneratedFunctionsFromCompiler(t *testing.T) { + f := newCompiledFixture(t, "") + user := func(needle string) string { return at(compiledUserRel, compiledUserSource, needle, t) } + + for _, tc := range []struct { + name, needle string + nth int + want []string + }{ + {"macro that calls a macro", "two_level(", 1, []string{user("outer :two_level")}}, + {"predicate from an atom", "active?(%{})", 1, []string{user("flag :active")}}, + {"@before_compile hook", "deferred(", 1, []string{user("later :deferred")}}, + {"declaring call next to a same-named call", "submit(", 1, []string{user("later :submit")}}, + {"def in a macro's quote", "handle_event(", 1, []string{user("on :click"), user("on :hover")}}, + {"line past the end of the file", "pinned_fun(", 1, []string{user("pinned :pinned_fun")}}, + {"one clause per call", "match(", 1, []string{user(`route :get`), user(`route :post`)}}, + {"one location: :keep clause per call", "handle(", 1, []string{user("keep_route :handle, :get"), user("keep_route :handle, :post")}}, + {"default arguments", "defaulted(", 1, []string{user("with_default :defaulted")}}, + {"module from Module.create", "run()", 1, []string{user("elsewhere :remote")}}, + {"generated defdelegate", "double(", 1, []string{user("delegate :double")}}, + {"one of many generated", "gen_1500(", 1, []string{user("many :gen")}}, + {"location: :keep", "kept(", 1, []string{user("kept_field :kept")}}, + {"@file stamp", "stamped(", 1, []string{user("stamped_field :stamped")}}, + {"comprehension", "loop_b(", 1, []string{user("def unquote(name)()")}}, + {"nested module", "nested?(", 1, []string{user("flag :nested")}}, + {"function of a module a macro nested", "run()", 2, []string{user("helper :audit")}}, + {"name of a module a macro nested", "Audit.run", 1, []string{user("helper :audit")}}, + {"function of a module a macro named", "run()", 3, []string{user("named_helper :report")}}, + {"name of a module a macro named", "Report.run", 1, []string{user("named_helper :report")}}, + {"bare call through import", "active?(%{})", 2, []string{user("flag :active")}}, + {"capture", "two_level/0", 1, []string{user("outer :two_level")}}, + } { + t.Run(tc.name, func(t *testing.T) { + expectDefinition(t, f.definitionOf(tc.needle, tc.nth), tc.want...) + }) + } + + t.Run("module that exists only as a BEAM", func(t *testing.T) { + locations, err := f.server.LookupName("Weird.Elsewhere.Remote", "", NameLookupOptions{}) + if err != nil { + t.Fatal(err) + } + want := filepath.Join(f.server.projectRoot, compiledUserRel) + if len(locations) != 1 || locations[0].FilePath != want || locations[0].Line != lineOf(t, compiledUserSource, "elsewhere :remote", 1) { + t.Errorf("module location = %#v, want the elsewhere call in %s", locations, want) + } + }) +} + +// Dexter cannot compile, so the declaring file can have edits the BEAM has not +// seen. The line from the last compile is still the answer: it is near, and +// the next compile makes it exact. A line the file no longer has is never +// returned, and the search for a declaring call reads the current text. +func TestDefinition_GeneratedFunctionsFromCompilerWithStaleSource(t *testing.T) { + f := newCompiledFixture(t, "") + compiled := func(needle string) string { return at(compiledUserRel, compiledUserSource, needle, t) } + + t.Run("lines added above, saved", func(t *testing.T) { + edited := "# one\n# two\n# three\n" + compiledUserSource + f.saveLater(compiledUserRel, edited) + t.Cleanup(func() { indexFile(t, f.server.store, f.server.projectRoot, compiledUserRel, compiledUserSource) }) + expectDefinition(t, f.definitionOf("two_level(", 1), compiled("outer :two_level")) + expectDefinition(t, f.definitionOf("match(", 1), compiled("route :get"), compiled("route :post")) + }) + + // The index then has the module below the recorded lines. The module + // line they are compared with comes from the same compile as they do, + // also for a module a macro nested in it. + t.Run("many lines added above, saved", func(t *testing.T) { + edited := strings.Repeat("# header\n", 40) + compiledUserSource + f.saveLater(compiledUserRel, edited) + t.Cleanup(func() { indexFile(t, f.server.store, f.server.projectRoot, compiledUserRel, compiledUserSource) }) + expectDefinition(t, f.definitionOf("two_level(", 1), compiled("outer :two_level")) + expectDefinition(t, f.definitionOf("run()", 2), compiled("helper :audit")) + }) + + t.Run("lines added in an unsaved buffer", func(t *testing.T) { + edited := strings.Replace(compiledUserSource, " outer :two_level\n", " # one\n # two\n outer :two_level\n", 1) + userURI := string(uri.File(filepath.Join(f.server.projectRoot, compiledUserRel))) + f.server.docs.Set(userURI, edited) + t.Cleanup(func() { f.server.docs.Close(userURI) }) + expectDefinition(t, f.definitionOf("two_level(", 1), compiled("outer :two_level")) + // The BEAM records only the module line here, so the declaring call + // is searched for, in the text as it is now. + expectDefinition(t, f.definitionOf("deferred(", 1), at(compiledUserRel, edited, "later :deferred", t)) + }) + + t.Run("file shorter than the recorded line", func(t *testing.T) { + edited := "defmodule Weird.User do\n use Weird.Dsl\n plug :match\n outer :two_level\nend\n" + f.saveLater(compiledUserRel, edited) + t.Cleanup(func() { indexFile(t, f.server.store, f.server.projectRoot, compiledUserRel, compiledUserSource) }) + expectDefinition(t, f.definitionOf("gen_1500(", 1), compiledUserRel+":1") + // The route clauses are gone; `plug :match` spells the name but is not + // where a clause was declared. + expectDefinition(t, f.definitionOf("match(", 1), compiledUserRel+":1") + expectDefinition(t, f.definitionOf("two_level(", 1), compiled("outer :two_level")) + }) +} + +// A project compiled without debug info still has its Docs and compile info +// chunks, and its source. +func TestDefinition_GeneratedFunctionsFromCompilerWithoutDebugInfo(t *testing.T) { + f := newCompiledFixture(t, "debug_info: false") + user := func(needle string) string { return at(compiledUserRel, compiledUserSource, needle, t) } + expectDefinition(t, f.definitionOf("two_level(", 1), user("outer :two_level")) + expectDefinition(t, f.definitionOf("deferred(", 1), user("later :deferred")) +} diff --git a/internal/lsp/generated_definition_test.go b/internal/lsp/generated_definition_test.go new file mode 100644 index 0000000..1ac136b --- /dev/null +++ b/internal/lsp/generated_definition_test.go @@ -0,0 +1,830 @@ +package lsp + +import ( + "context" + "encoding/binary" + "os" + "path/filepath" + "slices" + "strings" + "testing" + "time" + + "github.com/remoteoss/dexter/internal/beam" + "github.com/remoteoss/dexter/internal/store" + "go.lsp.dev/protocol" + "go.lsp.dev/uri" +) + +// A DSL that declares functions inside a module, the way an Ash domain's +// `define` calls do. Nothing in the source is a def, so the index has no row for +// get_room_by_slug!; only the compiled module knows it exists. +const generatedDomainSource = `# The chat domain. +defmodule MyApp.Chat do + use SharedLib.Domain + + resources do + resource MyApp.Chat.Room do + define :list_rooms, action: :read + define :get_room_by_slug, action: :read, get_by: [:slug] + end + end + + def run_local, do: get_room_by_slug!("lounge") +end +` + +const generatedCallerSource = `defmodule MyApp.Caller do + alias MyApp.Chat + + def run do + Chat.get_room_by_slug!("lounge") + end +end +` + +const ( + generatedDomainRel = "lib/my_app/chat.ex" + generatedCallerRel = "lib/my_app/caller.ex" + // generatedDefineLine is the 1-based line of `define :get_room_by_slug`. + generatedDefineLine = 8 + // generatedModuleLine is the 1-based line of `defmodule MyApp.Chat`. + generatedModuleLine = 2 +) + +// dbgiDefinition is one definition in a synthetic Dbgi chunk. +type dbgiDefinition struct { + name string + arity int + line int + keepFile string + keepLine int +} + +// dbgiTerm encodes the {:debug_info_v1, :elixir_erl, {:elixir_v1, map, specs}} +// term the Elixir compiler writes, with empty clause lists. +func dbgiTerm(file, relativeFile string, definitions ...dbgiDefinition) []byte { + var out []byte + tuple := func(arity int) { out = append(out, 104, byte(arity)) } + atom := func(name string) { out = append(out, etfAtom(name)...) } + list := func(count int) { out = append(out, etfListHeader(count)...) } + nilTerm := func() { out = append(out, 106) } + integer := func(value int) { + out = append(out, 98) + out = binary.BigEndian.AppendUint32(out, uint32(value)) + } + bin := func(text string) { + out = append(out, 109) + out = binary.BigEndian.AppendUint32(out, uint32(len(text))) + out = append(out, text...) + } + + tuple(3) + atom("debug_info_v1") + atom("elixir_erl") + tuple(3) + atom("elixir_v1") + out = append(out, 116, 0, 0, 0, 3) // map with three pairs + atom("definitions") + list(len(definitions)) + for _, definition := range definitions { + tuple(4) + tuple(2) + atom(definition.name) + integer(definition.arity) + atom("def") + pairs := 1 + if definition.keepFile != "" { + pairs++ + } + list(pairs) + tuple(2) + atom("line") + integer(definition.line) + if definition.keepFile != "" { + tuple(2) + atom("file") + tuple(2) + bin(definition.keepFile) + integer(definition.keepLine) + } + nilTerm() + nilTerm() // clauses + } + nilTerm() + atom("file") + bin(file) + atom("relative_file") + bin(relativeFile) + nilTerm() // specs + return out +} + +// newGeneratedDefinitionFixture indexes the domain and its caller, and writes +// the domain's BEAM with the given debug info. +func newGeneratedDefinitionFixture(t *testing.T, relativeFile string, definitions ...dbgiDefinition) (*Server, string) { + t.Helper() + return newGeneratedDefinitionFixtureWith(t, func(domainPath string) testBeamChunks { + return testBeamChunks{dbgi: dbgiTerm(domainPath, relativeFile, definitions...)} + }) +} + +// newGeneratedDefinitionFixtureWith is newGeneratedDefinitionFixture with the +// domain BEAM's chunks chosen by the test. +func newGeneratedDefinitionFixtureWith(t *testing.T, chunks func(domainPath string) testBeamChunks) (*Server, string) { + t.Helper() + server, cleanup := setupTestServer(t) + t.Cleanup(cleanup) + + indexFile(t, server.store, server.projectRoot, generatedDomainRel, generatedDomainSource) + indexFile(t, server.store, server.projectRoot, generatedCallerRel, generatedCallerSource) + domainPath := filepath.Join(server.projectRoot, generatedDomainRel) + writeGeneratedBeam(t, server, "MyApp.Chat", chunks(domainPath), + beamExport{"run_local", 0}, + beamExport{"list_rooms", 0}, + beamExport{"get_room_by_slug!", 1}, + beamExport{"get_room_by_slug!", 2}, + ) + return server, domainPath +} + +func writeGeneratedBeam(t *testing.T, server *Server, module string, chunks testBeamChunks, exports ...beamExport) { + t.Helper() + ebin := filepath.Join(server.projectRoot, "_build", "dev", "lib", "my_app", "ebin") + if err := os.MkdirAll(ebin, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(ebin, "Elixir."+module+".beam"), buildTestBeam(chunks, exports...), 0o644); err != nil { + t.Fatal(err) + } +} + +// docsEntry is one function in a synthetic docs_v1 term. +type docsEntry struct { + name string + arity int + anno int +} + +// docsTerm encodes the docs_v1 term the compiler writes in the Docs chunk. Each +// entry's anno is the line its def was expanded at. +func docsTerm(entries ...docsEntry) string { + var out []byte + tuple := func(arity int) { out = append(out, 104, byte(arity)) } + atom := func(name string) { out = append(out, etfAtom(name)...) } + emptyMap := func() { out = append(out, 116, 0, 0, 0, 0) } + integer := func(value int) { + out = append(out, 98) + out = binary.BigEndian.AppendUint32(out, uint32(value)) + } + bin := func(text string) { + out = append(out, 109) + out = binary.BigEndian.AppendUint32(out, uint32(len(text))) + out = append(out, text...) + } + + tuple(7) + atom("docs_v1") + integer(1) + atom("elixir") + bin("text/markdown") + atom("none") + emptyMap() + out = append(out, etfListHeader(len(entries))...) + for _, entry := range entries { + tuple(5) + tuple(3) + atom("function") + atom(entry.name) + integer(entry.arity) + integer(entry.anno) + out = append(out, etfListHeader(1)...) + bin(entry.name + "()") + out = append(out, 106) + atom("none") + emptyMap() + } + out = append(out, 106) + return string(out) +} + +// stampedDefinitions is what a generator writes when it stamps each function +// with `@file {file, line}` for the DSL call that declared it: :line is where +// the before-compile hook ran, and the file entry names the declaring line. +func stampedDefinitions() []dbgiDefinition { + return []dbgiDefinition{ + {name: "list_rooms", arity: 0, line: 1, keepFile: generatedDomainRel, keepLine: 7}, + {name: "get_room_by_slug!", arity: 1, line: 1, keepFile: generatedDomainRel, keepLine: generatedDefineLine}, + {name: "get_room_by_slug!", arity: 2, line: 1, keepFile: generatedDomainRel, keepLine: generatedDefineLine}, + } +} + +// lineDefinitions is what a generator writes when it expands each function at +// the line of the DSL call that declared it, as Ash's code interfaces do: :line +// is the declaring line, and the file entry still names the generator. +func lineDefinitions() []dbgiDefinition { + const generator = "deps/ash/lib/ash/code_interface.ex" + return []dbgiDefinition{ + {name: "list_rooms", arity: 0, line: 7, keepFile: generator, keepLine: 1112}, + {name: "get_room_by_slug!", arity: 1, line: generatedDefineLine, keepFile: generator, keepLine: 1112}, + {name: "get_room_by_slug!", arity: 2, line: generatedDefineLine, keepFile: generator, keepLine: 1112}, + } +} + +func generatedDefinitionAt(t *testing.T, server *Server, rel string, source string, line, character int) []protocol.Location { + t.Helper() + docURI := string(uri.File(filepath.Join(server.projectRoot, rel))) + server.docs.Set(docURI, source) + locations, err := server.Definition(context.Background(), &protocol.DefinitionParams{ + TextDocumentPositionParams: protocol.TextDocumentPositionParams{ + TextDocument: protocol.TextDocumentIdentifier{URI: protocol.DocumentURI(docURI)}, + Position: protocol.Position{Line: uint32(line), Character: uint32(character)}, + }, + }) + if err != nil { + t.Fatal(err) + } + return locations +} + +func expectSingleLocation(t *testing.T, locations []protocol.Location, path string, line int) { + t.Helper() + if len(locations) != 1 { + t.Fatalf("expected one location at %s:%d, got %#v", path, line, locations) + } + if got := uriToPath(locations[0].URI); got != path || int(locations[0].Range.Start.Line) != line-1 { + t.Fatalf("definition = %s:%d, want %s:%d", got, locations[0].Range.Start.Line+1, path, line) + } +} + +// Issue #108: a function a DSL generated resolved to the top of its module +// even when the compiled module records the line that declared it. +func TestDefinitionGeneratedFunctionUsesDebugInfoLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, stampedDefinitions()...) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) +} + +func TestDefinitionBareGeneratedFunctionUsesDebugInfoLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, stampedDefinitions()...) + locations := generatedDefinitionAt(t, server, generatedDomainRel, generatedDomainSource, 11, 24) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) +} + +func TestDefinitionGeneratedFunctionUsesLineWhenFileIsGenerator(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) +} + +// `dexter lookup` resolves through LookupName, and a recorded line is the +// function's own definition, so a strict lookup takes it too. +func TestLookupNameGeneratedFunctionUsesDebugInfoLine(t *testing.T) { + for _, tc := range []struct { + name string + opts NameLookupOptions + }{ + {"fallback", NameLookupOptions{FallbackToModule: true}}, + {"strict", NameLookupOptions{ExactModule: true}}, + } { + t.Run(tc.name, func(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) + locations, err := server.LookupName("MyApp.Chat", "get_room_by_slug!", tc.opts) + if err != nil { + t.Fatal(err) + } + if len(locations) != 1 || locations[0].FilePath != domainPath || locations[0].Line != generatedDefineLine { + t.Fatalf("expected %s:%d, got %#v", domainPath, generatedDefineLine, locations) + } + }) + } +} + +// Without a recorded line, the call that declared the function is found in +// the module, and a strict lookup takes it: it is the function's own place. +func TestLookupNameStrictGeneratedFunctionWithoutLineFindsDeclaration(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, + dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 1, keepFile: "deps/shared_lib/lib/interface.ex", keepLine: 1112}, + ) + locations, err := server.LookupName("MyApp.Chat", "get_room_by_slug!", NameLookupOptions{ExactModule: true}) + if err != nil { + t.Fatal(err) + } + if len(locations) != 1 || locations[0].FilePath != domainPath || locations[0].Line != generatedDefineLine { + t.Fatalf("expected %s:%d, got %#v", domainPath, generatedDefineLine, locations) + } +} + +// A library's `quote location: :keep` names the library's own file. That is +// the generator, not the declaration, and :line only says where the +// before-compile hook ran. The module's body still has the declaring call. +func TestDefinitionGeneratedFunctionForeignLocationFindsDeclaration(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, + dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 1, keepFile: "deps/shared_lib/lib/interface.ex", keepLine: 1112}, + ) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) +} + +// Dexter cannot compile the project, so a BEAM older than its source is the +// usual state while editing. Its line is from the last compile, which is +// still closer than the module line. +func TestDefinitionGeneratedFunctionStaleBEAMUsesRecordedLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, stampedDefinitions()...) + future := time.Now().Add(time.Hour) + if err := os.Chtimes(domainPath, future, future); err != nil { + t.Fatal(err) + } + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) +} + +// A module compiled without debug info still has its Docs chunk, whose anno is +// the line the def was expanded at. The compile info says which file that line +// is in. +func TestDefinitionGeneratedFunctionUsesDocsLineWithoutDebugInfo(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixtureWith(t, func(domainPath string) testBeamChunks { + return testBeamChunks{ + docs: docsTerm(docsEntry{name: "get_room_by_slug!", arity: 1, anno: generatedDefineLine}), + source: domainPath, + } + }) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) +} + +// Without compile info, a Docs anno does not say which file it is in. +func TestDefinitionGeneratedFunctionDocsLineWithoutSourceKeepsModuleLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixtureWith(t, func(string) testBeamChunks { + return testBeamChunks{docs: docsTerm(docsEntry{name: "get_room_by_slug!", arity: 1, anno: generatedDefineLine})} + }) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedModuleLine) +} + +// sourcelessModule is a generated module with no source row, as Spark creates +// for each DSL entity. It was compiled from a framework file on another machine. +const ( + sourcelessModule = "MyApp.Chat.Define" + sourcelessRecorded = "/build/agent/deps/shared_lib/lib/shared_lib/dsl/extension.ex" +) + +// The recorded path does not exist here, so it is rebased onto the project's +// deps, where the same file is. +func TestLookupNameSourcelessGeneratedModuleUsesCompiledSource(t *testing.T) { + for _, tc := range []struct { + name string + chunks testBeamChunks + }{ + {"debug info", testBeamChunks{ + source: sourcelessRecorded, + dbgi: dbgiTerm(sourcelessRecorded, "lib/shared_lib/dsl/extension.ex", dbgiDefinition{name: "build", arity: 1, line: 42}), + }}, + {"docs", testBeamChunks{ + source: sourcelessRecorded, + docs: docsTerm(docsEntry{name: "build", arity: 1, anno: 42}), + }}, + } { + t.Run(tc.name, func(t *testing.T) { + server, _ := newGeneratedDefinitionFixture(t, generatedDomainRel, stampedDefinitions()...) + source := filepath.Join(server.projectRoot, "deps", "shared_lib", "lib", "shared_lib", "dsl", "extension.ex") + if err := os.MkdirAll(filepath.Dir(source), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(source, []byte("defmodule SharedLib.Dsl.Extension do\n"+strings.Repeat("\n", 50)+"end\n"), 0o644); err != nil { + t.Fatal(err) + } + writeGeneratedBeam(t, server, sourcelessModule, tc.chunks, beamExport{"build", 1}) + + locations, err := server.LookupName(sourcelessModule, "build", NameLookupOptions{ExactModule: true}) + if err != nil { + t.Fatal(err) + } + if len(locations) != 1 || locations[0].FilePath != source || locations[0].Line != 42 { + t.Fatalf("expected %s:42, got %#v", source, locations) + } + }) + } +} + +// If the compiled source is not in this checkout, the lexical parent is still +// the answer. +func TestLookupNameSourcelessGeneratedModuleMissingSourceKeepsParent(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, stampedDefinitions()...) + writeGeneratedBeam(t, server, sourcelessModule, testBeamChunks{ + source: sourcelessRecorded, + docs: docsTerm(docsEntry{name: "build", arity: 1, anno: 42}), + }, beamExport{"build", 1}) + + locations, err := server.LookupName(sourcelessModule, "build", NameLookupOptions{FallbackToModule: true}) + if err != nil { + t.Fatal(err) + } + if len(locations) != 1 || locations[0].FilePath != domainPath || locations[0].Line != generatedModuleLine { + t.Fatalf("expected %s:%d, got %#v", domainPath, generatedModuleLine, locations) + } +} + +// The debug info's line belongs to the file it was compiled from. If that is +// not the file the index has for the module, the line means nothing there. +func TestDefinitionGeneratedFunctionOtherSourceKeepsModuleLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, "lib/my_app/other_chat.ex", + dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 1, keepFile: "lib/my_app/other_chat.ex", keepLine: generatedDefineLine}, + ) + // The absolute path must not match either. + beamPath := filepath.Join(server.projectRoot, "_build", "dev", "lib", "my_app", "ebin", "Elixir.MyApp.Chat.beam") + beamFile := minimalBeamWithDbgi(dbgiTerm(filepath.Join(server.projectRoot, "lib/my_app/other_chat.ex"), "lib/my_app/other_chat.ex", + dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 1, keepFile: "lib/my_app/other_chat.ex", keepLine: generatedDefineLine}), + beamExport{"get_room_by_slug!", 1}) + if err := os.WriteFile(beamPath, beamFile, 0o644); err != nil { + t.Fatal(err) + } + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedModuleLine) +} + +// Call hierarchy names the same place definition goes to. +func TestPrepareCallHierarchyGeneratedFunctionUsesDebugInfoLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, stampedDefinitions()...) + docURI := string(uri.File(filepath.Join(server.projectRoot, generatedCallerRel))) + server.docs.Set(docURI, generatedCallerSource) + items, err := server.PrepareCallHierarchy(context.Background(), &protocol.CallHierarchyPrepareParams{ + TextDocumentPositionParams: protocol.TextDocumentPositionParams{ + TextDocument: protocol.TextDocumentIdentifier{URI: protocol.DocumentURI(docURI)}, + Position: protocol.Position{Line: 4, Character: 10}, + }, + }) + if err != nil { + t.Fatal(err) + } + if len(items) != 1 || uriToPath(items[0].URI) != domainPath || int(items[0].Range.Start.Line) != generatedDefineLine-1 { + t.Fatalf("expected the call hierarchy item at %s:%d, got %#v", domainPath, generatedDefineLine, items) + } +} + +func TestLineDeclaresAtom(t *testing.T) { + names := []string{"get_room_by_slug!", "get_room_by_slug"} + for _, text := range []string{ + ` define :get_room_by_slug, action: :read`, + ` SharedLib.Interface.define(:get_room_by_slug)`, + `define :get_room_by_slug`, + } { + if !lineDeclaresAtom(text, names) { + t.Errorf("lineDeclaresAtom(%q) = false, want true", text) + } + } + // A call, a typespec, a keyword key or value, a longer atom, and an atom + // that is not the first argument are not declarations. + for _, text := range []string{ + `get_room_by_slug!("x")`, + `def typed(x :: :get_room_by_slug), do: x`, + `[key:get_room_by_slug]`, + `read action: :get_room_by_slug`, + `define :get_room_by_slug_extra`, + `define :other, as: :get_room_by_slug`, + `:get_room_by_slug`, + `define:get_room_by_slug`, + `@tag :get_room_by_slug`, + } { + if lineDeclaresAtom(text, names) { + t.Errorf("lineDeclaresAtom(%q) = true, want false", text) + } + } +} + +// Lines added above the declaration after the compile move it, but Dexter +// cannot compile: the line from the last compile is the answer until the next. +func TestDefinitionGeneratedFunctionStaleSourceKeepsCompiledLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) + indexFile(t, server.store, server.projectRoot, generatedDomainRel, "# one\n# two\n"+generatedDomainSource) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) +} + +// A current BEAM's lines are exact, even when the declaring line does not +// spell the name and another line does. +func TestDefinitionGeneratedFunctionCurrentBEAMKeepsRecordedLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, + dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 6, keepFile: "deps/ash/lib/ash/code_interface.ex", keepLine: 1112}, + ) + future := time.Now().Add(time.Hour) + beamPath := filepath.Join(server.projectRoot, "_build", "dev", "lib", "my_app", "ebin", "Elixir.MyApp.Chat.beam") + if err := os.Chtimes(beamPath, future, future); err != nil { + t.Fatal(err) + } + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, 6) +} + +func TestDefinitionGeneratedFunctionPastEndOfFileKeepsModuleLine(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) + short := "defmodule MyApp.Chat do\nend\n" + indexFile(t, server.store, server.projectRoot, generatedDomainRel, short) + server.docs.Set(string(uri.File(domainPath)), short) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, 1) +} + +// A `_build` copied from another checkout records that checkout's path, which +// usually still exists. The copy in this project wins; a path outside the +// project is used only when the project has no copy, as for a `path:` +// dependency. +func TestLookupNameSourcelessGeneratedModulePrefersThisCheckout(t *testing.T) { + for _, tc := range []struct { + name string + localCopy bool + }{ + {"other checkout", true}, + {"path dependency", false}, + } { + t.Run(tc.name, func(t *testing.T) { + server, _ := newGeneratedDefinitionFixture(t, generatedDomainRel, stampedDefinitions()...) + outside := filepath.Join(t.TempDir(), "other_checkout", "lib", "shared_lib", "dsl.ex") + local := filepath.Join(server.projectRoot, "lib", "shared_lib", "dsl.ex") + paths := []string{outside} + if tc.localCopy { + paths = append(paths, local) + } + for _, path := range paths { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte("defmodule SharedLib.Dsl do\n"+strings.Repeat("\n", 50)+"end\n"), 0o644); err != nil { + t.Fatal(err) + } + } + writeGeneratedBeam(t, server, sourcelessModule, testBeamChunks{ + source: outside, + docs: docsTerm(docsEntry{name: "build", arity: 1, anno: 42}), + }, beamExport{"build", 1}) + + want := outside + if tc.localCopy { + want = local + } + locations, err := server.LookupName(sourcelessModule, "build", NameLookupOptions{ExactModule: true}) + if err != nil { + t.Fatal(err) + } + if len(locations) != 1 || locations[0].FilePath != want || locations[0].Line != 42 { + t.Fatalf("expected %s:42, got %#v", want, locations) + } + }) + } +} + +func TestSourceRebaseCandidates(t *testing.T) { + root := filepath.FromSlash("/work/copy") + for _, tc := range []struct { + name, recorded, want string + }{ + {"moved umbrella app", "/work/orig/apps/gen_lib/lib/gen_lib.ex", "/work/copy/apps/gen_lib/lib/gen_lib.ex"}, + {"moved project", "/work/orig/lib/my_app/chat.ex", "/work/copy/lib/my_app/chat.ex"}, + {"dependency built elsewhere", "/build/agent/deps/spark/lib/spark/dsl/extension.ex", "/work/copy/deps/spark/lib/spark/dsl/extension.ex"}, + {"dependency built in its own repository", "/build/spark/lib/spark/dsl/extension.ex", "/work/copy/deps/spark/lib/spark/dsl/extension.ex"}, + } { + t.Run(tc.name, func(t *testing.T) { + got := sourceRebaseCandidates(filepath.FromSlash(tc.recorded), root) + if !slices.Contains(got, filepath.FromSlash(tc.want)) { + t.Errorf("candidates = %v, want %s among them", got, tc.want) + } + }) + } + // A bare file name must not match a file at the project root. + if got := sourceRebaseCandidates(filepath.FromSlash("/elsewhere/mix.exs"), root); slices.Contains(got, filepath.Join(root, "mix.exs")) { + t.Errorf("candidates = %v, must not include the root's mix.exs", got) + } +} + +// An example in a @moduledoc is not a declaration, and neither is the same +// call in a sibling module of the same file. +func TestDeclarationSkipsHeredocsAndSiblings(t *testing.T) { + text := `defmodule MyApp.Chat do + @moduledoc """ + Example: + + define :get_room_by_slug + """ + + resources do + define :get_room_by_slug + end +end + +defmodule MyApp.Archive do + define :get_room_by_slug +end +` + src := ¤tSource{text: text, lines: blankHeredocs(strings.Split(text, "\n"))} + functions := []beam.Function{{Name: "get_room_by_slug!", Arity: 1}} + if got, ok := src.declaration("MyApp.Chat", functions, nil); !ok || got != 9 { + t.Errorf("declaration = %d, %v; want 9, the one call in MyApp.Chat's body", got, ok) + } +} + +// A declaring call whose argument opens a heredoc still declares its name; +// only the heredoc's body is blanked. +func TestDeclarationKeepsCallThatOpensHeredoc(t *testing.T) { + text := `defmodule MyApp.Chat do + resources do + define :get_room_by_slug, description: """ + define :list_rooms + """ + end +end +` + src := ¤tSource{text: text, lines: blankHeredocs(strings.Split(text, "\n"))} + if got, ok := src.declaration("MyApp.Chat", []beam.Function{{Name: "get_room_by_slug!", Arity: 1}}, nil); !ok || got != 3 { + t.Errorf("declaration(get_room_by_slug!) = %d, %v; want 3, the call that opens the heredoc", got, ok) + } + if got, ok := src.declaration("MyApp.Chat", []beam.Function{{Name: "list_rooms", Arity: 0}}, nil); ok { + t.Errorf("declaration(list_rooms) = %d; want none, the only match is inside the heredoc", got) + } +} + +// When several calls spell a function's name, as an action and the code +// interface that runs it do, the macro whose calls name the most of the +// module's generated functions declared it. A tie gives no answer. +func TestDeclarationPrefersMacroThatNamesGeneratedFunctions(t *testing.T) { + text := `defmodule MyApp.Article do + actions do + create :create do + accept [:title] + end + + update :publish do + end + + read :published + end + + code_interface do + define :create, args: [:title] + define :publish + define :published + end + + later :twice + later :twice + + first :tied + second :tied +end +` + src := ¤tSource{text: text, lines: blankHeredocs(strings.Split(text, "\n"))} + var generated []beam.Function + for _, name := range []string{"create", "create!", "publish", "publish!", "published", "published!", "twice", "tied"} { + generated = append(generated, beam.Function{Name: name, Arity: 1}) + } + for _, tc := range []struct { + name string + want int + }{ + {"create", 14}, + {"publish!", 15}, + {"published", 16}, + } { + if got, ok := src.declaration("MyApp.Article", []beam.Function{{Name: tc.name, Arity: 1}}, generated); !ok || got != tc.want { + t.Errorf("declaration(%s) = %d, %v; want %d, the define call", tc.name, got, ok, tc.want) + } + } + for _, name := range []string{"twice", "tied"} { + if got, ok := src.declaration("MyApp.Article", []beam.Function{{Name: name, Arity: 0}}, generated); ok { + t.Errorf("declaration(%s) = %d; want none, two calls are equally likely", name, got) + } + } + // Without the module's generated functions, there is nothing to choose by. + if got, ok := src.declaration("MyApp.Article", []beam.Function{{Name: "publish", Arity: 1}}, nil); ok { + t.Errorf("declaration(publish) without generated functions = %d; want none", got) + } +} + +// A file that a request read from disk, without opening it, is only a cache. +// If the file then changes and is compiled again, the cache is old text, not +// unsaved edits, and a current BEAM's line must not be corrected against it. +func TestDefinitionGeneratedFunctionIgnoresCachedFile(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) + if _, ok := server.docs.GetOrLoad(string(uri.File(domainPath))); !ok { + t.Fatal("GetOrLoad did not cache the file") + } + edited := strings.Replace(generatedDomainSource, " resources do\n", " resources do\n # added\n", 1) + if err := os.WriteFile(domainPath, []byte(edited), 0o644); err != nil { + t.Fatal(err) + } + indexFile(t, server.store, server.projectRoot, generatedDomainRel, edited) + // Compiled after the edit: the define is now on line 9, and the BEAM says so. + const generator = "deps/ash/lib/ash/code_interface.ex" + writeGeneratedBeam(t, server, "MyApp.Chat", testBeamChunks{dbgi: dbgiTerm(domainPath, generatedDomainRel, + dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: generatedDefineLine + 1, keepFile: generator, keepLine: 1112}, + )}, beamExport{"get_room_by_slug!", 1}) + future := time.Now().Add(time.Hour) + beamPath := filepath.Join(server.projectRoot, "_build", "dev", "lib", "my_app", "ebin", "Elixir.MyApp.Chat.beam") + if err := os.Chtimes(beamPath, future, future); err != nil { + t.Fatal(err) + } + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine+1) +} + +func TestModuleOwnersByLine(t *testing.T) { + source := `defmodule A do + @moduledoc """ + defmodule NotAModule do + """ + defmodule Inner do + def x, do: :ok + end + + def y, do: "end" +end + +defmodule B do +end +` + owners := moduleOwnersByLine([]byte(source), strings.Count(source, "\n")+1) + for line, want := range map[int]string{1: "", 2: "A", 3: "A", 5: "A", 6: "A.Inner", 7: "A.Inner", 9: "A", 10: "A", 11: "", 13: "B"} { + if owners[line] != want { + t.Errorf("line %d owner = %q, want %q", line, owners[line], want) + } + } +} + +// A module defined in two files, as two umbrella apps can, has two rows. The +// recorded path decides between them; when it cannot, no line is trusted. +func TestBestDescribedResult(t *testing.T) { + a := store.LookupResult{FilePath: "/new/apps/a/lib/dup.ex", Line: 1} + b := store.LookupResult{FilePath: "/new/apps/b/lib/dup.ex", Line: 1} + lines := map[beam.FunctionKey]int{{Name: "f", Arity: 0}: 5} + for _, tc := range []struct { + name string + info beam.DebugInfo + want string + wantAmbiguous bool + }{ + {"exact path", beam.DebugInfo{File: b.FilePath, RelativeFile: "lib/dup.ex", Lines: lines}, b.FilePath, false}, + {"moved umbrella", beam.DebugInfo{File: "/old/apps/b/lib/dup.ex", RelativeFile: "lib/dup.ex", Lines: lines}, b.FilePath, false}, + {"only the relative path", beam.DebugInfo{RelativeFile: "lib/dup.ex", Lines: lines}, "", true}, + } { + t.Run(tc.name, func(t *testing.T) { + sources := generatedDefinitionSources{debugInfo: tc.info} + got, found, ambiguous := sources.bestDescribedResult([]store.LookupResult{a, b}) + if ambiguous != tc.wantAmbiguous || found != (tc.want != "") || (found && got.FilePath != tc.want) { + t.Errorf("bestDescribedResult = %s, found=%v, ambiguous=%v; want %q, ambiguous=%v", got.FilePath, found, ambiguous, tc.want, tc.wantAmbiguous) + } + }) + } +} + +// The declaration was deleted, and a module below declares the same name. The +// only declaration left is in another module, so the recorded line stays. +func TestDefinitionGeneratedFunctionDriftIgnoresModuleBelow(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, lineDefinitions()...) + withoutDefine := strings.Replace(generatedDomainSource, " define :get_room_by_slug, action: :read, get_by: [:slug]\n", "", 1) + edited := withoutDefine + ` +defmodule MyApp.Archive do + resources do + define :get_room_by_slug, action: :read + end +end +` + indexFile(t, server.store, server.projectRoot, generatedDomainRel, edited) + server.docs.Set(string(uri.File(domainPath)), edited) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) +} + +// A generator can give a def any line, such as `quote line: 99`. Even from a +// current BEAM, a line past the end of the file is not a place to go; the +// declaring call is. +func TestDefinitionGeneratedFunctionLinePastEndOfFileFindsDeclaration(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, + dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 99, keepFile: "deps/shared_lib/lib/interface.ex", keepLine: 1112}, + ) + future := time.Now().Add(time.Hour) + beamPath := filepath.Join(server.projectRoot, "_build", "dev", "lib", "my_app", "ebin", "Elixir.MyApp.Chat.beam") + if err := os.Chtimes(beamPath, future, future); err != nil { + t.Fatal(err) + } + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedDefineLine) +} + +// With no line from the BEAM and no call that declares the name, the module +// is still the answer. Two declaring calls are no better than none. +func TestDefinitionGeneratedFunctionWithoutDeclarationKeepsModuleLine(t *testing.T) { + for _, tc := range []struct { + name, source string + }{ + {"no declaring call", strings.Replace(generatedDomainSource, " define :get_room_by_slug, action: :read, get_by: [:slug]\n", "", 1)}, + {"two declaring calls", strings.Replace(generatedDomainSource, " define :list_rooms, action: :read\n", " define :get_room_by_slug, action: :read\n", 1)}, + } { + t.Run(tc.name, func(t *testing.T) { + server, domainPath := newGeneratedDefinitionFixture(t, generatedDomainRel, + dbgiDefinition{name: "get_room_by_slug!", arity: 1, line: 1, keepFile: "deps/shared_lib/lib/interface.ex", keepLine: 1112}, + ) + indexFile(t, server.store, server.projectRoot, generatedDomainRel, tc.source) + locations := generatedDefinitionAt(t, server, generatedCallerRel, generatedCallerSource, 4, 10) + expectSingleLocation(t, locations, domainPath, generatedModuleLine) + }) + } +} diff --git a/internal/lsp/name_navigation.go b/internal/lsp/name_navigation.go index 4e380f0..3719bcf 100644 --- a/internal/lsp/name_navigation.go +++ b/internal/lsp/name_navigation.go @@ -47,6 +47,11 @@ type NameReferenceOptions struct { func (s *Server) LookupName(module, function string, opts NameLookupOptions) ([]NameLocation, error) { if function == "" { results, err := s.store.LookupModule(module) + if err == nil && len(results) == 0 { + // A module that exists only as a BEAM, such as one Module.create + // made, is defined where the compiler recorded it. + results = s.generatedModuleLocation(module) + } return s.lookupLocations(results, opts.ExcludeStdlib), err } @@ -69,14 +74,16 @@ func (s *Server) LookupName(module, function string, opts NameLookupOptions) ([] } if len(results) == 0 && opts.Kind != NameKindType { if generated, found := s.generatedSymbol(module, "", function); found && len(generated) > 0 { - if opts.ExactModule { + // A line the compiled module records for the function is its own + // definition, so even an exact lookup takes it. + var precise bool + results, precise = s.generatedDefinitionResultsFor(module, "", generated) + if !precise && opts.ExactModule { if results, err = s.store.LookupModule(module); err != nil { return nil, err } - } else { - results = s.generatedDefinitionResults(module) } - if len(results) > 0 { + if !precise && len(results) > 0 { results[0].Arity = generated[0].Arity results[0].Kind = generated[0].Kind } diff --git a/internal/lsp/server.go b/internal/lsp/server.go index 0499a21..3e5cbfc 100644 --- a/internal/lsp/server.go +++ b/internal/lsp/server.go @@ -1208,11 +1208,11 @@ func (s *Server) Definition(ctx context.Context, params *protocol.DefinitionPara s.debugf("Definition: resolved bare %q -> %q", functionName, fullModule) if fullModule == "" { currentModule := s.store.LookupEnclosingModule(uriToPath(protocol.DocumentURI(docURI)), lineNum+1) - if provider, _, found := s.generatedSymbolInScope(currentModule, func() []string { + if provider, functions, found := s.generatedSymbolInScope(currentModule, func() []string { return s.enclosingBlockPath(docURI, lineNum, col) }, functionName); found { - if results := s.generatedDefinitionResults(provider.module); len(results) > 0 { - s.debugf("Definition: generated bare %q provider=%s", functionName, provider.module) + if results, precise := s.generatedDefinitionResultsFor(provider.module, provider.beamPath, functions); len(results) > 0 { + s.debugf("Definition: generated bare %q provider=%s precise=%t", functionName, provider.module, precise) return storeResultsToLocations(results), nil } } @@ -1256,11 +1256,11 @@ func (s *Server) Definition(ctx context.Context, params *protocol.DefinitionPara } currentModule = s.store.LookupEnclosingModule(uriToPath(protocol.DocumentURI(docURI)), lineNum+1) - if provider, _, found := s.generatedSymbolInScope(currentModule, func() []string { + if provider, functions, found := s.generatedSymbolInScope(currentModule, func() []string { return s.enclosingBlockPath(docURI, lineNum, col) }, functionName); found { - if results := s.generatedDefinitionResults(provider.module); len(results) > 0 { - s.debugf("Definition: generated fallback for bare %q provider=%s", functionName, provider.module) + if results, precise := s.generatedDefinitionResultsFor(provider.module, provider.beamPath, functions); len(results) > 0 { + s.debugf("Definition: generated fallback for bare %q provider=%s precise=%t", functionName, provider.module, precise) return storeResultsToLocations(results), nil } } @@ -3688,6 +3688,14 @@ func (s *Server) resolveBareFunctionModuleWithOrigin(filePath, text string, tf * return "Kernel", false } + // An imported module can export a function that a macro generated, which + // only its BEAM knows about. + for _, mod := range imports { + if _, found := s.generatedSymbol(mod, "", functionName); found { + return mod, false + } + } + // Slow fallback: function may be injected into an imported module via its // own use chain (e.g. MyApp.Factory uses ExMachina, which injects `insert`). for _, mod := range imports { @@ -7151,7 +7159,7 @@ func (s *Server) PrepareCallHierarchy(ctx context.Context, params *protocol.Call if len(generatedFunctions) == 0 { return nil, nil } - defResults = s.generatedDefinitionResults(fullModule) + defResults, _ = s.generatedDefinitionResultsFor(fullModule, "", generatedFunctions) if len(defResults) == 0 { return nil, nil } diff --git a/internal/parser/parser.go b/internal/parser/parser.go index fb2714c..e356b5d 100644 --- a/internal/parser/parser.go +++ b/internal/parser/parser.go @@ -237,6 +237,31 @@ func HasLinkedWorktreeGitFile(dir string, entries []fs.DirEntry) bool { return false } +// UnsettledGitFile reports whether dir holds a .git file that names no git +// directory yet. git rewrites a worktree's .git file in place when it moves the +// worktree, so for a moment after the rename the file is empty, and a directory +// that is a worktree looks like a plain one. Such a directory is checked again +// once git is done, rather than indexed. +func UnsettledGitFile(dir string) bool { + info, err := os.Lstat(filepath.Join(dir, ".git")) + if err != nil || !info.Mode().IsRegular() { + return false + } + _, ok := gitdirFromFile(filepath.Join(dir, ".git")) + return !ok +} + +// HasUnsettledGitFile is UnsettledGitFile for a directory whose entries are +// already read, so a directory without a .git file costs no syscall. +func HasUnsettledGitFile(dir string, entries []fs.DirEntry) bool { + for _, e := range entries { + if e.Name() == ".git" { + return !e.IsDir() && UnsettledGitFile(dir) + } + } + return false +} + // isLinkedWorktreeGitFile reports whether the .git file at path belongs to a // linked worktree. Submodules also have a .git file, and they stay indexed like // any other directory. Only a directory that has a .git file pays for this check. diff --git a/internal/parser/parser_test.go b/internal/parser/parser_test.go index 6ba6441..b09aa0f 100644 --- a/internal/parser/parser_test.go +++ b/internal/parser/parser_test.go @@ -233,6 +233,99 @@ end } } +// A def inside a macro's quote is code the macro generates in its caller, not a +// function of the macro's module. What __using__ injects is still indexed, and +// so is a quote in a helper function, which the use-chain resolves. +func TestParseFile_DoesNotIndexDefsInsideMacroQuotes(t *testing.T) { + path := writeTempFile(t, `defmodule MyApp.Dsl do + defmacro __using__(_opts) do + quote do + import MyApp.Dsl + def injected, do: :injected + end + end + + defmacro route(verb, path) do + quote do + def handle(unquote(verb), unquote(path)), do: :ok + end + end + + defmacro kept(name) do + quote bind_quoted: [ + name: name + ], + location: :keep do + def bound_def, do: name + end + end + + defmacro assigned do + ast = quote do + def assigned_def, do: :ok + end + ast + end + + defmacro inline do + quote do: def inline_def, do: :ok + end + + defmacro __before_compile__(_env) do + quote do + def before_compile_def, do: :ok + end + end + + defmacro same_line(name), do: quote do + def same_line_def, do: unquote(name) + end + + defmacro next_line(name), do: + quote do + def next_line_def, do: unquote(name) + end + + defmacro one_line do quote do + def one_line_def, do: :ok + end + end + + defmacro compact, do: quote do: def compact_def, do: :ok + + def controller do + quote do + def helper_quote_def, do: :ok + end + end + + def handle_request(conn), do: conn + defmacro real_macro(x), do: x +end +`) + + defs, _, err := ParseFile(path) + if err != nil { + t.Fatal(err) + } + found := map[string]bool{} + for _, d := range defs { + if d.Function != "" { + found[d.Function] = true + } + } + for _, name := range []string{"__using__", "injected", "route", "kept", "assigned", "inline", "__before_compile__", "same_line", "next_line", "one_line", "compact", "controller", "helper_quote_def", "handle_request", "real_macro"} { + if !found[name] { + t.Errorf("%s should be indexed; got %v", name, found) + } + } + for _, name := range []string{"handle", "bound_def", "assigned_def", "inline_def", "before_compile_def", "same_line_def", "next_line_def", "one_line_def", "compact_def"} { + if found[name] { + t.Errorf("%s is generated by a macro's quote and should not be indexed as MyApp.Dsl.%s", name, name) + } + } +} + func TestParseFile_DoesNotIndexUnquoteFragmentsAsDefinitions(t *testing.T) { path := writeTempFile(t, `defmodule MyApp.Generated do for name <- [:foo, :bar] do diff --git a/internal/parser/parser_tokenized.go b/internal/parser/parser_tokenized.go index efaffbf..b36e476 100644 --- a/internal/parser/parser_tokenized.go +++ b/internal/parser/parser_tokenized.go @@ -18,11 +18,42 @@ func parseTextFromTokens(path string, source []byte, tokens, interp []Token) ([] var moduleStack []moduleFrame depth := 0 + aliases := map[string]string{} injectors := map[string]bool{} n := len(tokens) + // A def inside a macro's quote is code the macro generates in its caller, + // not a function of the macro's module, so it is not indexed as one. The + // compiled caller's BEAM records where it was declared. `__using__` is the + // exception: what it injects is resolved through its body, and kept. + // macroBodies holds the depth inside each open do-block of such a macro, + // innermost last. quoteDepth is the depth inside an open `quote do` block + // in one, or zero. quoteLine is the line of an inline `quote do:`. + var macroBodies []int + quoteDepth, quoteLine := 0, 0 + // inlineMacroLine is the line of the last such macro written with `, do:`, + // whose quote can follow on the next line. + inlineMacroLine := 0 + markQuote := func(at, depthAt int) { + switch form, line := quoteForm(source, tokens, n, at+1); form { + case quoteBlock: + // The do is counted when the walk reaches it. + quoteDepth = depthAt + 1 + case quoteInline: + quoteLine = line + } + } + leaveBlocks := func() { + if quoteDepth > depth { + quoteDepth = 0 + } + for len(macroBodies) > 0 && macroBodies[len(macroBodies)-1] > depth { + macroBodies = macroBodies[:len(macroBodies)-1] + } + } + tokenText := func(t Token) string { return TokenText(source, t) } @@ -218,6 +249,7 @@ func parseTextFromTokens(path string, source []byte, tokens, interp []Token) ([] aliases = frame.savedAliases injectors = frame.savedInjectors } + leaveBlocks() } } } @@ -254,6 +286,7 @@ func parseTextFromTokens(path string, source []byte, tokens, interp []Token) ([] aliases = frame.savedAliases injectors = frame.savedInjectors } + leaveBlocks() i++ continue @@ -308,6 +341,43 @@ func parseTextFromTokens(path string, source []byte, tokens, interp []Token) ([] delegateTo, delegateAs = scanDelegateOpts(pj) } + // Decided before the head is scanned below, which can mark a + // quote on this same line (`defmacro m, do: quote do: ...`). + insideQuote := (quoteDepth > 0 && depth >= quoteDepth) || quoteLine == defLine + if (kind == "defmacro" || kind == "defmacrop") && funcName != "__using__" { + if opensDoBlock(tokens, n, pj) { + // The do on this line is counted when the line is walked + // below, so the body is one deeper than here. + macroBodies = append(macroBodies, depth+1) + } else { + inlineMacroLine = defLine + } + // The walk below takes the rest of this line, so a quote on + // it (`, do: quote do`) is found here. + if quoteDepth == 0 { + lineDepth := depth + for k := pj; k < n && tokens[k].Kind != TokEOL && tokens[k].Kind != TokEOF; k++ { + switch tokens[k].Kind { + case TokDo, TokFn: + lineDepth++ + case TokEnd: + lineDepth-- + case TokIdent: + if tokenText(tokens[k]) == "quote" { + markQuote(k, lineDepth) + } + } + if quoteDepth != 0 || quoteLine == defLine { + break + } + } + } + } + if insideQuote { + i = j + goto extractRefsForLine + } + minArity := maxArity - defaultCount for arity := minArity; arity <= maxArity; arity++ { params := JoinParams(paramNames, arity) @@ -636,6 +706,9 @@ func parseTextFromTokens(path string, source []byte, tokens, interp []Token) ([] continue case TokIdent: + if quoteDepth == 0 && tokenText(tok) == "quote" && (len(macroBodies) > 0 || followsInlineMacroHead(source, tokens, i, inlineMacroLine)) { + markQuote(i, depth) + } cm := currentModule() if cm != "" && len(injectors) > 0 { isStatementStart := i == 0 || tokens[i-1].Kind == TokEOL || tokens[i-1].Kind == TokComment @@ -1014,3 +1087,82 @@ func FixParamNames(names []string) []string { } return names } + +// opensDoBlock reports whether the declaration whose head ends before tokens[from] +// opens a do-block on its line, rather than taking `, do:` or having no body. +func opensDoBlock(tokens []Token, n, from int) bool { + for k := from; k < n; k++ { + switch tokens[k].Kind { + case TokDo: + return true + case TokComma, TokEOL, TokEOF: + return false + } + } + return false +} + +type quoteShape int + +const ( + quoteNone quoteShape = iota + quoteBlock + quoteInline +) + +// quoteForm reads what follows a `quote` identifier: options such as +// `location: :keep` or `bind_quoted: [...]`, which may span lines inside +// brackets, then either a do-block or an inline `do:`. For the inline form it +// also returns the line of the `do:`. +func quoteForm(source []byte, tokens []Token, n, from int) (quoteShape, int) { + balance := 0 + // A trailing comma carries the options on to the next line. + continued := false + for k := from; k < n; k++ { + kind := tokens[k].Kind + switch kind { + case TokOpenParen, TokOpenBracket, TokOpenBrace: + balance++ + case TokCloseParen, TokCloseBracket, TokCloseBrace: + balance-- + if balance < 0 { + return quoteNone, 0 + } + case TokDo: + if balance == 0 { + return quoteBlock, 0 + } + case TokIdent: + if TokenText(source, tokens[k]) == "do" && k+1 < n && tokens[k+1].Kind == TokColon { + return quoteInline, tokens[k].Line + } + case TokEOL: + if balance == 0 && !continued { + return quoteNone, 0 + } + case TokEOF: + return quoteNone, 0 + } + if kind != TokEOL && kind != TokComment { + continued = kind == TokComma + } + } + return quoteNone, 0 +} + +// followsInlineMacroHead reports whether the token at i comes right after the +// `do:` that ends the head of a macro on line, as in +// +// defmacro route(verb, path), do: +// quote do +func followsInlineMacroHead(source []byte, tokens []Token, i, line int) bool { + if line == 0 { + return false + } + k := i - 1 + for k >= 0 && (tokens[k].Kind == TokEOL || tokens[k].Kind == TokComment) { + k-- + } + return k >= 1 && tokens[k].Kind == TokColon && tokens[k-1].Kind == TokIdent && + TokenText(source, tokens[k-1]) == "do" && tokens[k-1].Line == line +} diff --git a/internal/version/version.go b/internal/version/version.go index 97903fe..b896a42 100644 --- a/internal/version/version.go +++ b/internal/version/version.go @@ -7,4 +7,4 @@ const Version = "0.7.2" // a change that makes existing indexes stale — and bump daemon.ContractVersion // in the same release, because a running daemon from the older build would // otherwise keep serving the new frontend from that stale index. -const IndexVersion = 14 +const IndexVersion = 15 diff --git a/internal/workspace/watch_fsnotify.go b/internal/workspace/watch_fsnotify.go index 8c8e29f..f40f670 100644 --- a/internal/workspace/watch_fsnotify.go +++ b/internal/workspace/watch_fsnotify.go @@ -136,6 +136,14 @@ func (w *fsnotifyWatcher) walkDirectories(root string, includeRoot bool) int { w.tops.add(dir) return } + // A worktree that git is still moving has an empty .git file. It is a + // top until the pending check says otherwise, so that its files are + // not reported in the meantime. + if dir != w.root && parser.HasUnsettledGitFile(dir, entries) { + w.tops.add(dir) + w.markPending(dir) + return + } for _, e := range entries { if e.IsDir() && !skipWatchDir(e.Name()) { walk(filepath.Join(dir, e.Name())) @@ -179,7 +187,8 @@ func (w *fsnotifyWatcher) checkPending() { if parser.IsLinkedWorktree(dir) { continue } - if recordedWorktree(w.root, dir) { + // git may still be writing the .git file of a worktree it moved. + if parser.UnsettledGitFile(dir) || recordedWorktree(w.root, dir) { recorded = append(recorded, dir) continue } diff --git a/internal/workspace/watch_platform_darwin.go b/internal/workspace/watch_platform_darwin.go index f07e2b8..ad0565b 100644 --- a/internal/workspace/watch_platform_darwin.go +++ b/internal/workspace/watch_platform_darwin.go @@ -150,6 +150,16 @@ func (w *fseventsWatcher) handle(event fsevents.Event) { } return } + // A worktree that git is still moving has an empty .git file. It is a + // top until it is checked again, so that no full reconcile indexes it + // in the meantime. + if flags&(fsevents.ItemCreated|fsevents.ItemRenamed) != 0 && parser.UnsettledGitFile(path) { + if w.tops.add(path) { + w.callbacks.PathChanged(path) + } + w.checkTopLater(path) + return + } if flags&(fsevents.ItemCreated|fsevents.ItemRemoved|fsevents.ItemRenamed) != 0 { w.callbacks.FullReconcile() } @@ -196,7 +206,8 @@ func (w *fseventsWatcher) checkTopLater(dir string) { if parser.IsLinkedWorktree(dir) { return } - if recordedWorktree(w.root, dir) { + // git may still be writing the .git file of a worktree it moved. + if parser.UnsettledGitFile(dir) || recordedWorktree(w.root, dir) { w.checkTopLater(dir) return } diff --git a/internal/workspace/watch_platform_darwin_test.go b/internal/workspace/watch_platform_darwin_test.go index 50b6052..aef744d 100644 --- a/internal/workspace/watch_platform_darwin_test.go +++ b/internal/workspace/watch_platform_darwin_test.go @@ -350,6 +350,29 @@ func TestFSEventsWorktreeLifecycle(t *testing.T) { git("worktree", "remove", wt) settle() + // git worktree move renames the directory, then writes its .git file again + // in place, so the watcher can find the file empty. + outside := filepath.Join(t.TempDir(), "feature") + git("worktree", "add", "-q", outside) + content, err := os.ReadFile(filepath.Join(outside, ".git")) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(outside, ".git"), nil, 0o644); err != nil { + t.Fatal(err) + } + if err := os.Rename(outside, wt); err != nil { + t.Fatal(err) + } + settle() + if err := os.WriteFile(filepath.Join(wt, ".git"), content, 0o644); err != nil { + t.Fatal(err) + } + git("worktree", "repair", wt) + settle() + git("worktree", "remove", "--force", wt) + settle() + mu.Lock() defer mu.Unlock() if full != 0 { diff --git a/internal/workspace/watch_test.go b/internal/workspace/watch_test.go index a99080d..99776c9 100644 --- a/internal/workspace/watch_test.go +++ b/internal/workspace/watch_test.go @@ -539,6 +539,32 @@ func TestFSNotifyWatcherSkipsWorktreeAddedWhileRunning(t *testing.T) { env.settle(t) env.git("worktree", "move", outside, wt) }}, + // git worktree move renames the directory, then writes its .git file + // again in place, so the watcher can find the file empty. + {"move in while git rewrites its .git file", func(t *testing.T, env *watchedRepo, wt string) { + outside := filepath.Join(t.TempDir(), "feature") + env.git("worktree", "add", "-q", outside) + dotgit := filepath.Join(outside, ".git") + content, err := os.ReadFile(dotgit) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(dotgit, nil, 0o644); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(filepath.Dir(wt), 0o755); err != nil { + t.Fatal(err) + } + env.settle(t) + if err := os.Rename(outside, wt); err != nil { + t.Fatal(err) + } + env.settle(t) + if err := os.WriteFile(filepath.Join(wt, ".git"), content, 0o644); err != nil { + t.Fatal(err) + } + env.git("worktree", "repair", wt) + }}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) {