Repository navigation
[Bulk index][Scala] Consume SemanticDB and TASTy snapshots through BSP #77
Description
Outcome
Implement separate Scala 2 and Scala 3 typed compiler exporters, driven by the
repository's BSP/Zinc build. Use direct SemanticDB ingestion for declarations,
types, annotations, overrides, diagnostics and source hashes, but collect
calls/accesses/construction from the same typed compiler pass rather than
starting a second TASTy analysis.
Current master baseline
No strict Scala provider is registered on current master. scip-java 0.13
removed Scala and Zinc support; the current registry correctly limits it to
Java and Kotlin. Scala therefore uses the ordinary language-server/static lane,
and PR #147 has no strict index-time cell to cite.
Do not resurrect the old scip-java Scala route. The last Scala-capable release
is a historical fallback and would pin the product to a removed pipeline.
What SemanticDB provides—and does not
The SemanticDB specification
is a strong per-source artifact:
- document text/md5, URI, language and build target;
- symbol kinds/properties, structured signatures, annotations and access;
- occurrences, diagnostics and synthetics;
- direct parent declarations and overridden symbols.
Direct protobuf ingestion retains more information than converting to SCIP.
It can prove declarations, containment, types, annotations, direct
extends/implements, overrides, source hashes and diagnostics.
Occurrences do not carry a call/read/write role. SemanticDB alone cannot prove
calls, accesses or instantiates. TASTy can fill Scala 3 after compilation,
but a separate TASTy pass duplicates semantic work and has no Scala 2
equivalent. The fastest common architecture is therefore a typed compiler
plugin that emits those facts during the compilation already producing
SemanticDB.
Decision: two typed plugins, one BSP/Zinc transaction
Scala 2
Build a scalac Global plugin with a phase immediately after typer and before
lowerings erase source-level intent. Walk typed trees once, resolve symbols and
types, and write per-source shards.
Scala 3
Build a Dotty StandardPlugin/PluginPhase after typer, following the official
Scala 3 compiler plugin model.
Walk typed trees before later transformations obscure givens, extension
methods, inline/macro origins and source spans.
Build orchestration
Use BSP to discover exact build targets, sources, scalac options, class
directories and dependency graph, then request compilation. Let Zinc/sbt/Bloop/
Mill determine affected targets and source recompilation. The exporter writes
changed shards into a target-scoped output and an atomic target manifest.
Enable SemanticDB in the same compile as a cross-check and source/diagnostic
input:
- Scala 3: compiler
-Xsemanticdb; - Scala 2: matching
semanticdb-scalac, source root/range/text/diagnostic
options; and - never run
cleanmerely to force files to re-emit. Existing validated shards
cover sources Zinc decides are unchanged.
Facts and source semantics
The typed plugins emit:
- packages, classes, traits, objects, enums, opaque/type aliases, methods,
constructors, fields, parameters, locals, extension methods, givens and
generated definitions with origin; - imports, exports and visibility;
- resolved
Apply,Select,New, eta-expansion and extension calls; - read/write accesses and type references;
- direct parents, trait implementations and overridden symbols;
- annotations;
- implicit/given resolution and inline/macro expansion as source-linked
evidence/unresolved facts; and - compiler diagnostics from the same target generation.
Do not invent an explicit source span for a synthetic implicit/given insertion.
Use SemanticDB synthetics and compiler origin trees to link the source site and
selected symbol.
Framework tests use a named ScalaTest/MUnit/specs2 enricher. renders is
unsupported unless a framework-specific producer is added.
Identity and target universes
SemanticDB symbols are useful interoperability keys but overload disambiguators
are source-order based. The final graph identity must use:
- BSP build target and module/artifact coordinate;
- fully qualified owner and symbol kind;
- structural signature, including receiver, parameter/type parameter lists and
result where needed; - Scala language/compiler major and relevant encoding; and
- source owner/lexical structure for locals and anonymous functions.
Exact fixtures insert/reorder overloads and constructors and compare Scala 2/3
encodings. Same-named symbols in main/test/cross-built targets must not collide.
The universe includes BSP target, Scala compiler and plugin version, platform
and Scala binary version, scalac options, classpath order/content, source roots,
SemanticDB options, compiler plugins, macro/inline dependencies, Zinc analysis
and generated sources. Cross builds are separate universes.
Incremental and performance behavior
- no-op: BSP/build-tool null compile plus manifest validation, or manifest load
without a producer invocation when inputs prove unchanged; - body edit: Zinc recompiles affected sources; replace only emitted shards;
- public signature/macro/inline edit: accept Zinc's dependent closure and record
it; - target/classpath/scalac/plugin change: invalidate that target universe;
- create/delete/rename: reconcile target source list and delete stale shards;
and - failed compile/cancel/crash: retain the prior committed target generation.
Do not call SemanticDB per symbol, do not inspect TASTy in a second cold process,
and do not hold compiler tree/symbol objects after the plugin phase.
Cold overhead is measured against the identical native BSP compile. Resident
latency is bounded by the build server's null/incremental compile plus shard
merge.
Implementation map
- Add [Bulk index] Replace symbol-by-symbol LSP scans with compiler-owned snapshots #63's protocol and target-scoped shard store.
- Implement direct SemanticDB protobuf parsing and conformance fixtures.
- Implement the Scala 2 plugin and exact fixture.
- Implement the Scala 3 plugin with the same normalized schema.
- Add BSP discovery/compile/diagnostic integration and Zinc invalidation
traces. - Register the strict provider only when the target's compiler/plugin pair is
compatible; otherwise decline to Metals/static.
Acceptance
- Scala 2 and Scala 3 have separate fixtures and compatibility matrices.
- Typed-tree extraction proves calls/accesses/construction without a second
semantic pass. - SemanticDB md5/build-target/kinds/signatures/annotations/overrides/
diagnostics are retained and cross-checked. - Givens/implicits, extensions, inline/macros, opaque types, enums,
overloads, constructors, generics, locals and tests have exact fixtures. - Overload insertion/reorder and cross-target twins keep stable identities.
- All 15 fact families have explicit coverage/unresolved rows.
- No-op, body/API/macro edit, create/delete/rename, target/classpath/
compiler-option/plugin change, diagnostic, cancel, crash and retry are
atomic. - Cold overhead is measured against the native BSP compile; no-op/edit
phases and Zinc invalidation are reported separately. - Metals/static remain truthful fallbacks; scip-java is not presented as
Scala-capable. - Focused tests, Scala experiments,
pnpm build,pnpm test, and
pnpm coveragepass.
Activity
samchon commentedon Jul 17, 2026
Provider recon at graph
71e35afinvalidates this issue's required bulk route. scip-java no longer indexes Scala at all.sourcegraph/scip-javanow redirects toscip-code/scip-java. Release v0.13.0 (2026-07-02), verbatim:PR #883 "Drop support for indexing Scala code" (merged 2026-05-27) deletes
SbtBuildTool.scala,MillBuildTool.scala,ScalaVersion.scala,ScalaCompilerClassLoader.scala. Theindex-semanticdbsubcommand is gone; the CLI is nowindex,aggregate,snapshot. The last Scala-capable release is v0.12.3 (2026-04-02). The Mill fallbackckipp01/mill-scipis archived (last release 2023-10-06).This is an opportunity, not a setback
Reading SemanticDB directly is strictly better than routing it through SCIP, and it is what this issue already asked for ("consume SemanticDB and TASTy snapshots through BSP").
SemanticDB
TextDocumentcarriesschema,uri,text,md5,language,build_target, plussymbols,occurrences,diagnostics,synthetics.SymbolInformationcarrieskind(CLASS/TRAIT/OBJECT/METHOD/FIELD/…), apropertiesbitmask (ABSTRACT, FINAL, SEALED, IMPLICIT, LAZY, CASE, INLINE, GIVEN…), structuredsignature(ClassSignature parents/declarations; MethodSignature type params/param lists/return),display_name,annotations,access, andoverridden_symbols.Stock SCIP from scip-dotnet and scip-kotlinc emits none of
kind,access,annotations, md5, or diagnostics (see the #69 comment). Routing Scala through SCIP would flatten all of it away.And no JVM is needed to parse it.
semanticdb.protois plain protobuf; this repository is TypeScript and can decode it with the same protobuf tooling #69 already requires forscip.proto. The JVM is needed only to compile via BSP.Revised route
Drop scip-java. Compile through the repo's own build to emit SemanticDB, then parse
semanticdb.protoin-process.-Xsemanticdb(in-compiler).org.scalameta:semanticdb-scalac:4.17.2(2026-07-09) +-Yrangepos,-P:semanticdb:{sourceroot,targetroot,text:on,diagnostics:on}.ThisBuild / semanticdbEnabled := trueconfigures both. Watch sbt#8082 —semanticdbIncludeInJarmust be enabled forsemanticdbEnabledto surviveclean.META-INF/semanticdb/**/*.semanticdb, one per source file, under the class directory.workspace/buildTargets(tagsincludestest),buildTarget/sources,buildTarget/scalacOptions(flags + classpath + classDirectory, i.e. where the SemanticDB lands),buildTarget/compile,build/publishDiagnostics. Servers: Bloop v2.1.1 (2026-07-07), sbt (built-in ≥1.4.0), Mill, bazel-bsp.What this changes for acceptance
Scala becomes the only language in the campaign where
extends,implements,overrides, anddecoratesare proven by the bulk artifact with no sidecar —ClassSignature.parentsis direct (not a transitive closure like scip-dotnet),kinddiscriminates class from trait,overridden_symbolsis first-class, andannotationsis first-class.It is also the best incremental story in the campaign: BSP owns affected-target invalidation, Bloop/sbt are already resident daemons, and SemanticDB is per-document with an md5 — so the provider watches
META-INF/semanticdb/**, replaces changed documents, deletes stale ones, and publishes a generation oncebuildTarget/compilesucceeds. That is genuinesemantic-incremental, needing no custom resident analyzer (unlike C#'s MSBuildWorkspace or Kotlin's Analysis API).And the best provenance: per-document md5 directly satisfies #69's stale-source rejection;
build_targetdisambiguates same-named symbols across targets, which is exactly what this issue's "same-named symbols in target scopes do not collide" row needs.synthetics(Range + Tree) is the truthful home for implicit/given expansions — directly supporting step 5's "never invent an explicit source span for an implicit/given expansion". scala/scala3#13135 is closed (2025-04-29), so Scala 3 now emits synthetics.Residual risk
callsandaccessesare still not proven — SemanticDB occurrences carry a REFERENCE role with no call role (SCIP has none either; see #69). Enrichment needs TASTy for Scala 3 (Apply/Select/Newnodes viascala3-tasty-inspector/tasty-query; no stability guarantee) and a scalac plugin for Scala 2 (TASTy has no Scala 2 equivalent). Two separate implementations — which is why this issue's "separate Scala 2 and 3 fixtures" requirement is non-negotiable.Identity: SemanticDB's overload disambiguator is still source-order-based (
+1,+2), so it churns on insert like every other producer — but unlike the others it is a specified contract with a reference implementation, andoverridden_symbolsgives an independent cross-check. See the #65 comment.Requested amendments: delete the scip-java dependency and the "share scip-java distribution work with Java/Kotlin" coordination line; adopt BSP + direct SemanticDB protobuf ingestion; record TASTy (Scala 3) and a scalac plugin (Scala 2) as the two required enrichers.
Full recon:
.wiki/bulk-index-providers/recon-csharp-kotlin-scala.md.samchon commentedon Jul 30, 2026
The body now selects typed compiler plugins for both Scala generations, with
SemanticDB as the declaration/type/annotation/override/source-hash cross-check.
Scala 2 uses a post-typer scalac plugin; Scala 3 uses a post-typer Dotty plugin;
BSP/Zinc owns target discovery and invalidation.
This refines the earlier “SemanticDB plus later TASTy/scalac enrichment” plan:
a second TASTy inspection would duplicate Scala 3 semantic work and offers no
common Scala 2 path. Calls/accesses/construction should be emitted during the
same typed compile that produces SemanticDB. scip-java remains excluded because
current upstream no longer indexes Scala.
feat: add resident Scala compiler snapshots