Skip to content

[Bulk index][Scala] Consume SemanticDB and TASTy snapshots through BSP #77

Open

Description

@samchon

Outcome

Implement separate Scala 2 and Scala 3 typed compiler exporters, driven by the
repository's BSP/Zinc build. Use direct SemanticDB ingestion for declarations,
types, annotations, overrides, diagnostics and source hashes, but collect
calls/accesses/construction from the same typed compiler pass rather than
starting a second TASTy analysis.

Current master baseline

No strict Scala provider is registered on current master. scip-java 0.13
removed Scala and Zinc support; the current registry correctly limits it to
Java and Kotlin. Scala therefore uses the ordinary language-server/static lane,
and PR #147 has no strict index-time cell to cite.

Do not resurrect the old scip-java Scala route. The last Scala-capable release
is a historical fallback and would pin the product to a removed pipeline.

What SemanticDB provides—and does not

The SemanticDB specification
is a strong per-source artifact:

  • document text/md5, URI, language and build target;
  • symbol kinds/properties, structured signatures, annotations and access;
  • occurrences, diagnostics and synthetics;
  • direct parent declarations and overridden symbols.

Direct protobuf ingestion retains more information than converting to SCIP.
It can prove declarations, containment, types, annotations, direct
extends/implements, overrides, source hashes and diagnostics.

Occurrences do not carry a call/read/write role. SemanticDB alone cannot prove
calls, accesses or instantiates. TASTy can fill Scala 3 after compilation,
but a separate TASTy pass duplicates semantic work and has no Scala 2
equivalent. The fastest common architecture is therefore a typed compiler
plugin that emits those facts during the compilation already producing
SemanticDB.

Decision: two typed plugins, one BSP/Zinc transaction

Scala 2

Build a scalac Global plugin with a phase immediately after typer and before
lowerings erase source-level intent. Walk typed trees once, resolve symbols and
types, and write per-source shards.

Scala 3

Build a Dotty StandardPlugin/PluginPhase after typer, following the official
Scala 3 compiler plugin model.
Walk typed trees before later transformations obscure givens, extension
methods, inline/macro origins and source spans.

Build orchestration

Use BSP to discover exact build targets, sources, scalac options, class
directories and dependency graph, then request compilation. Let Zinc/sbt/Bloop/
Mill determine affected targets and source recompilation. The exporter writes
changed shards into a target-scoped output and an atomic target manifest.

Enable SemanticDB in the same compile as a cross-check and source/diagnostic
input:

  • Scala 3: compiler -Xsemanticdb;
  • Scala 2: matching semanticdb-scalac, source root/range/text/diagnostic
    options; and
  • never run clean merely to force files to re-emit. Existing validated shards
    cover sources Zinc decides are unchanged.

Facts and source semantics

The typed plugins emit:

  • packages, classes, traits, objects, enums, opaque/type aliases, methods,
    constructors, fields, parameters, locals, extension methods, givens and
    generated definitions with origin;
  • imports, exports and visibility;
  • resolved Apply, Select, New, eta-expansion and extension calls;
  • read/write accesses and type references;
  • direct parents, trait implementations and overridden symbols;
  • annotations;
  • implicit/given resolution and inline/macro expansion as source-linked
    evidence/unresolved facts; and
  • compiler diagnostics from the same target generation.

Do not invent an explicit source span for a synthetic implicit/given insertion.
Use SemanticDB synthetics and compiler origin trees to link the source site and
selected symbol.

Framework tests use a named ScalaTest/MUnit/specs2 enricher. renders is
unsupported unless a framework-specific producer is added.

Identity and target universes

SemanticDB symbols are useful interoperability keys but overload disambiguators
are source-order based. The final graph identity must use:

  • BSP build target and module/artifact coordinate;
  • fully qualified owner and symbol kind;
  • structural signature, including receiver, parameter/type parameter lists and
    result where needed;
  • Scala language/compiler major and relevant encoding; and
  • source owner/lexical structure for locals and anonymous functions.

Exact fixtures insert/reorder overloads and constructors and compare Scala 2/3
encodings. Same-named symbols in main/test/cross-built targets must not collide.

The universe includes BSP target, Scala compiler and plugin version, platform
and Scala binary version, scalac options, classpath order/content, source roots,
SemanticDB options, compiler plugins, macro/inline dependencies, Zinc analysis
and generated sources. Cross builds are separate universes.

Incremental and performance behavior

  • no-op: BSP/build-tool null compile plus manifest validation, or manifest load
    without a producer invocation when inputs prove unchanged;
  • body edit: Zinc recompiles affected sources; replace only emitted shards;
  • public signature/macro/inline edit: accept Zinc's dependent closure and record
    it;
  • target/classpath/scalac/plugin change: invalidate that target universe;
  • create/delete/rename: reconcile target source list and delete stale shards;
    and
  • failed compile/cancel/crash: retain the prior committed target generation.

Do not call SemanticDB per symbol, do not inspect TASTy in a second cold process,
and do not hold compiler tree/symbol objects after the plugin phase.
Cold overhead is measured against the identical native BSP compile. Resident
latency is bounded by the build server's null/incremental compile plus shard
merge.

Implementation map

  1. Add [Bulk index] Replace symbol-by-symbol LSP scans with compiler-owned snapshots #63's protocol and target-scoped shard store.
  2. Implement direct SemanticDB protobuf parsing and conformance fixtures.
  3. Implement the Scala 2 plugin and exact fixture.
  4. Implement the Scala 3 plugin with the same normalized schema.
  5. Add BSP discovery/compile/diagnostic integration and Zinc invalidation
    traces.
  6. Register the strict provider only when the target's compiler/plugin pair is
    compatible; otherwise decline to Metals/static.

Acceptance

  • Scala 2 and Scala 3 have separate fixtures and compatibility matrices.
  • Typed-tree extraction proves calls/accesses/construction without a second
    semantic pass.
  • SemanticDB md5/build-target/kinds/signatures/annotations/overrides/
    diagnostics are retained and cross-checked.
  • Givens/implicits, extensions, inline/macros, opaque types, enums,
    overloads, constructors, generics, locals and tests have exact fixtures.
  • Overload insertion/reorder and cross-target twins keep stable identities.
  • All 15 fact families have explicit coverage/unresolved rows.
  • No-op, body/API/macro edit, create/delete/rename, target/classpath/
    compiler-option/plugin change, diagnostic, cancel, crash and retry are
    atomic.
  • Cold overhead is measured against the native BSP compile; no-op/edit
    phases and Zinc invalidation are reported separately.
  • Metals/static remain truthful fallbacks; scip-java is not presented as
    Scala-capable.
  • Focused tests, Scala experiments, pnpm build, pnpm test, and
    pnpm coverage pass.

Primary references

Activity

  1. samchon commented on Jul 17, 2026

    @samchon
    OwnerAuthor

    Provider recon at graph 71e35af invalidates this issue's required bulk route. scip-java no longer indexes Scala at all.

    sourcegraph/scip-java now redirects to scip-code/scip-java. Release v0.13.0 (2026-07-02), verbatim:

    The project has been substantially slimmed down and modernized: the CLI is rewritten in Kotlin, the compiler plugins emit SCIP directly without the SemanticDB intermediate step... Support for Java 8/11, Gradle < 8, Scala indexing, and Zinc has been dropped.

    PR #883 "Drop support for indexing Scala code" (merged 2026-05-27) deletes SbtBuildTool.scala, MillBuildTool.scala, ScalaVersion.scala, ScalaCompilerClassLoader.scala. The index-semanticdb subcommand is gone; the CLI is now index, aggregate, snapshot. The last Scala-capable release is v0.12.3 (2026-04-02). The Mill fallback ckipp01/mill-scip is archived (last release 2023-10-06).

    This is an opportunity, not a setback

    Reading SemanticDB directly is strictly better than routing it through SCIP, and it is what this issue already asked for ("consume SemanticDB and TASTy snapshots through BSP").

    SemanticDB TextDocument carries schema, uri, text, md5, language, build_target, plus symbols, occurrences, diagnostics, synthetics. SymbolInformation carries kind (CLASS/TRAIT/OBJECT/METHOD/FIELD/…), a properties bitmask (ABSTRACT, FINAL, SEALED, IMPLICIT, LAZY, CASE, INLINE, GIVEN…), structured signature (ClassSignature parents/declarations; MethodSignature type params/param lists/return), display_name, annotations, access, and overridden_symbols.

    Stock SCIP from scip-dotnet and scip-kotlinc emits none of kind, access, annotations, md5, or diagnostics (see the #69 comment). Routing Scala through SCIP would flatten all of it away.

    And no JVM is needed to parse it. semanticdb.proto is plain protobuf; this repository is TypeScript and can decode it with the same protobuf tooling #69 already requires for scip.proto. The JVM is needed only to compile via BSP.

    Revised route

    Drop scip-java. Compile through the repo's own build to emit SemanticDB, then parse semanticdb.proto in-process.

    • Scala 3: -Xsemanticdb (in-compiler).
    • Scala 2: plugin org.scalameta:semanticdb-scalac:4.17.2 (2026-07-09) + -Yrangepos, -P:semanticdb:{sourceroot,targetroot,text:on,diagnostics:on}.
    • sbt: ThisBuild / semanticdbEnabled := true configures both. Watch sbt#8082 — semanticdbIncludeInJar must be enabled for semanticdbEnabled to survive clean.
    • Output: META-INF/semanticdb/**/*.semanticdb, one per source file, under the class directory.
    • BSP (spec 2.2.0) drives it without mutating the checkout: workspace/buildTargets (tags includes test), buildTarget/sources, buildTarget/scalacOptions (flags + classpath + classDirectory, i.e. where the SemanticDB lands), buildTarget/compile, build/publishDiagnostics. Servers: Bloop v2.1.1 (2026-07-07), sbt (built-in ≥1.4.0), Mill, bazel-bsp.

    What this changes for acceptance

    Scala becomes the only language in the campaign where extends, implements, overrides, and decorates are proven by the bulk artifact with no sidecar — ClassSignature.parents is direct (not a transitive closure like scip-dotnet), kind discriminates class from trait, overridden_symbols is first-class, and annotations is first-class.

    It is also the best incremental story in the campaign: BSP owns affected-target invalidation, Bloop/sbt are already resident daemons, and SemanticDB is per-document with an md5 — so the provider watches META-INF/semanticdb/**, replaces changed documents, deletes stale ones, and publishes a generation once buildTarget/compile succeeds. That is genuine semantic-incremental, needing no custom resident analyzer (unlike C#'s MSBuildWorkspace or Kotlin's Analysis API).

    And the best provenance: per-document md5 directly satisfies #69's stale-source rejection; build_target disambiguates same-named symbols across targets, which is exactly what this issue's "same-named symbols in target scopes do not collide" row needs.

    synthetics (Range + Tree) is the truthful home for implicit/given expansions — directly supporting step 5's "never invent an explicit source span for an implicit/given expansion". scala/scala3#13135 is closed (2025-04-29), so Scala 3 now emits synthetics.

    Residual risk

    calls and accesses are still not proven — SemanticDB occurrences carry a REFERENCE role with no call role (SCIP has none either; see #69). Enrichment needs TASTy for Scala 3 (Apply/Select/New nodes via scala3-tasty-inspector/tasty-query; no stability guarantee) and a scalac plugin for Scala 2 (TASTy has no Scala 2 equivalent). Two separate implementations — which is why this issue's "separate Scala 2 and 3 fixtures" requirement is non-negotiable.

    Identity: SemanticDB's overload disambiguator is still source-order-based (+1, +2), so it churns on insert like every other producer — but unlike the others it is a specified contract with a reference implementation, and overridden_symbols gives an independent cross-check. See the #65 comment.

    Requested amendments: delete the scip-java dependency and the "share scip-java distribution work with Java/Kotlin" coordination line; adopt BSP + direct SemanticDB protobuf ingestion; record TASTy (Scala 3) and a scalac plugin (Scala 2) as the two required enrichers.

    Full recon: .wiki/bulk-index-providers/recon-csharp-kotlin-scala.md.

  2. samchon commented on Jul 30, 2026

    @samchon
    OwnerAuthor

    The body now selects typed compiler plugins for both Scala generations, with
    SemanticDB as the declaration/type/annotation/override/source-hash cross-check.
    Scala 2 uses a post-typer scalac plugin; Scala 3 uses a post-typer Dotty plugin;
    BSP/Zinc owns target discovery and invalidation.

    This refines the earlier “SemanticDB plus later TASTy/scalac enrichment” plan:
    a second TASTy inspection would duplicate Scala 3 semantic work and offers no
    common Scala 2 path. Calls/accesses/construction should be emitted during the
    same typed compile that produces SemanticDB. scip-java remains excluded because
    current upstream no longer indexes Scala.

  3. added a commit that references this issue on Sep 4, 2026
    57369d9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions