Repository navigation
[Bulk index][Java] Export compiler and JDT workspace snapshots without query fanout #74
Description
Outcome
Export Java graph shards from the compiler invocation that already type-checks
the project, using the public javac plugin/tree/model APIs after attribution.
Preserve Gradle/Maven incremental state and use a JDT workspace route only when
the resident IDE model is the semantic owner. Do not run clean, query JDT once
per declaration, or type-check the same source a second time for enrichment.
Current master baseline
Current master registers scip-java for Java and Kotlin. For Java it publishes
semantic-index authority with contains and references. The PR #147 Gson
cold cell took 88.653 s, compared with 231.398 s through generic JDT LS
fan-out.
scip-java 0.13.x changed its architecture:
- the project moved to
scip-code/scip-java; - javac/kotlinc plugins emit SCIP directly rather than SemanticDB;
- the supported floor is Java 17 and Gradle 8;
- Scala/Zinc support was removed; and
- the default Gradle/Maven path performs a real build and includes
clean.
Its artifact remains a useful navigation fallback, but SCIP has no call role.
Imports/accesses/implementation relationships are producer-specific, and the
order-based overload/constructor discriminator churns when declarations are
inserted or reordered.
Upstream mechanism
The public JDK compiler API is enough for the build-integrated route:
Plugin
is loaded byServiceLoaderon the processor path and receives the active
JavacTask;- a plugin registers a
TaskListener
for parse/enter/analyze/generate/compilation events; and Trees
maps attributedTreePaths toElement,TypeMirror, source positions,
scopes and declaration trees.
Elements/Types provide direct supertypes, member lookup and override checks.
No private javac AST parser is required for the initial strict provider.
Annotation-processor/generated source handling still needs lifecycle tests.
Decision: two semantic-owner lanes, one protocol
Build lane: javac plugin
Attach a versioned graph plugin to the project's ordinary Java compile tasks.
On ANALYZE completion for a compilation unit, scan its attributed tree once
and write a source shard. At final compilation completion, write the ordered
manifest/commit record only if diagnostics and the build universe satisfy the
configured policy.
Integrate through Gradle/Maven without replacing their task graph:
- Gradle: add the plugin/processor path and
-Xplugin:option to each actual
Java source-set compile task; retain daemon, build cache, configuration cache
and incremental compilation; - Maven: attach to the existing compiler execution and target directory; do not
prependclean; and - Bazel/other javac hosts: attach at the toolchain/action boundary only when the
exact target/action inputs are nameable.
If a task is UP-TO-DATE, load the previously committed shards. Deleted source,
classpath, compiler-option, annotation-processor or generated-source changes
must invalidate the affected manifest even when no plugin invocation occurs.
Resident lane: JDT
When an Eclipse/JDT workspace already owns unsaved/resident source, expose a
narrow bulk snapshot from that workspace and its reconciled compilation units.
It must produce the same graph schema and identities as the javac route and
must freeze one workspace generation. It is a fallback/IDE lane, not a second
pass after javac.
Fact extraction
Walk attributed trees and elements:
- declarations, visibility, nesting, packages/modules, imports/exports;
MethodInvocationTreeand member references to resolved executable elements;NewClassTree/constructor elements for instantiation;- identifiers/member selects plus assignment/update context for reads/writes;
TypeMirrorand tree paths for type references;- direct class/interface supertypes and
Elements.overridesfor inheritance and
overrides; - annotations with resolved annotation type and annotated target;
- diagnostics from the same compilation task; and
- test facts only through a named JUnit/TestNG enricher with framework
provenance.
Virtual/interface calls identify the statically selected method. Runtime
implementations remain candidate/unresolved unless final/sealed analysis proves
one exact target.
Identity and universe
Use module/artifact coordinate plus binary owner name, item kind and a canonical
structural signature:
- methods/constructors include erased or fully qualified parameter types,
generic arity and return/type-parameter context as required; - fields, record components, enum constants and annotation elements remain
distinct; - locals include source file, enclosing stable member and lexical structure;
- anonymous/local classes include a structural owner key that does not change
on unrelated line insertion; and - source/bridge/generated elements retain origin and generator/processor
provenance.
Do not trust scip-java's +N overload ordinal as the final ID. Exact fixtures
must insert/reorder overloads and constructors without renaming unaffected
members.
The universe includes module/source set, Java release/preview flags, classpath
and module path order/content, compiler/JDK version, annotation processors and
options, generated sources, Gradle/Maven target coordinates and relevant build
properties. One multi-module build can have several target universes.
Incremental and correctness behavior
- no-op: validate existing manifest; compiler need not run;
- private body edit: ordinary incremental compile writes only affected source
shards; - public ABI edit: build tool/compiler invalidates dependents, and the graph
records the affected target/source closure; - annotation processor or classpath change: invalidate generated/consumer
shards according to the build; - compiler error: publish diagnostics and either a clearly partial generation
or retain the prior strict generation according to the common policy—never
silently combine old and new facts; - delete/rename: explicit shard deletion; and
- cancellation/crash: no commit.
Cold acceptance is relative to the same build without the graph plugin. Target
overhead is at most 25% plus serialization, with no extra compilation.
Implementation map
- Implement the common shard and coverage schema from [Bulk index] Replace symbol-by-symbol LSP scans with compiler-owned snapshots #63.
- Create a separately versioned Java exporter artifact and Gradle/Maven fixture
integration; do not put a JVM inside the TypeScript package. - Add a provider beside the existing scip-java entry and make it preferred
only after a handshake proves target coverage. - Normalize javac/JDT identities through one conformance fixture.
- Retain scip-java as a compatibility/navigation fallback.
- Add phase-separated real Gson measurement and lifecycle rows.
Acceptance
- One compile traversal emits declarations, calls, construction,
accesses, types, direct inheritance, overrides, annotations, diagnostics
and source evidence. - No
clean, no disabled Gradle daemon/configuration cache, and no second
compiler pass. - Multi-module Gradle and Maven fixtures cover main/test source sets,
JPMS, records, sealed types, overloads, constructors, generics, lambdas,
anonymous/local classes, annotation processing and generated sources. - Overload insertion/reorder and multiple constructors keep stable IDs.
- All 15 fact families have explicit coverage/unresolved rows.
- No-op, body/API edit, create/delete/rename, classpath/config/processor
change, error, cancel, crash and retry are atomic. - JDT and javac lanes agree on shared source facts and report different
producer/universe provenance. - Cold overhead is measured against the native compile; warm no-op and edit
latency are reported separately. - scip-java and generic JDT LS remain truthful fallbacks.
- Focused tests, Java experiment,
pnpm build,pnpm test, and
pnpm coveragepass.
Activity
samchon commentedon Jul 17, 2026
Provider recon at graph
71e35af. This issue's premise is stale in two places, and one of its coordination notes now breaks a sibling issue.scip-java v0.13.0 is a breaking release
sourcegraph/scip-javamoved toscip-code/scip-java. Release v0.13.0 (2026-07-02), verbatim:Corrections:
index-semanticdbsubcommand is gone; the CLI is nowindex,aggregate,snapshot.SbtBuildTool.scala,MillBuildTool.scala,ScalaVersion.scala). This breaks [Bulk index][Scala] Consume SemanticDB and TASTy snapshots through BSP #77's assumed dependency — see the correction there; Scala should move to BSP + direct SemanticDB ingestion instead. The shared-plumbing note now holds for Java + Kotlin only.ghcr.io/scip-code/scip-java) or coursier only.Also: Java 8/11 and Gradle < 8 are no longer supported; JDK 17+ required (17/21/25), with
--add-exports jdk.compiler/com.sun.tools.javac.{model,api,tree,util,code}=ALL-UNNAMED.scip-java indexruns a real build, not an analysis passDefaults: Gradle
clean compileTestJava compileTestKotlin compileTestKotlinJvm; Maven--batch-mode clean verify -DskipTests. It runs the build — network dependency resolution, annotation processors, codegen — with a compiler plugin attached. This issue's "prepare dependencies without modifying tracked files" must contend with it writing totarget//build/.Build tool matrix: Maven ✅ Java, Gradle ✅ Java, Bazel ✅ Java, Ant ❌, Buck ❌. v0.13.1 fixed "duplicate documents in Bazel index aggregation" — a caution for the Bazel path.
The identity finding (feeds #65)
The overload disambiguator is order-of-appearance, not signature-derived. Per the SemanticDB spec whose grammar scip-java still follows:
with the Java-specific ordering:
Consequences:
static↔ non-static renumbers both partitions.<init>) get ordinal disambiguators too — an explicit acceptance row on [Bulk index] Introduce overload-safe provider-native symbol identities #65.Recommendation: derive a signature hash from
SymbolInformation.signaturerather than trusting+N. Note the campaign-wide asymmetry — scip-clang's disambiguator is a canonical-type hash (order-independent) while scip-java's is an ordinal; #65 cannot assume "provider-native key = stable" uniformly.Locals:
local <id>is "unique within the underlying document, but they don't have to be unique across multiple documents", and the suffix is "implementation-dependent" → must be paired withDocument.relative_pathand marked generation-scoped.Multi-module: SCIP
<package>=<manager> <package-name> <version>, so modules stay distinct without name merging. This issue's multi-module acceptance row is satisfied by the format.What the artifact proves
Java is the strongest SCIP story in the campaign, but still bounded by the schema.
SymbolRolehas noCallrole at all (see the #69 comment), so:contains(descriptor chain),references.imports(Importrole + occurrences on import statements — the best-supported edge beyond references);accesses(ReadAccess/WriteAccessexist in the schema — verify the plugin sets them);extends/implements/overrides(is_implementationrelationships; Java can be disentangled by target kindInterfacevsClassand method vs type — strictly better separable than C++);tests(Testrole exists and the Gradle default compilescompileTestJava, so test sources are indexed — verify the plugin setsTestper test source root, since this issue's Gson smoke requires non-zerotests).instantiates—new Foo()yields an occurrence on the constructor symbol withKind=Constructor. Still an inference from target kind rather than a role, but a semantic inference, not punctuation-guessing, so it survives [Bulk index] Add strict SCIP navigation-skeleton ingestion and enrichment gates #69's bar. Verify with a fixture.calls,type_refas such,dispatches,decorates(an annotation use is just a reference to the annotation type — needs enrichment to mark it asdecoratesand attach the target; annotations are an acceptance row here),exports(only JPMSmodule-infoexports anything).Required sidecar: extend the javac plugin
org.scip-code:scip-javac— do not fork JDT — walkingMethodInvocationTree/NewClassTree/MemberSelectTree/AnnotationTree, emitting facts keyed by the same SCIP symbol strings. Cheap relative to C++: it rides the compile that is already happening.Incremental
Cold-only at milestone 1, as this issue already states. Per-source shards exist but shard production is not dependency invalidation; any classpath/compiler-option/AP/module change forces full. Gradle's own incremental compile could in principle be leveraged (the plugin rides javac), but the
cleanin both default commands explicitly defeats it — worth overriding viascip-java index -- <custom>at milestone 2.Full recon:
.wiki/bulk-index-providers/recon-c-cpp-java.md.Serve Java from the compiler that already type-checked it
samchon commentedon Aug 26, 2026
Java consumer: written, fenced, and blocked on one producer staleness
Campaign chronology, from PR #164's cycle.
Consumer state. The route is implemented against the merged producer (samchon/scip-java
32eca21) and registered asjavac-graph: pinned handshake on schema/protocol/producer identity, committed per-target generations, target-scoped identity, shard-level deltas against the previous generation, all fifteen coverage rows, machine-readable unresolved facts, and an honest decline toscip-javawhen the installed launcher'sindex --helpdoes not publish--graph-output.Splitting
scip-javawas a precondition rather than a preference: a fallback owns the same atomic languages as the route it backs, so a Java-only strict route could take that entry as its fallback only by taking Kotlin's ownership with it. The lanes are now named for their producers —scip-java's javac plugin keeps the name, and the Kotlin facts come from the kotlinc plugin the same launcher ships, which is also what lets the Kotlin row state its own compiler instead of leaving the field empty under the Java name.Two contract findings worth carrying forward:
partialcoverage row to carry at least one unresolved site — "some sites are unproven" with no list reads exactly like "every site is proven". The aggregator marks six families partial as a property of the exporter rather than of any call site, so the consumer publishes aprovider-gaprow at the target's own coordinate for each partial family with no located site.hello.compilerVersionmust be non-empty, and a Gradle toolchain per source set can legitimately run two JDKs in one build. The generation publishes both, joined, rather than presenting the first shard's reading as the build's.Where it is blocked. The real lane reaches a real artifact and is refused at the coordinator's input fence, on the lifecycle's edit step:
Those two digests are identified.
b82fc2f4is the pristinefixtures/maven/basic/src/main/java/com/Example.java;05a4731fis the same file with the lifecycle's edit appended. The coordinator hashed the edited file before and after the refresh and got the same answer both times, so the checkout was stable — and the producer published a shard whosediskDigestdescribes the file as it was before the edit.So on the edit step the aggregate carries a committed generation that no longer describes the checkout, and
GraphAggregateRunner.parseShard's own freshness check — which re-reads each source and compares it to the shard'sdiskDigest— did not catch it. Whether the build was UP-TO-DATE and re-published a stale commit, or the store returned a generation the compile never refreshed, is the producer's to say; from here it is visible only as a digest that describes bytes that no longer exist.The consumer's behaviour is what it should be: it refuses to publish facts bound to bytes no reader can hash, and it now names which of the four ways a source comes unbound this was. It also requires a non-empty disk digest at the adapter, so a producer that never hashed a source is declined where the producer can be named rather than three refreshes later.
Where this leaves the issue. #74 stays open. The consumer half is written, its fixtures pass, and it publishes through
buildGraphDump— the coordinator fence included — against a fake launcher. The route cannot be called serving until the real producer's committed generation survives an ordinary edit.samchon commentedon Aug 27, 2026
Correction: the fence failure was the consumer's, and it is fixed
The previous comment attributed the edit-step fence failure to producer staleness. That was wrong, and the digests it quoted are what disprove it.
b82fc2f4is the pristineExample.java;05a4731fis that file with the lifecycle's edit appended; the coordinator read the edited file both before and after the refresh. So the checkout was stable and the producer had simply never been asked to rebuild — which means nothing stale was ever published.The cause is in the consumer's session wiring. A registry entry's
buildInputsis deliberately the build files alone: the files outside a provider's own language whose change invalidates it. A session'sinputsis the fingerprint that decides whether the build has to run at all, and it must cover everything the producer compiled. Rust keeps the two apart by name —rustInputswalks the sources,rustBuildInputsdoes not — and the Java route used the second where the first belongs. A.javaedit moved nothing, the session reused the snapshot taken before it, and the coordinator refused a generation describing bytes the file no longer had.Refused correctly, which is the part worth keeping: the input fence caught a route that would otherwise have served facts one edit out of date, and said so with digests precise enough to identify both files.
The fixture is what hid it. Every lifecycle case passed its own list of sources as
inputs, so the provider's real wiring was never under test. The selection case now edits a source and republishes throughbuildGraphDump, exercising the provider's own inputs and the coordinator fence together.GraphAggregateRunnerandMavenGraphGenerationStoreare not implicated; the retraction is filed at samchon/scip-java#1.#74 stays open until the real lane publishes a generation on CI, but the remaining risk is now an ordinary integration one rather than a suspected upstream defect.
samchon commentedon Aug 27, 2026
What the Java route now proves, and what it does not
The
javac-graphroute is implemented and its real-corpus lane is green in CI on358bbbb. Recording the evidence against this issue's own acceptance list rather than against the general impression of progress, because most of the list is still open.Supported now.
clean, no second compiler pass, no disabled daemon.partialwith no located site publishes aprovider-gaprow at the target's own coordinate — so "some sites are unproven" cannot read as "every site is proven".scip-javaandscip-kotlinc, because a fallback must own the same atomic languages as the route it backs. A launcher whoseindex --helpdoes not publish--graph-outputdeclines to that lane rather than running a whole build and finding nothing to read.pnpm build,pnpm test,pnpm coverage(100% line/function/branch) and the Java experiment lane all pass.Still open, and each for its own reason.
scip-javafallback evidence, labelled as such.One declared limitation, filed upstream. Restoring the original sources returns identical facts — five nodes and eight edges both times — under a different build universe, because
ScipOptionBuilder.writeGraphUniversedigests the raw javac invocation and that invocation names the per-run temporary directory the launcher unpacks its plugin jar into. It is declared in the catalog as aregenerationLimitationnaming the mechanism, so the lane proves everything else it owes rather than staying red on it, and it is filed at samchon/scip-java#1. The declaration comes off when that does.This issue stays open.
Link the repository by the name it actually has
samchon commentedon Aug 27, 2026
The Java lane ran, and here is what it did and did not prove
java LSPis green at run 33030412174 on358bbbb, against the producer merged at32eca21. The PR body had this lane down as "CI's to confirm" because the development checkout has no JVM. It is confirmed. The result artifact, in full:Provenance.
javac-graph, authoritycompiler, producerscip-java-javac-graph0.0.0-SNAPSHOTon javac21.0.12, schema 1, protocol 1. Capabilitiescoverage,diskDigests,incremental,sourceDigests,universe,unresolved.Lifecycle, every phase separately timed:
The cross-file claim the PR flagged as unverified.
lifecycleCreatedEdgeisinstantiates,samchonGraphExperiment→Example,crossFile: true. A new file's constructor call resolves to a type declared in another file, through attributed trees, in one compile.Identity is target-scoped as specified:
@v2/java/1aaf2fb4…#com.Example:classand@v2/java/43962d7a…#com.ExampleSuite.ExampleSuite:constructorcarry different target hashes for main and test source sets.One honest negative: regeneration is
limited, and it is upstream'sThe facts reproduce exactly — same node and edge counts, and
retryContentequals theeditphase's content digest. What does not reproduce is the universe, because the digest includes/tmp/scip-java<random>/bin/javac. That is a producer defect, not a consumer one, and it is filed on samchon/scip-java#1. The lane reports it as a named limitation rather than passing quietly or failing, which is the behaviour this issue asked for.Against the acceptance list
Proven by this run and the gate:
pnpm build,pnpm test,pnpm coveragepass. The lane is green and the gate is 100% line/function/branch.pnpm provider-supportagainstGRAPH_PROVIDERSand the experiment catalog; the provider declares 14 producible families andrenderscarries anunsupportedrow.scip-javais split into a Java-only lane andscip-kotlincso a fallback owns the same atomic languages as the route it backs.Partially proven, and I would rather say so than tick them:
mvn verifybaseline it should be compared against is not.Not proven, and each needs work this cycle did not do:
contains,exportsandinstantiatesonly.So the route is real, atomic and target-scoped, and the compiler-owned producer works end to end. The semantic breadth this issue specifies is still ahead of it, and #74 should stay open for that rather than close on a green lane.
samchon commentedon Sep 1, 2026
Not closed by #164 (
1bf9486).JavaGraphSnapshotAdapterlanded and thejava LSPexperiment is green.Still unmet from this issue's Acceptance list: there is no Maven fixture in the repository at all (no
pom.xmlundertests/), while the acceptance requires multi-module Gradle and Maven fixtures covering main/test source sets, JPMS, records, sealed types, overloads, constructors, generics, lambdas, anonymous/local classes, annotation processing and generated sources. The JDT/javac lane agreement on shared source facts and the cold-overhead-versus-native-compile measurement are also unreported.Leaving this open.
samchon commentedon Sep 1, 2026
Current-cycle revalidation separates the passing producer fixture from the failing Gson corpus:
instantiatesedge. Those observations exceed the shared no-op/edit targets and are not a native-build comparison.scip-java. The retained error contains only benignJAVA_TOOL_OPTIONSnotices because the shared batch session discards stdout whenever stderr is non-empty; [Provider diagnostics] Preserve both process streams on batch failure #169 will preserve both streams so the next run exposes the Maven cause.This issue remains open.
Close #74: [Bulk index][Java] Export compiler and JDT workspace snaps…
Advance #74: prove Java compiler acceptance and native baselines
Advance #74: prove Java compiler acceptance and native baselines
Advance #74: prove Java compiler acceptance and native baselines
Advance #74: add pinned resident JDT snapshot agreement
1 remaining item
Advance #74: prove JDT retry cursors remain transactional
Advance #74: make JDT retry coverage deterministic
Advance #74: pin the complete JDT source archive
Finish #74: pin verified JDT producer tree
Finish #74: execute producer pin regressions
Finish #74: pin deterministic JDT fixtures
Finish #74: pin binding-resolving JDT producer
Expose Java compiler universe drift
Fence Java universe evidence links
Name immutable Java universe differences
Keep long Java universe differences visible
Preserve Unicode in Java universe diagnostics
Compare Java compiler evidence by invocation
Prioritize Java invocation evidence
Close #74: Export Java compiler snapshots