[kabhishek1001]

Failure

Since #2024, the Lineage Builder loads its assets file and its lineage file by calling Asset Import's Importer.import. It builds both AssetImportCfgs without assetsPreviousFilePrefix, so Importer falls back to csa-asset-import and both copies are saved under csa-asset-import/<connection>/, the same folder Asset Import's own runs use.

When an Asset Import workflow on Full Replacement (assets_delta_semantic: full) and a Lineage Builder load into the same connection:

1. Asset Import saves its file as the newest copy in csa-asset-import/<connection>/.
2. The Lineage Builder runs and saves CSA_LB_assets.csv and then CSA_LB_lineage.csv into the same folder.
3. The next Full Replacement run picks the newest copy, which is the lineage file, and archives everything in it that isn't in the new file. So it archives the lineage processes, and none of the assets that dropped out of its own file.
4. The next Lineage Builder run restores the lineage.

I reproduced this on an internal tenant with Asset Import 2.7.0 and Lineage Builder 2.4.70 (both on 7.3.1). A second Full Replacement import straight after, with no Lineage Builder run in between, archived the dropped asset as expected. So Full Replacement works when the newest copy is its own.

Fix

Loader passes assetsPreviousFilePrefix = "csa-lineage-builder" in both AssetImportCfgs, the same way the Relational Assets Builder (csa-relational-assets-builder) and Cube Assets Builder (csa-cube-assets-builder) use their own prefix.

The Lineage Builder never sets a delta mode, so it always runs Incremental and only writes to this folder. It never reads from it. The connection cache update doesn't use the prefix.

Test

samples/packages/lineage-builder/src/test/kotlin/SeparatePreviousFilesTest.kt is a package test against a tenant, in the same style as the other Lineage Builder tests. On one new connection it runs:

1. Asset Import on Full Replacement with tables T1, T2 and T3.
2. The Lineage Builder with one process, T1 to T2.
3. Asset Import on Full Replacement again, with T3 dropped.

It checks that T3 is archived, T1 and T2 are still active, the lineage process is still active, and that Asset Import and the Lineage Builder each have their two saved files under their own prefix. It names the prefix as a string rather than using the new constant, so it also compiles without the fix. The test is in its own commit (66d78d8), before the fix (a010ad1).

What ran

• pr-build (assemble, shadowJar, spotlessCheck) and pr-test (testClasses test) passed.
• The Test workflow on this branch (run 42706040361) couldn't run the new test. Creating its connection returned ATLAN-JAVA-500-000 Server responded with an error -- 504. The two existing Lineage Builder tests that use the tenant, PartialAssetsTest and CustomConnectionTest, failed setup with the same 504, as they did in last night's nightly run on main.
• So I ran the new test with TestNG against an internal tenant instead, from CI's own builds. A local Gradle build can't configure, because the managed Gradle repository firewall doesn't serve the kotlin-dsl 5.2.0 plugin that buildSrc needs.
  • With the fix: the classes and jars in the build-artifacts of run 42706040361 (this branch, a010ad1).
  • Without the fix: the same, with only the Lineage Builder's compiled classes taken from last night's nightly build of main (run 42560186102, b44068a). Loader.class is the only Lineage Builder class that differs between the two builds.
  • Third-party libraries at the versions in gradle/libs.versions.toml, on Java 17.

Check	Without the fix	With the fix
droppedTableArchived	Fail: no archived T3 found	Pass
keptTablesStillActive	Fail: expected [T1, T2], actual [T1, T2, T3]	Pass
lineageStillActive	Fail: no active lineage process found	Pass
eachPackageSavesToItsOwnPrefix	Fail: expected 2 files under csa-asset-import, actual 4	Pass

Without the fix, the Lineage Builder uploaded CSA_LB_assets.csv and CSA_LB_lineage.csv to csa-asset-import/<connection>/, and the second import logged previous file: <the CSA_LB_lineage.csv copy> before Deleting (SOFT) 1 assets. With the fix, both went to csa-lineage-builder/<connection>/, and the second import's previous file was the first import's own copy. Both runs purged their test connection afterwards. The package test will still need a green run in the Test workflow once the CI tenant stops returning 504s.

Rollout note

After upgrading, the first Full Replacement import on a connection will still find the last Lineage Builder copy in csa-asset-import/<connection>/ and compare against it. Removals start from the run after that, unless those leftover copies are deleted first.