Skip to content

DeleteNotInSource: fix deleted count and @folder rules for objects only in ADF (#440) - #514

Merged
NowinskiK merged 3 commits into
developfrom
issue-440-delete-not-in-source
Oct 17, 2026
Merged

NowinskiK merged 3 commits into
developfrom
issue-440-delete-not-in-source

Conversation

@NowinskiK

@NowinskiK NowinskiK commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Closes #512
Closes #513
Part of #440

Problem

With DeleteNotInSource = $true and -FilterFilePath:

  • Objects skipped as excluded (DoNotDeleteExcludedObjects = $true) were added to DeletedObjectNames. The log reported Deleted N objects from ADF service. although they were still in ADF, and the next run tried them again. The skip was only logged with Write-Verbose.
  • An object that exists only in ADF was checked against the rules as type.name, without its folder, so @folder rules never matched it. With include rules, it was silently skipped. With exclude rules such as -pipeline.*@Legacy*, it was not protected and was deleted.

Changes

  • Remove-AdfObject adds an object to DeletedObjectNames only after it has really been removed. Skipped objects are logged with Write-Host. Remove-AdfObjectIfNotInSource skips objects that were already deleted (as referencing objects).
  • New Get-AdfObjectFolderFromService reads the folder of a pipeline, dataset or data flow via REST API (Az does not return pipeline folders). It is called only when a rule uses @. Temporary errors are retried; other errors stop with ADFT0036.
  • AdfObjectName.IsNameExcluded uses the same rule semantics as deploy-time AdfObject.IsNameMatch: the folder is checked only when the rule has @folder, so pipeline.PL_X matches PL_X in any folder.
  • Get-DryRunPlan uses the folder too, so -DryRun plans the same deletions.
  • Remove-AdfObject takes the factory and resource group names from $adfInstance instead of the caller's variables, and the type from $obj instead of $adfTargetObj.

Behaviour changes

  • A rule without @folder now matches an object in any folder at delete time, as it already did at deploy time. One case in adfObjectName-class.Tests.ps1 was updated.
  • Exclude rules with @folder now protect objects that exist only in ADF.
  • This also applies to Remove-AdfObjects -Option (Add public function to delete objects from ADF instance #30): pipeline.PL_* now matches pipelines in folders too.

Merge with develop (conflicts with #30 and #462)

develop got #515 (#462), #516 (#30, new Remove-AdfObjects) and #517 (#382) after this branch was created. Conflicts were in changelog.md and private/Remove-AdfObject.ps1:

Tests that need Azure are tagged Integration

Some tests called Azure but were not tagged Integration, so a run with ExcludeTag = 'Integration' still needed Connect-AzAccount. Pester runs code in the file or Describe body at discovery, before tag filters apply, so a tag alone was not enough:

  • Remove-AdfObject.Tests.ps1: published adf1 at file level. Now an Integration Describe with the setup in BeforeAll; Should exist stays Unit.
  • Publish-AdfV2FromJson-2.Tests.ps1: Get-AzDataFactoryV2 ran at file level, and an Azure test was tagged Unit. Split and tagged Integration, with the call in BeforeAll.
  • Publish-AdfV2FromJson-1.Tests.ps1: a Describe that deploys to Azure had no tag. Now Integration.
  • zGet-AdfFromService.Tests.ps1: Get-TargetEnv was in the Describe body. Now in BeforeAll.
  • TestHelper: Get-TargetEnv threw without an Azure context, which broke files with only unit tests. New Get-TestSubscriptionPrefix returns 00000000 then; Integration, Stop-Triggers, Triggers and Test-AdfLinkedService tests use it (or a dummy subscription ID) instead of Get-AzContext.
  • CLAUDE.md has the rule now: tests that need Azure are tagged Integration, and Azure calls go in BeforeAll/It.

Local developer setup

New tools/Install-DevPrerequisites.ps1 installs, for the current user, the modules missing or older than needed: Az.Accounts, Az.Resources, Az.DataFactory, Az.Storage, Pester 6+, PSScriptAnalyzer. It also checks Node.js/npm. CLAUDE.md points to it.

Tests

  • New test/Remove-AdfObjectIfNotInSource.Tests.ps1: 14 mocked unit tests that reproduce both bugs. All failed before the fix.
  • Incremental-Deployment.Tests.ps1: the Set-AzDataFactoryV2 mock declared $DataFactoryName instead of $Name, so the fake factory had no name.
  • After the merge, all tests except Integration were run locally (Pester 6.2.0, pwsh 7.6): 402 passed, 1 failed, with an expired Azure login and with no Azure context, under $ErrorActionPreference Stop and Continue.
  • The 1 failure is Export-AdfToArmTemplate "Should completed successfully": the local npm install of azure-data-factory-utilities misses commander (Node 23 on that machine). It does not need Azure and passes on CI.

🤖 Generated with Claude Code

Kamil Nowinski and others added 3 commits October 16, 2026 23:49
…440

- Count an object as deleted only when it was really removed; log skipped
  excluded objects with Write-Host #512
- Read the folder of pipelines/datasets/data flows that exist only in ADF
  (REST API, only when a rule uses @folder), so @folder rules match them in
  the delete step and the DryRun plan; a rule without @folder matches any
  folder, as at deploy time #513
- Remove-AdfObject takes factory/resource group from the ADF instance
  instead of the caller's variables

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve conflicts with #30 (Remove-AdfObjects) and #462:
- Remove-AdfObject: keep develop's structure (optional adfSource, Selection/Deleted,
  retry, throw ADFT0018); add folder-aware exclusion check and DeletedObjectNames
  update only when adfSource is set
- Renumber folder read error to ADFT0036 (ADFT0034/0035 are used by #30) and
  document ADFT0034-0036 in messages_index.md
- changelog: keep entries from both sides

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Tests that need Azure are in Describe blocks tagged 'Integration'
  (Remove-AdfObject, Publish-AdfV2FromJson-1/-2); their Azure calls moved
  from the file/Describe body (run at discovery) into BeforeAll
- Get-TargetEnv and new Get-TestSubscriptionPrefix no longer require an
  Azure context, so unit tests run without Connect-AzAccount
- tools/Install-DevPrerequisites.ps1 installs Az modules, Pester 6+ and
  PSScriptAnalyzer for local development
- CLAUDE.md: rule for Integration tests and the setup script

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@NowinskiK
NowinskiK merged commit 2de4219 into develop Oct 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant