Repository navigation
DeleteNotInSource: fix deleted count and @folder rules for objects only in ADF (#440) - #514
Merged
Merged
Conversation
…440 - Count an object as deleted only when it was really removed; log skipped excluded objects with Write-Host #512 - Read the folder of pipelines/datasets/data flows that exist only in ADF (REST API, only when a rule uses @folder), so @folder rules match them in the delete step and the DryRun plan; a rule without @folder matches any folder, as at deploy time #513 - Remove-AdfObject takes factory/resource group from the ADF instance instead of the caller's variables Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve conflicts with #30 (Remove-AdfObjects) and #462: - Remove-AdfObject: keep develop's structure (optional adfSource, Selection/Deleted, retry, throw ADFT0018); add folder-aware exclusion check and DeletedObjectNames update only when adfSource is set - Renumber folder read error to ADFT0036 (ADFT0034/0035 are used by #30) and document ADFT0034-0036 in messages_index.md - changelog: keep entries from both sides Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Tests that need Azure are in Describe blocks tagged 'Integration' (Remove-AdfObject, Publish-AdfV2FromJson-1/-2); their Azure calls moved from the file/Describe body (run at discovery) into BeforeAll - Get-TargetEnv and new Get-TestSubscriptionPrefix no longer require an Azure context, so unit tests run without Connect-AzAccount - tools/Install-DevPrerequisites.ps1 installs Az modules, Pester 6+ and PSScriptAnalyzer for local development - CLAUDE.md: rule for Integration tests and the setup script Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #512
Closes #513
Part of #440
Problem
With
DeleteNotInSource = $trueand-FilterFilePath:DoNotDeleteExcludedObjects = $true) were added toDeletedObjectNames. The log reportedDeleted N objects from ADF service.although they were still in ADF, and the next run tried them again. The skip was only logged withWrite-Verbose.type.name, without its folder, so@folderrules never matched it. With include rules, it was silently skipped. With exclude rules such as-pipeline.*@Legacy*, it was not protected and was deleted.Changes
Remove-AdfObjectadds an object toDeletedObjectNamesonly after it has really been removed. Skipped objects are logged withWrite-Host.Remove-AdfObjectIfNotInSourceskips objects that were already deleted (as referencing objects).Get-AdfObjectFolderFromServicereads the folder of a pipeline, dataset or data flow via REST API (Az does not return pipeline folders). It is called only when a rule uses@. Temporary errors are retried; other errors stop withADFT0036.AdfObjectName.IsNameExcludeduses the same rule semantics as deploy-timeAdfObject.IsNameMatch: the folder is checked only when the rule has@folder, sopipeline.PL_Xmatches PL_X in any folder.Get-DryRunPlanuses the folder too, so-DryRunplans the same deletions.Remove-AdfObjecttakes the factory and resource group names from$adfInstanceinstead of the caller's variables, and the type from$objinstead of$adfTargetObj.Behaviour changes
@foldernow matches an object in any folder at delete time, as it already did at deploy time. One case inadfObjectName-class.Tests.ps1was updated.@foldernow protect objects that exist only in ADF.Remove-AdfObjects -Option(Add public function to delete objects from ADF instance #30):pipeline.PL_*now matches pipelines in folders too.Merge with
develop(conflicts with #30 and #462)developgot #515 (#462), #516 (#30, newRemove-AdfObjects) and #517 (#382) after this branch was created. Conflicts were inchangelog.mdandprivate/Remove-AdfObject.ps1:Remove-AdfObject.ps1: Add public function to delete objects from ADF instance #30 rewrote the function (optional$adfSource,-Selection/-Deleted, retry,throwfor ADFT0018). The merge keeps that version and adds the changes of this PR. The folder check andDeletedObjectNamesrun only when$adfSourceis set, becauseRemove-AdfObjectscalls the function without it.ADFT0034andADFT0035, so the folder read error of this PR is nowADFT0036.messages_index.mddocuments all three.changelog.md: entries from both sides are kept.Get-AdfObjectFolder(Add public function to delete objects from ADF instance #30, lists all folders, no retry) andGet-AdfObjectFolderFromService(this PR, one object, with retry) now both exist. They can be merged into one function in a follow-up.Tests that need Azure are tagged
IntegrationSome tests called Azure but were not tagged
Integration, so a run withExcludeTag = 'Integration'still neededConnect-AzAccount. Pester runs code in the file orDescribebody at discovery, before tag filters apply, so a tag alone was not enough:Remove-AdfObject.Tests.ps1: publishedadf1at file level. Now anIntegrationDescribewith the setup inBeforeAll;Should existstaysUnit.Publish-AdfV2FromJson-2.Tests.ps1:Get-AzDataFactoryV2ran at file level, and an Azure test was taggedUnit. Split and taggedIntegration, with the call inBeforeAll.Publish-AdfV2FromJson-1.Tests.ps1: aDescribethat deploys to Azure had no tag. NowIntegration.zGet-AdfFromService.Tests.ps1:Get-TargetEnvwas in theDescribebody. Now inBeforeAll.TestHelper:Get-TargetEnvthrew without an Azure context, which broke files with only unit tests. NewGet-TestSubscriptionPrefixreturns00000000then;Integration,Stop-Triggers,TriggersandTest-AdfLinkedServicetests use it (or a dummy subscription ID) instead ofGet-AzContext.CLAUDE.mdhas the rule now: tests that need Azure are taggedIntegration, and Azure calls go inBeforeAll/It.Local developer setup
New
tools/Install-DevPrerequisites.ps1installs, for the current user, the modules missing or older than needed: Az.Accounts, Az.Resources, Az.DataFactory, Az.Storage, Pester 6+, PSScriptAnalyzer. It also checks Node.js/npm.CLAUDE.mdpoints to it.Tests
test/Remove-AdfObjectIfNotInSource.Tests.ps1: 14 mocked unit tests that reproduce both bugs. All failed before the fix.Incremental-Deployment.Tests.ps1: theSet-AzDataFactoryV2mock declared$DataFactoryNameinstead of$Name, so the fake factory had no name.Integrationwere run locally (Pester 6.2.0, pwsh 7.6): 402 passed, 1 failed, with an expired Azure login and with no Azure context, under$ErrorActionPreferenceStopandContinue.Export-AdfToArmTemplate"Should completed successfully": the local npm install ofazure-data-factory-utilitiesmissescommander(Node 23 on that machine). It does not need Azure and passes on CI.🤖 Generated with Claude Code