diff --git a/README.md b/README.md index 1f3a266..f671426 100644 --- a/README.md +++ b/README.md @@ -90,6 +90,7 @@ The main advantage of the module is the ability to publish all the Azure Data Fa - [DoNotStopStartExcludedTriggers](#donotstopstartexcludedtriggers) - [Excluded (collection)](#excluded-collection) - [DoNotDeleteExcludedObjects](#donotdeleteexcludedobjects) +- [Delete selected objects from ADF](#delete-selected-objects-from-adf) - [Build/Test Azure Data Factory code](#buildtest-azure-data-factory-code) - [Test connection of Linked Service (preview)](#test-connection-of-linked-service-preview) - [Test connection with Service Principal (SPN)](#test-connection-with-service-principal-spn) @@ -706,6 +707,8 @@ Furthermore, you can better control which objects you want to omit from removing > The step might be skipped when `DeleteNotInSource = false` in *Publish Options* +> Objects that are still in the source (also the excluded ones) are never deleted by this step. To delete chosen objects explicitly, use [Remove-AdfObjects](#delete-selected-objects-from-adf). + ## Step: Restarting triggers 💬 In log you'll see line: `STEP: Starting triggers...` @@ -796,6 +799,29 @@ Otherwise, nothing will happen. This option gives you a flexibility of deleting objects in the target, but still not touching objects from 'other' group. +# Delete selected objects from ADF + +`Remove-AdfObjects` deletes chosen objects from ADF service, regardless of what is in the source code. +Select objects with name patterns in the same format as `Includes`/`Excludes` (`type.name[@folder]`, see [Pattern (WildCard)](#pattern-wildcard)), or pass a *Publish Option* to delete the objects which a deployment with this option would deploy ("reverse deployment"). + +```powershell +# Delete selected pipelines and all datasets +Remove-AdfObjects -ResourceGroupName 'rg-devops-factory' -DataFactoryName 'SQLPlayerDemo' -Pattern 'pipeline.PL_Old*', 'dataset.*' + +# Delete all pipelines from folder 'Archive' and its subfolders - show only what would be deleted +Remove-AdfObjects -ResourceGroupName 'rg-devops-factory' -DataFactoryName 'SQLPlayerDemo' -Pattern 'pipeline.*@Archive*' -WhatIf + +# Reverse deployment: delete objects selected by rules file +$opt = New-AdfPublishOption -FilePath ".\deployment\rules.csv" +Remove-AdfObjects -ResourceGroupName 'rg-devops-factory' -DataFactoryName 'SQLPlayerDemo' -Option $opt +``` + +* Objects are deleted in order: triggers (started ones are stopped first), pipelines, data flows, datasets, linked services, credentials, integration runtimes. +* When a selected object is referenced by another object, the referencing object is deleted first only if it is selected too. Otherwise the cmdlet stops with an error and nothing else is deleted. +* Only pipelines, datasets and data flows have folders. +* When the *Publish Option* has no `Includes`, all objects except those in `Excludes` are deleted. Use `-WhatIf` to check first. +* Any failed deletion stops the cmdlet. It returns names (`type.name`) of deleted objects. + # Build/Test Azure Data Factory code Another very helpful function is `Test-AdfCode`. Use it to validate the code of your Azure Data Factory before you publish it onto target ADF service. diff --git a/azure.datafactory.tools.psd1 b/azure.datafactory.tools.psd1 index 23b9073..f04caba 100644 --- a/azure.datafactory.tools.psd1 +++ b/azure.datafactory.tools.psd1 @@ -93,6 +93,7 @@ , 'Export-AdfToArmTemplate' , 'Test-AdfArmTemplate' , 'Set-AdfToolsAuthToken' + , 'Remove-AdfObjects' ) # Cmdlets to export from this module, for best performance, do not use wildcards and do not delete the entry, use an empty array if there are no cmdlets to export. diff --git a/changelog.md b/changelog.md index d769218..2f2bf2c 100644 --- a/changelog.md +++ b/changelog.md @@ -8,7 +8,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ### Added * `Test-AdfCode` warns about enabled triggers without pipeline references, which ADF refuses to start #507 * Deployment of an object is retried after a temporary API error (time-out, throttling, server error); new publish option `MaxRetries` (default 3, 0 = off) #295 +* New cmdlet `Remove-AdfObjects` deletes chosen objects from ADF service by name/folder pattern or by publish option #30 ### Fixed +* Deleting objects not in source retries temporary API errors and stops on an unsupported object type whatever `$ErrorActionPreference` is #30 * A failed object deployment now stops `Publish-AdfV2FromJson` whatever `$ErrorActionPreference` is, instead of being skipped and the object marked as deployed #295 * Starting and stopping triggers now catches Az errors whatever `$ErrorActionPreference` is, so retries and warnings work; a failed start is a warning and the deployment carries on #508 * `Export-AdfToArmTemplate` calls `npm.cmd` on Windows (avoids npm.ps1 failing on Windows PowerShell 5.1) and stops when an npm command fails #462 diff --git a/private/Get-AdfObjectFolder.ps1 b/private/Get-AdfObjectFolder.ps1 new file mode 100644 index 0000000..1e095aa --- /dev/null +++ b/private/Get-AdfObjectFolder.ps1 @@ -0,0 +1,35 @@ +function Get-AdfObjectFolder { + [CmdletBinding()] + param ( + [parameter(Mandatory = $true)] $adfInstance + ) + + Write-Debug "BEGIN: Get-AdfObjectFolder()" + + # Az cmdlets don't return folders (e.g. PSPipeline has no such property), so read them via REST API. + # Only pipelines, datasets and data flows can be placed in folders. + $folders = @{} + $types = [ordered]@{ 'pipeline' = 'pipelines'; 'dataset' = 'datasets'; 'dataflow' = 'dataflows' } + foreach ($type in $types.Keys) { + $url = "$($script:BaseApiUrl)$($adfInstance.Id)/$($types[$type])?api-version=2018-06-01" + do { + $r = Invoke-AzRestMethod -Method 'GET' -Uri $url + if ($r.StatusCode -ne 200) { + throw "ADFT0035: Unexpected response code: $($r.StatusCode) from REST API when listing $($types[$type])." + } + $content = $r.Content | ConvertFrom-Json + $url = if ($content.PSObject.Properties['nextLink']) { $content.nextLink } else { $null } + $content.value | ForEach-Object { + $folder = '' + $p = $_.properties + if ($null -ne $p -and $p.PSObject.Properties['folder'] -and $null -ne $p.folder) { + $folder = [string] $p.folder.name + } + $folders["$type.$($_.name)"] = $folder + } + } while ($url) + } + + Write-Debug "END: Get-AdfObjectFolder()" + return $folders +} diff --git a/private/Remove-AdfObject.ps1 b/private/Remove-AdfObject.ps1 index f67aecd..5c7792d 100644 --- a/private/Remove-AdfObject.ps1 +++ b/private/Remove-AdfObject.ps1 @@ -1,140 +1,148 @@ function Remove-AdfObject { [CmdletBinding()] param ( - [parameter(Mandatory = $true)] [Adf] $adfSource, + [parameter(Mandatory = $false)] [Adf] $adfSource, [parameter(Mandatory = $true)] $obj, - [parameter(Mandatory = $true)] $adfInstance + [parameter(Mandatory = $true)] $adfInstance, + # When set, a referencing object is deleted first only if it is in this list; otherwise ADFT0034 is thrown + [parameter(Mandatory = $false)] [array] $Selection, + # When set, collects deleted objects; objects already in it are skipped + [parameter(Mandatory = $false)] [System.Collections.ArrayList] $Deleted ) Write-Debug "BEGIN: Remove-AdfObject()" + $ResourceGroupName = $adfInstance.ResourceGroupName + $DataFactoryName = $adfInstance.Name $name = $obj.Name - $err = $null $ErrorMessage = $null $simtype = Get-SimplifiedType -Type $obj.GetType().Name if ($simtype -eq 'AdfObject') { - $simtype = ConvertTo-AdfType -AzType $adfTargetObj.Type + $simtype = ConvertTo-AdfType -AzType $obj.Type } - [AdfObjectName] $oname = [AdfObjectName]::new("$simType.$name") - $IsExcluded = $oname.IsNameExcluded($adfSource.PublishOptions) - if (-not $IsExcluded) { + if ($null -ne $Deleted -and $Deleted.Contains($obj)) { + Write-Verbose "Object [$simtype].[$name] has already been deleted." + Write-Debug "END: Remove-AdfObject()" + return + } + + $action = $simtype + if ($null -eq $adfSource) { Write-Host "Removing object: [$simtype].[$name]" - $action = $simtype } else { - if ($adfSource.PublishOptions.DoNotDeleteExcludedObjects) { - Write-Verbose "Object [$simtype].[$name] won't be deleted as publish option 'DoNotDeleteExcludedObjects' = true." - $action = "DoNothing" + [AdfObjectName] $oname = [AdfObjectName]::new("$simType.$name") + $IsExcluded = $oname.IsNameExcluded($adfSource.PublishOptions) + if (-not $IsExcluded) { + Write-Host "Removing object: [$simtype].[$name]" } else { - Write-Host "Removing excluded object: [$simtype].[$name] as publish option 'DoNotDeleteExcludedObjects' = false." - $action = $simtype + if ($adfSource.PublishOptions.DoNotDeleteExcludedObjects) { + Write-Verbose "Object [$simtype].[$name] won't be deleted as publish option 'DoNotDeleteExcludedObjects' = true." + $action = "DoNothing" + } else { + Write-Host "Removing excluded object: [$simtype].[$name] as publish option 'DoNotDeleteExcludedObjects' = false." + } } } - Try + Try { - switch -Exact ($action) - { - "Dataset" { - Remove-AzDataFactoryV2Dataset ` - -ResourceGroupName $ResourceGroupName ` - -DataFactoryName $DataFactoryName ` - -Name $name ` - -Force -ErrorVariable err -ErrorAction Stop | Out-Null - } - "DataFlow" { - Remove-AzDataFactoryV2DataFlow ` - -ResourceGroupName $ResourceGroupName ` - -DataFactoryName $DataFactoryName ` - -Name $name ` - -Force -ErrorVariable err -ErrorAction Stop | Out-Null - } - "Pipeline" { - Remove-AzDataFactoryV2Pipeline ` - -ResourceGroupName $ResourceGroupName ` - -DataFactoryName $DataFactoryName ` - -Name $name ` - -Force -ErrorVariable err -ErrorAction Stop | Out-Null - } - "LinkedService" { - Remove-AzDataFactoryV2LinkedService ` - -ResourceGroupName $ResourceGroupName ` - -DataFactoryName $DataFactoryName ` - -Name $name ` - -Force -ErrorVariable err -ErrorAction Stop | Out-Null - } - "IntegrationRuntime" { - Remove-AzDataFactoryV2IntegrationRuntime ` - -ResourceGroupName $ResourceGroupName ` - -DataFactoryName $DataFactoryName ` - -Name $name ` - -Force -ErrorVariable err -ErrorAction Stop | Out-Null - } - "Trigger" { - # Stop trigger if enabled before delete it - if ($obj.RuntimeState -eq 'Started') { - Write-Verbose "Disabling trigger: $name..." - Stop-AzDataFactoryV2Trigger ` + Invoke-AdfWithRetry -Activity "Removing [$simtype].[$name]" -ScriptBlock { + switch -Exact ($action) + { + "Dataset" { + Remove-AzDataFactoryV2Dataset ` -ResourceGroupName $ResourceGroupName ` -DataFactoryName $DataFactoryName ` -Name $name ` - -Force -ErrorVariable err -ErrorAction Stop | Out-Null + -Force -ErrorAction Stop | Out-Null } - Remove-AzDataFactoryV2Trigger ` - -ResourceGroupName $ResourceGroupName ` - -DataFactoryName $DataFactoryName ` - -Name $name ` - -Force -ErrorVariable err -ErrorAction Stop | Out-Null - } - "Credential" { - Remove-AdfObjectRestAPI ` - -type_plural 'credentials' ` - -name $name ` - -adfInstance $adfInstance ` - -ErrorVariable err -ErrorAction Stop | Out-Null - } - "DoNothing" { + "DataFlow" { + Remove-AzDataFactoryV2DataFlow ` + -ResourceGroupName $ResourceGroupName ` + -DataFactoryName $DataFactoryName ` + -Name $name ` + -Force -ErrorAction Stop | Out-Null + } + "Pipeline" { + Remove-AzDataFactoryV2Pipeline ` + -ResourceGroupName $ResourceGroupName ` + -DataFactoryName $DataFactoryName ` + -Name $name ` + -Force -ErrorAction Stop | Out-Null + } + "LinkedService" { + Remove-AzDataFactoryV2LinkedService ` + -ResourceGroupName $ResourceGroupName ` + -DataFactoryName $DataFactoryName ` + -Name $name ` + -Force -ErrorAction Stop | Out-Null + } + "IntegrationRuntime" { + Remove-AzDataFactoryV2IntegrationRuntime ` + -ResourceGroupName $ResourceGroupName ` + -DataFactoryName $DataFactoryName ` + -Name $name ` + -Force -ErrorAction Stop | Out-Null + } + "Trigger" { + # Stop trigger if enabled before delete it + if ($obj.RuntimeState -eq 'Started') { + Write-Verbose "Disabling trigger: $name..." + Stop-AzDataFactoryV2Trigger ` + -ResourceGroupName $ResourceGroupName ` + -DataFactoryName $DataFactoryName ` + -Name $name ` + -Force -ErrorAction Stop | Out-Null + } + Remove-AzDataFactoryV2Trigger ` + -ResourceGroupName $ResourceGroupName ` + -DataFactoryName $DataFactoryName ` + -Name $name ` + -Force -ErrorAction Stop | Out-Null + } + "Credential" { + Remove-AdfObjectRestAPI ` + -type_plural 'credentials' ` + -name $name ` + -adfInstance $adfInstance ` + -ErrorAction Stop | Out-Null + } + "DoNothing" { - } - default - { - Write-Error "ADFT0018: Type $($obj.GetType().Name) is not supported." + } + default + { + throw "ADFT0018: Type $($obj.GetType().Name) is not supported." + } } } } Catch { - Write-Debug "Error caught when deleting:`n$err" $ErrorMessage = $_.Exception.Message + Write-Debug "Error caught when deleting:`n$ErrorMessage" } - # Finally { - # if ($null -eq $ErrorMessage -and $null -ne $err) { - # if ($err.Count -gt 0) { $ErrorMessage = $err[0] } - # } - # } - - # if ($ErrorMessage -match 'Error Code: TriggerEnabledCannotUpdate') - # { - # Write-host "Disabling trigger: $name..." - # Stop-AzDataFactoryV2Trigger ` - # -ResourceGroupName $ResourceGroupName ` - # -DataFactoryName $DataFactoryName ` - # -Name $name ` - # -Force | Out-Null - # Remove-AdfObject -obj $obj -adfInstance $adfInstance - # } if ($ErrorMessage -match 'deleted since it is referenced by (?.+)\.') { - Write-Verbose "The document cannot be deleted since it is referenced by $($Matches.RefName)." - #$Matches.RefName - $refobj = $adfInstance.AllObjects() | Where-Object { $_.Name -eq $Matches.RefName } + $refName = $Matches.RefName + Write-Verbose "The document cannot be deleted since it is referenced by $refName." + $refobj = @($adfInstance.AllObjects() | Where-Object { $_.Name -eq $refName }) + if ($null -ne $Selection) { + $notSelected = @($refobj | Where-Object { $Selection -notcontains $_ }) + if ($refobj.Count -eq 0 -or $notSelected.Count -gt 0) { + throw "ADFT0034: [$simtype].[$name] cannot be deleted since it is referenced by $refName, which is not selected for deletion." + } + } $refobj | ForEach-Object { - Remove-AdfObject -adfSource $adfSource -obj $_ -adfInstance $adfInstance + Remove-AdfObject -adfSource $adfSource -obj $_ -adfInstance $adfInstance -Selection $Selection -Deleted $Deleted } - Remove-AdfObject -adfSource $adfSource -obj $obj -adfInstance $adfInstance + Remove-AdfObject -adfSource $adfSource -obj $obj -adfInstance $adfInstance -Selection $Selection -Deleted $Deleted } elseif ($null -ne $ErrorMessage) { #Rethrow exception throw $ErrorMessage + } elseif ($null -ne $Deleted -and $action -ne 'DoNothing') { + $Deleted.Add($obj) | Out-Null } Write-Debug "END: Remove-AdfObject()" diff --git a/public/Remove-AdfObjects.ps1 b/public/Remove-AdfObjects.ps1 new file mode 100644 index 0000000..16ffb18 --- /dev/null +++ b/public/Remove-AdfObjects.ps1 @@ -0,0 +1,106 @@ +<# +.SYNOPSIS +Deletes selected objects from Azure Data Factory instance (service). + +.DESCRIPTION +Deletes objects from Azure Data Factory instance (service) that match given name patterns, or that a deployment with given publish option would deploy. +Objects are deleted in order: triggers, pipelines, data flows, datasets, linked services, credentials, integration runtimes. Started triggers are stopped first. +When an object is referenced by another one, the referencing object is deleted first, but only if it is selected too. Otherwise the function stops with an error. +Any failed deletion stops the function. + +.PARAMETER ResourceGroupName +Resource Group Name of target ADF service. + +.PARAMETER DataFactoryName +Name of target ADF service. + +.PARAMETER Pattern +One or more wildcard patterns of objects to be deleted, in format: type.name[@folder] +The same format as in Includes/Excludes of the publish option, e.g.: +'pipeline.PL_Old*', 'dataset.*', 'pipeline.*@Archive', 'pipeline.*@Archive*' (Archive and its subfolders) +Only pipelines, datasets and data flows have folders. + +.PARAMETER Option +Publish option (see New-AdfPublishOption). Deletes the objects that a deployment with this option would deploy, i.e. matching Includes and not matching Excludes. +Note: when the option has no Includes, all objects except those in Excludes are deleted. + +.EXAMPLE +Remove-AdfObjects -ResourceGroupName 'rg-devops-factory' -DataFactoryName 'SQLPlayerDemo' -Pattern 'pipeline.PL_Old*', 'dataset.DS_Old*' + +.EXAMPLE +# Delete all pipelines from folder 'Archive' and its subfolders, but show only what would be deleted +Remove-AdfObjects -ResourceGroupName 'rg-devops-factory' -DataFactoryName 'SQLPlayerDemo' -Pattern 'pipeline.*@Archive*' -WhatIf + +.EXAMPLE +# "Reverse deployment": delete objects selected by deployment config file +$opt = New-AdfPublishOption -FilePath '.\deployment\rules.csv' +Remove-AdfObjects -ResourceGroupName 'rg-devops-factory' -DataFactoryName 'SQLPlayerDemo' -Option $opt + +.OUTPUTS +Names (type.name) of deleted objects. + +.LINK +Online version: https://github.com/SQLPlayer/azure.datafactory.tools/ +#> +function Remove-AdfObjects { + [CmdletBinding(SupportsShouldProcess = $true, DefaultParameterSetName = 'Pattern')] + [OutputType([String[]])] + param ( + [parameter(Mandatory = $true)] [String] $ResourceGroupName, + [parameter(Mandatory = $true)] [String] $DataFactoryName, + [parameter(Mandatory = $true, ParameterSetName = 'Pattern')] [String[]] $Pattern, + [parameter(Mandatory = $true, ParameterSetName = 'Option')] [AdfPublishOption] $Option + ) + + Write-Debug "BEGIN: Remove-AdfObjects(ResourceGroupName=$ResourceGroupName, DataFactoryName=$DataFactoryName)" + + $adfIns = Get-AdfFromService -FactoryName $DataFactoryName -ResourceGroupName $ResourceGroupName + + if ($PSCmdlet.ParameterSetName -eq 'Pattern') { + $patterns = @($Pattern) + } else { + $patterns = @($Option.Includes.Keys) + @($Option.Excludes.Keys) + } + $folders = @{} + if (@($patterns | Where-Object { $_ -like '*@*' }).Count -gt 0) { + $folders = Get-AdfObjectFolder -adfInstance $adfIns + } + + $matching = @($adfIns.AllObjects() | Where-Object { + $simtype = Get-SimplifiedType -Type $_.GetType().Name + $fullName = "$simtype.$($_.Name)" + $folder = if ($folders.ContainsKey($fullName)) { $folders[$fullName] } else { '' } + if ($PSCmdlet.ParameterSetName -eq 'Pattern') { + $isMatch = $false + foreach ($p in $Pattern) { + $arr = $p.Split('@') + if ($fullName -like $arr[0] -and ($arr.Count -le 1 -or $folder -like $arr[1])) { $isMatch = $true } + } + $isMatch + } else { + $oname = [AdfObjectName]::new($_.Name, $simtype, $folder) + -not $oname.IsNameExcluded($Option) + } + }) + # Delete referencing objects before the objects they reference + $typeOrder = @('Trigger', 'Pipeline', 'DataFlow', 'Dataset', 'LinkedService', 'Credential', 'IntegrationRuntime') + $selected = @(foreach ($type in $typeOrder) { + $matching | Where-Object { (Get-SimplifiedType -Type $_.GetType().Name) -eq $type } + }) + + Write-Host "Found $($selected.Count) object(s) to delete." + + # Ask about all objects first, so a cascade never deletes an object that was not confirmed + $confirmed = @($selected | Where-Object { + $PSCmdlet.ShouldProcess("[$(Get-SimplifiedType -Type $_.GetType().Name)].[$($_.Name)] in $DataFactoryName", 'Remove') + }) + + $deleted = [System.Collections.ArrayList]::new() + foreach ($obj in $confirmed) { + Remove-AdfObject -obj $obj -adfInstance $adfIns -Selection $confirmed -Deleted $deleted + } + + Write-Host "Deleted $($deleted.Count) object(s) from ADF service." + Write-Debug "END: Remove-AdfObjects()" + return [String[]] @($deleted | ForEach-Object { "$(Get-SimplifiedType -Type $_.GetType().Name).$($_.Name)" }) +} diff --git a/test/Incremental-Deployment.Tests.ps1 b/test/Incremental-Deployment.Tests.ps1 index d5824c2..9a0e23e 100644 --- a/test/Incremental-Deployment.Tests.ps1 +++ b/test/Incremental-Deployment.Tests.ps1 @@ -185,6 +185,9 @@ InModuleScope azure.datafactory.tools { Mock Get-AdfFromService { param ($FactoryName, $ResourceGroupName) + # Like the real function: Remove-AdfObject takes the factory from the instance + $script:TargetAdf.Name = $FactoryName + $script:TargetAdf.ResourceGroupName = $ResourceGroupName return $script:TargetAdf } } diff --git a/test/Remove-AdfObjects.Tests.ps1 b/test/Remove-AdfObjects.Tests.ps1 new file mode 100644 index 0000000..b0da7e2 --- /dev/null +++ b/test/Remove-AdfObjects.Tests.ps1 @@ -0,0 +1,215 @@ +BeforeDiscovery { + $ModuleRootPath = $PSScriptRoot | Split-Path -Parent + $moduleManifestName = 'azure.datafactory.tools.psd1' + $moduleManifestPath = Join-Path -Path $ModuleRootPath -ChildPath $moduleManifestName + + Import-Module -Name $moduleManifestPath -Force -Verbose:$false +} + +InModuleScope azure.datafactory.tools { + + Describe 'Remove-AdfObject' -Tag 'Unit' { + + BeforeEach { + $ErrorActionPreference = 'Continue' + Mock Write-Host {} + $script:ins = [AdfInstance]::new() + $script:ins.Name = 'adf' + $script:ins.ResourceGroupName = 'rg' + } + + It 'Should take the factory from adfInstance and work without a source model' { + Mock Remove-AzDataFactoryV2Pipeline {} + $p = [AdfPSPipeline]::new('PL_A') + $script:ins.Pipelines.Add($p) | Out-Null + Remove-AdfObject -obj $p -adfInstance $script:ins + Should -Invoke Remove-AzDataFactoryV2Pipeline -Times 1 -Exactly -ParameterFilter { + $ResourceGroupName -eq 'rg' -and $DataFactoryName -eq 'adf' -and $Name -eq 'PL_A' + } + } + + It 'Should throw for an unsupported type' { + $o = [PSCustomObject]@{ Name = 'x' } + { Remove-AdfObject -obj $o -adfInstance $script:ins } | Should -Throw '*ADFT0018*' + } + + It 'Should throw when the Az cmdlet writes a non-terminating error' { + Mock Remove-AzDataFactoryV2Dataset { Write-Error 'BadRequest: something went wrong' } + $d = [AdfPSDataset]::new('DS_A') + { Remove-AdfObject -obj $d -adfInstance $script:ins } | Should -Throw '*BadRequest*' + } + + It 'Should delete the referencing object first when it is in the selection' { + $script:dsDeleteAttempt = 0 + Mock Remove-AzDataFactoryV2Dataset { + $script:dsDeleteAttempt++ + if ($script:dsDeleteAttempt -eq 1) { Write-Error 'The document cannot be deleted since it is referenced by PL_A.' } + } + Mock Remove-AzDataFactoryV2Pipeline {} + $d = [AdfPSDataset]::new('DS_A') + $p = [AdfPSPipeline]::new('PL_A') + $script:ins.DataSets.Add($d) | Out-Null + $script:ins.Pipelines.Add($p) | Out-Null + $deleted = [System.Collections.ArrayList]::new() + Remove-AdfObject -obj $d -adfInstance $script:ins -Selection @($d, $p) -Deleted $deleted + Should -Invoke Remove-AzDataFactoryV2Pipeline -Times 1 -Exactly + Should -Invoke Remove-AzDataFactoryV2Dataset -Times 2 -Exactly + $deleted.Count | Should -Be 2 + } + + It 'Should throw and not delete the referencing object when it is not in the selection' { + Mock Remove-AzDataFactoryV2Dataset { Write-Error 'The document cannot be deleted since it is referenced by PL_A.' } + Mock Remove-AzDataFactoryV2Pipeline {} + $d = [AdfPSDataset]::new('DS_A') + $p = [AdfPSPipeline]::new('PL_A') + $script:ins.DataSets.Add($d) | Out-Null + $script:ins.Pipelines.Add($p) | Out-Null + { Remove-AdfObject -obj $d -adfInstance $script:ins -Selection @($d) } | Should -Throw '*ADFT0034*PL_A*' + Should -Invoke Remove-AzDataFactoryV2Pipeline -Times 0 -Exactly + } + + It 'Should skip an object that has already been deleted' { + Mock Remove-AzDataFactoryV2Pipeline {} + $p = [AdfPSPipeline]::new('PL_A') + $deleted = [System.Collections.ArrayList]::new() + $deleted.Add($p) | Out-Null + Remove-AdfObject -obj $p -adfInstance $script:ins -Selection @($p) -Deleted $deleted + Should -Invoke Remove-AzDataFactoryV2Pipeline -Times 0 -Exactly + } + } + + Describe 'Remove-AdfObjects' -Tag 'Unit' { + + BeforeEach { + $ErrorActionPreference = 'Continue' + Mock Write-Host {} + $script:ins = [AdfInstance]::new() + $script:ins.Id = '/subscriptions/s/resourceGroups/rg/providers/Microsoft.DataFactory/factories/adf' + $script:ins.Name = 'adf' + $script:ins.ResourceGroupName = 'rg' + $script:ins.Pipelines.Add([AdfPSPipeline]::new('PL_Old1')) | Out-Null + $script:ins.Pipelines.Add([AdfPSPipeline]::new('PL_Old2')) | Out-Null + $script:ins.Pipelines.Add([AdfPSPipeline]::new('PL_Keep')) | Out-Null + $script:ins.DataSets.Add([AdfPSDataset]::new('DS_Old')) | Out-Null + $script:ins.Triggers.Add([AdfPSTrigger]::new('TR_Old', 'Started')) | Out-Null + Mock Get-AdfFromService { $script:ins } + Mock Get-AdfObjectFolder { + @{ 'pipeline.PL_Old1' = 'Archive'; 'pipeline.PL_Old2' = 'Archive/2020'; 'pipeline.PL_Keep' = 'Main'; 'dataset.DS_Old' = '' } + } + Mock Remove-AzDataFactoryV2Pipeline {} + Mock Remove-AzDataFactoryV2Dataset {} + Mock Remove-AzDataFactoryV2Trigger {} + Mock Stop-AzDataFactoryV2Trigger {} + } + + It 'Should be exported' { + Get-Command -Module azure.datafactory.tools -Name 'Remove-AdfObjects' | Should -Not -BeNullOrEmpty + } + + It 'Should delete only objects matching the patterns' { + $r = Remove-AdfObjects -ResourceGroupName 'rg' -DataFactoryName 'adf' -Pattern 'pipeline.PL_Old*', 'dataset.*' + Should -Invoke Remove-AzDataFactoryV2Pipeline -Times 2 -Exactly + Should -Invoke Remove-AzDataFactoryV2Pipeline -Times 0 -Exactly -ParameterFilter { $Name -eq 'PL_Keep' } + Should -Invoke Remove-AzDataFactoryV2Dataset -Times 1 -Exactly + Should -Invoke Remove-AzDataFactoryV2Trigger -Times 0 -Exactly + Should -Invoke Get-AdfObjectFolder -Times 0 -Exactly + $r.Count | Should -Be 3 + } + + It 'Should filter by folder when a pattern has @folder' { + Remove-AdfObjects -ResourceGroupName 'rg' -DataFactoryName 'adf' -Pattern 'pipeline.*@Archive' + Should -Invoke Remove-AzDataFactoryV2Pipeline -Times 1 -Exactly -ParameterFilter { $Name -eq 'PL_Old1' } + Should -Invoke Remove-AzDataFactoryV2Pipeline -Times 1 -Exactly + } + + It 'Should include subfolders with a wildcard folder pattern' { + Remove-AdfObjects -ResourceGroupName 'rg' -DataFactoryName 'adf' -Pattern 'pipeline.*@Archive*' + Should -Invoke Remove-AzDataFactoryV2Pipeline -Times 2 -Exactly + Should -Invoke Remove-AzDataFactoryV2Pipeline -Times 0 -Exactly -ParameterFilter { $Name -eq 'PL_Keep' } + } + + It 'Should delete triggers before pipelines and stop started triggers' { + $script:order = [System.Collections.ArrayList]::new() + Mock Remove-AzDataFactoryV2Pipeline { $script:order.Add("pipeline.$Name") | Out-Null } + Mock Remove-AzDataFactoryV2Trigger { $script:order.Add("trigger.$Name") | Out-Null } + Remove-AdfObjects -ResourceGroupName 'rg' -DataFactoryName 'adf' -Pattern 'pipeline.PL_Old1', 'trigger.*' + $script:order[0] | Should -Be 'trigger.TR_Old' + $script:order[1] | Should -Be 'pipeline.PL_Old1' + Should -Invoke Stop-AzDataFactoryV2Trigger -Times 1 -Exactly + } + + It 'Should not delete anything with -WhatIf' { + $r = Remove-AdfObjects -ResourceGroupName 'rg' -DataFactoryName 'adf' -Pattern '*.*' -WhatIf + Should -Invoke Remove-AzDataFactoryV2Pipeline -Times 0 -Exactly + Should -Invoke Remove-AzDataFactoryV2Dataset -Times 0 -Exactly + Should -Invoke Remove-AzDataFactoryV2Trigger -Times 0 -Exactly + $r | Should -BeNullOrEmpty + } + + It 'Should delete what a deployment with the given option would deploy' { + $opt = New-AdfPublishOption + $opt.Includes.Add('pipeline.PL_Old*', '') + $opt.Excludes.Add('pipeline.PL_Old2', '') + Remove-AdfObjects -ResourceGroupName 'rg' -DataFactoryName 'adf' -Option $opt + Should -Invoke Remove-AzDataFactoryV2Pipeline -Times 1 -Exactly -ParameterFilter { $Name -eq 'PL_Old1' } + Should -Invoke Remove-AzDataFactoryV2Pipeline -Times 1 -Exactly + Should -Invoke Remove-AzDataFactoryV2Dataset -Times 0 -Exactly + } + + It 'Should stop at the first failed delete' { + Mock Remove-AzDataFactoryV2Trigger { Write-Error 'BadRequest: cannot delete' } + { Remove-AdfObjects -ResourceGroupName 'rg' -DataFactoryName 'adf' -Pattern 'pipeline.PL_Old1', 'trigger.*' } | Should -Throw '*BadRequest*' + Should -Invoke Remove-AzDataFactoryV2Pipeline -Times 0 -Exactly + } + + It 'Should do nothing when no object matches' { + $r = Remove-AdfObjects -ResourceGroupName 'rg' -DataFactoryName 'adf' -Pattern 'pipeline.Nothing*' + Should -Invoke Remove-AzDataFactoryV2Pipeline -Times 0 -Exactly + $r | Should -BeNullOrEmpty + } + } + + Describe 'Get-AdfObjectFolder' -Tag 'Unit' { + + BeforeEach { + $ErrorActionPreference = 'Continue' + $script:ins = [AdfInstance]::new() + $script:ins.Id = '/subscriptions/s/resourceGroups/rg/providers/Microsoft.DataFactory/factories/adf' + } + + It 'Should map type.name to folder for pipelines, datasets and data flows' { + Mock Invoke-AzRestMethod { + $body = switch -Wildcard ($Uri) { + '*/pipelines?*' { '{"value":[{"name":"PL_A","properties":{"folder":{"name":"Archive/2020"}}},{"name":"PL_B","properties":{}}]}' } + '*/datasets?*' { '{"value":[{"name":"DS_A","properties":{"folder":{"name":"Sales"}}}]}' } + '*/dataflows?*' { '{"value":[]}' } + } + [PSCustomObject]@{ StatusCode = 200; Content = $body } + } + $r = Get-AdfObjectFolder -adfInstance $script:ins + $r['pipeline.PL_A'] | Should -Be 'Archive/2020' + $r['pipeline.PL_B'] | Should -Be '' + $r['dataset.DS_A'] | Should -Be 'Sales' + Should -Invoke Invoke-AzRestMethod -Times 3 -Exactly + } + + It 'Should follow nextLink' { + Mock Invoke-AzRestMethod { + $body = switch -Wildcard ($Uri) { + 'https://next*' { '{"value":[{"name":"PL_B","properties":{"folder":{"name":"F2"}}}]}' } + '*/pipelines?*' { '{"value":[{"name":"PL_A","properties":{"folder":{"name":"F1"}}}],"nextLink":"https://next/page2"}' } + default { '{"value":[]}' } + } + [PSCustomObject]@{ StatusCode = 200; Content = $body } + } + $r = Get-AdfObjectFolder -adfInstance $script:ins + $r['pipeline.PL_A'] | Should -Be 'F1' + $r['pipeline.PL_B'] | Should -Be 'F2' + } + + It 'Should throw when the API returns an error' { + Mock Invoke-AzRestMethod { [PSCustomObject]@{ StatusCode = 403; Content = '{}' } } + { Get-AdfObjectFolder -adfInstance $script:ins } | Should -Throw '*403*' + } + } +}