Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ The main advantage of the module is the ability to publish all the Azure Data Fa
- [DoNotStopStartExcludedTriggers](#donotstopstartexcludedtriggers)
- [Excluded (collection)](#excluded-collection)
- [DoNotDeleteExcludedObjects](#donotdeleteexcludedobjects)
- [Delete selected objects from ADF](#delete-selected-objects-from-adf)
- [Build/Test Azure Data Factory code](#buildtest-azure-data-factory-code)
- [Test connection of Linked Service (preview)](#test-connection-of-linked-service-preview)
- [Test connection with Service Principal (SPN)](#test-connection-with-service-principal-spn)
Expand Down Expand Up @@ -706,6 +707,8 @@ Furthermore, you can better control which objects you want to omit from removing

> The step might be skipped when `DeleteNotInSource = false` in *Publish Options*

> Objects that are still in the source (also the excluded ones) are never deleted by this step. To delete chosen objects explicitly, use [Remove-AdfObjects](#delete-selected-objects-from-adf).

## Step: Restarting triggers

💬 In log you'll see line: `STEP: Starting triggers...`
Expand Down Expand Up @@ -796,6 +799,29 @@ Otherwise, nothing will happen.
This option gives you a flexibility of deleting objects in the target, but still not touching objects from 'other' group.


# Delete selected objects from ADF

`Remove-AdfObjects` deletes chosen objects from ADF service, regardless of what is in the source code.
Select objects with name patterns in the same format as `Includes`/`Excludes` (`type.name[@folder]`, see [Pattern (WildCard)](#pattern-wildcard)), or pass a *Publish Option* to delete the objects which a deployment with this option would deploy ("reverse deployment").

```powershell
# Delete selected pipelines and all datasets
Remove-AdfObjects -ResourceGroupName 'rg-devops-factory' -DataFactoryName 'SQLPlayerDemo' -Pattern 'pipeline.PL_Old*', 'dataset.*'

# Delete all pipelines from folder 'Archive' and its subfolders - show only what would be deleted
Remove-AdfObjects -ResourceGroupName 'rg-devops-factory' -DataFactoryName 'SQLPlayerDemo' -Pattern 'pipeline.*@Archive*' -WhatIf

# Reverse deployment: delete objects selected by rules file
$opt = New-AdfPublishOption -FilePath ".\deployment\rules.csv"
Remove-AdfObjects -ResourceGroupName 'rg-devops-factory' -DataFactoryName 'SQLPlayerDemo' -Option $opt
```

* Objects are deleted in order: triggers (started ones are stopped first), pipelines, data flows, datasets, linked services, credentials, integration runtimes.
* When a selected object is referenced by another object, the referencing object is deleted first only if it is selected too. Otherwise the cmdlet stops with an error and nothing else is deleted.
* Only pipelines, datasets and data flows have folders.
* When the *Publish Option* has no `Includes`, all objects except those in `Excludes` are deleted. Use `-WhatIf` to check first.
* Any failed deletion stops the cmdlet. It returns names (`type.name`) of deleted objects.

# Build/Test Azure Data Factory code

Another very helpful function is `Test-AdfCode`. Use it to validate the code of your Azure Data Factory before you publish it onto target ADF service.
Expand Down
1 change: 1 addition & 0 deletions azure.datafactory.tools.psd1
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,7 @@
, 'Export-AdfToArmTemplate'
, 'Test-AdfArmTemplate'
, 'Set-AdfToolsAuthToken'
, 'Remove-AdfObjects'
)

# Cmdlets to export from this module, for best performance, do not use wildcards and do not delete the entry, use an empty array if there are no cmdlets to export.
Expand Down
2 changes: 2 additions & 0 deletions changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
### Added
* `Test-AdfCode` warns about enabled triggers without pipeline references, which ADF refuses to start #507
* Deployment of an object is retried after a temporary API error (time-out, throttling, server error); new publish option `MaxRetries` (default 3, 0 = off) #295
* New cmdlet `Remove-AdfObjects` deletes chosen objects from ADF service by name/folder pattern or by publish option #30
### Fixed
* Deleting objects not in source retries temporary API errors and stops on an unsupported object type whatever `$ErrorActionPreference` is #30
* A failed object deployment now stops `Publish-AdfV2FromJson` whatever `$ErrorActionPreference` is, instead of being skipped and the object marked as deployed #295
* Starting and stopping triggers now catches Az errors whatever `$ErrorActionPreference` is, so retries and warnings work; a failed start is a warning and the deployment carries on #508
* `Export-AdfToArmTemplate` calls `npm.cmd` on Windows (avoids npm.ps1 failing on Windows PowerShell 5.1) and stops when an npm command fails #462
Expand Down
35 changes: 35 additions & 0 deletions private/Get-AdfObjectFolder.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
function Get-AdfObjectFolder {
[CmdletBinding()]
param (
[parameter(Mandatory = $true)] $adfInstance
)

Write-Debug "BEGIN: Get-AdfObjectFolder()"

# Az cmdlets don't return folders (e.g. PSPipeline has no such property), so read them via REST API.
# Only pipelines, datasets and data flows can be placed in folders.
$folders = @{}
$types = [ordered]@{ 'pipeline' = 'pipelines'; 'dataset' = 'datasets'; 'dataflow' = 'dataflows' }
foreach ($type in $types.Keys) {
$url = "$($script:BaseApiUrl)$($adfInstance.Id)/$($types[$type])?api-version=2018-06-01"
do {
$r = Invoke-AzRestMethod -Method 'GET' -Uri $url
if ($r.StatusCode -ne 200) {
throw "ADFT0035: Unexpected response code: $($r.StatusCode) from REST API when listing $($types[$type])."
}
$content = $r.Content | ConvertFrom-Json
$url = if ($content.PSObject.Properties['nextLink']) { $content.nextLink } else { $null }
$content.value | ForEach-Object {
$folder = ''
$p = $_.properties
if ($null -ne $p -and $p.PSObject.Properties['folder'] -and $null -ne $p.folder) {
$folder = [string] $p.folder.name
}
$folders["$type.$($_.name)"] = $folder
}
} while ($url)
}

Write-Debug "END: Get-AdfObjectFolder()"
return $folders
}
202 changes: 105 additions & 97 deletions private/Remove-AdfObject.ps1
Original file line number Diff line number Diff line change
@@ -1,140 +1,148 @@
function Remove-AdfObject {
[CmdletBinding()]
param (
[parameter(Mandatory = $true)] [Adf] $adfSource,
[parameter(Mandatory = $false)] [Adf] $adfSource,
[parameter(Mandatory = $true)] $obj,
[parameter(Mandatory = $true)] $adfInstance
[parameter(Mandatory = $true)] $adfInstance,
# When set, a referencing object is deleted first only if it is in this list; otherwise ADFT0034 is thrown
[parameter(Mandatory = $false)] [array] $Selection,
# When set, collects deleted objects; objects already in it are skipped
[parameter(Mandatory = $false)] [System.Collections.ArrayList] $Deleted
)

Write-Debug "BEGIN: Remove-AdfObject()"

$ResourceGroupName = $adfInstance.ResourceGroupName
$DataFactoryName = $adfInstance.Name
$name = $obj.Name
$err = $null
$ErrorMessage = $null
$simtype = Get-SimplifiedType -Type $obj.GetType().Name
if ($simtype -eq 'AdfObject') {
$simtype = ConvertTo-AdfType -AzType $adfTargetObj.Type
$simtype = ConvertTo-AdfType -AzType $obj.Type
}

[AdfObjectName] $oname = [AdfObjectName]::new("$simType.$name")
$IsExcluded = $oname.IsNameExcluded($adfSource.PublishOptions)
if (-not $IsExcluded) {
if ($null -ne $Deleted -and $Deleted.Contains($obj)) {
Write-Verbose "Object [$simtype].[$name] has already been deleted."
Write-Debug "END: Remove-AdfObject()"
return
}

$action = $simtype
if ($null -eq $adfSource) {
Write-Host "Removing object: [$simtype].[$name]"
$action = $simtype
} else {
if ($adfSource.PublishOptions.DoNotDeleteExcludedObjects) {
Write-Verbose "Object [$simtype].[$name] won't be deleted as publish option 'DoNotDeleteExcludedObjects' = true."
$action = "DoNothing"
[AdfObjectName] $oname = [AdfObjectName]::new("$simType.$name")
$IsExcluded = $oname.IsNameExcluded($adfSource.PublishOptions)
if (-not $IsExcluded) {
Write-Host "Removing object: [$simtype].[$name]"
} else {
Write-Host "Removing excluded object: [$simtype].[$name] as publish option 'DoNotDeleteExcludedObjects' = false."
$action = $simtype
if ($adfSource.PublishOptions.DoNotDeleteExcludedObjects) {
Write-Verbose "Object [$simtype].[$name] won't be deleted as publish option 'DoNotDeleteExcludedObjects' = true."
$action = "DoNothing"
} else {
Write-Host "Removing excluded object: [$simtype].[$name] as publish option 'DoNotDeleteExcludedObjects' = false."
}
}
}

Try
Try
{
switch -Exact ($action)
{
"Dataset" {
Remove-AzDataFactoryV2Dataset `
-ResourceGroupName $ResourceGroupName `
-DataFactoryName $DataFactoryName `
-Name $name `
-Force -ErrorVariable err -ErrorAction Stop | Out-Null
}
"DataFlow" {
Remove-AzDataFactoryV2DataFlow `
-ResourceGroupName $ResourceGroupName `
-DataFactoryName $DataFactoryName `
-Name $name `
-Force -ErrorVariable err -ErrorAction Stop | Out-Null
}
"Pipeline" {
Remove-AzDataFactoryV2Pipeline `
-ResourceGroupName $ResourceGroupName `
-DataFactoryName $DataFactoryName `
-Name $name `
-Force -ErrorVariable err -ErrorAction Stop | Out-Null
}
"LinkedService" {
Remove-AzDataFactoryV2LinkedService `
-ResourceGroupName $ResourceGroupName `
-DataFactoryName $DataFactoryName `
-Name $name `
-Force -ErrorVariable err -ErrorAction Stop | Out-Null
}
"IntegrationRuntime" {
Remove-AzDataFactoryV2IntegrationRuntime `
-ResourceGroupName $ResourceGroupName `
-DataFactoryName $DataFactoryName `
-Name $name `
-Force -ErrorVariable err -ErrorAction Stop | Out-Null
}
"Trigger" {
# Stop trigger if enabled before delete it
if ($obj.RuntimeState -eq 'Started') {
Write-Verbose "Disabling trigger: $name..."
Stop-AzDataFactoryV2Trigger `
Invoke-AdfWithRetry -Activity "Removing [$simtype].[$name]" -ScriptBlock {
switch -Exact ($action)
{
"Dataset" {
Remove-AzDataFactoryV2Dataset `
-ResourceGroupName $ResourceGroupName `
-DataFactoryName $DataFactoryName `
-Name $name `
-Force -ErrorVariable err -ErrorAction Stop | Out-Null
-Force -ErrorAction Stop | Out-Null
}
Remove-AzDataFactoryV2Trigger `
-ResourceGroupName $ResourceGroupName `
-DataFactoryName $DataFactoryName `
-Name $name `
-Force -ErrorVariable err -ErrorAction Stop | Out-Null
}
"Credential" {
Remove-AdfObjectRestAPI `
-type_plural 'credentials' `
-name $name `
-adfInstance $adfInstance `
-ErrorVariable err -ErrorAction Stop | Out-Null
}
"DoNothing" {
"DataFlow" {
Remove-AzDataFactoryV2DataFlow `
-ResourceGroupName $ResourceGroupName `
-DataFactoryName $DataFactoryName `
-Name $name `
-Force -ErrorAction Stop | Out-Null
}
"Pipeline" {
Remove-AzDataFactoryV2Pipeline `
-ResourceGroupName $ResourceGroupName `
-DataFactoryName $DataFactoryName `
-Name $name `
-Force -ErrorAction Stop | Out-Null
}
"LinkedService" {
Remove-AzDataFactoryV2LinkedService `
-ResourceGroupName $ResourceGroupName `
-DataFactoryName $DataFactoryName `
-Name $name `
-Force -ErrorAction Stop | Out-Null
}
"IntegrationRuntime" {
Remove-AzDataFactoryV2IntegrationRuntime `
-ResourceGroupName $ResourceGroupName `
-DataFactoryName $DataFactoryName `
-Name $name `
-Force -ErrorAction Stop | Out-Null
}
"Trigger" {
# Stop trigger if enabled before delete it
if ($obj.RuntimeState -eq 'Started') {
Write-Verbose "Disabling trigger: $name..."
Stop-AzDataFactoryV2Trigger `
-ResourceGroupName $ResourceGroupName `
-DataFactoryName $DataFactoryName `
-Name $name `
-Force -ErrorAction Stop | Out-Null
}
Remove-AzDataFactoryV2Trigger `
-ResourceGroupName $ResourceGroupName `
-DataFactoryName $DataFactoryName `
-Name $name `
-Force -ErrorAction Stop | Out-Null
}
"Credential" {
Remove-AdfObjectRestAPI `
-type_plural 'credentials' `
-name $name `
-adfInstance $adfInstance `
-ErrorAction Stop | Out-Null
}
"DoNothing" {

}
default
{
Write-Error "ADFT0018: Type $($obj.GetType().Name) is not supported."
}
default
{
throw "ADFT0018: Type $($obj.GetType().Name) is not supported."
}
}
}
}
Catch {
Write-Debug "Error caught when deleting:`n$err"
$ErrorMessage = $_.Exception.Message
Write-Debug "Error caught when deleting:`n$ErrorMessage"
}
# Finally {
# if ($null -eq $ErrorMessage -and $null -ne $err) {
# if ($err.Count -gt 0) { $ErrorMessage = $err[0] }
# }
# }

# if ($ErrorMessage -match 'Error Code: TriggerEnabledCannotUpdate')
# {
# Write-host "Disabling trigger: $name..."
# Stop-AzDataFactoryV2Trigger `
# -ResourceGroupName $ResourceGroupName `
# -DataFactoryName $DataFactoryName `
# -Name $name `
# -Force | Out-Null
# Remove-AdfObject -obj $obj -adfInstance $adfInstance
# }

if ($ErrorMessage -match 'deleted since it is referenced by (?<RefName>.+)\.')
{
Write-Verbose "The document cannot be deleted since it is referenced by $($Matches.RefName)."
#$Matches.RefName
$refobj = $adfInstance.AllObjects() | Where-Object { $_.Name -eq $Matches.RefName }
$refName = $Matches.RefName
Write-Verbose "The document cannot be deleted since it is referenced by $refName."
$refobj = @($adfInstance.AllObjects() | Where-Object { $_.Name -eq $refName })
if ($null -ne $Selection) {
$notSelected = @($refobj | Where-Object { $Selection -notcontains $_ })
if ($refobj.Count -eq 0 -or $notSelected.Count -gt 0) {
throw "ADFT0034: [$simtype].[$name] cannot be deleted since it is referenced by $refName, which is not selected for deletion."
}
}
$refobj | ForEach-Object {
Remove-AdfObject -adfSource $adfSource -obj $_ -adfInstance $adfInstance
Remove-AdfObject -adfSource $adfSource -obj $_ -adfInstance $adfInstance -Selection $Selection -Deleted $Deleted
}
Remove-AdfObject -adfSource $adfSource -obj $obj -adfInstance $adfInstance
Remove-AdfObject -adfSource $adfSource -obj $obj -adfInstance $adfInstance -Selection $Selection -Deleted $Deleted
} elseif ($null -ne $ErrorMessage) {
#Rethrow exception
throw $ErrorMessage
} elseif ($null -ne $Deleted -and $action -ne 'DoNothing') {
$Deleted.Add($obj) | Out-Null
}

Write-Debug "END: Remove-AdfObject()"
Expand Down
Loading
Loading