CmdLets should emit an Warning or error if enabled trigger does not have pipeline references

Describe the bug

Enabled triggers without pipeline references are not handled properly

On UX Data Factory validation blocks publish for triggers that are enabled wihtout pipeline references:

Test-AdfCode does not generate any warning for the trigger
Publish-AdfV2FromJson tries to to start the trigger, fails

To Reproduce

trigger JSON

{
	"name": "fail",
	"properties": {
		"description": "Backend will fail with this",
		"annotations": [],
		"runtimeState": "Started",
		"pipelines": [
		],
		"type": "ScheduleTrigger",
		"typeProperties": {
			"recurrence": {
				"frequency": "Week",
				"interval": 1,
				"startTime": "2023-02-15T19:32:00",
				"timeZone": "FLE Standard Time",
				"schedule": {
					"minutes": [
						11
					],
					"hours": [
						5
					],
					"weekDays": [
						"Thursday",
						"Friday",
						"Monday",
						"Tuesday",
						"Wednesday"
					]
				}
			}
		}
	}
}

Test-AdfCode does not generate any warning for the trigger

Publish-AdfV2FromJson tries to to start the trigger, fails

The number of triggers to start: 9
Starting triggers:
- Enabling trigger: fail
Start-AzDataFactoryV2Trigger: /Users/janne.kujanpaa/.local/share/powershell/Modules/azure.datafactory.tools/1.11.2/private/Start-Trigger.ps1:17:13
Line |
  17 |              Start-AzDataFactoryV2Trigger `
     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | HTTP Status Code: BadRequest Error Code: BadRequest Error Message: Missing or invalid pipeline references for trigger fail Request Id:
     | 7f4324c5-5f97-4acf-9777-805d34405bb5 Timestamp (Utc):12/15/2024 17:54:44
Start-AzDataFactoryV2Trigger: /Users/janne.kujanpaa/.local/share/powershell/Modules/azure.datafactory.tools/1.11.2/private/Start-Trigger.ps1:17:13
Line |
  17 |              Start-AzDataFactoryV2Trigger `
     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | HTTP Status Code: BadRequest Error Code: BadRequest Error Message: Missing or invalid pipeline references for trigger fail Request Id:
     | e644ca62-59c0-42a5-9c8e-ba048608e220 Timestamp (Utc):12/15/2024 17:54:44
Start-AzDataFactoryV2Trigger: /Users/janne.kujanpaa/.local/share/powershell/Modules/azure.datafactory.tools/1.11.2/private/Start-Trigger.ps1:17:13
Line |
  17 |              Start-AzDataFactoryV2Trigger `
     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | HTTP Status Code: BadRequest Error Code: BadRequest Error Message: Missing or invalid pipeline references for trigger fail Request Id:
     | abca1459-a823-4904-908f-dad36a49da36 Timestamp (Utc):12/15/2024 17:54:45
Start-AzDataFactoryV2Trigger: /Users/janne.kujanpaa/.local/share/powershell/Modules/azure.datafactory.tools/1.11.2/private/Start-Trigger.ps1:17:13
Line |
  17 |              Start-AzDataFactoryV2Trigger `
     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | HTTP Status Code: BadRequest Error Code: BadRequest Error Message: Missing or invalid pipeline references for trigger fail Request Id:
     | 53b314c9-be6c-4472-a36e-85b3c1338043 Timestamp (Utc):12/15/2024 17:54:46
Start-AzDataFactoryV2Trigger: /Users/janne.kujanpaa/.local/share/powershell/Modules/azure.datafactory.tools/1.11.2/private/Start-Trigger.ps1:17:13
Line |
  17 |              Start-AzDataFactoryV2Trigger `
     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | HTTP Status Code: BadRequest Error Code: BadRequest Error Message: Missing or invalid pipeline references for trigger fail Request Id:
     | 970fbf7e-59ba-4aa4-b993-c71342849ae8 Timestamp (Utc):12/15/2024 17:54:46
- Enabling trigger: Monthly - End of the Month

Expected behaviour

Test-AdfCode should emit an warning
Publish-AdfV2FromJson should have better indications of the failure to fail the running pipeline

More details

For me it looks like Start-Trigger error handling is not working like it should.

E.g. Write-Host and Write-Warning output are missing

                Write-Host "Failed starting trigger after $attempts attempts."
                Write-Warning -Message $_.Exception.Message

More details 2

Variables are empty if passing -ErrorVariable ev -WarningVariable wv to Publish-AdfV2FromJson

More details 3

Publish-AdfV2FromJson -ErrorVariable ev -WarningVariable wv -ErrorAction 'Stop' is a partial workaround to emit a failure when underlying API calls fails. As side-effect other failing code on our use-case causes early termination and triggers are not started again.

AFAIK error handling code on Start-Trigger should catch the errors when -ErrorAction 'Stop' is used

Version of PowerShell module of:

1.11.2

The error handling of Start-AzDataFactoryV2Trigger might be following:

Normal invocation only prints errors into stream 2. At least I could not capture warning with -warningvariable but 2>warning.log works
If -ErrorAction "Stop" is being used Start-AzDataFactoryV2Trigger CmdLet emits terminating error

In this case ADF backend fails to enable triggers that are using Blob events becasue of missing RBAC:

@jikuja

Thanks @jikuja for raising this issue and describing it with so many useful details.
Let me restore my private environment to test the current behaviour and consider the best solution.

Prep for unit test #435

@jikuja

Thanks again @jikuja for the detailed report. Your observations (no retry, no warning, empty -WarningVariable, -ErrorAction Stop as a partial workaround) pointed straight at the root cause. Here is what we found and the approach we took. It is split into two sub-issues, #507 and #508.

Root cause

Start-AzDataFactoryV2Trigger and Stop-AzDataFactoryV2Trigger report failures as non-terminating errors. try/catch only catches terminating errors, so with the default $ErrorActionPreference = 'Continue':

the catch block in Start-Trigger never ran, so there was no retry, no "Failed starting trigger" message and no warning
the raw API error went straight to the error stream (that is why 2>warning.log worked but -WarningVariable did not)
with -ErrorAction Stop the error became terminating, so it was caught, but the same setting also stopped the whole publish on other errors

We confirmed this against Azure: the error is caught only when the cmdlet is called with -ErrorAction Stop. The same bug affected Stop-Trigger, including the retry released in v1.19.0 (#404).

Approach

1. Validate at build/test time: `Test-AdfCode` (#507)

Deployment is the wrong place to explain why a trigger is invalid. By then we can only pass on what the Azure API returns (Missing or invalid pipeline references for trigger <name>). The source code is available at build time, so that is where we can say precisely what is wrong, before anything is deployed. ADF Studio validation blocks publishing in the same way.

Test-AdfCode now warns about every trigger with runtimeState: Started and no pipeline reference. It handles both shapes: the pipelines array (Schedule, BlobEvents, CustomEvents) and the single pipeline object (TumblingWindowTrigger). Stopped triggers without pipelines are valid and are not reported.

2. Make errors catchable at deployment: `Start-Trigger` / `Stop-Trigger` (#508)

The Az calls now use -ErrorAction Stop inside the try, so retries and warnings work whatever the caller's $ErrorActionPreference is.
Missing or invalid pipeline references is a permanent error, so it is not retried. Temporary errors (concurrent operations, provisioning) are still retried up to 5 times.
Start-Trigger returns success or failure (like Stop-Trigger). Start-Triggers tries every trigger, then lists the ones that failed in one warning.

Why the deployment carries on

A trigger that fails to start does not make the deployed code wrong. Pipelines, datasets and the other triggers are already deployed and working. Failing the whole run at the last step would hide that, and with -ErrorAction Stop it would also leave the remaining triggers stopped. So a failed start or stop is a warning: clearly visible in the log and in -WarningVariable, while the rest of the deployment completes. Teams that want to block such code should run Test-AdfCode in the build stage and fail on its warnings.

Tests

Test-AdfCode: the TR_without_pipeline.json fixture (adf2) plus cases per trigger type and state.
Start-Trigger / Stop-Trigger: the mocks now write a non-terminating error, as the real Az cmdlets do. The old mocks used throw, and that is why the existing tests never caught this bug.

Merge pull request #509 from Azure-Player/issue-435-trigger-validation

Navigation Menu

Repository navigation

CmdLets should emit an Warning or error if enabled trigger does not have pipeline references #435

Root cause

Approach

1. Validate at build/test time: `Test-AdfCode` (#507)

2. Make errors catchable at deployment: `Start-Trigger` / `Stop-Trigger` (#508)

Why the deployment carries on

Tests

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CmdLets should emit an Warning or error if enabled trigger does not have pipeline references #435

Description

Activity

jikuja commented on Dec 16, 2024

jikuja commented on Jan 30, 2025

NowinskiK commented on Jan 30, 2025

NowinskiK commented on Oct 16, 2026

Root cause

Approach

1. Validate at build/test time: Test-AdfCode (#507)

2. Make errors catchable at deployment: Start-Trigger / Stop-Trigger (#508)

Why the deployment carries on

Tests

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1. Validate at build/test time: `Test-AdfCode` (#507)

2. Make errors catchable at deployment: `Start-Trigger` / `Stop-Trigger` (#508)