diff --git a/docs/ADVANCED-USAGE.md b/docs/ADVANCED-USAGE.md index ffe880c..30dc66a 100644 --- a/docs/ADVANCED-USAGE.md +++ b/docs/ADVANCED-USAGE.md @@ -12,7 +12,7 @@ The example targets trusted official providers and organisation-authored private Create a repository Actions secret named `TF_API_TOKEN` with read access to your HCP Terraform registry modules and providers. The processing job receives it as `TF_TOKEN_app_terraform_io`. Publication runs separately with repository write permissions and no registry token. -Terraform commands can need extra environment variables, such as a token for another private registry or credentials for a module source during `terraform init`. Put non-sensitive entries in the `terraform_env` input and sensitive ones in the optional `TERRAFORM_ENV` secret, one `NAME=VALUE` per line; both reach `terraform init`, `fmt` and `validate` only. Within a value, `\n` becomes a real newline and `\\` a literal backslash, so a multi-line credential such as a GitHub App private key fits on one line; any other backslash sequence passes through unchanged, and a literal carriage return is rejected. Secret values are registered with `::add-mask::`, which redacts them from the workflow console but not from the captured command logs inside the processing artefact; that artefact is retained for seven days and is downloadable by anyone with read access to the repository, so a credential a provider echoes into Terraform's output appears there in plaintext. A multi-line value is registered as a single mask, so the console may not redact it line by line. Names the automation or the runner sets, and names that would redirect the programs Terraform runs or its configuration, credentials, logging or plug-in sources, such as `PATH`, any `PROCESS_` variable, or `TF_LOG`, are rejected before any Terraform command runs and before any file in the checkout is modified; the [example's README](../examples/github-actions/README.md) lists the full set and shows a GitHub App example. +Terraform commands can need extra environment variables, such as a token for another private registry or credentials for a module source during `terraform init`. Put non-sensitive entries in the `terraform_env` input and sensitive ones in the optional `TERRAFORM_ENV` secret, one `NAME=VALUE` per line; both reach `terraform init`, `fmt` and `validate` only. Within a value, `\n` becomes a real newline and `\\` a literal backslash, so a multi-line credential such as a GitHub App private key fits on one line; any other backslash sequence passes through unchanged, and a literal carriage return is rejected. Secret values are registered with `::add-mask::`, which redacts them from the workflow console, including the output of a failed command that the processing step prints, but not from the captured command logs inside the processing artefact; that artefact is retained for seven days and is downloadable by anyone with read access to the repository, so a credential a provider echoes into Terraform's output appears there in plaintext. A multi-line value is registered as a single mask, so the console may not redact it line by line. Step logs are kept for the repository's log retention period, 90 days by default, so a credential the console fails to redact stays visible there for longer than in the artefact. Names the automation or the runner sets, and names that would redirect the programs Terraform runs or its configuration, credentials, logging or plug-in sources, such as `PATH`, any `PROCESS_` variable, or `TF_LOG`, are rejected before any Terraform command runs and before any file in the checkout is modified; the [example's README](../examples/github-actions/README.md) lists the full set and shows a GitHub App example. Each configured root runs the updater and `terraform init`. If the candidate changes and `terraform_fmt` is enabled, formatting runs recursively below every configured root. All roots then run `terraform validate` in the same initialised checkout, including unchanged candidates. The reusable workflow defaults formatting to `false`; both supplied callers enable it. @@ -20,7 +20,7 @@ The callers pin `tf-version-bump` to `v1.0.0-rc.17` and archive SHA-256 `137eac8 `terraform_init_upgrade` defaults to `false`. Enable the manual input, or set it to `true` in the caller's `with` block for scheduled runs, to add `-upgrade`. Direct script callers use `PROCESS_TERRAFORM_INIT_UPGRADE=true`. Ordinary initialisation preserves compatible locked provider versions and fails if updated constraints exclude them; it does not retry with upgrade. Upgrade can update all eligible providers within their constraints. Generated lock files are included in the candidate. Modules are resolved from their configured constraints on each fresh run. -Any processing failure fails its `process` job, so the jobs list shows which branches broke. When processing wrote a result, the run summary names the branch and classification, adding the failed stage and root for an update, initialisation, formatting or validation failure; a failure before the result exists writes no summary. The summary also shows the updater's log for each root, while Terraform's logs stay in the artefact, and in a dry run each publish summary states what a live run would have done. Update, initialisation, formatting and validation failures still publish, closing the marked pull request and then creating or refreshing the failure issue. An automation failure, or a missing result, stops without changing managed pull requests, issues or refs. +Any processing failure fails its `process` job, so the jobs list shows which branches broke. When processing wrote a result, the run summary names the branch and classification, adding the failed stage and root for an update, initialisation, formatting or validation failure; a failure before the result exists writes no summary. The summary also shows the updater's log for each root and links the branch's result artefact. Terraform's logs stay out of the summary; a failed command's output appears in the `Process candidate` step log instead. In a dry run, each publish summary states what a live run would have done. Update, initialisation, formatting and validation failures still publish, closing the marked pull request and then creating or refreshing the failure issue. An automation failure, or a missing result, stops without changing managed pull requests, issues or refs. An updater log can end with a write error naming a backup of the file's original content in the runner's temporary directory. Nothing needs recovering from it: the runner and its checkout are discarded when the job ends, a candidate is published only when processing succeeds, and the state branch keeps the content it had. Read the error for the cause, usually a full or failing disk on the runner, and run the workflow again. diff --git a/examples/github-actions/.github/scripts/process-state-branch.sh b/examples/github-actions/.github/scripts/process-state-branch.sh index 71b88b7..cf27be4 100755 --- a/examples/github-actions/.github/scripts/process-state-branch.sh +++ b/examples/github-actions/.github/scripts/process-state-branch.sh @@ -32,8 +32,10 @@ process. For an invalid environment it prints the name-only diagnostic, emits no masks and exits 0, so process reports the failure. Emits result.json and logs/, plus candidate.patch only for changed, validated -success. Plain init uses -backend=false -input=false; explicit upgrade adds --upgrade. Every root uses its own temporary TF_DATA_DIR. No commits are created. +success. When an update, init, fmt or validate command runs and fails, its log is +also written to standard error. Plain init uses -backend=false -input=false; +explicit upgrade adds -upgrade. Every root uses its own temporary TF_DATA_DIR. No +commits are created. EOF } @@ -244,7 +246,12 @@ branch_command() { shift 4 run_bounded "$RESULT_STAGE/logs/$log" "$@" || command_status=$? if [[ "$command_status" -ne 0 ]]; then + # Recorded before printing, so a failed print cannot turn this branch failure into an + # automation failure. Printed to the step log, where the workflow's masks apply, it + # shows the cause; awk ends an unterminated last line, and a stage past the deadline + # has no log. write_result "$classification" "$stage" "$root" "$command_status" + [[ ! -f "$PROCESS_RESULT_DIR/logs/$log" ]] || awk 1 "$PROCESS_RESULT_DIR/logs/$log" >&2 processing_status_error "$stage failed for Terraform root $root" fi } diff --git a/examples/github-actions/.github/workflows/tf-version-bump-reusable.yml b/examples/github-actions/.github/workflows/tf-version-bump-reusable.yml index cb2c340..32bb03c 100644 --- a/examples/github-actions/.github/workflows/tf-version-bump-reusable.yml +++ b/examples/github-actions/.github/workflows/tf-version-bump-reusable.yml @@ -157,6 +157,7 @@ jobs: "$CONTROL_CHECKOUT/.github/scripts/process-state-branch.sh" mask "$CONTROL_CHECKOUT/.github/scripts/process-state-branch.sh" process - name: Upload processing result + id: upload if: ${{ always() }} uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 timeout-minutes: 10 @@ -172,6 +173,7 @@ jobs: PROCESS_OUTCOME: ${{ steps.process.outcome }} RESULT_MANIFEST: ${{ runner.temp }}/result/result.json PREVIEW: ${{ inputs.preview }} + ARTIFACT_URL: ${{ steps.upload.outputs.artifact-url }} run: | if [[ ! -f "$RESULT_MANIFEST" ]]; then echo "Processing wrote no result manifest; publication will not change managed refs, pull requests or issues." >&2 @@ -184,6 +186,10 @@ jobs: else "\n- Automation failed with status \(.failure.status); the `Process candidate` step log names the cause, and `logs/` in the processing artefact holds any command output" end)' \ "$RESULT_MANIFEST" >>"$GITHUB_STEP_SUMMARY" + # Artefact names carry an opaque ref hash, so link the one this job uploaded. + if [[ -n "$ARTIFACT_URL" ]]; then + printf -- '- [Processing artefact](%s)\n' "$ARTIFACT_URL" >>"$GITHUB_STEP_SUMMARY" + fi # Appends a file in a fence one backtick longer than any run inside it, truncating past # 64 KiB at a line boundary so the closing fence stays on a line of its own. append_fenced_file() { @@ -199,7 +205,8 @@ jobs: printf '%s\n' "$fence" >>"$GITHUB_STEP_SUMMARY" fi } - # Only the updater's log is shown: Terraform's logs can carry a credential a provider echoed. + # Only the updater's log is shown: Terraform's logs can carry a credential a provider echoed, + # so a failed command's log goes to the processing step's log, where masks apply. index=0 while IFS= read -r root; do index=$((index + 1)) diff --git a/examples/github-actions/README.md b/examples/github-actions/README.md index 353f153..31d94f7 100644 --- a/examples/github-actions/README.md +++ b/examples/github-actions/README.md @@ -71,7 +71,7 @@ GITHUB_APP_INSTALLATION_ID=12435523 GITHUB_APP_PEM_FILE=-----BEGIN RSA PRIVATE KEY-----\nMIIEow...\n-----END RSA PRIVATE KEY-----\n ``` -Each of the secret's values is registered with `::add-mask::`, which redacts it from the workflow console only. A multi-line value is registered as a single mask, so the console may not redact it line by line. Every command's output is captured to log files inside the processing artefact, and those files are not redacted: the artefact is retained for seven days and can be downloaded by anyone with read access to the repository, so a credential a provider echoes into Terraform's output appears there in plaintext. Supplied variables at least cannot turn on Terraform's trace logging, because `TF_LOG` and `TF_LOG_PATH` are reserved. +Each of the secret's values is registered with `::add-mask::`, which redacts it from the workflow console only. A multi-line value is registered as a single mask, so the console may not redact it line by line. When an update, `terraform init`, `fmt` or `validate` command fails, its output is printed in the `Process candidate` step log, where those masks apply. Step logs are kept for the repository's log retention period, 90 days by default, rather than the artefact's seven, so a multi-line credential a provider echoes can stay visible there for longer. Every command's output is also captured to log files inside the processing artefact, and those files are not redacted: the artefact is retained for seven days and can be downloaded by anyone with read access to the repository, so a credential a provider echoes into Terraform's output appears there in plaintext. Supplied variables at least cannot turn on Terraform's trace logging, because `TF_LOG` and `TF_LOG_PATH` are reserved. A name may appear only once across both sources. Names the automation or the runner sets are rejected, as are names that would redirect the programs Terraform runs or its configuration, credentials, logging or plug-in sources. The reserved prefixes are `PROCESS_`, `RECONCILE_`, `DISCOVERY_`, `RUNNER_`, `ACTIONS_`, `LD_`, `DYLD_`, `TF_CLI_ARGS`, `TF_LOG`, `TF_PLUGIN_CACHE` and `GIT_`. The reserved exact names are `PATH`, `IFS`, `ENV`, `BASH_ENV`, `SHELLOPTS`, `BASHOPTS`, `TF_DATA_DIR`, `TF_IN_AUTOMATION`, `CHECKPOINT_DISABLE`, `TF_CLI_CONFIG_FILE`, `TERRAFORM_CONFIG`, `TF_WORKSPACE`, `HOME`, `TMPDIR`, `SSL_CERT_FILE`, `SSL_CERT_DIR`, `GITHUB_ENV`, `GITHUB_PATH`, `GITHUB_OUTPUT`, `GITHUB_STEP_SUMMARY`, `GITHUB_STATE` and `TF_TOKEN_app_terraform_io`. The last of those is reserved in any letter case or other spelling Terraform maps to the `app.terraform.io` host, because it would silently shadow the registry token the workflow injects from the `TF_API_TOKEN` secret; `TF_TOKEN_*` names for other registries remain allowed. The five `GITHUB_*` names are the runner's own command channels rather than provider configuration, which is why `GITHUB_APP_ID` and the provider's other `GITHUB_` variables are accepted while `GITHUB_ENV` is not. Treat the list as best effort rather than exhaustive. The structural protection is a separate rule: a file newly created during a run is only ever publishable if it is a `.terraform.lock.hcl` directly inside a configured Terraform root. A rejected entry never prints its value, and is rejected before any Terraform command runs and before any file in the checkout is modified. @@ -178,6 +178,6 @@ Both CSVs keep values as the audit reports them: as written, except that when a Start with a manual run from the default branch and select `dry_run`. This processes and validates candidates and checks publication locally, without pushing refs or changing PRs or issues. Inspect the candidate patch and logs before enabling live publication. -Open the workflow run to inspect `discover`, `process` and `publish`. Each `process` job's summary includes the updater's log for every configured root, so the run overview page shows which versions changed; a long log is truncated there and kept in full in the artefact. In a preview, the summary also shows each changed branch's candidate patch, truncated the same way. Terraform's own logs appear only in the artefact, because a provider can echo a credential into them. In a dry run, each `publish` job's summary states what a live run would have done to the branch's pull request and failure issue. Download its result artefacts to see `result.json`, `candidate.patch` and captured command logs. Artefacts are retained for seven days. Results belong to one run attempt; use **Re-run all jobs**, as partial job reruns are unsupported. +Open the workflow run to inspect `discover`, `process` and `publish`. Each `process` job's summary includes the updater's log for every configured root, so the run overview page shows which versions changed; a long log is truncated there and kept in full in the artefact. In a preview, the summary also shows each changed branch's candidate patch, truncated the same way. Terraform's own logs stay out of the summary, because a provider can echo a credential into them; when a command fails, the `Process candidate` step log shows its output, masked as described in [Supply environment variables](#supply-environment-variables). Each summary links that job's result artefact, whose name identifies the branch only by a hash. In a dry run, each `publish` job's summary states what a live run would have done to the branch's pull request and failure issue. Download its result artefacts to see `result.json`, `candidate.patch` and captured command logs. Artefacts are retained for seven days. Results belong to one run attempt; use **Re-run all jobs**, as partial job reruns are unsupported. Before enabling schedules, test in a disposable private repository: run a dry run, publish a valid change twice and check that the same PR is refreshed, then introduce a validation failure and confirm that the PR closes and one failure issue is maintained. Finally, test a valid no-change result and confirm that the issue closes. No live GitHub repository is created or mutated by this repository's local component tests. diff --git a/examples/github-actions/test.sh b/examples/github-actions/test.sh index 5c5c0b0..c0e59e1 100755 --- a/examples/github-actions/test.sh +++ b/examples/github-actions/test.sh @@ -117,11 +117,15 @@ ensure_processing_container() { sleep 1 done docker exec "$PROCESS_CONTAINER_ID" /bin/bash -c \ - 'command -v curl >/dev/null && command -v jq >/dev/null && command -v timeout >/dev/null' + 'command -v curl >/dev/null && command -v jq >/dev/null && command -v timeout >/dev/null' \ + || fail 'processing container lacks curl, jq or timeout' } setup_processing_workspace() { + # Also started here, not only on first use, because run_processing is usually called with its + # output redirected, which would hide why the container could not start. + ensure_processing_container cleanup_processing_workspace unset PROCESS_CONFIG_PATH PROCESS_TERRAFORM_ROOTS RUNNER_TEMP unset PROCESS_PREPARATION_DEADLINE_EPOCH @@ -359,6 +363,18 @@ assert_processing_failure() { } +# A failed command's captured log reaches the step log, where registered masks apply, +# followed only by the one-line diagnostic, so the cause is readable in the console. +assert_failure_prints_command_log() { + local log="$PROCESS_RESULT_DIR/logs/$1" stderr="$PROCESS_TMP_ROOT/failure.stderr" + [[ -s "$log" ]] || fail "the failed command captured no $1" + # Compared byte for byte, because command substitution would drop a log's trailing + # blank lines but not the same lines once the diagnostic follows them. + cmp -s <(awk 1 "$log") <(sed '$d' "$stderr") && [[ $(tail -n 1 "$stderr") == 'processing status error: '* ]] \ + || fail "the step log does not show $1 before the diagnostic: $(<"$stderr")" +} + + setup_discovery_repository() { cleanup_discovery_repository unset DISCOVERY_PREVIEW @@ -1047,6 +1063,7 @@ test_processing_init_upgrade_is_opt_in() { 'locked provider conflict without upgrade' grep -F 'does not match configured version constraint' "$PROCESS_RESULT_DIR/logs/init-1.log" >/dev/null \ || fail "plain init did not report the locked provider conflict" + assert_failure_prints_command_log init-1.log jq -e '.classification == "branch-init" and .failure.stage == "terraform init"' \ "$PROCESS_RESULT_DIR/result.json" >/dev/null \ @@ -1254,9 +1271,48 @@ test_processing_records_real_update_and_format_failures() { jq -e --arg c "$classification" '.classification == $c and .failure.status > 0' "$PROCESS_RESULT_DIR/result.json" >/dev/null [[ ! -e "$PROCESS_RESULT_DIR/candidate.patch" ]] || fail 'failed processing emitted patch' grep -E 'Error|error' "$PROCESS_RESULT_DIR/logs/$log" >/dev/null || fail 'missing command diagnostics' + assert_failure_prints_command_log "$log" done } +test_processing_records_a_deadline_that_expires_between_stages() { + # Production break caught: a stage that starts after the shared deadline has no log to + # print, and failing on that turns a branch failure into an automation failure that + # stops publication. + setup_processing_workspace + local shim_bin="$PROCESS_TMP_ROOT/deadline-bin" marker="$PROCESS_TMP_ROOT/init-finished" + local diagnostic='processing status error: terraform validate failed for Terraform root root' + mkdir "$shim_bin" + # timeout marks the end of terraform init; from then on date reports a time past any + # deadline, so the next stage finds none of it left. + cat >"$shim_bin/timeout" <<'EOF' +#!/usr/bin/env bash +marker="${BASH_SOURCE%/*}/../init-finished" +PATH=${PATH#*:} +command_status=0 +timeout "$@" || command_status=$? +[[ " $* " != *' init '* ]] || touch "$marker" +exit "$command_status" +EOF + cat >"$shim_bin/date" <<'EOF' +#!/usr/bin/env bash +marker="${BASH_SOURCE%/*}/../init-finished" +PATH=${PATH#*:} +if [[ -e "$marker" && "$*" == +%s ]]; then echo 99999999999; else exec date "$@"; fi +EOF + chmod 755 "$shim_bin/timeout" "$shim_bin/date" + PROCESS_PATH_PREFIX=$shim_bin + assert_processing_failure "$diagnostic" \ + 'validation after the deadline' + [[ -e "$marker" ]] || fail 'the deadline shim never saw terraform init finish' + [[ "$(<"$PROCESS_TMP_ROOT/failure.stderr")" == "$diagnostic" ]] \ + || fail "a stage that never started printed more than its diagnostic: $(<"$PROCESS_TMP_ROOT/failure.stderr")" + jq -e '.classification == "branch-validation" and .failure.stage == "terraform validate" and + .failure.status == 124' "$PROCESS_RESULT_DIR/result.json" >/dev/null \ + || fail "an expired deadline was not recorded as a validation failure: $(<"$PROCESS_RESULT_DIR/result.json")" + [[ ! -e "$PROCESS_RESULT_DIR/logs/validate-1.log" ]] || fail 'validation ran after the deadline' +} + test_processing_rejects_invalid_inputs_before_updates() { local mode expected for mode in boolean-upgrade boolean-format root-absolute root-traversal root-missing duplicate config-absolute symlink control-oid base-oid hash deadline; do @@ -1681,7 +1737,8 @@ run_workflow_step() { run_report_step() { - run_workflow_step process 'Report processing result' "$3" PROCESS_OUTCOME="$2" RESULT_MANIFEST="$1" + run_workflow_step process 'Report processing result' "$3" PROCESS_OUTCOME="$2" RESULT_MANIFEST="$1" \ + ARTIFACT_URL="${4-}" } @@ -1698,7 +1755,11 @@ test_workflow_reports_the_processing_result() { and .[0].env.PROCESS_OUTCOME == "${{ steps.process.outcome }}" and .[0].env.RESULT_MANIFEST == $result + "/result.json" and .[0].env.PREVIEW == "${{ inputs.preview }}" + and .[0].env.ARTIFACT_URL == "${{ steps.upload.outputs.artifact-url }}" ' >/dev/null || fail 'the report step is not wired to the processing step and its result' + yq -o=json '.jobs.process.steps' "$REUSABLE_WORKFLOW" | jq -e ' + [.[] | select(.id == "upload")] | length == 1 and (.[0].uses | startswith("actions/upload-artifact@")) + ' >/dev/null || fail 'the report step cannot link the artefact the upload step published' local work="$TEST_TMP_ROOT/report-step" rm -rf -- "$work" mkdir "$work" @@ -1713,6 +1774,15 @@ test_workflow_reports_the_processing_result() { [[ "$report" == *'state/nonproduction/example-thing'* && "$report" == *"\`branch-init\`"* \ && "$report" == *'terraform init'* && "$report" == *'environments/production'* ]] \ || fail "the branch failure summary omits the branch, classification, stage or root: $report" + [[ "$report" != *'Processing artefact'* ]] \ + || fail "the summary links an artefact the upload step did not report: $report" + + # The artefact's name is opaque, so the summary links the one this job uploaded. + local url='https://github.com/example/repo/actions/runs/1/artifacts/2' + assert_silent_success 'linking the processing artefact' "$work/stdout" "$diagnostics" \ + run_report_step "$manifest" failure "$summary" "$url" + grep -qxF -- "- [Processing artefact]($url)" "$summary" \ + || fail "the summary does not link the uploaded artefact: $(<"$summary")" # An automation failure's stage and root are unreliable: the EXIT trap records the # literal stage `processing` and the first configured root whatever failed, and only @@ -1788,7 +1858,7 @@ test_workflow_summarises_update_logs() { # fence on a line of its own. [[ "$report" == *$'\n```\n\nThis log was truncated'* ]] \ || fail "the truncated update log's closing fence is not on a line of its own: $report" - # Terraform's logs can carry a credential a provider echoed, so they stay in the artefact. + # Terraform's logs can carry a credential a provider echoed, so they stay out of the summary. [[ "$report" != *SENTINEL* ]] \ || fail 'the summary exposes a Terraform log' grep -qF 'truncated' "$summary" \ @@ -2066,7 +2136,9 @@ if [[ $# -eq 0 ]]; then test_processing_scopes_ignored_modules_to_the_state_branch test_processing_validates_unchanged_candidates test_processing_init_upgrade_is_opt_in test_workflow_runs_three_jobs_with_current_attempt_results - test_processing_records_real_update_and_format_failures test_processing_rejects_invalid_inputs_before_updates + test_processing_records_real_update_and_format_failures + test_processing_records_a_deadline_that_expires_between_stages + test_processing_rejects_invalid_inputs_before_updates test_processing_invalid_config_reconciles_branch_failure test_processing_formats_only_after_dependency_or_lock_changes test_processing_rejects_ignored_generated_lock test_processing_rejects_formatter_changes_outside_patch_policy