Skip to content

PowerShell provisioner: retrying the env-var upload sends an empty file, so scripts run without environment_vars #13734

Open

Description

@patrix87

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Overview of the Issue

Sometimes, one script in a long powershell provisioner step runs with none of its environment_vars set. Scripts before it in the same step got their variables, and so do scripts after it. The script then fails at its first use of a variable, with a misleading error such as:

Cannot bind argument to parameter 'Path' because it is null.

We have seen this 4 times in 120 days of Windows Server 2022/2025 Azure image builds over WinRM. A different script failed each time. We have never seen it on Linux builds that use the shell provisioner.

Root cause

Before each script, uploadEnvVars in provisioner/powershell/provisioner.go writes the flattened env vars to remote_env_var_path (default C:/Windows/Temp/packer-ps-env-vars-<uuid>.ps1). The script then dot-sources that file. The upload runs inside retry.Config{...}.Run, but the strings.Reader is created once, outside the retry closure:

envVarReader := strings.NewReader(flattenedEnvVars)
err = retry.Config{StartTimeout: p.config.StartRetryTimeout}.Run(ctx, func(context.Context) error {
    if err := p.communicator.Upload(p.config.RemoteEnvVarPath, envVarReader, nil); err != nil {

If an attempt fails after the communicator has read some or all of the reader, the retry reuses the same reader. That reader is now partly or fully consumed, so the retry uploads a truncated or empty file and returns success.

The WinRM communicator makes this likely. winrmcp reads the whole input into a temp file first, then runs two more WinRM shells: one to restore the file to its destination and one to remove the temp file. A transient failure in either of those steps fails the Upload after the reader is already at EOF. On the retry, winrmcp reads 0 bytes, writes no temp file, and its restore script falls into echo $null > $dest_file_path. That creates an empty destination file and exits 0. If the earlier attempt only failed while removing its temp file, the file it had already written correctly is overwritten with the empty one.

The script upload in the same provisioner does not have this problem: it calls f.Seek(0, 0) at the start of every retry attempt. The shell provisioner also rewinds its env var file inside its retry loop, which explains why Linux builds are not affected.

Why it is hard to diagnose

  • The retried upload reports success, so nothing fails at upload time.
  • Dot-sourcing an empty .ps1 file is valid PowerShell, so nothing fails there either.
  • The script then fails later, at whatever line first uses a variable. The error points at the user's script, not at Packer.
  • The only sign is a Retryable error: Error uploading ps script containing env vars: ... line in the PACKER_LOG=1 output just before the failing script.

Affected components: PowerShell provisioner (provisioner/powershell), both the non-elevated and the elevated_user paths, since both go through prepareEnvVars → uploadEnvVars. Communicator: WinRM.

I checked the other uploads on these paths. The elevated wrapper upload in packer-plugin-sdk/guestexec.GenerateElevatedRunner is a single Upload with no retry, and the remote cleanup script creates a new reader inside its retry closure. Neither has this problem.

Reproduction Steps

This is intermittent because it needs a transient WinRM failure during the env var upload. It shows up most under heavy guest load, for example during large installs.

  1. Build a Windows image over WinRM with a powershell provisioner that runs many scripts and sets environment_vars.
  2. When a transient WinRM error hits the env var upload after its data has been sent, the log shows Retryable error: Error uploading ps script containing env vars. The retry then "succeeds".
  3. The next script runs with no environment variables set.

It reproduces deterministically in a unit test, using a mock communicator whose first Upload reads part of the reader and then returns an error. The second Upload receives the unread tail of the text, or an empty string if the first attempt read everything, instead of the full text.

Packer version

1.16.0

Simplified Packer Template

build {
  sources = ["source.azure-arm.windows"]

  provisioner "powershell" {
    environment_vars = [
      "INSTALL_ROOT=C:\\install",
      "SOME_SETTING=value",
    ]
    scripts = [
      "scripts/01.ps1",
      "scripts/02.ps1",
      # ... many more scripts
      "scripts/30.ps1",
    ]
  }
}

Operating system and Environment details

  • Guests: Windows Server 2022 and Windows Server 2025 (Azure, azure-arm builder)
  • Communicator: WinRM
  • Not seen on Linux guests using the shell provisioner over SSH

Log Fragments and crash.log files

With PACKER_LOG=1, the failing script comes right after a line like this:

Retryable error: Error uploading ps script containing env vars: <winrm error>

The script then fails with a null-argument error at its first use of an environment variable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions