From a793cd9222aba478612fcee3f1dc099ce06c8402 Mon Sep 17 00:00:00 2001 From: robert Date: Wed, 5 Aug 2026 10:56:59 +1000 Subject: [PATCH 01/16] Add design spec for external tool test separation (infra + Terraform) Reimplements the custom tooling mechanism from feature/external-tool-test-separation against current main, scoped to infrastructure + Terraform. Supersedes the in-place categorisation branches as the canonical approach. --- ...05-external-tool-test-separation-design.md | 89 +++++++++++++++++++ 1 file changed, 89 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-05-external-tool-test-separation-design.md diff --git a/docs/superpowers/specs/2026-08-05-external-tool-test-separation-design.md b/docs/superpowers/specs/2026-08-05-external-tool-test-separation-design.md new file mode 100644 index 000000000..8134e85e9 --- /dev/null +++ b/docs/superpowers/specs/2026-08-05-external-tool-test-separation-design.md @@ -0,0 +1,89 @@ +# External Tool Test Separation — Design (Infrastructure + Terraform) + +## Problem + +Calamari's test suite mixes tests that depend on external CLI tools (Terraform, Helm, kubectl, Azure CLI, etc.) into the main pipeline. This: + +- Slows the main feedback loop and couples it to external tool/service availability +- Pins tool versions inline in fixtures with no central tracking or version lifecycle +- Has no systematic way to know what versions we support or test against + +Two prior efforts explored fixes and left the direction unreconciled: + +- `external-tool-test-categorisation` / `robe/terraform-external-tool-categorization` — in-place `[Category(ExternalToolIntegration)]` tagging, no new project. +- `feature/external-tool-test-separation` — a new `Calamari.ExternalTools.Tests` project with a tool-version manifest and download/resolution infrastructure ("custom tooling mechanism"). + +## Decision + +The separate-project approach (`feature/external-tool-test-separation`) is canonical. It supersedes the in-place categorisation branches, which are no longer pursued. This branch (`robe/external-tool-test-separation`) reimplements that approach fresh against current `main` rather than cherry-picking, because: + +- Main has drifted from where the feature branch forked (unrelated churn: NSubstitute/Shouldly swaps, AKS/EKS version bumps) +- Several feature-branch commits bundle multiple tools together (e.g. one commit adds download strategies for all 8 tools at once), so they don't cherry-pick cleanly for a Terraform-only slice + +The feature branch's own implementation plan (`docs/superpowers/plans/2026-06-15-external-tool-test-separation.md`, Tasks 1-7) already scopes almost exactly to infrastructure + Terraform, and is used as the reference blueprint. + +## Scope + +**In scope for this branch:** +- `Calamari.ExternalTools.Tests` project scaffold + `tool-manifest.json` +- Shared infrastructure: `ToolManifest`, `ToolResolver`, `ToolDownloader`, `ExternalToolFixture`, `CalamariCommandHelper` +- Terraform: `TerraformStrategy`, Terraform integration tests migrated into the new project, Terraform unit tests added to `Calamari.Tests`, old `Calamari.Terraform.Tests` fixture trimmed/removed + +**Explicitly out of scope for this branch** (left for follow-up branches/PRs): +- Helm, kubectl, Azure CLI, GCloud, AWS CLI, aws-iam-authenticator, kubelogin +- All `CloudIntegration/` (SDK-based, e.g. Azure App Service) tests +- Automated version-expansion scheduling, NUnit `[Category]` wiring into TeamCity/Nuke build targets (infrastructure for this can land, but pipeline wiring is future work) + +## Architecture + +A single new NUnit test project, isolated from the main pipeline: + +``` +Calamari.ExternalTools.Tests/ + tool-manifest.json + Infrastructure/ + ToolManifest.cs (manifest reader, version range support) + ToolResolver.cs (env var override -> PATH lookup -> download) + ToolDownloader.cs (download + cache, retry, platform/arch detection) + ExternalToolFixture.cs (base class for tool fixtures) + CalamariCommandHelper.cs (in-process Calamari command runner) + ToolStrategies/ + TerraformStrategy.cs + ExternalTools/ + Terraform/ + TerraformCommandsFixture.cs (~5 integration tests, [Category("ExternalTool")]) +``` + +`CloudIntegration/` is not created yet — nothing lands there this round — so the directory structure doesn't need to change shape when cloud tests are added later. + +**Tool resolution order** (in `ToolResolver`): +1. `CALAMARI_TOOL_TERRAFORM_VERSION=X.Y.Z` — pins an exact version (for future automated version discovery) +2. `CALAMARI_TOOL_SKIP_DOWNLOAD=true` — PATH-only, fails loudly if not found (local dev) +3. Default — downloads the manifest's `highest` version (CI default, reproducible) + +**Tool manifest** (`tool-manifest.json`) declares, per tool: `lowest` (contractual minimum), `highest` (latest verified), `source` (release endpoint for future automated discovery), `architectures` (`amd64`, `arm64`). + +## Terraform Migration + +- Add unit tests to `Calamari.Tests` (main pipeline) covering logic gaps identified in the original fixture: var-file argument construction, init command construction, version range checking. +- Delete `source/Calamari.Terraform.Tests/CommandsFixture.cs` and now-redundant resource fixtures/directories. +- Keep 2-3 "wiring" tests as integration tests in the new project — these validate things unit tests can't, like Octostache-substitution-before-Terraform ordering, sensitive-variable output parsing, and plan detailed exit code handling. +- Add ~5 integration tests in `ExternalTools/Terraform/`, tagged `[Category("ExternalTool")]`, run against the manifest's `highest` version by default. +- Remove `Calamari.Terraform.Tests` from `Calamari.sln`; add `Calamari.ExternalTools.Tests`. + +## CI + +`--filter "Category=ExternalTool"` isolates the new project's tests for a separate (nightly/on-demand) run, distinct from the default pipeline. Wiring this into TeamCity/Nuke build targets is follow-up work, not required for this branch to be mergeable — the category exists and is filterable even before a dedicated pipeline stage consumes it. + +## Testing + +- New unit tests run in the default `dotnet test` pipeline via `Calamari.Tests` — no infrastructure changes needed there. +- `Calamari.ExternalTools.Tests` requires either network access (to download Terraform) or a pre-installed Terraform on PATH with `CALAMARI_TOOL_SKIP_DOWNLOAD=true`; it is not expected to run in the default CI job yet. +- `ToolManifest` and `ToolResolver` get their own unit tests (manifest parsing, version range checks, resolution order) since they're new shared infrastructure other tools will depend on later. + +## Out-of-scope follow-up (tracked, not blocking) + +- Migrate Helm, kubectl, Azure CLI, GCloud, AWS CLI, aws-iam-authenticator, kubelogin tool tests +- Migrate Azure App Service, AzureResourceGroup, AzureWebApp, GoogleCloudScripting cloud tests into `CloudIntegration/` +- Automated version-expansion scheduled job (`[Explicit]` `ToolVersionExpansionFixture`, `LatestVersionFinder`) +- TeamCity pipeline stage + Nuke build target for the external-tool test run From 405426f8e8e6b07ec96cb62a75415d7f6d7154f2 Mon Sep 17 00:00:00 2001 From: robert Date: Wed, 5 Aug 2026 11:10:32 +1000 Subject: [PATCH 02/16] Add implementation plan for external tool test separation (infra + Terraform) Task-by-task TDD plan reimplementing the tool-manifest/resolver/downloader mechanism against current main, plus the Terraform migration. Adds real unit coverage to the existing Calamari.Terraform.Tests fixtures (which already have internals access) instead of duplicating logic elsewhere. --- ...026-08-05-external-tool-test-separation.md | 1549 +++++++++++++++++ 1 file changed, 1549 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-05-external-tool-test-separation.md diff --git a/docs/superpowers/plans/2026-08-05-external-tool-test-separation.md b/docs/superpowers/plans/2026-08-05-external-tool-test-separation.md new file mode 100644 index 000000000..f43884b38 --- /dev/null +++ b/docs/superpowers/plans/2026-08-05-external-tool-test-separation.md @@ -0,0 +1,1549 @@ +# External Tool Test Separation (Infrastructure + Terraform) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Create a new `Calamari.ExternalTools.Tests` project with a Terraform-only tool-version manifest and shared download/resolution infrastructure, migrate Terraform's tool-dependent integration tests into it, and add real unit-test coverage for the Terraform CLI argument-construction logic that currently has no coverage — all without touching the in-place categorisation branches. + +**Architecture:** A single new NUnit test project reads the Terraform entry from a co-located `tool-manifest.json`. Shared infrastructure resolves the tool via env var override → PATH lookup → download-and-cache. The existing `Calamari.Terraform.Tests` project is trimmed (its large cloud/tool integration fixture moves out) but not deleted — three already-mocked unit fixtures stay there and gain new coverage. + +**Tech Stack:** C# / .NET 8.0, NUnit 3.14.0, FluentAssertions 7.2.0, NSubstitute 6.0.0, System.Text.Json 9.0.16 + +## Global Constraints + +- Package versions must match what's already pinned elsewhere in the repo: `FluentAssertions 7.2.0`, `NUnit 3.14.0`, `NUnit3TestAdapter 5.2.0`, `Microsoft.NET.Test.Sdk 18.0.0`, `TeamCity.VSTest.TestAdapter 1.0.41`, `System.Text.Json 9.0.16`. +- No `SharpCompress` dependency is added in this plan — Terraform only needs zip extraction (`System.IO.Compression.ZipFile`, already in the BCL). Tar.gz support is added when a tool that needs it (Helm, GCloud, ...) lands in a later branch. +- `tool-manifest.json` contains only a `terraform` entry. Other tools are added when their migration branch lands. +- **Deviation from the approved design spec:** the spec said new Terraform unit tests would live in `Calamari.Tests`. Investigation during planning found `Calamari.Terraform.Tests` already has real (not reimplemented) access to `Calamari.Terraform`'s internals via `InternalsVisibleTo`, and an existing mocked fixture (`TerraformCliExecutorFixture`) already exercises the executor via NSubstitute. Adding new tests there — against the real production methods — is stronger coverage than duplicating the logic in a separate project, and `Calamari.Terraform.Tests` already runs in the default (main) pipeline, so the "unit tests land in the main pipeline before integration tests are trimmed" requirement is still met. Task 7 below reflects this. + +--- + +### Task 1: Scaffold the `Calamari.ExternalTools.Tests` project + +**Files:** +- Create: `source/Calamari.ExternalTools.Tests/Calamari.ExternalTools.Tests.csproj` +- Create: `source/Calamari.ExternalTools.Tests/tool-manifest.json` +- Modify: `source/Calamari.sln` +- Modify: `source/Calamari.Terraform/Properties/InternalsVisibleTo.cs` + +- [ ] **Step 1: Create the project directory structure** + +```bash +mkdir -p source/Calamari.ExternalTools.Tests/Infrastructure/ToolStrategies +mkdir -p source/Calamari.ExternalTools.Tests/ExternalTools/Terraform +``` + +- [ ] **Step 2: Create the .csproj file** + +Create `source/Calamari.ExternalTools.Tests/Calamari.ExternalTools.Tests.csproj`: + +```xml + + + + Calamari.ExternalTools.Tests + Calamari.ExternalTools.Tests + win-x64;linux-x64;osx-x64;linux-arm;linux-arm64 + false + net8.0 + + CS8632 + true + + + + + + + + + + + + + + + + + + + + PreserveNewest + + + PreserveNewest + + + PreserveNewest + + + PreserveNewest + + + + +``` + +- [ ] **Step 3: Create the tool manifest** + +Create `source/Calamari.ExternalTools.Tests/tool-manifest.json`. The range matches `TerraformCliExecutor`'s own `supportedVersionRange` (`source/Calamari.Terraform/TerraformCliExecutor.cs:37`) — `0.13.7` inclusive to `1.9` exclusive: + +```json +{ + "tools": { + "terraform": { + "lowest": "0.13.7", + "highest": "1.8.5", + "source": "https://releases.hashicorp.com/terraform/", + "architectures": ["amd64", "arm64"] + } + } +} +``` + +- [ ] **Step 4: Add the project to the solution** + +```bash +cd source && dotnet sln Calamari.sln add Calamari.ExternalTools.Tests/Calamari.ExternalTools.Tests.csproj +``` + +- [ ] **Step 5: Grant the new project visibility into Calamari.Terraform's internals** + +`TerraformSpecialVariables` (`source/Calamari.Terraform/TerraformSpecialVariables.cs:6`) is an internal static class used throughout the Terraform test fixtures. Modify `source/Calamari.Terraform/Properties/InternalsVisibleTo.cs`: + +```csharp +using System.Runtime.CompilerServices; + +[assembly: InternalsVisibleTo("Calamari.Terraform.Tests")] +[assembly: InternalsVisibleTo("Calamari.ExternalTools.Tests")] +``` + +- [ ] **Step 6: Verify the solution builds** + +```bash +cd source && dotnet build Calamari.ExternalTools.Tests/Calamari.ExternalTools.Tests.csproj +``` + +Expected: Build succeeds with no errors (no source files yet, just the empty scaffold). + +- [ ] **Step 7: Commit** + +```bash +git add source/Calamari.ExternalTools.Tests/ source/Calamari.sln source/Calamari.Terraform/Properties/InternalsVisibleTo.cs +git commit -m "feat: scaffold Calamari.ExternalTools.Tests project with Terraform tool manifest" +``` + +--- + +### Task 2: Implement the manifest reader + +**Files:** +- Create: `source/Calamari.ExternalTools.Tests/Infrastructure/ToolManifest.cs` +- Create: `source/Calamari.ExternalTools.Tests/Infrastructure/ToolManifestTests.cs` + +**Interfaces:** +- Produces: `ToolManifest.Load()` → `ToolManifest`; `ToolManifest.GetTool(string name)` → `ToolDefinition?`; `ToolManifest.ToolNames` → `IReadOnlyCollection`; `ToolDefinition.Lowest`/`.Highest` → `Version`; `ToolDefinition.IsInRange(Version)` → `bool`. Task 3 (`ToolResolver`) and Task 6 (`ExternalToolFixture`) consume these. + +- [ ] **Step 1: Write the failing tests** + +Create `source/Calamari.ExternalTools.Tests/Infrastructure/ToolManifestTests.cs`: + +```csharp +using FluentAssertions; +using NUnit.Framework; + +namespace Calamari.ExternalTools.Tests.Infrastructure +{ + [TestFixture] + public class ToolManifestTests + { + [Test] + public void ShouldLoadManifestFromEmbeddedFile() + { + var manifest = ToolManifest.Load(); + + manifest.Should().NotBeNull(); + manifest.GetTool("terraform").Should().NotBeNull(); + manifest.GetTool("terraform")!.Lowest.ToString().Should().Be("0.13.7"); + manifest.GetTool("terraform")!.Highest.ToString().Should().Be("1.8.5"); + } + + [Test] + public void ShouldReturnNullForUnknownTool() + { + var manifest = ToolManifest.Load(); + + manifest.GetTool("nonexistent-tool").Should().BeNull(); + } + + [Test] + public void ShouldCheckVersionIsInRange() + { + var manifest = ToolManifest.Load(); + var terraform = manifest.GetTool("terraform")!; + + terraform.IsInRange(new System.Version(1, 0, 0)).Should().BeTrue(); + terraform.IsInRange(new System.Version(0, 12, 0)).Should().BeFalse(); + terraform.IsInRange(new System.Version(2, 0, 0)).Should().BeFalse(); + } + + [Test] + public void ShouldListAllTools() + { + var manifest = ToolManifest.Load(); + + manifest.ToolNames.Should().Contain("terraform"); + manifest.ToolNames.Should().HaveCount(1); + } + } +} +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +```bash +cd source && dotnet test Calamari.ExternalTools.Tests/ --filter "FullyQualifiedName~ToolManifestTests" -v minimal +``` + +Expected: Compilation error — `ToolManifest` does not exist. + +- [ ] **Step 3: Implement the manifest reader** + +Create `source/Calamari.ExternalTools.Tests/Infrastructure/ToolManifest.cs`: + +```csharp +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text.Json; +using System.Text.Json.Serialization; +using Calamari.Testing.Helpers; + +namespace Calamari.ExternalTools.Tests.Infrastructure +{ + public class ToolManifest + { + readonly Dictionary tools; + + ToolManifest(Dictionary tools) + { + this.tools = tools; + } + + public IReadOnlyCollection ToolNames => tools.Keys.ToList(); + + public ToolDefinition? GetTool(string name) + { + return tools.TryGetValue(name, out var tool) ? tool : null; + } + + public static ToolManifest Load() + { + var manifestPath = Path.Combine(TestEnvironment.CurrentWorkingDirectory, "tool-manifest.json"); + var json = File.ReadAllText(manifestPath); + var doc = JsonSerializer.Deserialize(json) + ?? throw new InvalidOperationException("Failed to deserialize tool-manifest.json"); + + var tools = new Dictionary(); + foreach (var (name, entry) in doc.Tools) + { + tools[name] = new ToolDefinition( + name, + ParseVersion(entry.Lowest), + ParseVersion(entry.Highest), + entry.Source, + entry.Architectures); + } + + return new ToolManifest(tools); + } + + static Version ParseVersion(string version) + { + var clean = version.TrimStart('v'); + return Version.Parse(clean); + } + + class ManifestDocument + { + [JsonPropertyName("tools")] + public Dictionary Tools { get; set; } = new(); + } + + class ManifestEntry + { + [JsonPropertyName("lowest")] + public string Lowest { get; set; } = ""; + + [JsonPropertyName("highest")] + public string Highest { get; set; } = ""; + + [JsonPropertyName("source")] + public string Source { get; set; } = ""; + + [JsonPropertyName("architectures")] + public string[] Architectures { get; set; } = Array.Empty(); + } + } + + public class ToolDefinition + { + public ToolDefinition(string name, Version lowest, Version highest, string source, string[] architectures) + { + Name = name; + Lowest = lowest; + Highest = highest; + Source = source; + Architectures = architectures; + } + + public string Name { get; } + public Version Lowest { get; } + public Version Highest { get; } + public string Source { get; } + public string[] Architectures { get; } + + public bool IsInRange(Version version) + { + return version >= Lowest && version <= Highest; + } + } +} +``` + +- [ ] **Step 4: Run the tests to verify they pass** + +```bash +cd source && dotnet test Calamari.ExternalTools.Tests/ --filter "FullyQualifiedName~ToolManifestTests" -v minimal +``` + +Expected: All 4 tests pass. + +- [ ] **Step 5: Commit** + +```bash +git add source/Calamari.ExternalTools.Tests/Infrastructure/ToolManifest* +git commit -m "feat: implement ToolManifest reader with version range support" +``` + +--- + +### Task 3: Implement the tool resolver + +**Files:** +- Create: `source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolver.cs` +- Create: `source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolverTests.cs` + +**Interfaces:** +- Consumes: `ToolManifest.GetTool(string)` (Task 2). +- Produces: `new ToolResolver(ToolManifest, Action log)`; `.ResolveVersion(string toolName)` → `string`; `static ToolResolver.GetOverrideEnvVar(string toolName)` → `string`; `static ToolResolver.FindOnPath(string toolName)` → `string?`; `static ToolResolver.GetInstalledVersion(string executablePath, string versionArg = "--version")` → `string?`. Task 6 (`ExternalToolFixture`) consumes `ResolveVersion` and `FindOnPath`. + +- [ ] **Step 1: Write the failing tests** + +Create `source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolverTests.cs`: + +```csharp +using FluentAssertions; +using NUnit.Framework; + +namespace Calamari.ExternalTools.Tests.Infrastructure +{ + [TestFixture] + public class ToolResolverTests + { + [Test] + public void ShouldBuildEnvironmentVariableOverrideName() + { + var envVarName = ToolResolver.GetOverrideEnvVar("terraform"); + envVarName.Should().Be("CALAMARI_TOOL_TERRAFORM_VERSION"); + } + + [Test] + public void ShouldResolveToManifestHighestWhenNoOverrideSet() + { + var manifest = ToolManifest.Load(); + var resolver = new ToolResolver(manifest, _ => { }); + + var version = resolver.ResolveVersion("terraform"); + + version.Should().Be("1.8.5"); + } + + [Test] + public void ShouldDetectToolOnPath() + { + // 'dotnet' is always on PATH in a .NET test run + var result = ToolResolver.FindOnPath("dotnet"); + result.Should().NotBeNullOrEmpty(); + } + + [Test] + public void ShouldReturnNullForToolNotOnPath() + { + var result = ToolResolver.FindOnPath("definitely-not-a-real-tool-abc123"); + result.Should().BeNull(); + } + } +} +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +```bash +cd source && dotnet test Calamari.ExternalTools.Tests/ --filter "FullyQualifiedName~ToolResolverTests" -v minimal +``` + +Expected: Compilation error — `ToolResolver` does not exist. + +- [ ] **Step 3: Implement the tool resolver** + +Create `source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolver.cs`: + +```csharp +using System; +using System.Text; +using Calamari.Common.Features.Processes; +using Calamari.Common.Plumbing; + +namespace Calamari.ExternalTools.Tests.Infrastructure +{ + /// + /// Resolves the version to use for a tool: env var override, else the manifest's highest. + /// Resolution of the actual executable (PATH vs download) is done by ExternalToolFixture. + /// + public class ToolResolver + { + readonly ToolManifest manifest; + readonly Action log; + + public ToolResolver(ToolManifest manifest, Action log) + { + this.manifest = manifest; + this.log = log; + } + + public static string GetOverrideEnvVar(string toolName) + { + return $"CALAMARI_TOOL_{toolName.Replace("-", "_").ToUpperInvariant()}_VERSION"; + } + + public string ResolveVersion(string toolName) + { + var envVar = GetOverrideEnvVar(toolName); + var overrideVersion = Environment.GetEnvironmentVariable(envVar); + + if (!string.IsNullOrEmpty(overrideVersion)) + { + log($"Using override version {overrideVersion} for {toolName} (from {envVar})"); + return overrideVersion; + } + + var tool = manifest.GetTool(toolName); + if (tool == null) + throw new InvalidOperationException($"Tool '{toolName}' not found in manifest"); + + return tool.Highest.ToString(); + } + + public static string? FindOnPath(string toolName) + { + try + { + var command = CalamariEnvironment.IsRunningOnWindows ? "where" : "which"; + var executableName = CalamariEnvironment.IsRunningOnWindows + ? $"{toolName}.exe" + : toolName; + + var stdOut = new StringBuilder(); + var result = SilentProcessRunner.ExecuteCommand( + command, + executableName, + ".", + s => stdOut.AppendLine(s), + _ => { }); + + if (result.ExitCode == 0) + { + var path = stdOut.ToString().Trim().Split(new[] { '\r', '\n' }, StringSplitOptions.RemoveEmptyEntries); + return path.Length > 0 ? path[0] : null; + } + } + catch + { + // Tool not found + } + + return null; + } + + public static string? GetInstalledVersion(string executablePath, string versionArg = "--version") + { + try + { + var stdOut = new StringBuilder(); + var result = SilentProcessRunner.ExecuteCommand( + executablePath, + versionArg, + ".", + s => stdOut.AppendLine(s), + _ => { }); + + return result.ExitCode == 0 ? stdOut.ToString().Trim() : null; + } + catch + { + return null; + } + } + } +} +``` + +- [ ] **Step 4: Run the tests to verify they pass** + +```bash +cd source && dotnet test Calamari.ExternalTools.Tests/ --filter "FullyQualifiedName~ToolResolverTests" -v minimal +``` + +Expected: All 4 tests pass. + +- [ ] **Step 5: Commit** + +```bash +git add source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolver* +git commit -m "feat: implement ToolResolver with env var override and manifest fallback" +``` + +--- + +### Task 4: Implement the tool downloader + +**Files:** +- Create: `source/Calamari.ExternalTools.Tests/Infrastructure/ToolDownloader.cs` + +**Interfaces:** +- Produces: `new ToolDownloader(Action log)`; `.Download(string toolName, string version, Func> downloadAction)` → `Task` (the resolved executable path); `static ToolDownloader.DownloadFile(url, destinationPath, client)`; `static ToolDownloader.DownloadAndExtractZip(url, destinationDir, client)`; `static ToolDownloader.GetPlatform()` → `"windows"|"darwin"|"linux"`; `static ToolDownloader.GetArchitecture()` → `"amd64"|"arm64"`. Task 5 (`TerraformStrategy`) consumes `DownloadAndExtractZip`, `GetPlatform`, `GetArchitecture`. Task 6 (`ExternalToolFixture`) consumes `Download`. + +No SharpCompress/tar.gz support here — Terraform only needs zip. Add tar.gz extraction when a tool that needs it lands. + +- [ ] **Step 1: Implement the downloader** + +Create `source/Calamari.ExternalTools.Tests/Infrastructure/ToolDownloader.cs`: + +```csharp +using System; +using System.IO; +using System.IO.Compression; +using System.Linq; +using System.Net.Http; +using System.Threading.Tasks; +using Calamari.Common.Features.Processes; +using Calamari.Common.Plumbing; +using Calamari.Common.Plumbing.Retry; +using Calamari.Testing.Helpers; + +namespace Calamari.ExternalTools.Tests.Infrastructure +{ + /// + /// Downloads and caches external tool binaries. + /// Cache location: {TestOutputDir}/Tools/{toolName}/{version}/ + /// + public class ToolDownloader + { + readonly Action log; + + public ToolDownloader(Action log) + { + this.log = log; + } + + public async Task Download(string toolName, string version, Func> downloadAction) + { + var destinationDir = TestEnvironment.GetTestPath("Tools", toolName, version); + + var existing = FindExistingExecutable(toolName, destinationDir); + if (existing != null) + { + log($"Using cached {toolName} {version} at {existing}"); + return existing; + } + + log($"Downloading {toolName} {version}..."); + Directory.CreateDirectory(destinationDir); + + var retry = new RetryTracker(4, TimeSpan.MaxValue, new LimitedExponentialRetryInterval(3000, 30000, 2)); + string? executablePath = null; + + while (retry.Try()) + { + try + { + using var client = CreateHttpClient(); + executablePath = await downloadAction(destinationDir, version, client); + AddExecutePermission(executablePath); + break; + } + catch + { + if (!retry.CanRetry()) + throw; + + await Task.Delay(retry.Sleep()); + } + } + + log($"Downloaded {toolName} {version} to {executablePath}"); + return executablePath!; + } + + string? FindExistingExecutable(string toolName, string destinationDir) + { + if (!Directory.Exists(destinationDir)) + return null; + + var path = Directory.EnumerateFiles(destinationDir, "*", SearchOption.AllDirectories) + .FirstOrDefault(f => + { + var name = Path.GetFileNameWithoutExtension(f).ToLowerInvariant(); + return name.Contains(toolName.ToLowerInvariant().Replace("-", "")); + }); + + return path != null && File.Exists(path) ? path : null; + } + + public static async Task DownloadFile(string url, string destinationPath, HttpClient client) + { + using var fileStream = new FileStream(destinationPath, FileMode.Create, FileAccess.Write, FileShare.None); + using var stream = await client.GetStreamAsync(url); + await stream.CopyToAsync(fileStream); + } + + public static async Task DownloadAndExtractZip(string url, string destinationDir, HttpClient client) + { + var tempPath = Path.Combine(Path.GetTempPath(), $"{Guid.NewGuid()}.zip"); + try + { + await DownloadFile(url, tempPath, client); + ZipFile.ExtractToDirectory(tempPath, destinationDir); + } + finally + { + if (File.Exists(tempPath)) + File.Delete(tempPath); + } + } + + static void AddExecutePermission(string exePath) + { + if (CalamariEnvironment.IsRunningOnWindows || string.IsNullOrEmpty(exePath)) + return; + + SilentProcessRunner.ExecuteCommand( + "chmod", $"+x {exePath}", + Path.GetDirectoryName(exePath) ?? ".", + _ => { }, _ => { }); + } + + static HttpClient CreateHttpClient() + { + var client = new HttpClient(); + client.DefaultRequestHeaders.UserAgent.ParseAdd( + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"); + return client; + } + + public static string GetPlatform() + { + if (CalamariEnvironment.IsRunningOnWindows) return "windows"; + if (CalamariEnvironment.IsRunningOnMac) return "darwin"; + return "linux"; + } + + public static string GetArchitecture() + { + return System.Runtime.InteropServices.RuntimeInformation.OSArchitecture switch + { + System.Runtime.InteropServices.Architecture.Arm64 => "arm64", + _ => "amd64" + }; + } + } +} +``` + +- [ ] **Step 2: Verify the project builds** + +```bash +cd source && dotnet build Calamari.ExternalTools.Tests/Calamari.ExternalTools.Tests.csproj +``` + +Expected: Build succeeds. + +- [ ] **Step 3: Commit** + +```bash +git add source/Calamari.ExternalTools.Tests/Infrastructure/ToolDownloader.cs +git commit -m "feat: implement ToolDownloader with retry, caching, and platform detection" +``` + +--- + +### Task 5: Create the Terraform download strategy + +**Files:** +- Create: `source/Calamari.ExternalTools.Tests/Infrastructure/ToolStrategies/TerraformStrategy.cs` + +**Interfaces:** +- Consumes: `ToolDownloader.GetPlatform()`, `ToolDownloader.GetArchitecture()`, `ToolDownloader.DownloadAndExtractZip` (Task 4). +- Produces: `static TerraformStrategy.Download(string destinationDir, string version, HttpClient client)` → `Task`. Task 6/8 consume this as the fixture's download delegate. + +- [ ] **Step 1: Create the Terraform download strategy** + +Create `source/Calamari.ExternalTools.Tests/Infrastructure/ToolStrategies/TerraformStrategy.cs`: + +```csharp +using System.IO; +using System.Linq; +using System.Net.Http; +using System.Threading.Tasks; + +namespace Calamari.ExternalTools.Tests.Infrastructure.ToolStrategies +{ + public static class TerraformStrategy + { + public static async Task Download(string destinationDir, string version, HttpClient client) + { + var platform = ToolDownloader.GetPlatform(); + var arch = ToolDownloader.GetArchitecture(); + var fileName = $"terraform_{version}_{platform}_{arch}.zip"; + var url = $"https://releases.hashicorp.com/terraform/{version}/{fileName}"; + + await ToolDownloader.DownloadAndExtractZip(url, destinationDir, client); + + return Directory.EnumerateFiles(destinationDir) + .First(f => Path.GetFileName(f).Contains("terraform")); + } + } +} +``` + +- [ ] **Step 2: Verify the project builds** + +```bash +cd source && dotnet build Calamari.ExternalTools.Tests/Calamari.ExternalTools.Tests.csproj +``` + +- [ ] **Step 3: Commit** + +```bash +git add source/Calamari.ExternalTools.Tests/Infrastructure/ToolStrategies/ +git commit -m "feat: add Terraform download strategy" +``` + +--- + +### Task 6: Create the ExternalToolFixture base class + +**Files:** +- Create: `source/Calamari.ExternalTools.Tests/Infrastructure/ExternalToolFixture.cs` + +**Interfaces:** +- Consumes: `ToolManifest.Load()` (Task 2), `ToolResolver` (Task 3), `ToolDownloader` (Task 4). +- Produces: abstract base with `protected string ToolExecutablePath { get; }`, `protected string ToolVersion { get; }`, `protected abstract string PrimaryToolName { get; }`, `protected abstract Task DownloadTool(string destinationDir, string version, HttpClient client)`. Task 8's `TerraformCommandsFixture` derives from this. + +- [ ] **Step 1: Create the base fixture** + +Create `source/Calamari.ExternalTools.Tests/Infrastructure/ExternalToolFixture.cs`: + +```csharp +using System.Net.Http; +using System.Threading.Tasks; +using NUnit.Framework; + +namespace Calamari.ExternalTools.Tests.Infrastructure +{ + /// + /// Base class for test fixtures that depend on an external tool. + /// Resolves the tool via: env var override -> PATH -> download. + /// Subclasses set PrimaryToolName and provide a download strategy. + /// + public abstract class ExternalToolFixture + { + static readonly ToolManifest Manifest = ToolManifest.Load(); + + protected string ToolExecutablePath { get; private set; } = ""; + protected string ToolVersion { get; private set; } = ""; + + protected abstract string PrimaryToolName { get; } + + protected abstract Task DownloadTool(string destinationDir, string version, HttpClient client); + + [OneTimeSetUp] + public async Task ResolveTool() + { + var resolver = new ToolResolver(Manifest, Log); + var downloader = new ToolDownloader(Log); + + ToolVersion = resolver.ResolveVersion(PrimaryToolName); + + var pathResult = ToolResolver.FindOnPath(PrimaryToolName); + if (pathResult != null) + { + Log($"Found {PrimaryToolName} on PATH at {pathResult}"); + ToolExecutablePath = pathResult; + return; + } + + ToolExecutablePath = await downloader.Download(PrimaryToolName, ToolVersion, DownloadTool); + } + + protected void Log(string message) + { + TestContext.Progress.WriteLine($"[{PrimaryToolName}] {message}"); + } + } +} +``` + +- [ ] **Step 2: Verify the project builds** + +```bash +cd source && dotnet build Calamari.ExternalTools.Tests/Calamari.ExternalTools.Tests.csproj +``` + +- [ ] **Step 3: Commit** + +```bash +git add source/Calamari.ExternalTools.Tests/Infrastructure/ExternalToolFixture.cs +git commit -m "feat: add ExternalToolFixture base class for tool-dependent tests" +``` + +--- + +### Task 7: Add real unit-test coverage for Terraform CLI argument construction + +**Files:** +- Modify: `source/Calamari.Terraform/Behaviours/TerraformDeployBehaviour.cs:62` (visibility bump only) +- Modify: `source/Calamari.Terraform.Tests/TerraformCliExecutorFixture.cs` + +This closes the "logic gaps" the previous effort identified before removing the Terraform integration fixture: init command construction (the `-get-plugins` flag, version-gated at 0.15.0) and version-range checking (the "untested version" warning). `TerraformCliExecutorFixture` already tests `TerraformVariableFiles` (var-file args) directly against the real `TerraformCliExecutor` via NSubstitute — these new tests follow the exact same pattern, no test doubles for the logic itself. + +**Interfaces:** +- Consumes: `TerraformCliExecutor` (existing, `source/Calamari.Terraform/TerraformCliExecutor.cs`), `TerraformDeployBehaviour.GetEnvironmentVariableArgs` (visibility bumped in Step 1). + +- [ ] **Step 1: Bump `GetEnvironmentVariableArgs` from private to internal** + +In `source/Calamari.Terraform/Behaviours/TerraformDeployBehaviour.cs:62`, change: + +```csharp + static Dictionary GetEnvironmentVariableArgs(IVariables variables) +``` + +to: + +```csharp + internal static Dictionary GetEnvironmentVariableArgs(IVariables variables) +``` + +This is a visibility-only change — no logic changes, no other call sites are affected. `Calamari.Terraform.Tests` already has `InternalsVisibleTo` access to `Calamari.Terraform`. + +- [ ] **Step 2: Verify the project still builds** + +```bash +cd source && dotnet build Calamari.Terraform/Calamari.Terraform.csproj +``` + +- [ ] **Step 3: Write the failing tests** + +Add to `source/Calamari.Terraform.Tests/TerraformCliExecutorFixture.cs`, inside the existing `TerraformCliExecutorFixture` class (after `InitializePlugins_ThrowsAfterRetriesExhausted`): + +```csharp + [Test] + public void InitCommand_PreV015_IncludesGetPluginsFlagTrue() + { + var capturedArguments = new List(); + var testVariables = Substitute.For(); + testVariables.GetStrings(KnownVariables.EnabledFeatureToggles).Returns(new List()); + testVariables.GetFlag(TerraformSpecialVariables.Action.Terraform.AllowPluginDownloads, true).Returns(true); + + var commandLineRunner = Substitute.For(); + commandLineRunner.Execute(Arg.Do(invocation => + { + capturedArguments.Add(invocation.Arguments); + if (capturedArguments.Count == 1) + invocation.AdditionalInvocationOutputSink.WriteInfo("Terraform v0.14.0"); + })).Returns(new CommandResult("terraform", 0)); + + new TerraformCliExecutor(Substitute.For(), Substitute.For(), commandLineRunner, new RunningDeployment("blah", testVariables), new Dictionary()); + + capturedArguments[1].Should().Contain("-get-plugins=true"); + } + + [Test] + public void InitCommand_PreV015_PluginDownloadsDisabled_IncludesGetPluginsFlagFalse() + { + var capturedArguments = new List(); + var testVariables = Substitute.For(); + testVariables.GetStrings(KnownVariables.EnabledFeatureToggles).Returns(new List()); + testVariables.GetFlag(TerraformSpecialVariables.Action.Terraform.AllowPluginDownloads, true).Returns(false); + + var commandLineRunner = Substitute.For(); + commandLineRunner.Execute(Arg.Do(invocation => + { + capturedArguments.Add(invocation.Arguments); + if (capturedArguments.Count == 1) + invocation.AdditionalInvocationOutputSink.WriteInfo("Terraform v0.14.0"); + })).Returns(new CommandResult("terraform", 0)); + + new TerraformCliExecutor(Substitute.For(), Substitute.For(), commandLineRunner, new RunningDeployment("blah", testVariables), new Dictionary()); + + capturedArguments[1].Should().Contain("-get-plugins=false"); + } + + [Test] + public void InitCommand_V015AndAbove_OmitsGetPluginsFlag() + { + var capturedArguments = new List(); + var testVariables = Substitute.For(); + testVariables.GetStrings(KnownVariables.EnabledFeatureToggles).Returns(new List()); + + var commandLineRunner = Substitute.For(); + commandLineRunner.Execute(Arg.Do(invocation => + { + capturedArguments.Add(invocation.Arguments); + if (capturedArguments.Count == 1) + invocation.AdditionalInvocationOutputSink.WriteInfo("Terraform v0.15.0"); + })).Returns(new CommandResult("terraform", 0)); + + new TerraformCliExecutor(Substitute.For(), Substitute.For(), commandLineRunner, new RunningDeployment("blah", testVariables), new Dictionary()); + + capturedArguments[1].Should().NotContain("-get-plugins"); + } + + [Test] + public void InitCommand_IncludesAdditionalInitParams() + { + var capturedArguments = new List(); + var testVariables = Substitute.For(); + testVariables.GetStrings(KnownVariables.EnabledFeatureToggles).Returns(new List()); + testVariables.Get(TerraformSpecialVariables.Action.Terraform.AdditionalInitParams).Returns("-backend-config=\"key=value\""); + + var commandLineRunner = Substitute.For(); + commandLineRunner.Execute(Arg.Do(invocation => + { + capturedArguments.Add(invocation.Arguments); + if (capturedArguments.Count == 1) + invocation.AdditionalInvocationOutputSink.WriteInfo("Terraform v1.0.0"); + })).Returns(new CommandResult("terraform", 0)); + + new TerraformCliExecutor(Substitute.For(), Substitute.For(), commandLineRunner, new RunningDeployment("blah", testVariables), new Dictionary()); + + capturedArguments[1].Should().Contain("-backend-config=\"key=value\""); + capturedArguments[1].Should().NotContain("-get-plugins"); + } + + [Test] + public void UntestedVersion_AboveSupportedRange_LogsInfoOnSuccessfulCommand() + { + var log = Substitute.For(); + var testVariables = Substitute.For(); + testVariables.GetStrings(KnownVariables.EnabledFeatureToggles).Returns(new List()); + + var commandLineRunner = Substitute.For(); + var callCount = 0; + commandLineRunner.Execute(Arg.Do(invocation => + { + callCount++; + if (callCount == 1) + invocation.AdditionalInvocationOutputSink.WriteInfo("Terraform v2.0.0"); + })).Returns(new CommandResult("terraform", 0)); + + var executor = new TerraformCliExecutor(log, Substitute.For(), commandLineRunner, new RunningDeployment("blah", testVariables), new Dictionary()); + executor.ExecuteCommand("plan"); + + log.Received(1).Info(Arg.Is(s => s.Contains("has not been tested"))); + } + + [Test] + public void SupportedVersion_WithinRange_DoesNotLogUntestedMessage() + { + var log = Substitute.For(); + var testVariables = Substitute.For(); + testVariables.GetStrings(KnownVariables.EnabledFeatureToggles).Returns(new List()); + + var commandLineRunner = Substitute.For(); + var callCount = 0; + commandLineRunner.Execute(Arg.Do(invocation => + { + callCount++; + if (callCount == 1) + invocation.AdditionalInvocationOutputSink.WriteInfo("Terraform v1.0.0"); + })).Returns(new CommandResult("terraform", 0)); + + var executor = new TerraformCliExecutor(log, Substitute.For(), commandLineRunner, new RunningDeployment("blah", testVariables), new Dictionary()); + executor.ExecuteCommand("plan"); + + log.DidNotReceive().Info(Arg.Is(s => s.Contains("has not been tested"))); + log.DidNotReceive().Warn(Arg.Is(s => s.Contains("has not been tested"))); + } + + [Test] + public void EnvironmentVariables_ParsedFromJson() + { + var variables = new CalamariVariables(); + variables.Set(TerraformSpecialVariables.Action.Terraform.EnvironmentVariables, + JsonConvert.SerializeObject(new Dictionary { { "TF_VAR_ami", "test-value" }, { "TF_LOG", "DEBUG" } })); + + var result = TerraformDeployBehaviour.GetEnvironmentVariableArgs(variables); + + result.Should().ContainKey("TF_VAR_ami").WhoseValue.Should().Be("test-value"); + result.Should().ContainKey("TF_LOG").WhoseValue.Should().Be("DEBUG"); + } + + [Test] + public void EnvironmentVariables_NotSet_ReturnsEmptyDictionary() + { + var variables = new CalamariVariables(); + + var result = TerraformDeployBehaviour.GetEnvironmentVariableArgs(variables); + + result.Should().BeEmpty(); + } +``` + +Add these `using` statements to the top of `source/Calamari.Terraform.Tests/TerraformCliExecutorFixture.cs` (alongside the existing ones): + +```csharp +using Calamari.Terraform.Behaviours; +using Newtonsoft.Json; +``` + +- [ ] **Step 4: Run the tests to verify they fail** + +```bash +cd source && dotnet test Calamari.Terraform.Tests/ --filter "FullyQualifiedName~TerraformCliExecutorFixture" -v minimal +``` + +Expected: Compilation error — `TerraformDeployBehaviour.GetEnvironmentVariableArgs` inaccessible until Step 1 is applied; new test methods fail to compile/run until present. + +- [ ] **Step 5: Run the tests to verify they pass** + +```bash +cd source && dotnet test Calamari.Terraform.Tests/ --filter "FullyQualifiedName~TerraformCliExecutorFixture" -v minimal +``` + +Expected: All tests pass (existing 8 + 8 new = 16). + +- [ ] **Step 6: Commit** + +```bash +git add source/Calamari.Terraform/Behaviours/TerraformDeployBehaviour.cs source/Calamari.Terraform.Tests/TerraformCliExecutorFixture.cs +git commit -m "test: add unit coverage for Terraform init command construction and version range checks" +``` + +--- + +### Task 8: Migrate the Terraform integration tests into `Calamari.ExternalTools.Tests` + +**Files:** +- Create: `source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/TerraformCommandsFixture.cs` +- Move: `source/Calamari.Terraform.Tests/{AWS,Azure,GoogleCloud,PlanDetailedExitCode,Simple,WithOutputSensitiveVariables,WithVariablesSubstitution}/*` → `source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/{same}/*` +- Move: `source/Calamari.Terraform.Tests/CommonTemplates/SingleVariable.json` → `source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/CommonTemplates/SingleVariable.json` +- Delete: `source/Calamari.Terraform.Tests/CommandsFixture.cs` +- Delete: `source/Calamari.Terraform.Tests/{AdditionalParams,TemplateDirectory,WithVariables}/` (unused once `CommandsFixture.cs` is gone) +- Delete: `source/Calamari.Terraform.Tests/CommonTemplates/{HclWithVariables.hcl,InlineJsonWithVariables.json,TemplateLoader.cs}` (only `SingleVariable.json` is used by the migrated tests; the rest backed tests that were dropped, not migrated) + +This keeps one end-to-end test, the wiring tests that catch pipeline/DI regressions unit tests can't, and the 3 cloud tests — matching what was already validated for this exact migration on `feature/external-tool-test-separation`. `Calamari.Terraform.Tests` (`CommandResolutionTests.cs`, `TerraformCliExecutorFixture.cs`, `TerraformPlanVariableFixture.cs`) stays in the solution and in the default pipeline. + +**Interfaces:** +- Consumes: `ExternalToolFixture` (Task 6), `TerraformStrategy.Download` (Task 5), `TestEnvironment.GetTestPath` (`Calamari.Testing.Helpers`), `CommandTestBuilder` (`Calamari.Testing`, already used by the original fixture — no new dependency). + +- [ ] **Step 1: Move the Terraform resource directories** + +```bash +mkdir -p source/Calamari.ExternalTools.Tests/ExternalTools/Terraform +git mv source/Calamari.Terraform.Tests/AWS source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/AWS +git mv source/Calamari.Terraform.Tests/Azure source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/Azure +git mv source/Calamari.Terraform.Tests/GoogleCloud source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/GoogleCloud +git mv source/Calamari.Terraform.Tests/PlanDetailedExitCode source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/PlanDetailedExitCode +git mv source/Calamari.Terraform.Tests/Simple source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/Simple +git mv source/Calamari.Terraform.Tests/WithOutputSensitiveVariables source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/WithOutputSensitiveVariables +git mv source/Calamari.Terraform.Tests/WithVariablesSubstitution source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/WithVariablesSubstitution +mkdir -p source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/CommonTemplates +git mv source/Calamari.Terraform.Tests/CommonTemplates/SingleVariable.json source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/CommonTemplates/SingleVariable.json +``` + +- [ ] **Step 2: Delete resources that only backed dropped tests** + +```bash +git rm -r source/Calamari.Terraform.Tests/AdditionalParams +git rm -r source/Calamari.Terraform.Tests/TemplateDirectory +git rm -r source/Calamari.Terraform.Tests/WithVariables +git rm source/Calamari.Terraform.Tests/CommonTemplates/HclWithVariables.hcl +git rm source/Calamari.Terraform.Tests/CommonTemplates/InlineJsonWithVariables.json +git rm source/Calamari.Terraform.Tests/CommonTemplates/TemplateLoader.cs +``` + +- [ ] **Step 3: Create the migrated fixture** + +Create `source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/TerraformCommandsFixture.cs`: + +```csharp +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Net; +using System.Net.Http; +using System.Net.Sockets; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Calamari.CloudAccounts; +using Calamari.Common.Plumbing.FileSystem; +using Calamari.Common.Plumbing.Variables; +using Calamari.ExternalTools.Tests.Infrastructure; +using Calamari.ExternalTools.Tests.Infrastructure.ToolStrategies; +using Calamari.Terraform.Commands; +using Calamari.Testing; +using Calamari.Testing.Azure; +using Calamari.Testing.Helpers; +using FluentAssertions; +using Newtonsoft.Json.Linq; +using NUnit.Framework; + +namespace Calamari.ExternalTools.Tests.ExternalTools.Terraform +{ + [TestFixture] + [Category("ExternalTool")] + public class TerraformCommandsFixture : ExternalToolFixture + { + protected override string PrimaryToolName => "terraform"; + + protected override Task DownloadTool(string destinationDir, string version, HttpClient client) + => TerraformStrategy.Download(destinationDir, version, client); + + readonly string planCommand = GetCommandFromType(typeof(PlanCommand)); + readonly string applyCommand = GetCommandFromType(typeof(ApplyCommand)); + readonly string destroyCommand = GetCommandFromType(typeof(DestroyCommand)); + + const string ResourceRoot = "ExternalTools/Terraform"; + + static string GetTestResourcePath(string relativePath) + => TestEnvironment.GetTestPath(ResourceRoot, relativePath); + + static string LoadTextTemplate(string templateName) + => File.ReadAllText(GetTestResourcePath(Path.Combine("CommonTemplates", templateName))); + + [OneTimeTearDown] + public static void OneTimeTearDown() + { + ClearTestDirectories(); + } + + static void ClearTestDirectories() + { + static void TryDeleteFile(string path) + { + try { File.Delete(path); } + catch (IOException) { } + } + + static void TryDeleteDirectory(string path, bool recursive) + { + try { Directory.Delete(path, recursive); } + catch (IOException) { } + } + + static void ClearTerraformDirectory(string directory) + { + var fullPath = GetTestResourcePath(directory); + TryDeleteFile(Path.Combine(fullPath, "terraform.tfstate")); + TryDeleteFile(Path.Combine(fullPath, "terraform.tfstate.backup")); + TryDeleteFile(Path.Combine(fullPath, "terraform.log")); + TryDeleteDirectory(Path.Combine(fullPath, ".terraform"), true); + TryDeleteDirectory(Path.Combine(fullPath, "terraform.tfstate.d"), true); + TryDeleteDirectory(Path.Combine(fullPath, "terraformplugins"), true); + } + + ClearTerraformDirectory("AWS"); + ClearTerraformDirectory("Azure"); + ClearTerraformDirectory("GoogleCloud"); + ClearTerraformDirectory("PlanDetailedExitCode"); + ClearTerraformDirectory("Simple"); + ClearTerraformDirectory("WithOutputSensitiveVariables"); + ClearTerraformDirectory("WithVariablesSubstitution"); + } + + /// Single end-to-end test validating the pipeline works with the manifest's Terraform version. + [Test] + public void ApplySimple_Succeeds() + { + ExecuteAndReturnLogOutput(applyCommand, _ => { }, "Simple") + .Should() + .NotContain("Error"); + } + + [Test] + public void InlineJsonTemplate_ProducesExpectedOutput() + { + string template = LoadTextTemplate("SingleVariable.json"); + var randomNumber = new Random().Next().ToString(); + + ExecuteAndReturnLogOutput(applyCommand, + _ => + { + _.Variables.Add("RandomNumber", randomNumber); + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.Template, template); + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.TemplateParameters, "{\"ami\":\"test-value\"}"); + _.Variables.Add(ScriptVariables.ScriptSource, ScriptVariables.ScriptSourceOptions.Inline); + }, + String.Empty, + _ => + { + _.OutputVariables.ContainsKey("TerraformValueOutputs[ami]").Should().BeTrue(); + _.OutputVariables["TerraformValueOutputs[ami]"].Value.Should().Be("test-value"); + }); + } + + /// Wiring test: Octostache substitution runs on variable files before terraform uses them. + [Test] + public void OutputAndSubstituteOctopusVariables() + { + ExecuteAndReturnLogOutput(applyCommand, + _ => + { + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.VarFiles, "example.txt"); + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.FileSubstitution, "example.txt"); + _.Variables.Add("Octopus.Action.StepName", "Step Name"); + _.Variables.Add("Should_Be_Substituted", "Hello World"); + _.Variables.Add("Should_Be_Substituted_in_txt", "Hello World from text"); + }, + "WithVariablesSubstitution", + result => + { + result.OutputVariables["TerraformValueOutputs[my_output]"].Value.Should().Be("Hello World"); + result.OutputVariables["TerraformValueOutputs[my_output_from_txt_file]"].Value.Should().Be("Hello World from text"); + }); + } + + /// Wiring test: terraform's sensitive outputs are marked IsSensitive in Calamari's output variables. + [Test] + public void WithOutputSensitiveVariables() + { + ExecuteAndReturnLogOutput(applyCommand, + _ => { }, + "WithOutputSensitiveVariables", + result => result.OutputVariables.Values.Should().OnlyContain(variable => variable.IsSensitive)); + } + + /// Wiring test: plan -> apply -> plan cycle with state file management (exit code 2 = changes, 0 = no changes). + [Test] + public async Task PlanDetailedExitCode() + { + using var stateFileFolder = TemporaryDirectory.Create(); + + var output = await ExecuteAndReturnResult(planCommand, PopulateVariables, "PlanDetailedExitCode"); + output.OutputVariables["TerraformPlanDetailedExitCode"].Value.Should().Be("2"); + + output = await ExecuteAndReturnResult(applyCommand, PopulateVariables, "PlanDetailedExitCode"); + output.FullLog.Should().Contain("apply -auto-approve"); + + output = await ExecuteAndReturnResult(planCommand, PopulateVariables, "PlanDetailedExitCode"); + output.OutputVariables["TerraformPlanDetailedExitCode"].Value.Should().Be("0"); + return; + + void PopulateVariables(CommandTestBuilderContext _) + { + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.AdditionalActionParams, + $"-state=\"{Path.Combine(stateFileFolder.DirectoryPath, "terraform.tfstate")}\" -refresh=false"); + } + } + + [Test] + public async Task GoogleCloudIntegration() + { + var bucketName = $"e2e-tf-{Guid.NewGuid().ToString("N").Substring(0, 6)}"; + + using var temporaryFolder = TemporaryDirectory.Create(); + CopyAllFiles(GetTestResourcePath("GoogleCloud"), temporaryFolder.DirectoryPath); + + var environmentJsonKey = await ExternalVariables.Get(ExternalVariable.GoogleCloudJsonKeyfile, CancellationToken.None); + var jsonKey = Convert.ToBase64String(Encoding.UTF8.GetBytes(environmentJsonKey)); + + void PopulateVariables(CommandTestBuilderContext _) + { + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.FileSubstitution, "test.txt"); + _.Variables.Add("Hello", "Hello World from Google Cloud"); + _.Variables.Add("bucket_name", bucketName); + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.VarFiles, "example.tfvars"); + _.Variables.Add("Octopus.Action.Terraform.GoogleCloudAccount", bool.TrueString); + _.Variables.Add("Octopus.Action.GoogleCloudAccount.JsonKey", jsonKey); + _.Variables.Add(KnownVariables.OriginalPackageDirectoryPath, temporaryFolder.DirectoryPath); + } + + var output = await ExecuteAndReturnResult(planCommand, PopulateVariables, temporaryFolder.DirectoryPath); + output.OutputVariables.ContainsKey("TerraformPlanOutput").Should().BeTrue(); + + output = await ExecuteAndReturnResult(applyCommand, PopulateVariables, temporaryFolder.DirectoryPath); + var requestUri = output.OutputVariables["TerraformValueOutputs[url]"].Value; + + string fileData; + var strategy = TestingRetryPolicies.CreateGoogleCloudHttpRetryPipeline(); + using (var client = new HttpClient()) + { + var response = await strategy.ExecuteAsync(async _ => await client.GetAsync(requestUri)); + response.IsSuccessStatusCode.Should().BeTrue(); + fileData = await response.Content.ReadAsStringAsync(); + } + + fileData.Should().Be("Hello World from Google Cloud"); + + await ExecuteAndReturnResult(destroyCommand, PopulateVariables, temporaryFolder.DirectoryPath); + using (var client = new HttpClient()) + { + var response = await strategy.ExecuteAsync(async _ => await client.GetAsync($"{requestUri}&bust_cache")); + response.StatusCode.Should().Be(HttpStatusCode.NotFound); + } + } + + [Test] + public async Task AzureIntegration() + { + var resourceGroupName = AzureTestResourceHelpers.GetResourceGroupName(); + var resourceGroupLocation = RandomAzureRegion.GetRandomRegionWithExclusions(); + + var subscriptionId = await ExternalVariables.Get(ExternalVariable.AzureSubscriptionId, CancellationToken.None); + var tenantId = await ExternalVariables.Get(ExternalVariable.AzureSubscriptionTenantId, CancellationToken.None); + var clientId = await ExternalVariables.Get(ExternalVariable.AzureSubscriptionClientId, CancellationToken.None); + var clientPassword = await ExternalVariables.Get(ExternalVariable.AzureSubscriptionPassword, CancellationToken.None); + + var random = Guid.NewGuid().ToString("N").Substring(0, 6); + var appName = $"cfe2e-{random}"; + var expectedHostName = $"{appName}.azurewebsites.net"; + + using var temporaryFolder = TemporaryDirectory.Create(); + CopyAllFiles(GetTestResourcePath("Azure"), temporaryFolder.DirectoryPath, ToolVersion); + + var output = await ExecuteAndReturnResult(planCommand, PopulateVariables, temporaryFolder.DirectoryPath); + output.OutputVariables.ContainsKey("TerraformPlanOutput").Should().BeTrue(); + + output = await ExecuteAndReturnResult(applyCommand, PopulateVariables, temporaryFolder.DirectoryPath); + output.OutputVariables["TerraformValueOutputs[url]"].Value.Should().Be(expectedHostName); + await AssertRequestResponse(HttpStatusCode.Forbidden); + + await ExecuteAndReturnResult(destroyCommand, PopulateVariables, temporaryFolder.DirectoryPath); + await AssertResponseIsNotReachable(); + return; + + void PopulateVariables(CommandTestBuilderContext _) + { + _.Variables.Add(AzureAccountVariables.SubscriptionId, subscriptionId); + _.Variables.Add(AzureAccountVariables.TenantId, tenantId); + _.Variables.Add(AzureAccountVariables.ClientId, clientId); + _.Variables.Add(AzureAccountVariables.Password, clientPassword); + _.Variables.Add("app_name", appName); + _.Variables.Add("resource_group_name", resourceGroupName); + _.Variables.Add("resource_group_location", resourceGroupLocation); + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.VarFiles, "example.tfvars"); + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.AzureManagedAccount, Boolean.TrueString); + _.Variables.Add(KnownVariables.OriginalPackageDirectoryPath, temporaryFolder.DirectoryPath); + } + + async Task AssertResponseIsNotReachable() + { + try + { + await AssertRequestResponse(HttpStatusCode.NotFound); + } + catch (HttpRequestException ex) + { + switch (ex.InnerException) + { + case SocketException socketException: + socketException.Message.Should().BeOneOf( + "No such host is known.", "Name or service not known", "nodename nor servname provided, or not known"); + break; + case WebException webException: + webException.Message.Should().StartWith("The remote name could not be resolved"); + break; + default: + throw; + } + } + } + + async Task AssertRequestResponse(HttpStatusCode expectedStatusCode) + { + using var client = new HttpClient(); + var response = await client.GetAsync($"https://{expectedHostName}").ConfigureAwait(false); + response.StatusCode.Should().Be(expectedStatusCode); + } + } + + [Test] + [Ignore("Test needs to be updated because s3 bucket doesn't seem to support ACLs anymore.")] + public async Task AWSIntegration() + { + var bucketName = $"cfe2e-tf-{Guid.NewGuid().ToString("N").Substring(0, 6)}"; + var expectedUrl = $"https://{bucketName}.s3.amazonaws.com/test.txt"; + + using var temporaryFolder = TemporaryDirectory.Create(); + CopyAllFiles(GetTestResourcePath("AWS"), temporaryFolder.DirectoryPath); + + var accessKey = await ExternalVariables.Get(ExternalVariable.AwsCloudFormationAndS3AccessKey, CancellationToken.None); + var secretKey = await ExternalVariables.Get(ExternalVariable.AwsCloudFormationAndS3SecretKey, CancellationToken.None); + + var output = await ExecuteAndReturnResult(planCommand, PopulateVariables, temporaryFolder.DirectoryPath); + output.OutputVariables.ContainsKey("TerraformPlanOutput").Should().BeTrue(); + + output = await ExecuteAndReturnResult(applyCommand, PopulateVariables, temporaryFolder.DirectoryPath); + output.OutputVariables["TerraformValueOutputs[url]"].Value.Should().Be(expectedUrl); + + string fileData; + using (var client = new HttpClient()) + fileData = await client.GetStringAsync(expectedUrl).ConfigureAwait(false); + + fileData.Should().Be("Hello World from AWS"); + + await ExecuteAndReturnResult(destroyCommand, PopulateVariables, temporaryFolder.DirectoryPath); + using (var client = new HttpClient()) + { + var response = await client.GetAsync(expectedUrl).ConfigureAwait(false); + response.StatusCode.Should().Be(HttpStatusCode.NotFound); + } + + return; + + void PopulateVariables(CommandTestBuilderContext _) + { + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.FileSubstitution, "test.txt"); + _.Variables.Add("Octopus.Action.Amazon.AccessKey", accessKey); + _.Variables.Add("Octopus.Action.Amazon.SecretKey", secretKey); + _.Variables.Add("Octopus.Action.Aws.Region", "ap-southeast-1"); + _.Variables.Add("Hello", "Hello World from AWS"); + _.Variables.Add("bucket_name", bucketName); + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.VarFiles, "example.tfvars"); + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.AWSManagedAccount, "AWS"); + _.Variables.Add(KnownVariables.OriginalPackageDirectoryPath, temporaryFolder.DirectoryPath); + } + } + + static void CopyAllFiles(string sourceFolderPath, string destinationFolderPath, string? terraformVersion = null) + { + if (!Directory.Exists(sourceFolderPath)) + throw new Exception($"'{nameof(sourceFolderPath)}' ({sourceFolderPath}) does not exist!"); + + if (terraformVersion != null && Directory.Exists(Path.Combine(sourceFolderPath, terraformVersion))) + sourceFolderPath = Path.Combine(sourceFolderPath, terraformVersion); + + foreach (var filePath in Directory.GetFiles(sourceFolderPath)) + { + var destFilePath = Path.Combine(destinationFolderPath, Path.GetFileName(filePath)); + File.Copy(filePath, destFilePath, true); + } + } + + string ExecuteAndReturnLogOutput(string command, Action populateVariables, string folderName, Action? assert = null) + { + return ExecuteAndReturnResult(command, populateVariables, folderName, assert).Result.FullLog; + } + + async Task ExecuteAndReturnResult(string command, Action populateVariables, string folderName, Action? assert = null) + { + var assertResult = assert ?? (_ => { }); + var terraformFiles = Path.IsPathRooted(folderName) ? folderName : GetTestResourcePath(folderName); + + var result = await CommandTestBuilder.CreateAsync(command) + .WithArrange(context => + { + context.Variables.Add(ScriptVariables.ScriptSource, ScriptVariables.ScriptSourceOptions.Package); + context.Variables.Add(TerraformSpecialVariables.Packages.PackageId, terraformFiles); + context.Variables.Add(TerraformSpecialVariables.Calamari.TerraformCliPath, Path.GetDirectoryName(ToolExecutablePath)); + context.Variables.Add(TerraformSpecialVariables.Action.Terraform.CustomTerraformExecutable, ToolExecutablePath); + + populateVariables(context); + + var isInline = context.Variables.Get(ScriptVariables.ScriptSource)! + .Equals(ScriptVariables.ScriptSourceOptions.Inline, StringComparison.InvariantCultureIgnoreCase); + if (isInline) + { + var template = context.Variables.Get(TerraformSpecialVariables.Action.Terraform.Template); + var templateParameters = context.Variables.Get(TerraformSpecialVariables.Action.Terraform.TemplateParameters); + var isJsonFormat = true; + + try { JToken.Parse(template); } + catch { isJsonFormat = false; } + + context.WithDataFileNoBom(template!, isJsonFormat ? TerraformSpecialVariables.JsonTemplateFile : TerraformSpecialVariables.HclTemplateFile); + context.WithDataFileNoBom(templateParameters!, isJsonFormat ? TerraformSpecialVariables.JsonVariablesFile : TerraformSpecialVariables.HclVariablesFile); + } + + if (!String.IsNullOrEmpty(folderName)) + context.WithFilesToCopy(terraformFiles); + }) + .Execute(); + + assertResult(result); + return result; + } + + static string GetCommandFromType(Type commandType) + { + return commandType.CustomAttributes.Where(t => t.AttributeType == typeof(Calamari.Common.Commands.CommandAttribute)) + .Select(c => c.ConstructorArguments.First().Value) + .Single() + ?.ToString()!; + } + } +} +``` + +- [ ] **Step 4: Verify the migrated project builds** + +```bash +cd source && dotnet build Calamari.ExternalTools.Tests/Calamari.ExternalTools.Tests.csproj +``` + +Expected: Build succeeds. If `TerraformSpecialVariables` or other internal Terraform types are inaccessible, verify Task 1 Step 5's `InternalsVisibleTo` change is present. + +- [ ] **Step 5: Verify `Calamari.Terraform.Tests` still builds after the deletions** + +```bash +cd source && dotnet build Calamari.Terraform.Tests/Calamari.Terraform.Tests.csproj +``` + +Expected: Build succeeds — `CommandResolutionTests.cs`, `TerraformCliExecutorFixture.cs`, `TerraformPlanVariableFixture.cs` are unaffected by the deletions in this task. + +- [ ] **Step 6: Run the non-cloud tests to validate the infrastructure works end-to-end** + +```bash +cd source && dotnet test Calamari.ExternalTools.Tests/ --filter "FullyQualifiedName~TerraformCommandsFixture.ApplySimple_Succeeds" -v normal +``` + +Expected: Terraform is downloaded per the manifest (or found on PATH), and the test passes. This is the first real exercise of the full resolve → download → run pipeline. + +- [ ] **Step 7: Run the remaining non-cloud tests** + +```bash +cd source && dotnet test Calamari.ExternalTools.Tests/ --filter "FullyQualifiedName~TerraformCommandsFixture&TestCategory!=Cloud" -v normal +``` + +Expected: `ApplySimple_Succeeds`, `InlineJsonTemplate_ProducesExpectedOutput`, `OutputAndSubstituteOctopusVariables`, `WithOutputSensitiveVariables`, `PlanDetailedExitCode` all pass. `GoogleCloudIntegration` and `AzureIntegration` require cloud credentials (`ExternalVariables`) and are expected to fail/skip locally without them; `AWSIntegration` is `[Ignore]`d. + +- [ ] **Step 8: Commit** + +```bash +git add source/Calamari.ExternalTools.Tests/ source/Calamari.Terraform.Tests/ +git commit -m "feat: migrate Terraform integration tests to Calamari.ExternalTools.Tests" +``` + +--- + +### Future Tasks (not in this plan) + +- Migrate Helm, kubectl, Azure CLI, GCloud, AWS CLI, aws-iam-authenticator, kubelogin tool tests (add their manifest entries, download strategies, and fixtures) +- Migrate Azure App Service, AzureResourceGroup, AzureWebApp, GoogleCloudScripting cloud tests into a new `CloudIntegration/` subdir +- Add a Nuke build target and TeamCity pipeline stage that runs `--filter "Category=ExternalTool"` on a nightly/on-demand schedule (the category attribute itself already exists on `TerraformCommandsFixture`, so no CI job currently isolates or runs it — it simply isn't excluded from a manual full-project `dotnet test` run either) +- Automated version-expansion scheduled job (`[Explicit]` `ToolVersionExpansionFixture`, `LatestVersionFinder`) that bumps `tool-manifest.json`'s `highest` automatically +- Add tar.gz extraction support to `ToolDownloader` when a tool that needs it (Helm, GCloud, kubelogin) is migrated From e4d09efd2320bb57d0ae205ca1635a2645096b8a Mon Sep 17 00:00:00 2001 From: robert Date: Wed, 5 Aug 2026 11:59:29 +1000 Subject: [PATCH 03/16] feat: scaffold Calamari.ExternalTools.Tests project with Terraform tool manifest --- .../Calamari.ExternalTools.Tests.csproj | 44 +++++++++++++++++++ .../tool-manifest.json | 10 +++++ .../Properties/InternalsVisibleTo.cs | 3 +- source/Calamari.sln | 32 ++++++++++---- 4 files changed, 80 insertions(+), 9 deletions(-) create mode 100644 source/Calamari.ExternalTools.Tests/Calamari.ExternalTools.Tests.csproj create mode 100644 source/Calamari.ExternalTools.Tests/tool-manifest.json diff --git a/source/Calamari.ExternalTools.Tests/Calamari.ExternalTools.Tests.csproj b/source/Calamari.ExternalTools.Tests/Calamari.ExternalTools.Tests.csproj new file mode 100644 index 000000000..323bede4b --- /dev/null +++ b/source/Calamari.ExternalTools.Tests/Calamari.ExternalTools.Tests.csproj @@ -0,0 +1,44 @@ + + + + Calamari.ExternalTools.Tests + Calamari.ExternalTools.Tests + win-x64;linux-x64;osx-x64;linux-arm;linux-arm64 + false + net8.0 + + CS8632 + true + + + + + + + + + + + + + + + + + + + + PreserveNewest + + + PreserveNewest + + + PreserveNewest + + + PreserveNewest + + + + diff --git a/source/Calamari.ExternalTools.Tests/tool-manifest.json b/source/Calamari.ExternalTools.Tests/tool-manifest.json new file mode 100644 index 000000000..1df787f36 --- /dev/null +++ b/source/Calamari.ExternalTools.Tests/tool-manifest.json @@ -0,0 +1,10 @@ +{ + "tools": { + "terraform": { + "lowest": "0.13.7", + "highest": "1.8.5", + "source": "https://releases.hashicorp.com/terraform/", + "architectures": ["amd64", "arm64"] + } + } +} diff --git a/source/Calamari.Terraform/Properties/InternalsVisibleTo.cs b/source/Calamari.Terraform/Properties/InternalsVisibleTo.cs index c4b36a8ca..6cf631d54 100644 --- a/source/Calamari.Terraform/Properties/InternalsVisibleTo.cs +++ b/source/Calamari.Terraform/Properties/InternalsVisibleTo.cs @@ -1,3 +1,4 @@ using System.Runtime.CompilerServices; -[assembly: InternalsVisibleTo("Calamari.Terraform.Tests")] \ No newline at end of file +[assembly: InternalsVisibleTo("Calamari.Terraform.Tests")] +[assembly: InternalsVisibleTo("Calamari.ExternalTools.Tests")] \ No newline at end of file diff --git a/source/Calamari.sln b/source/Calamari.sln index c776abd64..d03568df1 100644 --- a/source/Calamari.sln +++ b/source/Calamari.sln @@ -88,6 +88,10 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Calamari.Contracts", "Calam EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Calamari.DockerCredentialHelper", "Calamari.DockerCredentialHelper\Calamari.DockerCredentialHelper.csproj", "{B34DBEEC-7AC2-4BFE-ACDD-1788828925BD}" EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Calamari.AiAgent.Tests", "Calamari.AiAgent.Tests\Calamari.AiAgent.Tests.csproj", "{8D3FCBF5-369E-44B3-BD72-4C11E8058027}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Calamari.ExternalTools.Tests", "Calamari.ExternalTools.Tests\Calamari.ExternalTools.Tests.csproj", "{028A8E26-0E75-48C2-A639-2FE0D8B45DEE}" +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Any CPU = Debug|Any CPU @@ -212,14 +216,14 @@ Global {D8DEC40C-948F-4806-AE87-1A7502E41A06}.Debug|Any CPU.Build.0 = Debug|Any CPU {D8DEC40C-948F-4806-AE87-1A7502E41A06}.Release|Any CPU.ActiveCfg = Release|Any CPU {D8DEC40C-948F-4806-AE87-1A7502E41A06}.Release|Any CPU.Build.0 = Release|Any CPU - {74F2739F-A501-4F30-BF9E-75A087C3B917}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {74F2739F-A501-4F30-BF9E-75A087C3B917}.Debug|Any CPU.Build.0 = Debug|Any CPU - {74F2739F-A501-4F30-BF9E-75A087C3B917}.Release|Any CPU.ActiveCfg = Release|Any CPU - {74F2739F-A501-4F30-BF9E-75A087C3B917}.Release|Any CPU.Build.0 = Release|Any CPU - {A3BFFA0E-7514-41DE-A141-E124133D0665}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {A3BFFA0E-7514-41DE-A141-E124133D0665}.Debug|Any CPU.Build.0 = Debug|Any CPU - {A3BFFA0E-7514-41DE-A141-E124133D0665}.Release|Any CPU.ActiveCfg = Release|Any CPU - {A3BFFA0E-7514-41DE-A141-E124133D0665}.Release|Any CPU.Build.0 = Release|Any CPU + {74F2739F-A501-4F30-BF9E-75A087C3B917}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {74F2739F-A501-4F30-BF9E-75A087C3B917}.Debug|Any CPU.Build.0 = Debug|Any CPU + {74F2739F-A501-4F30-BF9E-75A087C3B917}.Release|Any CPU.ActiveCfg = Release|Any CPU + {74F2739F-A501-4F30-BF9E-75A087C3B917}.Release|Any CPU.Build.0 = Release|Any CPU + {A3BFFA0E-7514-41DE-A141-E124133D0665}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {A3BFFA0E-7514-41DE-A141-E124133D0665}.Debug|Any CPU.Build.0 = Debug|Any CPU + {A3BFFA0E-7514-41DE-A141-E124133D0665}.Release|Any CPU.ActiveCfg = Release|Any CPU + {A3BFFA0E-7514-41DE-A141-E124133D0665}.Release|Any CPU.Build.0 = Release|Any CPU {B4EB8110-50D8-4C7E-8DBF-102AE57FD943}.Debug|Any CPU.ActiveCfg = Debug|Any CPU {B4EB8110-50D8-4C7E-8DBF-102AE57FD943}.Debug|Any CPU.Build.0 = Debug|Any CPU {B4EB8110-50D8-4C7E-8DBF-102AE57FD943}.Release|Any CPU.ActiveCfg = Release|Any CPU @@ -232,6 +236,18 @@ Global {B34DBEEC-7AC2-4BFE-ACDD-1788828925BD}.Debug|Any CPU.Build.0 = Debug|Any CPU {B34DBEEC-7AC2-4BFE-ACDD-1788828925BD}.Release|Any CPU.ActiveCfg = Release|Any CPU {B34DBEEC-7AC2-4BFE-ACDD-1788828925BD}.Release|Any CPU.Build.0 = Release|Any CPU + {767EB703-FF66-4955-9AE2-322A93FB69EE}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {767EB703-FF66-4955-9AE2-322A93FB69EE}.Debug|Any CPU.Build.0 = Debug|Any CPU + {767EB703-FF66-4955-9AE2-322A93FB69EE}.Release|Any CPU.ActiveCfg = Release|Any CPU + {767EB703-FF66-4955-9AE2-322A93FB69EE}.Release|Any CPU.Build.0 = Release|Any CPU + {8D3FCBF5-369E-44B3-BD72-4C11E8058027}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {8D3FCBF5-369E-44B3-BD72-4C11E8058027}.Debug|Any CPU.Build.0 = Debug|Any CPU + {8D3FCBF5-369E-44B3-BD72-4C11E8058027}.Release|Any CPU.ActiveCfg = Release|Any CPU + {8D3FCBF5-369E-44B3-BD72-4C11E8058027}.Release|Any CPU.Build.0 = Release|Any CPU + {028A8E26-0E75-48C2-A639-2FE0D8B45DEE}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {028A8E26-0E75-48C2-A639-2FE0D8B45DEE}.Debug|Any CPU.Build.0 = Debug|Any CPU + {028A8E26-0E75-48C2-A639-2FE0D8B45DEE}.Release|Any CPU.ActiveCfg = Release|Any CPU + {028A8E26-0E75-48C2-A639-2FE0D8B45DEE}.Release|Any CPU.Build.0 = Release|Any CPU EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE From fc5508a7e3a184a7a76011e47c6d0837e5ace5fe Mon Sep 17 00:00:00 2001 From: robert Date: Wed, 5 Aug 2026 12:03:16 +1000 Subject: [PATCH 04/16] feat: implement ToolManifest reader with version range support --- .../Infrastructure/ToolManifest.cs | 98 +++++++++++++++++++ .../Infrastructure/ToolManifestTests.cs | 48 +++++++++ 2 files changed, 146 insertions(+) create mode 100644 source/Calamari.ExternalTools.Tests/Infrastructure/ToolManifest.cs create mode 100644 source/Calamari.ExternalTools.Tests/Infrastructure/ToolManifestTests.cs diff --git a/source/Calamari.ExternalTools.Tests/Infrastructure/ToolManifest.cs b/source/Calamari.ExternalTools.Tests/Infrastructure/ToolManifest.cs new file mode 100644 index 000000000..241ac0c6a --- /dev/null +++ b/source/Calamari.ExternalTools.Tests/Infrastructure/ToolManifest.cs @@ -0,0 +1,98 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text.Json; +using System.Text.Json.Serialization; +using Calamari.Testing.Helpers; + +namespace Calamari.ExternalTools.Tests.Infrastructure +{ + public class ToolManifest + { + readonly Dictionary tools; + + ToolManifest(Dictionary tools) + { + this.tools = tools; + } + + public IReadOnlyCollection ToolNames => tools.Keys.ToList(); + + public ToolDefinition? GetTool(string name) + { + return tools.TryGetValue(name, out var tool) ? tool : null; + } + + public static ToolManifest Load() + { + var manifestPath = Path.Combine(TestEnvironment.CurrentWorkingDirectory, "tool-manifest.json"); + var json = File.ReadAllText(manifestPath); + var doc = JsonSerializer.Deserialize(json) + ?? throw new InvalidOperationException("Failed to deserialize tool-manifest.json"); + + var tools = new Dictionary(); + foreach (var (name, entry) in doc.Tools) + { + tools[name] = new ToolDefinition( + name, + ParseVersion(entry.Lowest), + ParseVersion(entry.Highest), + entry.Source, + entry.Architectures); + } + + return new ToolManifest(tools); + } + + static Version ParseVersion(string version) + { + var clean = version.TrimStart('v'); + return Version.Parse(clean); + } + + class ManifestDocument + { + [JsonPropertyName("tools")] + public Dictionary Tools { get; set; } = new(); + } + + class ManifestEntry + { + [JsonPropertyName("lowest")] + public string Lowest { get; set; } = ""; + + [JsonPropertyName("highest")] + public string Highest { get; set; } = ""; + + [JsonPropertyName("source")] + public string Source { get; set; } = ""; + + [JsonPropertyName("architectures")] + public string[] Architectures { get; set; } = Array.Empty(); + } + } + + public class ToolDefinition + { + public ToolDefinition(string name, Version lowest, Version highest, string source, string[] architectures) + { + Name = name; + Lowest = lowest; + Highest = highest; + Source = source; + Architectures = architectures; + } + + public string Name { get; } + public Version Lowest { get; } + public Version Highest { get; } + public string Source { get; } + public string[] Architectures { get; } + + public bool IsInRange(Version version) + { + return version >= Lowest && version <= Highest; + } + } +} diff --git a/source/Calamari.ExternalTools.Tests/Infrastructure/ToolManifestTests.cs b/source/Calamari.ExternalTools.Tests/Infrastructure/ToolManifestTests.cs new file mode 100644 index 000000000..201c3d9b9 --- /dev/null +++ b/source/Calamari.ExternalTools.Tests/Infrastructure/ToolManifestTests.cs @@ -0,0 +1,48 @@ +using FluentAssertions; +using NUnit.Framework; + +namespace Calamari.ExternalTools.Tests.Infrastructure +{ + [TestFixture] + public class ToolManifestTests + { + [Test] + public void ShouldLoadManifestFromEmbeddedFile() + { + var manifest = ToolManifest.Load(); + + manifest.Should().NotBeNull(); + manifest.GetTool("terraform").Should().NotBeNull(); + manifest.GetTool("terraform")!.Lowest.ToString().Should().Be("0.13.7"); + manifest.GetTool("terraform")!.Highest.ToString().Should().Be("1.8.5"); + } + + [Test] + public void ShouldReturnNullForUnknownTool() + { + var manifest = ToolManifest.Load(); + + manifest.GetTool("nonexistent-tool").Should().BeNull(); + } + + [Test] + public void ShouldCheckVersionIsInRange() + { + var manifest = ToolManifest.Load(); + var terraform = manifest.GetTool("terraform")!; + + terraform.IsInRange(new System.Version(1, 0, 0)).Should().BeTrue(); + terraform.IsInRange(new System.Version(0, 12, 0)).Should().BeFalse(); + terraform.IsInRange(new System.Version(2, 0, 0)).Should().BeFalse(); + } + + [Test] + public void ShouldListAllTools() + { + var manifest = ToolManifest.Load(); + + manifest.ToolNames.Should().Contain("terraform"); + manifest.ToolNames.Should().HaveCount(1); + } + } +} From c6907c967e960eaf7da47e6c2f74105676ad0e04 Mon Sep 17 00:00:00 2001 From: robert Date: Wed, 5 Aug 2026 12:06:08 +1000 Subject: [PATCH 05/16] feat: implement ToolResolver with env var override and manifest fallback --- .../Infrastructure/ToolResolver.cs | 97 +++++++++++++++++++ .../Infrastructure/ToolResolverTests.cs | 42 ++++++++ 2 files changed, 139 insertions(+) create mode 100644 source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolver.cs create mode 100644 source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolverTests.cs diff --git a/source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolver.cs b/source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolver.cs new file mode 100644 index 000000000..1fccd2e19 --- /dev/null +++ b/source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolver.cs @@ -0,0 +1,97 @@ +using System; +using System.Text; +using Calamari.Common.Features.Processes; +using Calamari.Common.Plumbing; + +namespace Calamari.ExternalTools.Tests.Infrastructure +{ + /// + /// Resolves the version to use for a tool: env var override, else the manifest's highest. + /// Resolution of the actual executable (PATH vs download) is done by ExternalToolFixture. + /// + public class ToolResolver + { + readonly ToolManifest manifest; + readonly Action log; + + public ToolResolver(ToolManifest manifest, Action log) + { + this.manifest = manifest; + this.log = log; + } + + public static string GetOverrideEnvVar(string toolName) + { + return $"CALAMARI_TOOL_{toolName.Replace("-", "_").ToUpperInvariant()}_VERSION"; + } + + public string ResolveVersion(string toolName) + { + var envVar = GetOverrideEnvVar(toolName); + var overrideVersion = Environment.GetEnvironmentVariable(envVar); + + if (!string.IsNullOrEmpty(overrideVersion)) + { + log($"Using override version {overrideVersion} for {toolName} (from {envVar})"); + return overrideVersion; + } + + var tool = manifest.GetTool(toolName); + if (tool == null) + throw new InvalidOperationException($"Tool '{toolName}' not found in manifest"); + + return tool.Highest.ToString(); + } + + public static string? FindOnPath(string toolName) + { + try + { + var command = CalamariEnvironment.IsRunningOnWindows ? "where" : "which"; + var executableName = CalamariEnvironment.IsRunningOnWindows + ? $"{toolName}.exe" + : toolName; + + var stdOut = new StringBuilder(); + var result = SilentProcessRunner.ExecuteCommand( + command, + executableName, + ".", + s => stdOut.AppendLine(s), + _ => { }); + + if (result.ExitCode == 0) + { + var path = stdOut.ToString().Trim().Split(new[] { '\r', '\n' }, StringSplitOptions.RemoveEmptyEntries); + return path.Length > 0 ? path[0] : null; + } + } + catch + { + // Tool not found + } + + return null; + } + + public static string? GetInstalledVersion(string executablePath, string versionArg = "--version") + { + try + { + var stdOut = new StringBuilder(); + var result = SilentProcessRunner.ExecuteCommand( + executablePath, + versionArg, + ".", + s => stdOut.AppendLine(s), + _ => { }); + + return result.ExitCode == 0 ? stdOut.ToString().Trim() : null; + } + catch + { + return null; + } + } + } +} diff --git a/source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolverTests.cs b/source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolverTests.cs new file mode 100644 index 000000000..dd74abbb5 --- /dev/null +++ b/source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolverTests.cs @@ -0,0 +1,42 @@ +using FluentAssertions; +using NUnit.Framework; + +namespace Calamari.ExternalTools.Tests.Infrastructure +{ + [TestFixture] + public class ToolResolverTests + { + [Test] + public void ShouldBuildEnvironmentVariableOverrideName() + { + var envVarName = ToolResolver.GetOverrideEnvVar("terraform"); + envVarName.Should().Be("CALAMARI_TOOL_TERRAFORM_VERSION"); + } + + [Test] + public void ShouldResolveToManifestHighestWhenNoOverrideSet() + { + var manifest = ToolManifest.Load(); + var resolver = new ToolResolver(manifest, _ => { }); + + var version = resolver.ResolveVersion("terraform"); + + version.Should().Be("1.8.5"); + } + + [Test] + public void ShouldDetectToolOnPath() + { + // 'dotnet' is always on PATH in a .NET test run + var result = ToolResolver.FindOnPath("dotnet"); + result.Should().NotBeNullOrEmpty(); + } + + [Test] + public void ShouldReturnNullForToolNotOnPath() + { + var result = ToolResolver.FindOnPath("definitely-not-a-real-tool-abc123"); + result.Should().BeNull(); + } + } +} From e7bcf2f7d044d398156fa31606563e2045e16d96 Mon Sep 17 00:00:00 2001 From: robert Date: Wed, 5 Aug 2026 12:08:47 +1000 Subject: [PATCH 06/16] feat: implement ToolDownloader with retry, caching, and platform detection --- .../Infrastructure/ToolDownloader.cs | 138 ++++++++++++++++++ 1 file changed, 138 insertions(+) create mode 100644 source/Calamari.ExternalTools.Tests/Infrastructure/ToolDownloader.cs diff --git a/source/Calamari.ExternalTools.Tests/Infrastructure/ToolDownloader.cs b/source/Calamari.ExternalTools.Tests/Infrastructure/ToolDownloader.cs new file mode 100644 index 000000000..f6fcb2d8b --- /dev/null +++ b/source/Calamari.ExternalTools.Tests/Infrastructure/ToolDownloader.cs @@ -0,0 +1,138 @@ +using System; +using System.IO; +using System.IO.Compression; +using System.Linq; +using System.Net.Http; +using System.Threading.Tasks; +using Calamari.Common.Features.Processes; +using Calamari.Common.Plumbing; +using Calamari.Common.Plumbing.Retry; +using Calamari.Testing.Helpers; + +namespace Calamari.ExternalTools.Tests.Infrastructure +{ + /// + /// Downloads and caches external tool binaries. + /// Cache location: {TestOutputDir}/Tools/{toolName}/{version}/ + /// + public class ToolDownloader + { + readonly Action log; + + public ToolDownloader(Action log) + { + this.log = log; + } + + public async Task Download(string toolName, string version, Func> downloadAction) + { + var destinationDir = TestEnvironment.GetTestPath("Tools", toolName, version); + + var existing = FindExistingExecutable(toolName, destinationDir); + if (existing != null) + { + log($"Using cached {toolName} {version} at {existing}"); + return existing; + } + + log($"Downloading {toolName} {version}..."); + Directory.CreateDirectory(destinationDir); + + var retry = new RetryTracker(4, TimeSpan.MaxValue, new LimitedExponentialRetryInterval(3000, 30000, 2)); + string? executablePath = null; + + while (retry.Try()) + { + try + { + using var client = CreateHttpClient(); + executablePath = await downloadAction(destinationDir, version, client); + AddExecutePermission(executablePath); + break; + } + catch + { + if (!retry.CanRetry()) + throw; + + await Task.Delay(retry.Sleep()); + } + } + + log($"Downloaded {toolName} {version} to {executablePath}"); + return executablePath!; + } + + string? FindExistingExecutable(string toolName, string destinationDir) + { + if (!Directory.Exists(destinationDir)) + return null; + + var path = Directory.EnumerateFiles(destinationDir, "*", SearchOption.AllDirectories) + .FirstOrDefault(f => + { + var name = Path.GetFileNameWithoutExtension(f).ToLowerInvariant(); + return name.Contains(toolName.ToLowerInvariant().Replace("-", "")); + }); + + return path != null && File.Exists(path) ? path : null; + } + + public static async Task DownloadFile(string url, string destinationPath, HttpClient client) + { + using var fileStream = new FileStream(destinationPath, FileMode.Create, FileAccess.Write, FileShare.None); + using var stream = await client.GetStreamAsync(url); + await stream.CopyToAsync(fileStream); + } + + public static async Task DownloadAndExtractZip(string url, string destinationDir, HttpClient client) + { + var tempPath = Path.Combine(Path.GetTempPath(), $"{Guid.NewGuid()}.zip"); + try + { + await DownloadFile(url, tempPath, client); + ZipFile.ExtractToDirectory(tempPath, destinationDir); + } + finally + { + if (File.Exists(tempPath)) + File.Delete(tempPath); + } + } + + static void AddExecutePermission(string exePath) + { + if (CalamariEnvironment.IsRunningOnWindows || string.IsNullOrEmpty(exePath)) + return; + + SilentProcessRunner.ExecuteCommand( + "chmod", $"+x {exePath}", + Path.GetDirectoryName(exePath) ?? ".", + _ => { }, _ => { }); + } + + static HttpClient CreateHttpClient() + { + var client = new HttpClient(); + client.DefaultRequestHeaders.UserAgent.ParseAdd( + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"); + return client; + } + + public static string GetPlatform() + { + if (CalamariEnvironment.IsRunningOnWindows) return "windows"; + if (CalamariEnvironment.IsRunningOnMac) return "darwin"; + return "linux"; + } + + public static string GetArchitecture() + { + return System.Runtime.InteropServices.RuntimeInformation.OSArchitecture switch + { + System.Runtime.InteropServices.Architecture.Arm64 => "arm64", + _ => "amd64" + }; + } + } +} From bd941ba48cf92088a1afa91ec530c252981bc320 Mon Sep 17 00:00:00 2001 From: robert Date: Wed, 5 Aug 2026 12:12:29 +1000 Subject: [PATCH 07/16] feat: add Terraform download strategy --- .../ToolStrategies/TerraformStrategy.cs | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 source/Calamari.ExternalTools.Tests/Infrastructure/ToolStrategies/TerraformStrategy.cs diff --git a/source/Calamari.ExternalTools.Tests/Infrastructure/ToolStrategies/TerraformStrategy.cs b/source/Calamari.ExternalTools.Tests/Infrastructure/ToolStrategies/TerraformStrategy.cs new file mode 100644 index 000000000..971d33016 --- /dev/null +++ b/source/Calamari.ExternalTools.Tests/Infrastructure/ToolStrategies/TerraformStrategy.cs @@ -0,0 +1,23 @@ +using System.IO; +using System.Linq; +using System.Net.Http; +using System.Threading.Tasks; + +namespace Calamari.ExternalTools.Tests.Infrastructure.ToolStrategies +{ + public static class TerraformStrategy + { + public static async Task Download(string destinationDir, string version, HttpClient client) + { + var platform = ToolDownloader.GetPlatform(); + var arch = ToolDownloader.GetArchitecture(); + var fileName = $"terraform_{version}_{platform}_{arch}.zip"; + var url = $"https://releases.hashicorp.com/terraform/{version}/{fileName}"; + + await ToolDownloader.DownloadAndExtractZip(url, destinationDir, client); + + return Directory.EnumerateFiles(destinationDir) + .First(f => Path.GetFileName(f).Contains("terraform")); + } + } +} From 6fcc67963821dff2f9990fa3db5c449f6afc64fc Mon Sep 17 00:00:00 2001 From: robert Date: Wed, 5 Aug 2026 12:48:13 +1000 Subject: [PATCH 08/16] feat: add ExternalToolFixture base class for tool-dependent tests --- .../Infrastructure/ExternalToolFixture.cs | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 source/Calamari.ExternalTools.Tests/Infrastructure/ExternalToolFixture.cs diff --git a/source/Calamari.ExternalTools.Tests/Infrastructure/ExternalToolFixture.cs b/source/Calamari.ExternalTools.Tests/Infrastructure/ExternalToolFixture.cs new file mode 100644 index 000000000..9ee45da31 --- /dev/null +++ b/source/Calamari.ExternalTools.Tests/Infrastructure/ExternalToolFixture.cs @@ -0,0 +1,47 @@ +using System.Net.Http; +using System.Threading.Tasks; +using NUnit.Framework; + +namespace Calamari.ExternalTools.Tests.Infrastructure +{ + /// + /// Base class for test fixtures that depend on an external tool. + /// Resolves the tool via: env var override -> PATH -> download. + /// Subclasses set PrimaryToolName and provide a download strategy. + /// + public abstract class ExternalToolFixture + { + static readonly ToolManifest Manifest = ToolManifest.Load(); + + protected string ToolExecutablePath { get; private set; } = ""; + protected string ToolVersion { get; private set; } = ""; + + protected abstract string PrimaryToolName { get; } + + protected abstract Task DownloadTool(string destinationDir, string version, HttpClient client); + + [OneTimeSetUp] + public async Task ResolveTool() + { + var resolver = new ToolResolver(Manifest, Log); + var downloader = new ToolDownloader(Log); + + ToolVersion = resolver.ResolveVersion(PrimaryToolName); + + var pathResult = ToolResolver.FindOnPath(PrimaryToolName); + if (pathResult != null) + { + Log($"Found {PrimaryToolName} on PATH at {pathResult}"); + ToolExecutablePath = pathResult; + return; + } + + ToolExecutablePath = await downloader.Download(PrimaryToolName, ToolVersion, DownloadTool); + } + + protected void Log(string message) + { + TestContext.Progress.WriteLine($"[{PrimaryToolName}] {message}"); + } + } +} \ No newline at end of file From 150dbdf011922c923d09e494ebd0fc2781df481f Mon Sep 17 00:00:00 2001 From: robert Date: Wed, 5 Aug 2026 13:07:44 +1000 Subject: [PATCH 09/16] test: add unit coverage for Terraform init command construction and version range checks --- .../TerraformCliExecutorFixture.cs | 154 ++++++++++++++++++ .../Behaviours/TerraformDeployBehaviour.cs | 2 +- 2 files changed, 155 insertions(+), 1 deletion(-) diff --git a/source/Calamari.Terraform.Tests/TerraformCliExecutorFixture.cs b/source/Calamari.Terraform.Tests/TerraformCliExecutorFixture.cs index 5853f3cc5..fa1ba9f10 100644 --- a/source/Calamari.Terraform.Tests/TerraformCliExecutorFixture.cs +++ b/source/Calamari.Terraform.Tests/TerraformCliExecutorFixture.cs @@ -5,7 +5,9 @@ using Calamari.Common.Plumbing.FileSystem; using Calamari.Common.Plumbing.Logging; using Calamari.Common.Plumbing.Variables; +using Calamari.Terraform.Behaviours; using FluentAssertions; +using Newtonsoft.Json; using NSubstitute; using NUnit.Framework; @@ -137,5 +139,157 @@ public void InitializePlugins_ThrowsAfterRetriesExhausted() act.Should().Throw(); commandLineRunner.Received(5).Execute(Arg.Any()); } + + [Test] + public void InitCommand_PreV015_IncludesGetPluginsFlagTrue() + { + var capturedArguments = new List(); + var testVariables = Substitute.For(); + testVariables.GetStrings(KnownVariables.EnabledFeatureToggles).Returns(new List()); + testVariables.GetFlag(TerraformSpecialVariables.Action.Terraform.AllowPluginDownloads, true).Returns(true); + + var commandLineRunner = Substitute.For(); + commandLineRunner.Execute(Arg.Do(invocation => + { + capturedArguments.Add(invocation.Arguments); + if (capturedArguments.Count == 1) + invocation.AdditionalInvocationOutputSink.WriteInfo("Terraform v0.14.0"); + })).Returns(new CommandResult("terraform", 0)); + + new TerraformCliExecutor(Substitute.For(), Substitute.For(), commandLineRunner, new RunningDeployment("blah", testVariables), new Dictionary()); + + capturedArguments[1].Should().Contain("-get-plugins=true"); + } + + [Test] + public void InitCommand_PreV015_PluginDownloadsDisabled_IncludesGetPluginsFlagFalse() + { + var capturedArguments = new List(); + var testVariables = Substitute.For(); + testVariables.GetStrings(KnownVariables.EnabledFeatureToggles).Returns(new List()); + testVariables.GetFlag(TerraformSpecialVariables.Action.Terraform.AllowPluginDownloads, true).Returns(false); + + var commandLineRunner = Substitute.For(); + commandLineRunner.Execute(Arg.Do(invocation => + { + capturedArguments.Add(invocation.Arguments); + if (capturedArguments.Count == 1) + invocation.AdditionalInvocationOutputSink.WriteInfo("Terraform v0.14.0"); + })).Returns(new CommandResult("terraform", 0)); + + new TerraformCliExecutor(Substitute.For(), Substitute.For(), commandLineRunner, new RunningDeployment("blah", testVariables), new Dictionary()); + + capturedArguments[1].Should().Contain("-get-plugins=false"); + } + + [Test] + public void InitCommand_V015AndAbove_OmitsGetPluginsFlag() + { + var capturedArguments = new List(); + var testVariables = Substitute.For(); + testVariables.GetStrings(KnownVariables.EnabledFeatureToggles).Returns(new List()); + + var commandLineRunner = Substitute.For(); + commandLineRunner.Execute(Arg.Do(invocation => + { + capturedArguments.Add(invocation.Arguments); + if (capturedArguments.Count == 1) + invocation.AdditionalInvocationOutputSink.WriteInfo("Terraform v0.15.0"); + })).Returns(new CommandResult("terraform", 0)); + + new TerraformCliExecutor(Substitute.For(), Substitute.For(), commandLineRunner, new RunningDeployment("blah", testVariables), new Dictionary()); + + capturedArguments[1].Should().NotContain("-get-plugins"); + } + + [Test] + public void InitCommand_IncludesAdditionalInitParams() + { + var capturedArguments = new List(); + var testVariables = Substitute.For(); + testVariables.GetStrings(KnownVariables.EnabledFeatureToggles).Returns(new List()); + testVariables.Get(TerraformSpecialVariables.Action.Terraform.AdditionalInitParams).Returns("-backend-config=\"key=value\""); + + var commandLineRunner = Substitute.For(); + commandLineRunner.Execute(Arg.Do(invocation => + { + capturedArguments.Add(invocation.Arguments); + if (capturedArguments.Count == 1) + invocation.AdditionalInvocationOutputSink.WriteInfo("Terraform v1.0.0"); + })).Returns(new CommandResult("terraform", 0)); + + new TerraformCliExecutor(Substitute.For(), Substitute.For(), commandLineRunner, new RunningDeployment("blah", testVariables), new Dictionary()); + + capturedArguments[1].Should().Contain("-backend-config=\"key=value\""); + capturedArguments[1].Should().NotContain("-get-plugins"); + } + + [Test] + public void UntestedVersion_AboveSupportedRange_LogsInfoOnSuccessfulCommand() + { + var log = Substitute.For(); + var testVariables = Substitute.For(); + testVariables.GetStrings(KnownVariables.EnabledFeatureToggles).Returns(new List()); + + var commandLineRunner = Substitute.For(); + var callCount = 0; + commandLineRunner.Execute(Arg.Do(invocation => + { + callCount++; + if (callCount == 1) + invocation.AdditionalInvocationOutputSink.WriteInfo("Terraform v2.0.0"); + })).Returns(new CommandResult("terraform", 0)); + + var executor = new TerraformCliExecutor(log, Substitute.For(), commandLineRunner, new RunningDeployment("blah", testVariables), new Dictionary()); + executor.ExecuteCommand("plan"); + + log.Received(1).Info(Arg.Is(s => s.Contains("has not been tested"))); + } + + [Test] + public void SupportedVersion_WithinRange_DoesNotLogUntestedMessage() + { + var log = Substitute.For(); + var testVariables = Substitute.For(); + testVariables.GetStrings(KnownVariables.EnabledFeatureToggles).Returns(new List()); + + var commandLineRunner = Substitute.For(); + var callCount = 0; + commandLineRunner.Execute(Arg.Do(invocation => + { + callCount++; + if (callCount == 1) + invocation.AdditionalInvocationOutputSink.WriteInfo("Terraform v1.0.0"); + })).Returns(new CommandResult("terraform", 0)); + + var executor = new TerraformCliExecutor(log, Substitute.For(), commandLineRunner, new RunningDeployment("blah", testVariables), new Dictionary()); + executor.ExecuteCommand("plan"); + + log.DidNotReceive().Info(Arg.Is(s => s.Contains("has not been tested"))); + log.DidNotReceive().Warn(Arg.Is(s => s.Contains("has not been tested"))); + } + + [Test] + public void EnvironmentVariables_ParsedFromJson() + { + var variables = new CalamariVariables(); + variables.Set(TerraformSpecialVariables.Action.Terraform.EnvironmentVariables, + JsonConvert.SerializeObject(new Dictionary { { "TF_VAR_ami", "test-value" }, { "TF_LOG", "DEBUG" } })); + + var result = TerraformDeployBehaviour.GetEnvironmentVariableArgs(variables); + + result.Should().ContainKey("TF_VAR_ami").WhoseValue.Should().Be("test-value"); + result.Should().ContainKey("TF_LOG").WhoseValue.Should().Be("DEBUG"); + } + + [Test] + public void EnvironmentVariables_NotSet_ReturnsEmptyDictionary() + { + var variables = new CalamariVariables(); + + var result = TerraformDeployBehaviour.GetEnvironmentVariableArgs(variables); + + result.Should().BeEmpty(); + } } } diff --git a/source/Calamari.Terraform/Behaviours/TerraformDeployBehaviour.cs b/source/Calamari.Terraform/Behaviours/TerraformDeployBehaviour.cs index 5e6dd578c..a747d7dc6 100644 --- a/source/Calamari.Terraform/Behaviours/TerraformDeployBehaviour.cs +++ b/source/Calamari.Terraform/Behaviours/TerraformDeployBehaviour.cs @@ -59,7 +59,7 @@ public async Task Execute(RunningDeployment context) await Execute(context, environmentVariables); } - static Dictionary GetEnvironmentVariableArgs(IVariables variables) + internal static Dictionary GetEnvironmentVariableArgs(IVariables variables) { var rawJson = variables.Get(TerraformSpecialVariables.Action.Terraform.EnvironmentVariables); if (string.IsNullOrEmpty(rawJson)) From f166fb5d5336a650cf4cfc17093fd6a620f5be88 Mon Sep 17 00:00:00 2001 From: robert Date: Wed, 5 Aug 2026 15:10:59 +1000 Subject: [PATCH 10/16] feat: migrate Terraform integration tests to Calamari.ExternalTools.Tests --- .../ExternalTools/Terraform}/AWS/example.tf | 0 .../Terraform}/AWS/example.tfvars | 0 .../ExternalTools/Terraform}/AWS/test.txt | 0 .../ExternalTools/Terraform}/Azure/example.tf | 0 .../Terraform}/Azure/example.tfvars | 0 .../Terraform}/Azure/versions.tf | 0 .../CommonTemplates/SingleVariable.json | 0 .../Terraform}/GoogleCloud/example.tf | 0 .../Terraform}/GoogleCloud/example.tfvars | 0 .../Terraform}/GoogleCloud/test.txt | 0 .../Terraform}/GoogleCloud/versions.tf | 0 .../PlanDetailedExitCode/example.tf | 0 .../Terraform}/Simple/backend.tfvars | 0 .../Terraform}/Simple/example.tf | 0 .../Terraform/TerraformCommandsFixture.cs | 409 ++++++++ .../WithOutputSensitiveVariables/example.tf | 0 .../WithVariablesSubstitution/example.tf | 0 .../WithVariablesSubstitution/example.txt | 0 .../terraform.tfvars | 0 .../AdditionalParams/example.tf | 7 - .../CommandsFixture.cs | 883 ------------------ .../CommonTemplates/HclWithVariables.hcl | 43 - .../InlineJsonWithVariables.json | 29 - .../CommonTemplates/TemplateLoader.cs | 15 - .../TemplateDirectory/SubFolder/example.tf | 3 - .../WithVariables/example.tf | 7 - .../WithVariables/example.tfvars | 1 - 27 files changed, 409 insertions(+), 988 deletions(-) rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/AWS/example.tf (100%) rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/AWS/example.tfvars (100%) rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/AWS/test.txt (100%) rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/Azure/example.tf (100%) rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/Azure/example.tfvars (100%) rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/Azure/versions.tf (100%) rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/CommonTemplates/SingleVariable.json (100%) rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/GoogleCloud/example.tf (100%) rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/GoogleCloud/example.tfvars (100%) rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/GoogleCloud/test.txt (100%) rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/GoogleCloud/versions.tf (100%) rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/PlanDetailedExitCode/example.tf (100%) rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/Simple/backend.tfvars (100%) rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/Simple/example.tf (100%) create mode 100644 source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/TerraformCommandsFixture.cs rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/WithOutputSensitiveVariables/example.tf (100%) rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/WithVariablesSubstitution/example.tf (100%) rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/WithVariablesSubstitution/example.txt (100%) rename source/{Calamari.Terraform.Tests => Calamari.ExternalTools.Tests/ExternalTools/Terraform}/WithVariablesSubstitution/terraform.tfvars (100%) delete mode 100644 source/Calamari.Terraform.Tests/AdditionalParams/example.tf delete mode 100644 source/Calamari.Terraform.Tests/CommandsFixture.cs delete mode 100644 source/Calamari.Terraform.Tests/CommonTemplates/HclWithVariables.hcl delete mode 100644 source/Calamari.Terraform.Tests/CommonTemplates/InlineJsonWithVariables.json delete mode 100644 source/Calamari.Terraform.Tests/CommonTemplates/TemplateLoader.cs delete mode 100644 source/Calamari.Terraform.Tests/TemplateDirectory/SubFolder/example.tf delete mode 100644 source/Calamari.Terraform.Tests/WithVariables/example.tf delete mode 100644 source/Calamari.Terraform.Tests/WithVariables/example.tfvars diff --git a/source/Calamari.Terraform.Tests/AWS/example.tf b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/AWS/example.tf similarity index 100% rename from source/Calamari.Terraform.Tests/AWS/example.tf rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/AWS/example.tf diff --git a/source/Calamari.Terraform.Tests/AWS/example.tfvars b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/AWS/example.tfvars similarity index 100% rename from source/Calamari.Terraform.Tests/AWS/example.tfvars rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/AWS/example.tfvars diff --git a/source/Calamari.Terraform.Tests/AWS/test.txt b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/AWS/test.txt similarity index 100% rename from source/Calamari.Terraform.Tests/AWS/test.txt rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/AWS/test.txt diff --git a/source/Calamari.Terraform.Tests/Azure/example.tf b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/Azure/example.tf similarity index 100% rename from source/Calamari.Terraform.Tests/Azure/example.tf rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/Azure/example.tf diff --git a/source/Calamari.Terraform.Tests/Azure/example.tfvars b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/Azure/example.tfvars similarity index 100% rename from source/Calamari.Terraform.Tests/Azure/example.tfvars rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/Azure/example.tfvars diff --git a/source/Calamari.Terraform.Tests/Azure/versions.tf b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/Azure/versions.tf similarity index 100% rename from source/Calamari.Terraform.Tests/Azure/versions.tf rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/Azure/versions.tf diff --git a/source/Calamari.Terraform.Tests/CommonTemplates/SingleVariable.json b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/CommonTemplates/SingleVariable.json similarity index 100% rename from source/Calamari.Terraform.Tests/CommonTemplates/SingleVariable.json rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/CommonTemplates/SingleVariable.json diff --git a/source/Calamari.Terraform.Tests/GoogleCloud/example.tf b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/GoogleCloud/example.tf similarity index 100% rename from source/Calamari.Terraform.Tests/GoogleCloud/example.tf rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/GoogleCloud/example.tf diff --git a/source/Calamari.Terraform.Tests/GoogleCloud/example.tfvars b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/GoogleCloud/example.tfvars similarity index 100% rename from source/Calamari.Terraform.Tests/GoogleCloud/example.tfvars rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/GoogleCloud/example.tfvars diff --git a/source/Calamari.Terraform.Tests/GoogleCloud/test.txt b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/GoogleCloud/test.txt similarity index 100% rename from source/Calamari.Terraform.Tests/GoogleCloud/test.txt rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/GoogleCloud/test.txt diff --git a/source/Calamari.Terraform.Tests/GoogleCloud/versions.tf b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/GoogleCloud/versions.tf similarity index 100% rename from source/Calamari.Terraform.Tests/GoogleCloud/versions.tf rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/GoogleCloud/versions.tf diff --git a/source/Calamari.Terraform.Tests/PlanDetailedExitCode/example.tf b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/PlanDetailedExitCode/example.tf similarity index 100% rename from source/Calamari.Terraform.Tests/PlanDetailedExitCode/example.tf rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/PlanDetailedExitCode/example.tf diff --git a/source/Calamari.Terraform.Tests/Simple/backend.tfvars b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/Simple/backend.tfvars similarity index 100% rename from source/Calamari.Terraform.Tests/Simple/backend.tfvars rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/Simple/backend.tfvars diff --git a/source/Calamari.Terraform.Tests/Simple/example.tf b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/Simple/example.tf similarity index 100% rename from source/Calamari.Terraform.Tests/Simple/example.tf rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/Simple/example.tf diff --git a/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/TerraformCommandsFixture.cs b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/TerraformCommandsFixture.cs new file mode 100644 index 000000000..f7211a20b --- /dev/null +++ b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/TerraformCommandsFixture.cs @@ -0,0 +1,409 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Net; +using System.Net.Http; +using System.Net.Sockets; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Calamari.CloudAccounts; +using Calamari.Common.Plumbing.FileSystem; +using Calamari.Common.Plumbing.Variables; +using Calamari.ExternalTools.Tests.Infrastructure; +using Calamari.ExternalTools.Tests.Infrastructure.ToolStrategies; +using Calamari.Terraform; +using Calamari.Terraform.Commands; +using Calamari.Testing; +using Calamari.Testing.Azure; +using Calamari.Testing.Helpers; +using FluentAssertions; +using Newtonsoft.Json.Linq; +using NUnit.Framework; + +namespace Calamari.ExternalTools.Tests.ExternalTools.Terraform +{ + [TestFixture] + [Category("ExternalTool")] + public class TerraformCommandsFixture : ExternalToolFixture + { + protected override string PrimaryToolName => "terraform"; + + protected override Task DownloadTool(string destinationDir, string version, HttpClient client) + => TerraformStrategy.Download(destinationDir, version, client); + + readonly string planCommand = GetCommandFromType(typeof(PlanCommand)); + readonly string applyCommand = GetCommandFromType(typeof(ApplyCommand)); + readonly string destroyCommand = GetCommandFromType(typeof(DestroyCommand)); + + const string ResourceRoot = "ExternalTools/Terraform"; + + static string GetTestResourcePath(string relativePath) + => TestEnvironment.GetTestPath(ResourceRoot, relativePath); + + static string LoadTextTemplate(string templateName) + => File.ReadAllText(GetTestResourcePath(Path.Combine("CommonTemplates", templateName))); + + [OneTimeTearDown] + public static void OneTimeTearDown() + { + ClearTestDirectories(); + } + + static void ClearTestDirectories() + { + static void TryDeleteFile(string path) + { + try { File.Delete(path); } + catch (IOException) { } + } + + static void TryDeleteDirectory(string path, bool recursive) + { + try { Directory.Delete(path, recursive); } + catch (IOException) { } + } + + static void ClearTerraformDirectory(string directory) + { + var fullPath = GetTestResourcePath(directory); + TryDeleteFile(Path.Combine(fullPath, "terraform.tfstate")); + TryDeleteFile(Path.Combine(fullPath, "terraform.tfstate.backup")); + TryDeleteFile(Path.Combine(fullPath, "terraform.log")); + TryDeleteDirectory(Path.Combine(fullPath, ".terraform"), true); + TryDeleteDirectory(Path.Combine(fullPath, "terraform.tfstate.d"), true); + TryDeleteDirectory(Path.Combine(fullPath, "terraformplugins"), true); + } + + ClearTerraformDirectory("AWS"); + ClearTerraformDirectory("Azure"); + ClearTerraformDirectory("GoogleCloud"); + ClearTerraformDirectory("PlanDetailedExitCode"); + ClearTerraformDirectory("Simple"); + ClearTerraformDirectory("WithOutputSensitiveVariables"); + ClearTerraformDirectory("WithVariablesSubstitution"); + } + + /// Single end-to-end test validating the pipeline works with the manifest's Terraform version. + [Test] + public void ApplySimple_Succeeds() + { + ExecuteAndReturnLogOutput(applyCommand, _ => { }, "Simple") + .Should() + .NotContain("Error"); + } + + [Test] + public void InlineJsonTemplate_ProducesExpectedOutput() + { + string template = LoadTextTemplate("SingleVariable.json"); + var randomNumber = new Random().Next().ToString(); + + ExecuteAndReturnLogOutput(applyCommand, + _ => + { + _.Variables.Add("RandomNumber", randomNumber); + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.Template, template); + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.TemplateParameters, "{\"ami\":\"test-value\"}"); + _.Variables.Add(ScriptVariables.ScriptSource, ScriptVariables.ScriptSourceOptions.Inline); + }, + String.Empty, + _ => + { + _.OutputVariables.ContainsKey("TerraformValueOutputs[ami]").Should().BeTrue(); + _.OutputVariables["TerraformValueOutputs[ami]"].Value.Should().Be("test-value"); + }); + } + + /// Wiring test: Octostache substitution runs on variable files before terraform uses them. + [Test] + public void OutputAndSubstituteOctopusVariables() + { + ExecuteAndReturnLogOutput(applyCommand, + _ => + { + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.VarFiles, "example.txt"); + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.FileSubstitution, "example.txt"); + _.Variables.Add("Octopus.Action.StepName", "Step Name"); + _.Variables.Add("Should_Be_Substituted", "Hello World"); + _.Variables.Add("Should_Be_Substituted_in_txt", "Hello World from text"); + }, + "WithVariablesSubstitution", + result => + { + result.OutputVariables["TerraformValueOutputs[my_output]"].Value.Should().Be("Hello World"); + result.OutputVariables["TerraformValueOutputs[my_output_from_txt_file]"].Value.Should().Be("Hello World from text"); + }); + } + + /// Wiring test: terraform's sensitive outputs are marked IsSensitive in Calamari's output variables. + [Test] + public void WithOutputSensitiveVariables() + { + ExecuteAndReturnLogOutput(applyCommand, + _ => { }, + "WithOutputSensitiveVariables", + result => result.OutputVariables.Values.Should().OnlyContain(variable => variable.IsSensitive)); + } + + /// Wiring test: plan -> apply -> plan cycle with state file management (exit code 2 = changes, 0 = no changes). + [Test] + public async Task PlanDetailedExitCode() + { + using var stateFileFolder = TemporaryDirectory.Create(); + + var output = await ExecuteAndReturnResult(planCommand, PopulateVariables, "PlanDetailedExitCode"); + output.OutputVariables["TerraformPlanDetailedExitCode"].Value.Should().Be("2"); + + output = await ExecuteAndReturnResult(applyCommand, PopulateVariables, "PlanDetailedExitCode"); + output.FullLog.Should().Contain("apply -auto-approve"); + + output = await ExecuteAndReturnResult(planCommand, PopulateVariables, "PlanDetailedExitCode"); + output.OutputVariables["TerraformPlanDetailedExitCode"].Value.Should().Be("0"); + return; + + void PopulateVariables(CommandTestBuilderContext _) + { + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.AdditionalActionParams, + $"-state=\"{Path.Combine(stateFileFolder.DirectoryPath, "terraform.tfstate")}\" -refresh=false"); + } + } + + [Test] + public async Task GoogleCloudIntegration() + { + var bucketName = $"e2e-tf-{Guid.NewGuid().ToString("N").Substring(0, 6)}"; + + using var temporaryFolder = TemporaryDirectory.Create(); + CopyAllFiles(GetTestResourcePath("GoogleCloud"), temporaryFolder.DirectoryPath); + + var environmentJsonKey = await ExternalVariables.Get(ExternalVariable.GoogleCloudJsonKeyfile, CancellationToken.None); + var jsonKey = Convert.ToBase64String(Encoding.UTF8.GetBytes(environmentJsonKey)); + + void PopulateVariables(CommandTestBuilderContext _) + { + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.FileSubstitution, "test.txt"); + _.Variables.Add("Hello", "Hello World from Google Cloud"); + _.Variables.Add("bucket_name", bucketName); + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.VarFiles, "example.tfvars"); + _.Variables.Add("Octopus.Action.Terraform.GoogleCloudAccount", bool.TrueString); + _.Variables.Add("Octopus.Action.GoogleCloudAccount.JsonKey", jsonKey); + _.Variables.Add(KnownVariables.OriginalPackageDirectoryPath, temporaryFolder.DirectoryPath); + } + + var output = await ExecuteAndReturnResult(planCommand, PopulateVariables, temporaryFolder.DirectoryPath); + output.OutputVariables.ContainsKey("TerraformPlanOutput").Should().BeTrue(); + + output = await ExecuteAndReturnResult(applyCommand, PopulateVariables, temporaryFolder.DirectoryPath); + var requestUri = output.OutputVariables["TerraformValueOutputs[url]"].Value; + + string fileData; + var strategy = TestingRetryPolicies.CreateGoogleCloudHttpRetryPipeline(); + using (var client = new HttpClient()) + { + var response = await strategy.ExecuteAsync(async _ => await client.GetAsync(requestUri)); + response.IsSuccessStatusCode.Should().BeTrue(); + fileData = await response.Content.ReadAsStringAsync(); + } + + fileData.Should().Be("Hello World from Google Cloud"); + + await ExecuteAndReturnResult(destroyCommand, PopulateVariables, temporaryFolder.DirectoryPath); + using (var client = new HttpClient()) + { + var response = await strategy.ExecuteAsync(async _ => await client.GetAsync($"{requestUri}&bust_cache")); + response.StatusCode.Should().Be(HttpStatusCode.NotFound); + } + } + + [Test] + public async Task AzureIntegration() + { + var resourceGroupName = AzureTestResourceHelpers.GetResourceGroupName(); + var resourceGroupLocation = RandomAzureRegion.GetRandomRegionWithExclusions(); + + var subscriptionId = await ExternalVariables.Get(ExternalVariable.AzureSubscriptionId, CancellationToken.None); + var tenantId = await ExternalVariables.Get(ExternalVariable.AzureSubscriptionTenantId, CancellationToken.None); + var clientId = await ExternalVariables.Get(ExternalVariable.AzureSubscriptionClientId, CancellationToken.None); + var clientPassword = await ExternalVariables.Get(ExternalVariable.AzureSubscriptionPassword, CancellationToken.None); + + var random = Guid.NewGuid().ToString("N").Substring(0, 6); + var appName = $"cfe2e-{random}"; + var expectedHostName = $"{appName}.azurewebsites.net"; + + using var temporaryFolder = TemporaryDirectory.Create(); + CopyAllFiles(GetTestResourcePath("Azure"), temporaryFolder.DirectoryPath, ToolVersion); + + var output = await ExecuteAndReturnResult(planCommand, PopulateVariables, temporaryFolder.DirectoryPath); + output.OutputVariables.ContainsKey("TerraformPlanOutput").Should().BeTrue(); + + output = await ExecuteAndReturnResult(applyCommand, PopulateVariables, temporaryFolder.DirectoryPath); + output.OutputVariables["TerraformValueOutputs[url]"].Value.Should().Be(expectedHostName); + await AssertRequestResponse(HttpStatusCode.Forbidden); + + await ExecuteAndReturnResult(destroyCommand, PopulateVariables, temporaryFolder.DirectoryPath); + await AssertResponseIsNotReachable(); + return; + + void PopulateVariables(CommandTestBuilderContext _) + { + _.Variables.Add(AzureAccountVariables.SubscriptionId, subscriptionId); + _.Variables.Add(AzureAccountVariables.TenantId, tenantId); + _.Variables.Add(AzureAccountVariables.ClientId, clientId); + _.Variables.Add(AzureAccountVariables.Password, clientPassword); + _.Variables.Add("app_name", appName); + _.Variables.Add("resource_group_name", resourceGroupName); + _.Variables.Add("resource_group_location", resourceGroupLocation); + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.VarFiles, "example.tfvars"); + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.AzureManagedAccount, Boolean.TrueString); + _.Variables.Add(KnownVariables.OriginalPackageDirectoryPath, temporaryFolder.DirectoryPath); + } + + async Task AssertResponseIsNotReachable() + { + try + { + await AssertRequestResponse(HttpStatusCode.NotFound); + } + catch (HttpRequestException ex) + { + switch (ex.InnerException) + { + case SocketException socketException: + socketException.Message.Should().BeOneOf( + "No such host is known.", "Name or service not known", "nodename nor servname provided, or not known"); + break; + case WebException webException: + webException.Message.Should().StartWith("The remote name could not be resolved"); + break; + default: + throw; + } + } + } + + async Task AssertRequestResponse(HttpStatusCode expectedStatusCode) + { + using var client = new HttpClient(); + var response = await client.GetAsync($"https://{expectedHostName}").ConfigureAwait(false); + response.StatusCode.Should().Be(expectedStatusCode); + } + } + + [Test] + [Ignore("Test needs to be updated because s3 bucket doesn't seem to support ACLs anymore.")] + public async Task AWSIntegration() + { + var bucketName = $"cfe2e-tf-{Guid.NewGuid().ToString("N").Substring(0, 6)}"; + var expectedUrl = $"https://{bucketName}.s3.amazonaws.com/test.txt"; + + using var temporaryFolder = TemporaryDirectory.Create(); + CopyAllFiles(GetTestResourcePath("AWS"), temporaryFolder.DirectoryPath); + + var accessKey = await ExternalVariables.Get(ExternalVariable.AwsCloudFormationAndS3AccessKey, CancellationToken.None); + var secretKey = await ExternalVariables.Get(ExternalVariable.AwsCloudFormationAndS3SecretKey, CancellationToken.None); + + var output = await ExecuteAndReturnResult(planCommand, PopulateVariables, temporaryFolder.DirectoryPath); + output.OutputVariables.ContainsKey("TerraformPlanOutput").Should().BeTrue(); + + output = await ExecuteAndReturnResult(applyCommand, PopulateVariables, temporaryFolder.DirectoryPath); + output.OutputVariables["TerraformValueOutputs[url]"].Value.Should().Be(expectedUrl); + + string fileData; + using (var client = new HttpClient()) + fileData = await client.GetStringAsync(expectedUrl).ConfigureAwait(false); + + fileData.Should().Be("Hello World from AWS"); + + await ExecuteAndReturnResult(destroyCommand, PopulateVariables, temporaryFolder.DirectoryPath); + using (var client = new HttpClient()) + { + var response = await client.GetAsync(expectedUrl).ConfigureAwait(false); + response.StatusCode.Should().Be(HttpStatusCode.NotFound); + } + + return; + + void PopulateVariables(CommandTestBuilderContext _) + { + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.FileSubstitution, "test.txt"); + _.Variables.Add("Octopus.Action.Amazon.AccessKey", accessKey); + _.Variables.Add("Octopus.Action.Amazon.SecretKey", secretKey); + _.Variables.Add("Octopus.Action.Aws.Region", "ap-southeast-1"); + _.Variables.Add("Hello", "Hello World from AWS"); + _.Variables.Add("bucket_name", bucketName); + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.VarFiles, "example.tfvars"); + _.Variables.Add(TerraformSpecialVariables.Action.Terraform.AWSManagedAccount, "AWS"); + _.Variables.Add(KnownVariables.OriginalPackageDirectoryPath, temporaryFolder.DirectoryPath); + } + } + + static void CopyAllFiles(string sourceFolderPath, string destinationFolderPath, string? terraformVersion = null) + { + if (!Directory.Exists(sourceFolderPath)) + throw new Exception($"'{nameof(sourceFolderPath)}' ({sourceFolderPath}) does not exist!"); + + if (terraformVersion != null && Directory.Exists(Path.Combine(sourceFolderPath, terraformVersion))) + sourceFolderPath = Path.Combine(sourceFolderPath, terraformVersion); + + foreach (var filePath in Directory.GetFiles(sourceFolderPath)) + { + var destFilePath = Path.Combine(destinationFolderPath, Path.GetFileName(filePath)); + File.Copy(filePath, destFilePath, true); + } + } + + string ExecuteAndReturnLogOutput(string command, Action populateVariables, string folderName, Action? assert = null) + { + return ExecuteAndReturnResult(command, populateVariables, folderName, assert).Result.FullLog; + } + + async Task ExecuteAndReturnResult(string command, Action populateVariables, string folderName, Action? assert = null) + { + var assertResult = assert ?? (_ => { }); + var terraformFiles = Path.IsPathRooted(folderName) ? folderName : GetTestResourcePath(folderName); + + var result = await CommandTestBuilder.CreateAsync(command) + .WithArrange(context => + { + context.Variables.Add(ScriptVariables.ScriptSource, ScriptVariables.ScriptSourceOptions.Package); + context.Variables.Add(TerraformSpecialVariables.Packages.PackageId, terraformFiles); + context.Variables.Add(TerraformSpecialVariables.Calamari.TerraformCliPath, Path.GetDirectoryName(ToolExecutablePath)); + context.Variables.Add(TerraformSpecialVariables.Action.Terraform.CustomTerraformExecutable, ToolExecutablePath); + + populateVariables(context); + + var isInline = context.Variables.Get(ScriptVariables.ScriptSource)! + .Equals(ScriptVariables.ScriptSourceOptions.Inline, StringComparison.InvariantCultureIgnoreCase); + if (isInline) + { + var template = context.Variables.Get(TerraformSpecialVariables.Action.Terraform.Template); + var templateParameters = context.Variables.Get(TerraformSpecialVariables.Action.Terraform.TemplateParameters); + var isJsonFormat = true; + + try { JToken.Parse(template); } + catch { isJsonFormat = false; } + + context.WithDataFileNoBom(template!, isJsonFormat ? TerraformSpecialVariables.JsonTemplateFile : TerraformSpecialVariables.HclTemplateFile); + context.WithDataFileNoBom(templateParameters!, isJsonFormat ? TerraformSpecialVariables.JsonVariablesFile : TerraformSpecialVariables.HclVariablesFile); + } + + if (!String.IsNullOrEmpty(folderName)) + context.WithFilesToCopy(terraformFiles); + }) + .Execute(); + + assertResult(result); + return result; + } + + static string GetCommandFromType(Type commandType) + { + return commandType.CustomAttributes.Where(t => t.AttributeType == typeof(Calamari.Common.Commands.CommandAttribute)) + .Select(c => c.ConstructorArguments.First().Value) + .Single() + ?.ToString()!; + } + } +} diff --git a/source/Calamari.Terraform.Tests/WithOutputSensitiveVariables/example.tf b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/WithOutputSensitiveVariables/example.tf similarity index 100% rename from source/Calamari.Terraform.Tests/WithOutputSensitiveVariables/example.tf rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/WithOutputSensitiveVariables/example.tf diff --git a/source/Calamari.Terraform.Tests/WithVariablesSubstitution/example.tf b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/WithVariablesSubstitution/example.tf similarity index 100% rename from source/Calamari.Terraform.Tests/WithVariablesSubstitution/example.tf rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/WithVariablesSubstitution/example.tf diff --git a/source/Calamari.Terraform.Tests/WithVariablesSubstitution/example.txt b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/WithVariablesSubstitution/example.txt similarity index 100% rename from source/Calamari.Terraform.Tests/WithVariablesSubstitution/example.txt rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/WithVariablesSubstitution/example.txt diff --git a/source/Calamari.Terraform.Tests/WithVariablesSubstitution/terraform.tfvars b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/WithVariablesSubstitution/terraform.tfvars similarity index 100% rename from source/Calamari.Terraform.Tests/WithVariablesSubstitution/terraform.tfvars rename to source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/WithVariablesSubstitution/terraform.tfvars diff --git a/source/Calamari.Terraform.Tests/AdditionalParams/example.tf b/source/Calamari.Terraform.Tests/AdditionalParams/example.tf deleted file mode 100644 index 41a30c108..000000000 --- a/source/Calamari.Terraform.Tests/AdditionalParams/example.tf +++ /dev/null @@ -1,7 +0,0 @@ -variable "my_var" { - description = "the var passed in" -} - -output "my_output" { - value = "boo" -} \ No newline at end of file diff --git a/source/Calamari.Terraform.Tests/CommandsFixture.cs b/source/Calamari.Terraform.Tests/CommandsFixture.cs deleted file mode 100644 index a5b885c05..000000000 --- a/source/Calamari.Terraform.Tests/CommandsFixture.cs +++ /dev/null @@ -1,883 +0,0 @@ -using System; -using System.Collections.Generic; -using System.IO; -using System.IO.Compression; -using System.Linq; -using System.Net; -using System.Net.Http; -using System.Net.Sockets; -using System.Text; -using System.Threading; -using System.Threading.Tasks; -using Calamari.CloudAccounts; -using Calamari.Common.Features.Processes; -using Calamari.Common.Plumbing; -using Calamari.Common.Plumbing.FileSystem; -using Calamari.Common.Plumbing.Retry; -using Calamari.Common.Plumbing.Variables; -using Calamari.Terraform.Commands; -using Calamari.Terraform.Tests.CommonTemplates; -using Calamari.Testing; -using Calamari.Testing.Azure; -using Calamari.Testing.Helpers; -using FluentAssertions; -using Newtonsoft.Json; -using Newtonsoft.Json.Linq; -using NUnit.Framework; - -namespace Calamari.Terraform.Tests -{ - [TestFixture("0.13.7")] - [TestFixture("1.8.5")] - public class CommandsFixture - { - string? customTerraformExecutable; - readonly string terraformCliVersion; - readonly string planCommand = GetCommandFromType(typeof(PlanCommand)); - readonly string applyCommand = GetCommandFromType(typeof(ApplyCommand)); - readonly string destroyCommand = GetCommandFromType(typeof(DestroyCommand)); - readonly string destroyPlanCommand = GetCommandFromType(typeof(DestroyPlanCommand)); - - Version TerraformCliVersionAsObject => new(terraformCliVersion); - - public CommandsFixture(string version) - { - terraformCliVersion = version; - InstallTools().GetAwaiter().GetResult(); - } - - [OneTimeTearDown] - public static void OneTimeTearDown() - { - ClearTestDirectories(); - } - - static void ClearTestDirectories() - { - static void TryDeleteFile(string path) - { - try - { - File.Delete(TestEnvironment.GetTestPath(path)); - } - catch (IOException) - { - } - } - - static void TryDeleteDirectory(string path, bool recursive) - { - try - { - Directory.Delete(TestEnvironment.GetTestPath(path), recursive); - } - catch (IOException) - { - } - } - - static void ClearTerraformDirectory(string directory) - { - TryDeleteFile(Path.Combine(directory, "terraform.tfstate")); - TryDeleteFile(Path.Combine(directory, "terraform.tfstate.backup")); - TryDeleteFile(Path.Combine(directory, "terraform.log")); - TryDeleteDirectory(Path.Combine(directory, ".terraform"), true); - TryDeleteDirectory(Path.Combine(directory, "terraform.tfstate.d"), true); - TryDeleteDirectory(Path.Combine(directory, "terraformplugins"), true); - } - - ClearTerraformDirectory("AdditionalParams"); - ClearTerraformDirectory("AWS"); - ClearTerraformDirectory("Azure"); - ClearTerraformDirectory("GoogleCloud"); - ClearTerraformDirectory("PlanDetailedExitCode"); - ClearTerraformDirectory("Simple"); - ClearTerraformDirectory($"TemplateDirectory{Path.DirectorySeparatorChar}SubFolder"); - ClearTerraformDirectory("TemplateDirectory"); - ClearTerraformDirectory("WithOutputSensitiveVariables"); - ClearTerraformDirectory("WithVariables"); - ClearTerraformDirectory("WithVariablesSubstitution"); - } - - public async Task InstallTools() - { - ClearTestDirectories(); // pre-emptively clear test directories for better dev experience - - static string GetTerraformFileName(string currentVersion) - { - if (CalamariEnvironment.IsRunningOnNix) - return $"terraform_{currentVersion}_linux_amd64.zip"; - if (CalamariEnvironment.IsRunningOnMac) - return $"terraform_{currentVersion}_darwin_amd64.zip"; - - return $"terraform_{currentVersion}_windows_amd64.zip"; - } - - static async Task DownloadTerraform(string fileName, - HttpClient client, - string downloadBaseUrl, - string destination) - { - var zipPath = Path.Combine(Path.GetTempPath(), fileName); - using (new TemporaryFile(zipPath)) - { - using (var fileStream = - new FileStream(zipPath, FileMode.Create, FileAccess.Write, FileShare.None)) - using (var stream = await client.GetStreamAsync($"{downloadBaseUrl}{fileName}")) - { - await stream.CopyToAsync(fileStream); - } - - ZipFile.ExtractToDirectory(zipPath, destination); - } - } - - async Task DownloadCli(string destination, string version) - { - Console.WriteLine("Downloading terraform cli..."); - - var retry = new RetryTracker(3, TimeSpan.MaxValue, new LimitedExponentialRetryInterval(1000, 30000, 2)); - while (retry.Try()) - { - try - { - using (var client = new HttpClient()) - { - var downloadBaseUrl = $"https://releases.hashicorp.com/terraform/{version}/"; - var fileName = GetTerraformFileName(version); - - await DownloadTerraform(fileName, client, downloadBaseUrl, destination); - } - - customTerraformExecutable = Directory.EnumerateFiles(destination) - .FirstOrDefault(f => Path.GetFileName(f).Contains("terraform")); - Console.WriteLine($"Downloaded terraform to {customTerraformExecutable}"); - - AddExecutePermission(customTerraformExecutable!); - break; - } - catch - { - if (!retry.CanRetry()) - { - throw; - } - - await Task.Delay(retry.Sleep()); - } - } - } - - var destinationDirectoryName = Path.Combine(TestEnvironment.GetTestPath("TerraformCLIPath"), terraformCliVersion); - - if (Directory.Exists(destinationDirectoryName)) - { - var path = Directory.EnumerateFiles(destinationDirectoryName).FirstOrDefault(f => Path.GetFileName(f).Contains("terraform")); - if (path != null) - { - customTerraformExecutable = path; - Console.WriteLine($"Using existing terraform located in {customTerraformExecutable}"); - return; - } - } - - await DownloadCli(destinationDirectoryName, terraformCliVersion); - } - - [Test] - public void OverridingCacheFolder_WithNonsense_ThrowsAnError() - { - ExecuteAndReturnLogOutput("apply-terraform", - _ => - { - _.Variables.Add(ScriptVariables.ScriptSource, - ScriptVariables.ScriptSourceOptions.Package); - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.EnvironmentVariables, - JsonConvert.SerializeObject(new Dictionary { { "TF_PLUGIN_CACHE_DIR", "Nonsense" } })); - }, - "Simple") - .Should() - .ContainAll("The specified plugin cache dir", "cannot be opened"); - } - - [Test] - public void NotProvidingEnvVariables_DoesNotCrashEverything() - { - ExecuteAndReturnLogOutput("apply-terraform", - _ => - { - _.Variables.Add(ScriptVariables.ScriptSource, - ScriptVariables.ScriptSourceOptions.Package); - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.EnvironmentVariables, null); - }, - "Simple") - .Should() - .NotContain("Error"); - } - - [Test] - public void UserDefinedEnvVariables_OverrideDefaultBehaviour() - { - string template = TemplateLoader.LoadTextTemplate("SingleVariable.json"); - - ExecuteAndReturnLogOutput(applyCommand, - _ => - { - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.Template, template); - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.TemplateParameters, "{}"); - _.Variables.Add(ScriptVariables.ScriptSource, - ScriptVariables.ScriptSourceOptions.Inline); - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.EnvironmentVariables, - JsonConvert.SerializeObject(new Dictionary { { "TF_VAR_ami", "new ami value" } })); - }, - String.Empty, - _ => - { - _.OutputVariables.ContainsKey("TerraformValueOutputs[ami]").Should().BeTrue(); - _.OutputVariables["TerraformValueOutputs[ami]"].Value.Should().Be("new ami value"); - }); - } - - [Test] - public void ExtraInitParametersAreSet() - { - IgnoreIfVersionIsNotInRange("0.0.0", "1.0.0", "-get-plugins was removed in 0.15.0/1.0.0"); - var additionalParams = "-var-file=\"backend.tfvars\""; - ExecuteAndReturnLogOutput(planCommand, - _ => - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.AdditionalInitParams, additionalParams), - "Simple") - .Should() - .Contain($"init -get-plugins=true {additionalParams}"); - } - - [Test] - public void AllowPluginDownloadsShouldBeDisabled() - { - IgnoreIfVersionIsNotInRange("0.0.0", "0.15.0", "-get-plugins was removed in 0.15.0/1.0.0"); - ExecuteAndReturnLogOutput(planCommand, - _ => - { - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.AllowPluginDownloads, - false.ToString()); - }, - "Simple") - .Should() - .Contain("init -get-plugins=false"); - } - - [Test] - public void AttachLogFile() - { - ExecuteAndReturnLogOutput(planCommand, - _ => - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.AttachLogFile, true.ToString()), - "Simple", - result => - { - result.Artifacts.Count.Should().Be(1); - }); - } - - [Test] - [TestCase(typeof(PlanCommand), "plan -detailed-exitcode -var my_var=\"Hello world\"")] - [TestCase(typeof(ApplyCommand), "apply -auto-approve -var my_var=\"Hello world\"")] - [TestCase(typeof(DestroyPlanCommand), "plan -detailed-exitcode -destroy -var my_var=\"Hello world\"")] - [TestCase(typeof(DestroyCommand), "destroy -auto-approve -var my_var=\"Hello world\"")] - public void AdditionalActionParams(Type commandType, string expected) - { - var command = GetCommandFromType(commandType); - - ExecuteAndReturnLogOutput(command, - _ => - { - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.AdditionalActionParams, "-var my_var=\"Hello world\""); - }, - "AdditionalParams") - .Should() - .Contain(expected); - } - - [Test] - [TestCase(typeof(PlanCommand), "plan -detailed-exitcode -var-file=\"example.tfvars\"")] - [TestCase(typeof(ApplyCommand), "apply -auto-approve -var-file=\"example.tfvars\"")] - [TestCase(typeof(DestroyPlanCommand), "plan -detailed-exitcode -destroy -var-file=\"example.tfvars\"")] - [TestCase(typeof(DestroyCommand), "destroy -auto-approve -var-file=\"example.tfvars\"")] - public void VarFiles(Type commandType, string actual) - { - ExecuteAndReturnLogOutput(GetCommandFromType(commandType), _ => { _.Variables.Add(TerraformSpecialVariables.Action.Terraform.VarFiles, "example.tfvars"); }, "WithVariables") - .Should() - .Contain(actual); - } - - [Test] - public void WithOutputSensitiveVariables() - { - ExecuteAndReturnLogOutput(applyCommand, - _ => { }, - "WithOutputSensitiveVariables", - result => - { - result.OutputVariables.Values.Should().OnlyContain(variable => variable.IsSensitive); - }); - } - - [Test] - public void OutputAndSubstituteOctopusVariables() - { - ExecuteAndReturnLogOutput(applyCommand, - _ => - { - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.VarFiles, "example.txt"); - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.FileSubstitution, "example.txt"); - _.Variables.Add("Octopus.Action.StepName", "Step Name"); - _.Variables.Add("Should_Be_Substituted", "Hello World"); - _.Variables.Add("Should_Be_Substituted_in_txt", "Hello World from text"); - }, - "WithVariablesSubstitution", - result => - { - result.OutputVariables - .ContainsKey("TerraformValueOutputs[my_output]") - .Should() - .BeTrue(); - result.OutputVariables["TerraformValueOutputs[my_output]"] - .Value - .Should() - .Be("Hello World"); - result.OutputVariables - .ContainsKey("TerraformValueOutputs[my_output_from_txt_file]") - .Should() - .BeTrue(); - result.OutputVariables["TerraformValueOutputs[my_output_from_txt_file]"] - .Value - .Should() - .Be("Hello World from text"); - }); - } - - [Test] - public void EnableNoMatchWarningIsNotSet() - { - ExecuteAndReturnLogOutput(applyCommand, _ => { }, "Simple") - .Should() - .NotContain("No files were found that match the substitution target pattern"); - } - - [Test] - public void EnableNoMatchWarningIsNotSetWithAdditionSubstitution() - { - ExecuteAndReturnLogOutput(applyCommand, - _ => - { - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.FileSubstitution, "doesNotExist.txt"); - }, - "Simple") - .Should() - .MatchRegex("No files were found in (.*) that match the substitution target pattern '\\*\\*/\\*\\.tfvars\\.json'") - .And - .MatchRegex("No files were found in (.*) that match the substitution target pattern 'doesNotExist.txt'"); - } - - [Test] - public void EnableNoMatchWarningIsTrue() - { - ExecuteAndReturnLogOutput(applyCommand, - _ => - { - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.FileSubstitution, "doesNotExist.txt"); - _.Variables.Add("Octopus.Action.SubstituteInFiles.EnableNoMatchWarning", "true"); - }, - "Simple") - .Should() - .MatchRegex("No files were found in (.*) that match the substitution target pattern '\\*\\*/\\*\\.tfvars\\.json'") - .And - .MatchRegex("No files were found in (.*) that match the substitution target pattern 'doesNotExist.txt'"); - } - - [Test] - public void EnableNoMatchWarningIsFalse() - { - ExecuteAndReturnLogOutput(applyCommand, - _ => - { - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.FileSubstitution, "doesNotExist.txt"); - _.Variables.Add("Octopus.Action.SubstituteInFiles.EnableNoMatchWarning", "False"); - }, - "Simple") - .Should() - .NotContain("No files were found that match the substitution target pattern"); - } - - [Test] - [TestCase(typeof(PlanCommand))] - [TestCase(typeof(DestroyPlanCommand))] - public void TerraformPlanOutput(Type commandType) - { - ExecuteAndReturnLogOutput(GetCommandFromType(commandType), - _ => { _.Variables.Add("Octopus.Action.StepName", "Step Name"); }, - "Simple", - result => - { - result.OutputVariables - .ContainsKey("TerraformPlanOutput") - .Should() - .BeTrue(); - }); - } - - [Test] - public void UsesWorkSpace() - { - ExecuteAndReturnLogOutput(applyCommand, _ => { _.Variables.Add(TerraformSpecialVariables.Action.Terraform.Workspace, "myspace"); }, "Simple") - .Should() - .Contain("workspace new \"myspace\""); - } - - [Test] - public void UsesTemplateDirectory() - { - ExecuteAndReturnLogOutput(applyCommand, _ => { _.Variables.Add(TerraformSpecialVariables.Action.Terraform.TemplateDirectory, "SubFolder"); }, "TemplateDirectory") - .Should() - .Contain($"SubFolder{Path.DirectorySeparatorChar}example.tf"); - } - - [Test] - public async Task GoogleCloudIntegration() - { - var bucketName = $"e2e-tf-{Guid.NewGuid().ToString("N").Substring(0, 6)}"; - - using var temporaryFolder = TemporaryDirectory.Create(); - CopyAllFiles(TestEnvironment.GetTestPath("GoogleCloud"), temporaryFolder.DirectoryPath); - - var environmentJsonKey = await ExternalVariables.Get(ExternalVariable.GoogleCloudJsonKeyfile, CancellationToken.None); - var jsonKey = Convert.ToBase64String(Encoding.UTF8.GetBytes(environmentJsonKey)); - - void PopulateVariables(CommandTestBuilderContext _) - { - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.FileSubstitution, "test.txt"); - _.Variables.Add("Hello", "Hello World from Google Cloud"); - _.Variables.Add("bucket_name", bucketName); - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.VarFiles, "example.tfvars"); - _.Variables.Add("Octopus.Action.Terraform.GoogleCloudAccount", bool.TrueString); - _.Variables.Add("Octopus.Action.GoogleCloudAccount.JsonKey", jsonKey); - _.Variables.Add(KnownVariables.OriginalPackageDirectoryPath, temporaryFolder.DirectoryPath); - } - - var output = await ExecuteAndReturnResult(planCommand, PopulateVariables, temporaryFolder.DirectoryPath); - output.OutputVariables.ContainsKey("TerraformPlanOutput").Should().BeTrue(); - - output = await ExecuteAndReturnResult(applyCommand, PopulateVariables, temporaryFolder.DirectoryPath); - output.OutputVariables.ContainsKey("TerraformValueOutputs[url]").Should().BeTrue(); - var requestUri = output.OutputVariables["TerraformValueOutputs[url]"].Value; - - string fileData; - // This intermittently throws a 401, requiring authorization. These buckets are public by default and the client has no authorization so this looks to be a race condition in the bucket configuration. - var strategy = TestingRetryPolicies.CreateGoogleCloudHttpRetryPipeline(); - using (var client = new HttpClient()) - { - //we perform checking in a retry as sometimes it's not quite ready by the time we want to request it - var response = await strategy.ExecuteAsync(async _ => await client.GetAsync(requestUri)); - response.IsSuccessStatusCode.Should().BeTrue(); - fileData = await response.Content.ReadAsStringAsync(); - } - - fileData.Should().Be("Hello World from Google Cloud"); - - await ExecuteAndReturnResult(destroyCommand, PopulateVariables, temporaryFolder.DirectoryPath); - using (var client = new HttpClient()) - { - var response = await strategy.ExecuteAsync(async _ => await client.GetAsync($"{requestUri}&bust_cache")); - response.StatusCode.Should().Be(HttpStatusCode.NotFound); - } - } - - [Test] - public async Task AzureIntegration() - { - var resourceGroupName = AzureTestResourceHelpers.GetResourceGroupName(); - var resourceGroupLocation = RandomAzureRegion.GetRandomRegionWithExclusions(); - - var subscriptionId = await ExternalVariables.Get(ExternalVariable.AzureSubscriptionId, CancellationToken.None); - var tenantId = await ExternalVariables.Get(ExternalVariable.AzureSubscriptionTenantId, CancellationToken.None); - var clientId = await ExternalVariables.Get(ExternalVariable.AzureSubscriptionClientId, CancellationToken.None); - var clientPassword = await ExternalVariables.Get(ExternalVariable.AzureSubscriptionPassword, CancellationToken.None); - - var random = Guid.NewGuid().ToString("N").Substring(0, 6); - var appName = $"cfe2e-{random}"; - var expectedHostName = $"{appName}.azurewebsites.net"; - - using var temporaryFolder = TemporaryDirectory.Create(); - CopyAllFiles(TestEnvironment.GetTestPath("Azure"), temporaryFolder.DirectoryPath, terraformCliVersion); - - var output = await ExecuteAndReturnResult(planCommand, PopulateVariables, temporaryFolder.DirectoryPath); - output.OutputVariables.ContainsKey("TerraformPlanOutput").Should().BeTrue(); - - output = await ExecuteAndReturnResult(applyCommand, PopulateVariables, temporaryFolder.DirectoryPath); - output.OutputVariables.ContainsKey("TerraformValueOutputs[url]").Should().BeTrue(); - output.OutputVariables["TerraformValueOutputs[url]"].Value.Should().Be(expectedHostName); - await AssertRequestResponse(HttpStatusCode.Forbidden); - - await ExecuteAndReturnResult(destroyCommand, PopulateVariables, temporaryFolder.DirectoryPath); - - await AssertResponseIsNotReachable(); - return; - - void PopulateVariables(CommandTestBuilderContext _) - { - _.Variables.Add(AzureAccountVariables.SubscriptionId,subscriptionId ); - _.Variables.Add(AzureAccountVariables.TenantId,tenantId); - _.Variables.Add(AzureAccountVariables.ClientId,clientId); - _.Variables.Add(AzureAccountVariables.Password, clientPassword); - _.Variables.Add("app_name", appName); - _.Variables.Add("resource_group_name", resourceGroupName); - _.Variables.Add("resource_group_location", resourceGroupLocation); - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.VarFiles, "example.tfvars"); - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.AzureManagedAccount, Boolean.TrueString); - _.Variables.Add(KnownVariables.OriginalPackageDirectoryPath, temporaryFolder.DirectoryPath); - } - - async Task AssertResponseIsNotReachable() - { - //This will throw on some platforms and return "NotFound" on others - try - { - await AssertRequestResponse(HttpStatusCode.NotFound); - } - catch (HttpRequestException ex) - { - switch (ex.InnerException) - { - case SocketException socketException: - socketException.Message.Should() - .BeOneOf( - "No such host is known.", - "Name or service not known", //Some Linux distros - "nodename nor servname provided, or not known" //Mac - ); - break; - case WebException webException: - webException.Message.Should() - .StartWith("The remote name could not be resolved"); - break; - default: - throw; - } - } - } - - async Task AssertRequestResponse(HttpStatusCode expectedStatusCode) - { - using var client = new HttpClient(); - var response = await client.GetAsync($"https://{expectedHostName}").ConfigureAwait(false); - response.StatusCode.Should().Be(expectedStatusCode); - } - } - - //TODO: #team-modern-deployments-requests-and-discussion - [Test] - [Ignore("Test needs to be updated because s3 bucket doesn't seem to support ACLs anymore.")] - public async Task AWSIntegration() - { - var bucketName = $"cfe2e-tf-{Guid.NewGuid().ToString("N").Substring(0, 6)}"; - var expectedUrl = $"https://{bucketName}.s3.amazonaws.com/test.txt"; - - using var temporaryFolder = TemporaryDirectory.Create(); - CopyAllFiles(TestEnvironment.GetTestPath("AWS"), temporaryFolder.DirectoryPath); - - var accessKey = await ExternalVariables.Get(ExternalVariable.AwsCloudFormationAndS3AccessKey, CancellationToken.None); - var secretKey = await ExternalVariables.Get(ExternalVariable.AwsCloudFormationAndS3SecretKey, CancellationToken.None); - - var output = await ExecuteAndReturnResult(planCommand, PopulateVariables, temporaryFolder.DirectoryPath); - output.OutputVariables.ContainsKey("TerraformPlanOutput").Should().BeTrue(); - - output = await ExecuteAndReturnResult(applyCommand, PopulateVariables, temporaryFolder.DirectoryPath); - output.OutputVariables.ContainsKey("TerraformValueOutputs[url]").Should().BeTrue(); - output.OutputVariables["TerraformValueOutputs[url]"].Value.Should().Be(expectedUrl); - - string fileData; - using (var client = new HttpClient()) - fileData = await client.GetStringAsync(expectedUrl).ConfigureAwait(false); - - fileData.Should().Be("Hello World from AWS"); - - await ExecuteAndReturnResult(destroyCommand, PopulateVariables, temporaryFolder.DirectoryPath); - using (var client = new HttpClient()) - { - var response = await client.GetAsync(expectedUrl).ConfigureAwait(false); - response.StatusCode.Should().Be(HttpStatusCode.NotFound); - } - - return; - - void PopulateVariables(CommandTestBuilderContext _) - { - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.FileSubstitution, "test.txt"); - _.Variables.Add("Octopus.Action.Amazon.AccessKey", accessKey); - _.Variables.Add("Octopus.Action.Amazon.SecretKey",secretKey); - _.Variables.Add("Octopus.Action.Aws.Region", "ap-southeast-1"); - _.Variables.Add("Hello", "Hello World from AWS"); - _.Variables.Add("bucket_name", bucketName); - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.VarFiles, "example.tfvars"); - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.AWSManagedAccount, "AWS"); - _.Variables.Add(KnownVariables.OriginalPackageDirectoryPath, temporaryFolder.DirectoryPath); - } - } - - [Test] - public async Task PlanDetailedExitCode() - { - using var stateFileFolder = TemporaryDirectory.Create(); - - var output = await ExecuteAndReturnResult(planCommand, PopulateVariables, "PlanDetailedExitCode"); - output.OutputVariables.ContainsKey("TerraformPlanDetailedExitCode").Should().BeTrue(); - output.OutputVariables["TerraformPlanDetailedExitCode"].Value.Should().Be("2"); - - output = await ExecuteAndReturnResult(applyCommand, PopulateVariables, "PlanDetailedExitCode"); - output.FullLog.Should() - .Contain("apply -auto-approve"); - - output = await ExecuteAndReturnResult(planCommand, PopulateVariables, "PlanDetailedExitCode"); - output.OutputVariables.ContainsKey("TerraformPlanDetailedExitCode").Should().BeTrue(); - output.OutputVariables["TerraformPlanDetailedExitCode"].Value.Should().Be("0"); - return; - - void PopulateVariables(CommandTestBuilderContext _) - { - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.AdditionalActionParams, - $"-state=\"{Path.Combine(stateFileFolder.DirectoryPath, "terraform.tfstate")}\" -refresh=false"); - } - } - - [Test] - public void InlineHclTemplateAndVariables() - { - const string variables = "stringvar = \"default string\""; - string template = TemplateLoader.LoadTextTemplate("HclWithVariables.hcl"); - - ExecuteAndReturnLogOutput(applyCommand, - _ => - { - _.Variables.Add("RandomNumber", new Random().Next().ToString()); - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.Template, template); - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.TemplateParameters, variables); - _.Variables.Add(ScriptVariables.ScriptSource, - ScriptVariables.ScriptSourceOptions.Inline); - }, - String.Empty, - _ => - { - _.OutputVariables.ContainsKey("TerraformValueOutputs[nestedlist]").Should().BeTrue(); - _.OutputVariables.ContainsKey("TerraformValueOutputs[nestedmap]").Should().BeTrue(); - }); - } - - [Test] - public void InlineHclTemplateWithMultilineOutput() - { - const string expected = @"apiVersion: v1 -kind: ConfigMap -metadata: - name: aws-auth - namespace: kube-system -data: - mapRoles: | - - rolearn: arbitrary text - username: system:node:username - groups: - - system:bootstrappers - - system:nodes"; - string template = $@"locals {{ - config-map-aws-auth = < - { - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.Template, template); - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.TemplateParameters, ""); - _.Variables.Add(ScriptVariables.ScriptSource, - ScriptVariables.ScriptSourceOptions.Inline); - }, - String.Empty, - _ => - { - _.OutputVariables.ContainsKey("TerraformValueOutputs[config-map-aws-auth]").Should().BeTrue(); - _.OutputVariables["TerraformValueOutputs[config-map-aws-auth]"] - .Value?.TrimEnd() - .Replace("\r\n", "\n") - .Should() - .Be($"{expected.Replace("\r\n", "\n")}"); - }); - } - - [Test] - public void CanDetermineTerraformVersion() - { - ExecuteAndReturnLogOutput(applyCommand, _ => { _.Variables.Add(TerraformSpecialVariables.Action.Terraform.Workspace, "testversionspace"); }, "Simple") - .Should() - .NotContain("Could not parse Terraform CLI version"); - } - - [Test] - public void InlineJsonTemplateAndVariables() - { - const string variables = - "{\"ami\":\"new ami value\"}"; - string template = TemplateLoader.LoadTextTemplate("InlineJsonWithVariables.json"); - - var randomNumber = new Random().Next().ToString(); - - ExecuteAndReturnLogOutput(applyCommand, - _ => - { - _.Variables.Add("RandomNumber", randomNumber); - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.Template, template); - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.TemplateParameters, variables); - _.Variables.Add(ScriptVariables.ScriptSource, - ScriptVariables.ScriptSourceOptions.Inline); - }, - String.Empty, - _ => - { - _.OutputVariables.ContainsKey("TerraformValueOutputs[ami]").Should().BeTrue(); - _.OutputVariables["TerraformValueOutputs[ami]"].Value.Should().Be("new ami value"); - _.OutputVariables.ContainsKey("TerraformValueOutputs[random]").Should().BeTrue(); - _.OutputVariables["TerraformValueOutputs[random]"].Value.Should().Be(randomNumber); - }); - } - - static void CopyAllFiles(string sourceFolderPath, string destinationFolderPath, string terraformVersion = null) - { - if (Directory.Exists(sourceFolderPath)) - { - //if there is version specific folder, use that - if (terraformVersion != null && Directory.Exists(Path.Combine(sourceFolderPath, terraformVersion))) - { - sourceFolderPath = Path.Combine(sourceFolderPath, terraformVersion); - } - - var filePaths = Directory.GetFiles(sourceFolderPath); - - // Copy the files and overwrite destination files if they already exist. - foreach (var filePath in filePaths) - { - var fileName = Path.GetFileName(filePath); - var destFilePath = Path.Combine(destinationFolderPath, fileName); - File.Copy(filePath, destFilePath, true); - } - } - else - { - throw new Exception($"'{nameof(sourceFolderPath)}' ({sourceFolderPath}) does not exist!"); - } - } - - string ExecuteAndReturnLogOutput(string command, - Action populateVariables, - string folderName, - Action? assert = null) - { - return ExecuteAndReturnResult(command, populateVariables, folderName, assert).Result.FullLog; - } - - async Task ExecuteAndReturnResult(string command, Action populateVariables, string folderName, Action? assert = null) - { - var assertResult = assert ?? (_ => { }); - - var terraformFiles = Path.IsPathRooted(folderName) ? folderName : TestEnvironment.GetTestPath(folderName); - - var result = await CommandTestBuilder.CreateAsync(command) - .WithArrange(context => - { - context.Variables.Add(ScriptVariables.ScriptSource, - ScriptVariables.ScriptSourceOptions.Package); - context.Variables.Add(TerraformSpecialVariables.Packages.PackageId, terraformFiles); - context.Variables.Add(TerraformSpecialVariables.Calamari.TerraformCliPath, - Path.GetDirectoryName(customTerraformExecutable)); - context.Variables.Add(TerraformSpecialVariables.Action.Terraform.CustomTerraformExecutable, - customTerraformExecutable); - - populateVariables(context); - - var isInline = context.Variables.Get(ScriptVariables.ScriptSource)! - .Equals(ScriptVariables.ScriptSourceOptions.Inline, StringComparison.InvariantCultureIgnoreCase); - if (isInline) - { - var template = context.Variables.Get(TerraformSpecialVariables.Action.Terraform.Template); - var variables = context.Variables.Get(TerraformSpecialVariables.Action.Terraform.TemplateParameters); - var isJsonFormat = true; - - try - { - JToken.Parse(template); - } - catch - { - isJsonFormat = false; - } - - context.WithDataFileNoBom( - template!, - isJsonFormat ? TerraformSpecialVariables.JsonTemplateFile : TerraformSpecialVariables.HclTemplateFile); - context.WithDataFileNoBom( - variables!, - isJsonFormat ? TerraformSpecialVariables.JsonVariablesFile : TerraformSpecialVariables.HclVariablesFile); - } - - if (!String.IsNullOrEmpty(folderName)) - { - context.WithFilesToCopy(terraformFiles); - } - }) - .Execute(); - - assertResult(result); - return result; - } - - static string GetCommandFromType(Type commandType) - { - return commandType.CustomAttributes.Where(t => t.AttributeType == typeof(Calamari.Common.Commands.CommandAttribute)) - .Select(c => c.ConstructorArguments.First().Value) - .Single() - ?.ToString(); - } - - void IgnoreIfVersionIsNotInRange(string minimum, string maximum, string because) - { - var minimumVersion = new Version(minimum); - var maximumVersion = new Version(maximum ?? "999.0.0"); - - if (TerraformCliVersionAsObject.CompareTo(minimumVersion) < 0 - || TerraformCliVersionAsObject.CompareTo(maximumVersion) >= 0) - { - var becauseText = because is not null ? $" because {because}" : null; - Assert.Ignore($"Test ignored as terraform version is not between {minimumVersion} and {maximumVersion}{becauseText}"); - } - } - - //TODO: This is ported over from the ExecutableHelper in Sashimi.Tests.Shared. This project doesn't have a valid nuget package for net452 - static void AddExecutePermission(string exePath) - { - if (CalamariEnvironment.IsRunningOnWindows) - return; - StringBuilder stdOut = new StringBuilder(); - StringBuilder stdError = new StringBuilder(); - if (SilentProcessRunner.ExecuteCommand("chmod", - "+x " + exePath, - Path.GetDirectoryName(exePath) ?? string.Empty, - (Action)(s => stdOut.AppendLine(s)), - (Action)(s => stdError.AppendLine(s))) - .ExitCode - != 0) - throw new Exception(stdOut.ToString() + stdError?.ToString()); - } - } -} \ No newline at end of file diff --git a/source/Calamari.Terraform.Tests/CommonTemplates/HclWithVariables.hcl b/source/Calamari.Terraform.Tests/CommonTemplates/HclWithVariables.hcl deleted file mode 100644 index e40c14b18..000000000 --- a/source/Calamari.Terraform.Tests/CommonTemplates/HclWithVariables.hcl +++ /dev/null @@ -1,43 +0,0 @@ -variable stringvar { - type = string - default = "default string" -} -variable "images" { - type = map(string) - default = { - us-east-1 = "image-1234" - us-west-2 = "image-4567" - } -} -variable "test2" { - type = map - default = { - val1 = [ - "hi"] - } -} -variable "test3" { - type = map - default = { - val1 = { - val2 = "#{RandomNumber}" - } - } -} -variable "test4" { - type = map - default = { - val1 = { - val2 = [ - "hi"] - } - } -} -# Example of getting an element from a list in a map -output "nestedlist" { - value = "${element(var.test2["val1"], 0)}" -} -# Example of getting an element from a nested map -output "nestedmap" { - value = "${lookup(var.test3["val1"], "val2")}" -} \ No newline at end of file diff --git a/source/Calamari.Terraform.Tests/CommonTemplates/InlineJsonWithVariables.json b/source/Calamari.Terraform.Tests/CommonTemplates/InlineJsonWithVariables.json deleted file mode 100644 index ad1785fc6..000000000 --- a/source/Calamari.Terraform.Tests/CommonTemplates/InlineJsonWithVariables.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "variable": { - "ami": { - "type": "string", - "description": "the AMI to use", - "default": "1234567890" - } - }, - "output": { - "test": { - "value": "hi there" - }, - "test2": { - "value": [ - "hi there", - "hi again" - ] - }, - "test3": { - "value": "${tomap({ a = \"hi\" })}" - }, - "ami": { - "value": "${var.ami}" - }, - "random": { - "value": "#{RandomNumber}" - } - } -} \ No newline at end of file diff --git a/source/Calamari.Terraform.Tests/CommonTemplates/TemplateLoader.cs b/source/Calamari.Terraform.Tests/CommonTemplates/TemplateLoader.cs deleted file mode 100644 index 02316d22f..000000000 --- a/source/Calamari.Terraform.Tests/CommonTemplates/TemplateLoader.cs +++ /dev/null @@ -1,15 +0,0 @@ -using System; -using System.IO; - -namespace Calamari.Terraform.Tests.CommonTemplates -{ - public static class TemplateLoader - { - private const string TemplatesFolder = "CommonTemplates"; - - public static string LoadTextTemplate(string templateName) - { - return File.ReadAllText(Path.Combine(TemplatesFolder, templateName)); - } - } -} diff --git a/source/Calamari.Terraform.Tests/TemplateDirectory/SubFolder/example.tf b/source/Calamari.Terraform.Tests/TemplateDirectory/SubFolder/example.tf deleted file mode 100644 index f51db9540..000000000 --- a/source/Calamari.Terraform.Tests/TemplateDirectory/SubFolder/example.tf +++ /dev/null @@ -1,3 +0,0 @@ -output "my_output" { - value = "boo" -} \ No newline at end of file diff --git a/source/Calamari.Terraform.Tests/WithVariables/example.tf b/source/Calamari.Terraform.Tests/WithVariables/example.tf deleted file mode 100644 index 3f490b320..000000000 --- a/source/Calamari.Terraform.Tests/WithVariables/example.tf +++ /dev/null @@ -1,7 +0,0 @@ -variable "my_var" { - description = "the var passed in" -} - -output "my_output" { - value = "${var.my_var}" -} \ No newline at end of file diff --git a/source/Calamari.Terraform.Tests/WithVariables/example.tfvars b/source/Calamari.Terraform.Tests/WithVariables/example.tfvars deleted file mode 100644 index 731f5cd54..000000000 --- a/source/Calamari.Terraform.Tests/WithVariables/example.tfvars +++ /dev/null @@ -1 +0,0 @@ -my_var = "Hello World" \ No newline at end of file From 337c57fe8c27d8dddc3194ecf20e54f2284dee68 Mon Sep 17 00:00:00 2001 From: robert Date: Wed, 5 Aug 2026 16:01:41 +1000 Subject: [PATCH 11/16] fix: remove Calamari.AiAgent.Tests entries resurrected by rebase Rebasing onto main (which removed Calamari.AiAgent/.Tests as part of the Claude consolidation, #2073) reintroduced the AiAgent.Tests sln entries as a side effect of the original malformed-EndProject fix in Task 1. The project no longer exists on disk; drop the dangling entry. --- source/Calamari.sln | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/source/Calamari.sln b/source/Calamari.sln index d03568df1..2e6a13448 100644 --- a/source/Calamari.sln +++ b/source/Calamari.sln @@ -88,8 +88,6 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Calamari.Contracts", "Calam EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Calamari.DockerCredentialHelper", "Calamari.DockerCredentialHelper\Calamari.DockerCredentialHelper.csproj", "{B34DBEEC-7AC2-4BFE-ACDD-1788828925BD}" EndProject -Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Calamari.AiAgent.Tests", "Calamari.AiAgent.Tests\Calamari.AiAgent.Tests.csproj", "{8D3FCBF5-369E-44B3-BD72-4C11E8058027}" -EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Calamari.ExternalTools.Tests", "Calamari.ExternalTools.Tests\Calamari.ExternalTools.Tests.csproj", "{028A8E26-0E75-48C2-A639-2FE0D8B45DEE}" EndProject Global @@ -236,14 +234,6 @@ Global {B34DBEEC-7AC2-4BFE-ACDD-1788828925BD}.Debug|Any CPU.Build.0 = Debug|Any CPU {B34DBEEC-7AC2-4BFE-ACDD-1788828925BD}.Release|Any CPU.ActiveCfg = Release|Any CPU {B34DBEEC-7AC2-4BFE-ACDD-1788828925BD}.Release|Any CPU.Build.0 = Release|Any CPU - {767EB703-FF66-4955-9AE2-322A93FB69EE}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {767EB703-FF66-4955-9AE2-322A93FB69EE}.Debug|Any CPU.Build.0 = Debug|Any CPU - {767EB703-FF66-4955-9AE2-322A93FB69EE}.Release|Any CPU.ActiveCfg = Release|Any CPU - {767EB703-FF66-4955-9AE2-322A93FB69EE}.Release|Any CPU.Build.0 = Release|Any CPU - {8D3FCBF5-369E-44B3-BD72-4C11E8058027}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {8D3FCBF5-369E-44B3-BD72-4C11E8058027}.Debug|Any CPU.Build.0 = Debug|Any CPU - {8D3FCBF5-369E-44B3-BD72-4C11E8058027}.Release|Any CPU.ActiveCfg = Release|Any CPU - {8D3FCBF5-369E-44B3-BD72-4C11E8058027}.Release|Any CPU.Build.0 = Release|Any CPU {028A8E26-0E75-48C2-A639-2FE0D8B45DEE}.Debug|Any CPU.ActiveCfg = Debug|Any CPU {028A8E26-0E75-48C2-A639-2FE0D8B45DEE}.Debug|Any CPU.Build.0 = Debug|Any CPU {028A8E26-0E75-48C2-A639-2FE0D8B45DEE}.Release|Any CPU.ActiveCfg = Release|Any CPU From 3568b15241a0a20ca769d2efa26f11301a346baa Mon Sep 17 00:00:00 2001 From: robert Date: Wed, 5 Aug 2026 16:06:22 +1000 Subject: [PATCH 12/16] fix: enforce CALAMARI_TOOL_SKIP_DOWNLOAD gate per design spec Closes a final-review finding: ExternalToolFixture checked PATH unconditionally before downloading, so any tool found on PATH silently won over the manifest-pinned version regardless of CALAMARI_TOOL_SKIP_DOWNLOAD, defeating reproducibility. PATH is now only consulted when the flag is set, and downloading is the default; if the flag is set but the tool isn't on PATH, resolution now fails loudly instead of falling through to download. --- .../Infrastructure/ExternalToolFixture.cs | 24 +++++++++++++++---- .../Infrastructure/ToolResolver.cs | 12 ++++++++++ .../Infrastructure/ToolResolverTests.cs | 24 +++++++++++++++++++ 3 files changed, 56 insertions(+), 4 deletions(-) diff --git a/source/Calamari.ExternalTools.Tests/Infrastructure/ExternalToolFixture.cs b/source/Calamari.ExternalTools.Tests/Infrastructure/ExternalToolFixture.cs index 9ee45da31..73573d6a7 100644 --- a/source/Calamari.ExternalTools.Tests/Infrastructure/ExternalToolFixture.cs +++ b/source/Calamari.ExternalTools.Tests/Infrastructure/ExternalToolFixture.cs @@ -1,3 +1,4 @@ +using System; using System.Net.Http; using System.Threading.Tasks; using NUnit.Framework; @@ -6,7 +7,8 @@ namespace Calamari.ExternalTools.Tests.Infrastructure { /// /// Base class for test fixtures that depend on an external tool. - /// Resolves the tool via: env var override -> PATH -> download. + /// Resolves the tool via: env var override -> manifest highest (downloaded), unless + /// CALAMARI_TOOL_SKIP_DOWNLOAD is set, in which case the tool must be found on PATH. /// Subclasses set PrimaryToolName and provide a download strategy. /// public abstract class ExternalToolFixture @@ -28,11 +30,25 @@ public async Task ResolveTool() ToolVersion = resolver.ResolveVersion(PrimaryToolName); - var pathResult = ToolResolver.FindOnPath(PrimaryToolName); - if (pathResult != null) + if (ToolResolver.ShouldSkipDownload()) { - Log($"Found {PrimaryToolName} on PATH at {pathResult}"); + var pathResult = ToolResolver.FindOnPath(PrimaryToolName); + if (pathResult == null) + throw new InvalidOperationException($"{ToolResolver.SkipDownloadEnvVar} was set but '{PrimaryToolName}' was not found on PATH."); + + Log($"{ToolResolver.SkipDownloadEnvVar} is set; using {PrimaryToolName} found on PATH at {pathResult}"); ToolExecutablePath = pathResult; + + var installedVersion = ToolResolver.GetInstalledVersion(pathResult); + if (!string.IsNullOrEmpty(installedVersion)) + { + ToolVersion = installedVersion; + } + else + { + Log($"Could not determine the installed version of {PrimaryToolName} at {pathResult}; falling back to resolved version {ToolVersion}"); + } + return; } diff --git a/source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolver.cs b/source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolver.cs index 1fccd2e19..e7dec56ff 100644 --- a/source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolver.cs +++ b/source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolver.cs @@ -20,11 +20,23 @@ public ToolResolver(ToolManifest manifest, Action log) this.log = log; } + public const string SkipDownloadEnvVar = "CALAMARI_TOOL_SKIP_DOWNLOAD"; + public static string GetOverrideEnvVar(string toolName) { return $"CALAMARI_TOOL_{toolName.Replace("-", "_").ToUpperInvariant()}_VERSION"; } + public static bool ShouldSkipDownload() + { + return ShouldSkipDownload(Environment.GetEnvironmentVariable(SkipDownloadEnvVar)); + } + + public static bool ShouldSkipDownload(string? envVarValue) + { + return string.Equals(envVarValue, bool.TrueString, StringComparison.OrdinalIgnoreCase); + } + public string ResolveVersion(string toolName) { var envVar = GetOverrideEnvVar(toolName); diff --git a/source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolverTests.cs b/source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolverTests.cs index dd74abbb5..d68b7bdbe 100644 --- a/source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolverTests.cs +++ b/source/Calamari.ExternalTools.Tests/Infrastructure/ToolResolverTests.cs @@ -38,5 +38,29 @@ public void ShouldReturnNullForToolNotOnPath() var result = ToolResolver.FindOnPath("definitely-not-a-real-tool-abc123"); result.Should().BeNull(); } + + [Test] + public void ShouldSkipDownloadWhenFlagIsExactlyTrue() + { + ToolResolver.ShouldSkipDownload("true").Should().BeTrue(); + } + + [Test] + public void ShouldSkipDownloadWhenFlagIsTrueIgnoringCase() + { + ToolResolver.ShouldSkipDownload("TRUE").Should().BeTrue(); + } + + [Test] + public void ShouldNotSkipDownloadWhenFlagIsNotSet() + { + ToolResolver.ShouldSkipDownload(null).Should().BeFalse(); + } + + [Test] + public void ShouldNotSkipDownloadWhenFlagIsAnyOtherValue() + { + ToolResolver.ShouldSkipDownload("1").Should().BeFalse(); + } } } From 794f47f51056e9c856f0357ec0dfd25336090251 Mon Sep 17 00:00:00 2001 From: robert Date: Thu, 6 Aug 2026 20:19:49 +1000 Subject: [PATCH 13/16] test: add mocked-CLI coverage for ApplyBehaviour and PlanBehaviour Neither had unit coverage of their Execute() logic - apply arg construction, output-JSON parsing and sensitivity flagging, and the plan detailed-exit-code handling. Mocks ICommandLineRunner the same way TerraformCliExecutorFixture does, so no real terraform binary is needed. --- .../Terraform/PlanDetailedExitCode/example.tf | 4 - .../WithOutputSensitiveVariables/example.tf | 4 - .../ApplyBehaviourFixture.cs | 98 +++++++++++++++++++ .../PlanBehaviourFixture.cs | 79 +++++++++++++++ 4 files changed, 177 insertions(+), 8 deletions(-) delete mode 100644 source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/PlanDetailedExitCode/example.tf delete mode 100644 source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/WithOutputSensitiveVariables/example.tf create mode 100644 source/Calamari.Terraform.Tests/ApplyBehaviourFixture.cs create mode 100644 source/Calamari.Terraform.Tests/PlanBehaviourFixture.cs diff --git a/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/PlanDetailedExitCode/example.tf b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/PlanDetailedExitCode/example.tf deleted file mode 100644 index cb16583e4..000000000 --- a/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/PlanDetailedExitCode/example.tf +++ /dev/null @@ -1,4 +0,0 @@ -resource "local_file" "foo" { - content = "foo!" - filename = "${path.module}/foo.txt" -} \ No newline at end of file diff --git a/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/WithOutputSensitiveVariables/example.tf b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/WithOutputSensitiveVariables/example.tf deleted file mode 100644 index b4ba09567..000000000 --- a/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/WithOutputSensitiveVariables/example.tf +++ /dev/null @@ -1,4 +0,0 @@ -output "my_output" { - value = "Top Secret" - sensitive = true -} \ No newline at end of file diff --git a/source/Calamari.Terraform.Tests/ApplyBehaviourFixture.cs b/source/Calamari.Terraform.Tests/ApplyBehaviourFixture.cs new file mode 100644 index 000000000..f2b86deab --- /dev/null +++ b/source/Calamari.Terraform.Tests/ApplyBehaviourFixture.cs @@ -0,0 +1,98 @@ +using System.Collections.Generic; +using System.Threading.Tasks; +using Calamari.Common.Commands; +using Calamari.Common.Features.Processes; +using Calamari.Common.Plumbing.FileSystem; +using Calamari.Common.Plumbing.Logging; +using Calamari.Common.Plumbing.Variables; +using Calamari.Terraform.Behaviours; +using FluentAssertions; +using NSubstitute; +using NUnit.Framework; + +namespace Calamari.Terraform.Tests +{ + // Exercises ApplyBehaviour.Execute() against a mocked ICommandLineRunner - no real terraform binary + // involved. TerraformCliExecutor always issues "version --json" then "init ..." on construction, so + // captured argument index 2 is the "apply" command and index 3 is the "output -json" command. + public class ApplyBehaviourFixture + { + IVariables variables; + ILog log; + ICommandLineRunner commandLineRunner; + List capturedArguments; + + [SetUp] + public void SetUp() + { + variables = Substitute.For(); + variables.GetStrings(KnownVariables.EnabledFeatureToggles).Returns(new List()); + + log = Substitute.For(); + commandLineRunner = Substitute.For(); + capturedArguments = new List(); + } + + void ConfigureCommandLineRunner(System.Func outputFor = null, System.Func resultFor = null) + { + outputFor ??= _ => null; + resultFor ??= _ => new CommandResult("terraform", 0); + + var callCount = 0; + commandLineRunner.Execute(Arg.Do(invocation => + { + callCount++; + capturedArguments.Add(invocation.Arguments); + var output = callCount == 1 ? "Terraform v1.0.0" : outputFor(callCount); + if (output != null) + invocation.AdditionalInvocationOutputSink.WriteInfo(output); + })).Returns(_ => resultFor(callCount)); + } + + ApplyBehaviour CreateBehaviour() => new ApplyBehaviour(log, Substitute.For(), commandLineRunner); + + [Test] + public async Task Execute_ConstructsApplyArgs_WithVarFilesAndActionParams() + { + variables.Get(TerraformSpecialVariables.Action.Terraform.VarFiles).Returns("foo.tfvars"); + variables.Get(TerraformSpecialVariables.Action.Terraform.AdditionalActionParams).Returns("-lock=false"); + ConfigureCommandLineRunner(resultFor: callCount => callCount == 4 ? new CommandResult("terraform output", 1) : new CommandResult("terraform", 0)); + + await CreateBehaviour().Execute(new RunningDeployment("blah", variables)); + + capturedArguments[2].Should().Contain("apply").And.Contain("-auto-approve").And.Contain("-var-file=\"foo.tfvars\"").And.Contain("-lock=false"); + } + + [Test] + public async Task Execute_OutputCommandFails_DoesNotSetAnyOutputVariables() + { + ConfigureCommandLineRunner(resultFor: callCount => callCount == 4 ? new CommandResult("terraform output", 1) : new CommandResult("terraform", 0)); + + await CreateBehaviour().Execute(new RunningDeployment("blah", variables)); + + log.DidNotReceive().SetOutputVariable(Arg.Any(), Arg.Any(), Arg.Any(), Arg.Any()); + } + + [Test] + public async Task Execute_ParsesNonSensitiveOutputVariable() + { + const string outputJson = "{\"ami\":{\"value\":\"test-value\",\"type\":\"string\",\"sensitive\":false}}"; + ConfigureCommandLineRunner(outputFor: callCount => callCount == 4 ? outputJson : null); + + await CreateBehaviour().Execute(new RunningDeployment("blah", variables)); + + log.Received(1).SetOutputVariable("TerraformValueOutputs[ami]", "test-value", variables, false); + } + + [Test] + public async Task Execute_ParsesSensitiveOutputVariable() + { + const string outputJson = "{\"password\":{\"value\":\"s3cr3t\",\"type\":\"string\",\"sensitive\":true}}"; + ConfigureCommandLineRunner(outputFor: callCount => callCount == 4 ? outputJson : null); + + await CreateBehaviour().Execute(new RunningDeployment("blah", variables)); + + log.Received(1).SetOutputVariable("TerraformValueOutputs[password]", "s3cr3t", variables, true); + } + } +} diff --git a/source/Calamari.Terraform.Tests/PlanBehaviourFixture.cs b/source/Calamari.Terraform.Tests/PlanBehaviourFixture.cs new file mode 100644 index 000000000..433b4d628 --- /dev/null +++ b/source/Calamari.Terraform.Tests/PlanBehaviourFixture.cs @@ -0,0 +1,79 @@ +using System.Collections.Generic; +using System.Threading.Tasks; +using Calamari.Common.Commands; +using Calamari.Common.Features.Processes; +using Calamari.Common.Plumbing.FileSystem; +using Calamari.Common.Plumbing.Logging; +using Calamari.Common.Plumbing.Variables; +using Calamari.Terraform.Behaviours; +using FluentAssertions; +using NSubstitute; +using NUnit.Framework; + +namespace Calamari.Terraform.Tests +{ + // Exercises PlanBehaviour.Execute() against a mocked ICommandLineRunner. Terraform's own "plan" + // exit-code contract (0 = no changes, 2 = changes pending, anything else = real error) is not what's + // under test here - that's Terraform's behaviour, not Calamari's. What's under test is Calamari's + // handling of that contract: does exit code 2 get treated as success rather than a failure, and does + // it get captured into the TerraformPlanDetailedExitCode output variable correctly. + public class PlanBehaviourFixture + { + IVariables variables; + ILog log; + ICommandLineRunner commandLineRunner; + + [SetUp] + public void SetUp() + { + variables = Substitute.For(); + variables.GetStrings(KnownVariables.EnabledFeatureToggles).Returns(new List()); + + log = Substitute.For(); + commandLineRunner = Substitute.For(); + } + + void ConfigureCommandLineRunner(int planExitCode) + { + var callCount = 0; + commandLineRunner.Execute(Arg.Do(invocation => + { + callCount++; + if (callCount == 1) + invocation.AdditionalInvocationOutputSink.WriteInfo("Terraform v1.0.0"); + })).Returns(_ => callCount == 3 ? new CommandResult("terraform plan", planExitCode) : new CommandResult("terraform", 0)); + } + + PlanBehaviour CreateBehaviour() => new PlanBehaviour(log, Substitute.For(), commandLineRunner); + + [Test] + public async Task Execute_ChangesPending_ExitCode2_DoesNotThrow_AndCapturesDetailedExitCode() + { + ConfigureCommandLineRunner(planExitCode: 2); + + await CreateBehaviour().Execute(new RunningDeployment("blah", variables)); + + log.Received(1).SetOutputVariable(TerraformSpecialVariables.Action.Terraform.PlanDetailedExitCode, "2", variables); + } + + [Test] + public async Task Execute_NoChanges_ExitCode0_CapturesDetailedExitCode() + { + ConfigureCommandLineRunner(planExitCode: 0); + + await CreateBehaviour().Execute(new RunningDeployment("blah", variables)); + + log.Received(1).SetOutputVariable(TerraformSpecialVariables.Action.Terraform.PlanDetailedExitCode, "0", variables); + } + + [Test] + public async Task Execute_RealError_ExitCode1_Throws() + { + ConfigureCommandLineRunner(planExitCode: 1); + + var act = () => CreateBehaviour().Execute(new RunningDeployment("blah", variables)); + + await act.Should().ThrowAsync(); + } + } +} From c7b20728faca24619bd70c34007a6cd4530ffc8b Mon Sep 17 00:00:00 2001 From: robert Date: Thu, 6 Aug 2026 20:20:25 +1000 Subject: [PATCH 14/16] test: drop TerraformCommandsFixture tests now covered by mocked-CLI tests WithOutputSensitiveVariables and PlanDetailedExitCode only exercised Calamari's own output/exit-code handling, not terraform's behaviour - that's now covered by ApplyBehaviourFixture/PlanBehaviourFixture without needing a real terraform binary. --- .../Terraform/TerraformCommandsFixture.cs | 35 ------------------- 1 file changed, 35 deletions(-) diff --git a/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/TerraformCommandsFixture.cs b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/TerraformCommandsFixture.cs index f7211a20b..d2fffaf13 100644 --- a/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/TerraformCommandsFixture.cs +++ b/source/Calamari.ExternalTools.Tests/ExternalTools/Terraform/TerraformCommandsFixture.cs @@ -79,9 +79,7 @@ static void ClearTerraformDirectory(string directory) ClearTerraformDirectory("AWS"); ClearTerraformDirectory("Azure"); ClearTerraformDirectory("GoogleCloud"); - ClearTerraformDirectory("PlanDetailedExitCode"); ClearTerraformDirectory("Simple"); - ClearTerraformDirectory("WithOutputSensitiveVariables"); ClearTerraformDirectory("WithVariablesSubstitution"); } @@ -137,39 +135,6 @@ public void OutputAndSubstituteOctopusVariables() }); } - /// Wiring test: terraform's sensitive outputs are marked IsSensitive in Calamari's output variables. - [Test] - public void WithOutputSensitiveVariables() - { - ExecuteAndReturnLogOutput(applyCommand, - _ => { }, - "WithOutputSensitiveVariables", - result => result.OutputVariables.Values.Should().OnlyContain(variable => variable.IsSensitive)); - } - - /// Wiring test: plan -> apply -> plan cycle with state file management (exit code 2 = changes, 0 = no changes). - [Test] - public async Task PlanDetailedExitCode() - { - using var stateFileFolder = TemporaryDirectory.Create(); - - var output = await ExecuteAndReturnResult(planCommand, PopulateVariables, "PlanDetailedExitCode"); - output.OutputVariables["TerraformPlanDetailedExitCode"].Value.Should().Be("2"); - - output = await ExecuteAndReturnResult(applyCommand, PopulateVariables, "PlanDetailedExitCode"); - output.FullLog.Should().Contain("apply -auto-approve"); - - output = await ExecuteAndReturnResult(planCommand, PopulateVariables, "PlanDetailedExitCode"); - output.OutputVariables["TerraformPlanDetailedExitCode"].Value.Should().Be("0"); - return; - - void PopulateVariables(CommandTestBuilderContext _) - { - _.Variables.Add(TerraformSpecialVariables.Action.Terraform.AdditionalActionParams, - $"-state=\"{Path.Combine(stateFileFolder.DirectoryPath, "terraform.tfstate")}\" -refresh=false"); - } - } - [Test] public async Task GoogleCloudIntegration() { From 93043160bec085c02f40c2cfcbc7ccadf05e3ee8 Mon Sep 17 00:00:00 2001 From: robert Date: Thu, 6 Aug 2026 20:24:42 +1000 Subject: [PATCH 15/16] style: drop redundant explanatory comment in ApplyBehaviourFixture --- source/Calamari.Terraform.Tests/ApplyBehaviourFixture.cs | 3 --- 1 file changed, 3 deletions(-) diff --git a/source/Calamari.Terraform.Tests/ApplyBehaviourFixture.cs b/source/Calamari.Terraform.Tests/ApplyBehaviourFixture.cs index f2b86deab..ed737335c 100644 --- a/source/Calamari.Terraform.Tests/ApplyBehaviourFixture.cs +++ b/source/Calamari.Terraform.Tests/ApplyBehaviourFixture.cs @@ -12,9 +12,6 @@ namespace Calamari.Terraform.Tests { - // Exercises ApplyBehaviour.Execute() against a mocked ICommandLineRunner - no real terraform binary - // involved. TerraformCliExecutor always issues "version --json" then "init ..." on construction, so - // captured argument index 2 is the "apply" command and index 3 is the "output -json" command. public class ApplyBehaviourFixture { IVariables variables; From 064fb568cb8037682a381381367a71208ed63f88 Mon Sep 17 00:00:00 2001 From: robert Date: Thu, 6 Aug 2026 20:55:03 +1000 Subject: [PATCH 16/16] build: publish Calamari.ExternalTools.Tests and add a nightly test target Folds it into the existing publish/zip pipeline (no signing, same as other .Tests projects) so a build breakage is caught on every commit instead of only when the nightly TestCalamariExternalTools run happens to build it. That new target follows TestCalamariFlavourProject's shape - points at the pre-built CalamariTests/ dll, filtered to TestCategory=ExternalTool. --- build/Build.PackageCalamariProjects.cs | 8 +++++++- build/Build.TestCalamariExternalTools.cs | 21 +++++++++++++++++++++ 2 files changed, 28 insertions(+), 1 deletion(-) create mode 100644 build/Build.TestCalamariExternalTools.cs diff --git a/build/Build.PackageCalamariProjects.cs b/build/Build.PackageCalamariProjects.cs index 000ae8b3a..ba365f89d 100644 --- a/build/Build.PackageCalamariProjects.cs +++ b/build/Build.PackageCalamariProjects.cs @@ -23,7 +23,13 @@ public partial class Build //its assumed each project has a corresponding test project var testProjectNames = projectNames.Select(f => $"{f}.Tests"); - var allProjectNames = projectNames.Concat(testProjectNames).ToHashSet(); + + // Calamari.ExternalTools.Tests has no corresponding non-test flavour - it's a standalone + // test project covering tools (Terraform, etc.) that need a real external CLI binary. + // Included here (rather than via the flavour+".Tests" convention above) purely so it + // gets published/zipped for the nightly TestCalamariExternalTools build to consume - + // it is never run as part of this (default) pipeline. + var allProjectNames = projectNames.Concat(testProjectNames).Append("Calamari.ExternalTools.Tests").ToHashSet(); var calamariProjects = Solution.Projects .Where(project => allProjectNames.Contains(project.Name)) diff --git a/build/Build.TestCalamariExternalTools.cs b/build/Build.TestCalamariExternalTools.cs new file mode 100644 index 000000000..06d17dd12 --- /dev/null +++ b/build/Build.TestCalamariExternalTools.cs @@ -0,0 +1,21 @@ +using JetBrains.Annotations; + +namespace Calamari.Build; + +partial class Build +{ + [PublicAPI] + Target TestCalamariExternalTools => + target => target + .Executes(async () => + { + var dotnetPath = await LocateOrInstallDotNetSdk(); + + // Runs against a real external CLI tool (e.g. terraform), downloaded or found on PATH - + // not part of the default per-commit pipeline, invoked as its own nightly/on-demand build. + CreateTestRun("CalamariTests/Calamari.ExternalTools.Tests.dll") + .WithDotNetPath(dotnetPath) + .WithFilter("TestCategory=ExternalTool") + .Execute(); + }); +}