diff --git a/CHANGELOG.md b/CHANGELOG.md index 71b5fdb..ab1e489 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Changed +- **Template: resources keyed by Dev Container instance** (internal refactor, no behavior change; + plan 004 Phase A). The agent, its scripts, apps and environment, the PVC and the Deployment are + now `for_each` over `local.instances`, for now a list of one, `main`, the image the build + produced. `main` keeps its agent name, its Kubernetes object names (`coder-`, + `coder--data`) and its labels and selector. `moved` blocks map every old address + to `["main"]`, so updating an existing workspace to this version only renames resources in the + state: nothing is replaced, and the pod isn't restarted. `moved` blocks need Terraform >= 1.1; + the template already requires >= 1.5, and Coder's bundled Terraform is newer. Forwarded-port + apps past slot 10 (a `max_forwarded_ports` above its default) are recreated, not moved. - **Service: an image per devcontainer.json in sub-folders** ([ADR-0016](docs/decisions/0016-one-image-per-devcontainer-json.md)). Each `.devcontainer//devcontainer.json` is built with `--config` as an image of its own, `-`, after the root config's `main` image. `POST /build` lists them all in a new diff --git a/docs/claude/plans/004-devcontainer-instance-list.md b/docs/claude/plans/004-devcontainer-instance-list.md index a1a5a4c..b2225ae 100644 --- a/docs/claude/plans/004-devcontainer-instance-list.md +++ b/docs/claude/plans/004-devcontainer-instance-list.md @@ -5,7 +5,37 @@ created: 2026-10-06 # List support: `/build` returns a list of images → one Coder agent per image -## Status (2026-10-07): Phase B step 1 (service) implemented; provider, module and template still open +## Status (2026-10-07): Phase A and Phase B step 1 (service) implemented; provider, module and template step 4 still open + +**Phase A** is on `feat/template-per-instance` (template only). It differs +from this plan in these ways: + +- **The agent stays `coder_agent.main`, not `coder_agent.dev`.** Coder names + an agent after its Terraform *resource name*, not a `name` argument or the + `for_each` key (`provisioner/terraform/resources.go`, `ConvertState`: + `Name: tfResource.Name`, and `duplicate agent name` when two share it). A + rename to `dev` would rename every existing workspace's agent to "dev". +- **Blocker for Phase B step 4:** for the same reason, a `for_each` over + `coder_agent` with two or more keys fails in Coder with + `duplicate agent name: main`. One agent per image needs another shape: + for example a fixed number of statically declared agent blocks, or + Coder's sub-agents (`coder_devcontainer`). The per-instance PVC, Deployment, + scripts and apps from Phase A still apply to either. App slugs are unique + per workspace across agents too: `coder_app.forwarded_port` already + prefixes non-`main` slugs with `-`, but the vscode-web and + vscode-desktop modules' slugs would collide. +- **`main`'s Kubernetes labels are unchanged** (a Deployment's selector is + immutable, and a pod-template label change would restart the pod). Other + instances get `-` in their names *and* in `app.kubernetes.io/instance`, + plus `com.deepspacecartel.devcontainer-builder/instance=`: without the + former, `main`'s selector would also match their pods. +- **Forwarded-port apps** are keyed `"/"`. `moved` blocks cover + slots 0-9, the default `max_forwarded_ports`. +- **Verified offline** with mock providers: origin/main's template applied, + then this one planned against the same state (`terraform test` + `state_key`), for a running and a stopped workspace: only moves, + `0 to add, 0 to change, 0 to destroy`. Still to check live: updating an + existing workspace in Coder. The service side is on `feat/multi-config-builds`, recorded in [ADR-0016](../../decisions/0016-one-image-per-devcontainer-json.md) (0013 was @@ -32,7 +62,7 @@ already taken). It differs from this plan in these ways: Next: the provider's `images`/`instances`/dry run in `ModifyPlan` (step 2), then the module (step 3), the template (step 4), and the guide and README -(step 5). Phase A (the template's list-of-one refactor) is still open. +(step 5). Step 4 first needs a way around the agent-naming blocker above. ## Context diff --git a/templates/coder-kubernetes/main.tf b/templates/coder-kubernetes/main.tf index 26ee581..52a55bb 100644 --- a/templates/coder-kubernetes/main.tf +++ b/templates/coder-kubernetes/main.tf @@ -10,7 +10,10 @@ # the platform prerequisites this template itself does NOT set up (a # running devcontainer-builder instance, BuildKit). # -# Persistence: one PVC per workspace (coder--data), mounted twice via +# Instances: the agent, its scripts and apps, the PVC and the Deployment +# are per Dev Container instance (local.instances), for now only `main`. +# +# Persistence: one PVC per instance (coder--data for `main`), mounted twice via # subPaths - `home/` at the remote user's home and `workspaces/` at # /workspaces - so both survive a pod restart. A `seed-home` init container # copies the image's own home into the PVC on first start, so the mount @@ -289,16 +292,46 @@ data "coder_workspace" "me" {} data "coder_workspace_owner" "me" {} locals { - # Only read while the workspace is starting (count = start_count below), - # so stopping a workspace never depends on devcontainer-builder being up. - dc = one(data.devcontainerbuilder_devcontainer.workspace[*]) + # The Dev Container instances this workspace runs, keyed by instance id: + # each gets its own agent, PVC and Deployment (plan 004). For now a list + # of one, `main`, the image devcontainerbuilder_build built; it becomes + # devcontainerbuilder_build.workspace.images once the provider returns + # one image per devcontainer.json (ADR-0016), which has these fields. + instances = { + main = { + image = devcontainerbuilder_build.workspace.image + registry = devcontainerbuilder_build.workspace.resolved_registry + name = devcontainerbuilder_build.workspace.resolved_name + tag = devcontainerbuilder_build.workspace.resolved_tag + } + } + instance_keys = keys(local.instances) + # The ones whose resources only exist while the workspace runs (what + # count = start_count did for the single instance). + started_instances = { for inst, i in local.instances : inst => i if data.coder_workspace.me.start_count > 0 } + + # Kubernetes object names and labels. `main` keeps the names and labels + # it had before instances existed (a Deployment's selector can't change, + # and a renamed PVC would be a new, empty one); any other instance gets + # an - suffix and its own instance label, so no selector matches + # another instance's pods. + k8s_suffix = { for inst in local.instance_keys : inst => inst == "main" ? "" : "-${inst}" } + k8s_instance_labels = { for inst in local.instance_keys : inst => inst == "main" ? {} : { "com.deepspacecartel.devcontainer-builder/instance" = inst } } + + # Only read while the workspace is starting (for_each = + # local.started_instances below), so stopping a workspace never depends + # on devcontainer-builder being up. Per instance, as is everything below + # that's derived from it. one() marks the whole object sensitive (the + # data source has a sensitive input), as one(...[*]) did before + # instances - so plans show exactly what they did. + dc = { for inst in local.instance_keys : inst => one([for k, d in data.devcontainerbuilder_devcontainer.workspace : d if k == inst]) } # The user tools, IDEs and hooks run as (remoteUser, else containerUser, # else the image's USER, else root - resolved by devcontainer-builder) # and its uid/gid/home, recorded by the build: the pod runs as that user # from the start, and that home is persisted. - remote_user = try(coalesce(local.dc.remote_user, local.dc.container_user), "root") - runtime = try(local.dc.runtime, null) + remote_user = { for inst in local.instance_keys : inst => try(coalesce(local.dc[inst].remote_user, local.dc[inst].container_user), "root") } + runtime = { for inst in local.instance_keys : inst => try(local.dc[inst].runtime, null) } # An image built before devcontainer-builder 0.3.0 has no recorded # account. Fall back to uid/gid 1000 - what this template used before it # recorded them - so a workspace upgraded from an older template version @@ -306,66 +339,74 @@ locals { # takes effect once a start has succeeded on this template version # (replace_triggered_by doesn't fire when terraform_data.rebuild is # created, only when it changes). - recorded_uid = try(local.runtime.remote_user_uid, null) - recorded_gid = try(local.runtime.remote_user_gid, null) - uid = local.recorded_uid != null ? local.recorded_uid : 1000 - gid = local.recorded_gid != null ? local.recorded_gid : 1000 - home_dir = try(coalesce(local.runtime.remote_user_home), local.remote_user == "root" ? "/root" : "/home/${local.remote_user}") - - lifecycle_scripts = try(local.dc.lifecycle_scripts, {}) - env_scripts = try(local.dc.env_scripts, {}) - vscode_extensions = try(local.dc.extensions, []) - vscode_settings = try(local.dc.settings_json, "{}") - host_requirements = try(local.dc.host_requirements, null) - warnings = try(local.dc.warnings, []) - variables = try(local.dc.variables, []) + recorded_uid = { for inst in local.instance_keys : inst => try(local.runtime[inst].remote_user_uid, null) } + recorded_gid = { for inst in local.instance_keys : inst => try(local.runtime[inst].remote_user_gid, null) } + uid = { for inst in local.instance_keys : inst => local.recorded_uid[inst] != null ? local.recorded_uid[inst] : 1000 } + gid = { for inst in local.instance_keys : inst => local.recorded_gid[inst] != null ? local.recorded_gid[inst] : 1000 } + home_dir = { for inst in local.instance_keys : inst => try(coalesce(local.runtime[inst].remote_user_home), local.remote_user[inst] == "root" ? "/root" : "/home/${local.remote_user[inst]}") } + + lifecycle_scripts = { for inst in local.instance_keys : inst => try(local.dc[inst].lifecycle_scripts, {}) } + env_scripts = { for inst in local.instance_keys : inst => try(local.dc[inst].env_scripts, {}) } + vscode_extensions = { for inst in local.instance_keys : inst => try(local.dc[inst].extensions, []) } + vscode_settings = { for inst in local.instance_keys : inst => try(local.dc[inst].settings_json, "{}") } + host_requirements = { for inst in local.instance_keys : inst => try(local.dc[inst].host_requirements, null) } + warnings = { for inst in local.instance_keys : inst => try(local.dc[inst].warnings, []) } + variables = { for inst in local.instance_keys : inst => try(local.dc[inst].variables, []) } # devcontainer.json's workspaceFolder (default: Dev Containers' # "Clone Repository in Container Volume" convention, # /workspaces/), with the workspace placeholders filled in. # The repository is cloned there. repo_name = trimsuffix(basename(trimsuffix(data.coder_parameter.repository.value, "/")), ".git") - raw_workspace_folder = coalesce(try(local.dc.workspace_folder, null), "/workspaces/$${localWorkspaceFolderBasename}") - workspace_folder = replace(replace(replace(local.raw_workspace_folder, - "$${localWorkspaceFolderBasename}", local.repo_name), - "$${containerWorkspaceFolderBasename}", local.repo_name), - "$${devcontainerId}", data.coder_workspace.me.id) + raw_workspace_folder = { for inst in local.instance_keys : inst => coalesce(try(local.dc[inst].workspace_folder, null), "/workspaces/$${localWorkspaceFolderBasename}") } + workspace_folder = { for inst in local.instance_keys : inst => ( + replace(replace(replace(local.raw_workspace_folder[inst], + "$${localWorkspaceFolderBasename}", local.repo_name), + "$${containerWorkspaceFolderBasename}", local.repo_name), + "$${devcontainerId}", data.coder_workspace.me.id) + ) } # The workspace folder gets its own directory on the PVC unless it's # already persisted by the /workspaces mount (at or below it) - or is the # home itself, which a second mount at the same path would collide with. - workspace_folder_mounted = !(local.workspace_folder == "/workspaces" || startswith(local.workspace_folder, "/workspaces/") || local.workspace_folder == local.home_dir) + workspace_folder_mounted = { for inst in local.instance_keys : inst => !(local.workspace_folder[inst] == "/workspaces" || startswith(local.workspace_folder[inst], "/workspaces/") || local.workspace_folder[inst] == local.home_dir[inst]) } # Mount targets may use the workspace placeholders too. - requested_mounts = [for i, m in try(local.dc.mounts, []) : merge(m, { - index = i - target = replace(replace(replace(replace(m.target, - "$${containerWorkspaceFolder}", local.workspace_folder), - "$${localWorkspaceFolder}", local.workspace_folder), - "$${containerWorkspaceFolderBasename}", basename(local.workspace_folder)), - "$${localWorkspaceFolderBasename}", basename(local.workspace_folder)) - })] + requested_mounts = { for inst in local.instance_keys : inst => ( + [for i, m in try(local.dc[inst].mounts, []) : merge(m, { + index = i + target = replace(replace(replace(replace(m.target, + "$${containerWorkspaceFolder}", local.workspace_folder[inst]), + "$${localWorkspaceFolder}", local.workspace_folder[inst]), + "$${containerWorkspaceFolderBasename}", basename(local.workspace_folder[inst])), + "$${localWorkspaceFolderBasename}", basename(local.workspace_folder[inst])) + })] + ) } # A pod with two volume mounts at the same path is rejected, so a mount # whose target is one of the template's own mount points (or an earlier # mount's target) is dropped, with a warning. Compared without a # trailing slash. - reserved_mount_paths = compact(["/workspaces", local.home_dir, "/dev/shm", local.workspace_folder_mounted ? local.workspace_folder : ""]) - mount_keys = [for m in local.requested_mounts : trimsuffix(m.target, "/")] - mount_collides = [for i, key in local.mount_keys : - contains(local.reserved_mount_paths, key) || contains(slice(local.mount_keys, 0, i), key) - ] - mounts = [for i, m in local.requested_mounts : m if !local.mount_collides[i]] - colliding_mounts = [for i, m in local.requested_mounts : m if local.mount_collides[i]] - mount_warnings = [for m in local.colliding_mounts : - "mounts: the ${m.kind} mount at ${m.target} is not mounted - that path is already a mount point (the home, /workspaces, the workspace folder, /dev/shm or another mount)" - ] - volume_mounts = [for m in local.mounts : m if m.kind == "volume"] - tmpfs_mounts = [for m in local.mounts : m if m.kind == "tmpfs"] + reserved_mount_paths = { for inst in local.instance_keys : inst => compact(["/workspaces", local.home_dir[inst], "/dev/shm", local.workspace_folder_mounted[inst] ? local.workspace_folder[inst] : ""]) } + mount_keys = { for inst in local.instance_keys : inst => [for m in local.requested_mounts[inst] : trimsuffix(m.target, "/")] } + mount_collides = { for inst in local.instance_keys : inst => ( + [for i, key in local.mount_keys[inst] : + contains(local.reserved_mount_paths[inst], key) || contains(slice(local.mount_keys[inst], 0, i), key) + ] + ) } + mounts = { for inst in local.instance_keys : inst => [for i, m in local.requested_mounts[inst] : m if !local.mount_collides[inst][i]] } + colliding_mounts = { for inst in local.instance_keys : inst => [for i, m in local.requested_mounts[inst] : m if local.mount_collides[inst][i]] } + mount_warnings = { for inst in local.instance_keys : inst => ( + [for m in local.colliding_mounts[inst] : + "mounts: the ${m.kind} mount at ${m.target} is not mounted - that path is already a mount point (the home, /workspaces, the workspace folder, /dev/shm or another mount)" + ] + ) } + volume_mounts = { for inst in local.instance_keys : inst => [for m in local.mounts[inst] : m if m.kind == "volume"] } + tmpfs_mounts = { for inst in local.instance_keys : inst => [for m in local.mounts[inst] : m if m.kind == "tmpfs"] } # PVC directory per named volume (volumes/); anonymous volumes # get one per position. - volume_dirs = [for m in local.volume_mounts : "volumes/${coalesce(m.source, "anonymous-${m.index}")}"] + volume_dirs = { for inst in local.instance_keys : inst => [for m in local.volume_mounts[inst] : "volumes/${coalesce(m.source, "anonymous-${m.index}")}"] } - ports = try(local.dc.forward_ports, []) + ports = { for inst in local.instance_keys : inst => try(local.dc[inst].forward_ports, []) } # customizations.kubernetes (ADR-0015): Kubernetes-native resources and # node placement, read from the image's merged configuration. That has one @@ -375,19 +416,21 @@ locals { # that isn't a string or number becomes "" (invalid, so dropped with a # warning below). Maps and lists of the wrong shape are skipped. # Nothing here fails the plan. - dc_configuration = try(jsondecode(local.dc.configuration_json), {}) - k8s_entries = try([for e in local.dc_configuration.customizations.kubernetes : e if can(keys(e))], []) - k8s_requests_raw = merge(concat([{}], [for e in local.k8s_entries : try({ for k, v in e.resources.requests : k => v == null ? "" : try(tostring(v), "") if can(keys(e.resources.requests)) }, {})])...) - k8s_limits_raw = merge(concat([{}], [for e in local.k8s_entries : try({ for k, v in e.resources.limits : k => v == null ? "" : try(tostring(v), "") if can(keys(e.resources.limits)) }, {})])...) - k8s_storage_raw = try(reverse(compact([for e in local.k8s_entries : try(tostring(e.storage), null)]))[0], null) - k8s_node_selector = merge(concat([{}], [for e in local.k8s_entries : try({ for k, v in e.nodeSelector : k => v == null ? "" : try(tostring(v), "") if can(keys(e.nodeSelector)) }, {})])...) - k8s_tolerations = distinct(flatten([for e in local.k8s_entries : try([for t in e.tolerations : { - key = try(tostring(t.key), null) - operator = try(tostring(t.operator), null) - value = try(tostring(t.value), null) - effect = try(tostring(t.effect), null) - toleration_seconds = try(tostring(t.tolerationSeconds), null) - } if !can(keys(e.tolerations))], [])])) + dc_configuration = { for inst in local.instance_keys : inst => try(jsondecode(local.dc[inst].configuration_json), {}) } + k8s_entries = { for inst in local.instance_keys : inst => try([for e in local.dc_configuration[inst].customizations.kubernetes : e if can(keys(e))], []) } + k8s_requests_raw = { for inst in local.instance_keys : inst => merge(concat([{}], [for e in local.k8s_entries[inst] : try({ for k, v in e.resources.requests : k => v == null ? "" : try(tostring(v), "") if can(keys(e.resources.requests)) }, {})])...) } + k8s_limits_raw = { for inst in local.instance_keys : inst => merge(concat([{}], [for e in local.k8s_entries[inst] : try({ for k, v in e.resources.limits : k => v == null ? "" : try(tostring(v), "") if can(keys(e.resources.limits)) }, {})])...) } + k8s_storage_raw = { for inst in local.instance_keys : inst => try(reverse(compact([for e in local.k8s_entries[inst] : try(tostring(e.storage), null)]))[0], null) } + k8s_node_selector = { for inst in local.instance_keys : inst => merge(concat([{}], [for e in local.k8s_entries[inst] : try({ for k, v in e.nodeSelector : k => v == null ? "" : try(tostring(v), "") if can(keys(e.nodeSelector)) }, {})])...) } + k8s_tolerations = { for inst in local.instance_keys : inst => ( + distinct(flatten([for e in local.k8s_entries[inst] : try([for t in e.tolerations : { + key = try(tostring(t.key), null) + operator = try(tostring(t.operator), null) + value = try(tostring(t.value), null) + effect = try(tostring(t.effect), null) + toleration_seconds = try(tostring(t.tolerationSeconds), null) + } if !can(keys(e.tolerations))], [])])) + ) } # Kubernetes quantities ("500m", "2", "1.5", "512Mi", "4Gi", "1G", plain # bytes; no exponent form), parsed to numbers to compare and cap them. @@ -398,23 +441,25 @@ locals { # resource (/, e.g. nvidia.com/gpu). Also keeps the names # safe to echo in the agent metadata script. resource_name_re = "^(cpu|memory|ephemeral-storage|hugepages-[0-9]+[KMG]i|[a-z0-9]([-a-z0-9.]*[a-z0-9])?/[A-Za-z0-9]([-A-Za-z0-9_.]*[A-Za-z0-9])?)$" - k8s_requests_m = { for k, v in local.k8s_requests_raw : k => regex(local.quantity_re, v) if can(regex(local.resource_name_re, k)) && can(regex(local.quantity_re, v)) } - k8s_limits_m = { for k, v in local.k8s_limits_raw : k => regex(local.quantity_re, v) if can(regex(local.resource_name_re, k)) && can(regex(local.quantity_re, v)) } - k8s_requests = { for k, m in local.k8s_requests_m : k => local.k8s_requests_raw[k] } - k8s_limits = { for k, m in local.k8s_limits_m : k => local.k8s_limits_raw[k] } - k8s_requests_n = { for k, m in local.k8s_requests_m : k => tonumber(startswith(m[0], ".") ? "0${m[0]}" : m[0]) * local.quantity_factors[m[1] == null ? "" : m[1]] } - k8s_limits_n = { for k, m in local.k8s_limits_m : k => tonumber(startswith(m[0], ".") ? "0${m[0]}" : m[0]) * local.quantity_factors[m[1] == null ? "" : m[1]] } - k8s_storage_m = try(regex(local.quantity_re, local.k8s_storage_raw), null) - k8s_storage_n = try(tonumber(startswith(local.k8s_storage_m[0], ".") ? "0${local.k8s_storage_m[0]}" : local.k8s_storage_m[0]) * local.quantity_factors[local.k8s_storage_m[1] == null ? "" : local.k8s_storage_m[1]], null) + k8s_requests_m = { for inst in local.instance_keys : inst => { for k, v in local.k8s_requests_raw[inst] : k => regex(local.quantity_re, v) if can(regex(local.resource_name_re, k)) && can(regex(local.quantity_re, v)) } } + k8s_limits_m = { for inst in local.instance_keys : inst => { for k, v in local.k8s_limits_raw[inst] : k => regex(local.quantity_re, v) if can(regex(local.resource_name_re, k)) && can(regex(local.quantity_re, v)) } } + k8s_requests = { for inst in local.instance_keys : inst => { for k, m in local.k8s_requests_m[inst] : k => local.k8s_requests_raw[inst][k] } } + k8s_limits = { for inst in local.instance_keys : inst => { for k, m in local.k8s_limits_m[inst] : k => local.k8s_limits_raw[inst][k] } } + k8s_requests_n = { for inst in local.instance_keys : inst => { for k, m in local.k8s_requests_m[inst] : k => tonumber(startswith(m[0], ".") ? "0${m[0]}" : m[0]) * local.quantity_factors[m[1] == null ? "" : m[1]] } } + k8s_limits_n = { for inst in local.instance_keys : inst => { for k, m in local.k8s_limits_m[inst] : k => tonumber(startswith(m[0], ".") ? "0${m[0]}" : m[0]) * local.quantity_factors[m[1] == null ? "" : m[1]] } } + k8s_storage_m = { for inst in local.instance_keys : inst => try(regex(local.quantity_re, local.k8s_storage_raw[inst]), null) } + k8s_storage_n = { for inst in local.instance_keys : inst => try(tonumber(startswith(local.k8s_storage_m[inst][0], ".") ? "0${local.k8s_storage_m[inst][0]}" : local.k8s_storage_m[inst][0]) * local.quantity_factors[local.k8s_storage_m[inst][1] == null ? "" : local.k8s_storage_m[inst][1]], null) } # Any valid request or limit makes customizations.kubernetes.resources # authoritative for CPU and memory: hostRequirements' cpus/memory (and # runArgs --cpus/--memory) are then ignored. - k8s_resources_set = length(local.k8s_requests) + length(local.k8s_limits) > 0 - k8s_invalid_resources = concat( - [for k, v in local.k8s_requests_raw : "customizations.kubernetes.resources.requests: ${can(regex(local.resource_name_re, k)) ? "${k} \"${v}\" is not a Kubernetes quantity" : "\"${k}\" is not a Kubernetes resource name"} - ignored" if !contains(keys(local.k8s_requests), k)], - [for k, v in local.k8s_limits_raw : "customizations.kubernetes.resources.limits: ${can(regex(local.resource_name_re, k)) ? "${k} \"${v}\" is not a Kubernetes quantity" : "\"${k}\" is not a Kubernetes resource name"} - ignored" if !contains(keys(local.k8s_limits), k)], - local.k8s_storage_raw != null && local.k8s_storage_n == null ? ["customizations.kubernetes.storage \"${local.k8s_storage_raw}\" is not a Kubernetes quantity - ignored"] : [], - ) + k8s_resources_set = { for inst in local.instance_keys : inst => length(local.k8s_requests[inst]) + length(local.k8s_limits[inst]) > 0 } + k8s_invalid_resources = { for inst in local.instance_keys : inst => ( + concat( + [for k, v in local.k8s_requests_raw[inst] : "customizations.kubernetes.resources.requests: ${can(regex(local.resource_name_re, k)) ? "${k} \"${v}\" is not a Kubernetes quantity" : "\"${k}\" is not a Kubernetes resource name"} - ignored" if !contains(keys(local.k8s_requests[inst]), k)], + [for k, v in local.k8s_limits_raw[inst] : "customizations.kubernetes.resources.limits: ${can(regex(local.resource_name_re, k)) ? "${k} \"${v}\" is not a Kubernetes quantity" : "\"${k}\" is not a Kubernetes resource name"} - ignored" if !contains(keys(local.k8s_limits[inst]), k)], + local.k8s_storage_raw[inst] != null && local.k8s_storage_n[inst] == null ? ["customizations.kubernetes.storage \"${local.k8s_storage_raw[inst]}\" is not a Kubernetes quantity - ignored"] : [], + ) + ) } # The minimums: customizations.kubernetes' requests when it sets any # resources, else hostRequirements (ADR-0014), which are minimums as in @@ -429,168 +474,194 @@ locals { param_cpu = tonumber(data.coder_parameter.cpu.value) param_memory = tonumber(data.coder_parameter.memory.value) * local.gib param_disk = tonumber(data.coder_parameter.disk_size.value) * local.gib - hr_cpu = try(local.host_requirements.cpus, null) - hr_mem = try(local.host_requirements.memory_bytes, null) - hr_disk = try(local.host_requirements.storage_bytes, null) - required_cpu = local.k8s_resources_set ? lookup(local.k8s_requests_n, "cpu", null) : local.hr_cpu - required_mem = local.k8s_resources_set ? lookup(local.k8s_requests_n, "memory", null) : local.hr_mem - required_disk = local.k8s_storage_n != null ? local.k8s_storage_n : local.hr_disk - cpu_capped = local.required_cpu != null && try(local.required_cpu > var.max_cpu, false) - mem_capped = local.required_mem != null && try(local.required_mem > var.max_memory * local.gib, false) - reserved_cpu = local.required_cpu != null ? min(local.required_cpu, var.max_cpu) : null - reserved_mem = local.required_mem != null ? floor(min(local.required_mem, var.max_memory * local.gib)) : null - k8s_cpu_limit = lookup(local.k8s_limits_n, "cpu", null) - k8s_mem_limit = lookup(local.k8s_limits_n, "memory", null) + hr_cpu = { for inst in local.instance_keys : inst => try(local.host_requirements[inst].cpus, null) } + hr_mem = { for inst in local.instance_keys : inst => try(local.host_requirements[inst].memory_bytes, null) } + hr_disk = { for inst in local.instance_keys : inst => try(local.host_requirements[inst].storage_bytes, null) } + required_cpu = { for inst in local.instance_keys : inst => local.k8s_resources_set[inst] ? lookup(local.k8s_requests_n[inst], "cpu", null) : local.hr_cpu[inst] } + required_mem = { for inst in local.instance_keys : inst => local.k8s_resources_set[inst] ? lookup(local.k8s_requests_n[inst], "memory", null) : local.hr_mem[inst] } + required_disk = { for inst in local.instance_keys : inst => local.k8s_storage_n[inst] != null ? local.k8s_storage_n[inst] : local.hr_disk[inst] } + cpu_capped = { for inst in local.instance_keys : inst => local.required_cpu[inst] != null && try(local.required_cpu[inst] > var.max_cpu, false) } + mem_capped = { for inst in local.instance_keys : inst => local.required_mem[inst] != null && try(local.required_mem[inst] > var.max_memory * local.gib, false) } + reserved_cpu = { for inst in local.instance_keys : inst => local.required_cpu[inst] != null ? min(local.required_cpu[inst], var.max_cpu) : null } + reserved_mem = { for inst in local.instance_keys : inst => local.required_mem[inst] != null ? floor(min(local.required_mem[inst], var.max_memory * local.gib)) : null } + k8s_cpu_limit = { for inst in local.instance_keys : inst => lookup(local.k8s_limits_n[inst], "cpu", null) } + k8s_mem_limit = { for inst in local.instance_keys : inst => lookup(local.k8s_limits_n[inst], "memory", null) } # Without a reservation: 250m / 512Mi, or a smaller limit the repository # set (a request can't exceed its limit). - cpu_request_n = local.reserved_cpu != null ? local.reserved_cpu : try(min(local.k8s_cpu_limit, 0.25), 0.25) - memory_request_n = local.reserved_mem != null ? local.reserved_mem : try(min(local.k8s_mem_limit, 536870912), 536870912) - cpu_request = ( - local.reserved_cpu == null ? (try(local.k8s_cpu_limit < 0.25, false) ? lookup(local.k8s_limits, "cpu", "") : "250m") : - local.cpu_capped ? tostring(var.max_cpu) : - local.k8s_resources_set ? lookup(local.k8s_requests, "cpu", "") : tostring(local.reserved_cpu) - ) - memory_request = ( - local.reserved_mem == null ? (try(local.k8s_mem_limit < 536870912, false) ? lookup(local.k8s_limits, "memory", "") : "512Mi") : - local.mem_capped ? (local.k8s_resources_set ? "${var.max_memory}Gi" : tostring(local.reserved_mem)) : - local.k8s_resources_set ? lookup(local.k8s_requests, "memory", "") : tostring(local.reserved_mem) - ) - cpu_limit_low = try(local.k8s_cpu_limit < local.cpu_request_n, false) - mem_limit_low = try(local.k8s_mem_limit < local.memory_request_n, false) - cpu_limit = ( - local.k8s_cpu_limit != null ? (local.cpu_limit_low ? local.cpu_request : lookup(local.k8s_limits, "cpu", "")) : - local.reserved_cpu != null && try(local.reserved_cpu > local.param_cpu, false) ? local.cpu_request : data.coder_parameter.cpu.value - ) - memory_limit = ( - local.k8s_mem_limit != null ? (local.mem_limit_low ? local.memory_request : lookup(local.k8s_limits, "memory", "")) : - local.reserved_mem != null && try(local.reserved_mem > local.param_memory, false) ? local.memory_request : "${data.coder_parameter.memory.value}Gi" - ) - disk_size = local.required_disk != null && try(local.required_disk > local.param_disk, false) ? (local.k8s_storage_n != null ? local.k8s_storage_raw : tostring(floor(local.required_disk))) : "${data.coder_parameter.disk_size.value}Gi" + cpu_request_n = { for inst in local.instance_keys : inst => local.reserved_cpu[inst] != null ? local.reserved_cpu[inst] : try(min(local.k8s_cpu_limit[inst], 0.25), 0.25) } + memory_request_n = { for inst in local.instance_keys : inst => local.reserved_mem[inst] != null ? local.reserved_mem[inst] : try(min(local.k8s_mem_limit[inst], 536870912), 536870912) } + cpu_request = { for inst in local.instance_keys : inst => ( + ( + local.reserved_cpu[inst] == null ? (try(local.k8s_cpu_limit[inst] < 0.25, false) ? lookup(local.k8s_limits[inst], "cpu", "") : "250m") : + local.cpu_capped[inst] ? tostring(var.max_cpu) : + local.k8s_resources_set[inst] ? lookup(local.k8s_requests[inst], "cpu", "") : tostring(local.reserved_cpu[inst]) + ) + ) } + memory_request = { for inst in local.instance_keys : inst => ( + ( + local.reserved_mem[inst] == null ? (try(local.k8s_mem_limit[inst] < 536870912, false) ? lookup(local.k8s_limits[inst], "memory", "") : "512Mi") : + local.mem_capped[inst] ? (local.k8s_resources_set[inst] ? "${var.max_memory}Gi" : tostring(local.reserved_mem[inst])) : + local.k8s_resources_set[inst] ? lookup(local.k8s_requests[inst], "memory", "") : tostring(local.reserved_mem[inst]) + ) + ) } + cpu_limit_low = { for inst in local.instance_keys : inst => try(local.k8s_cpu_limit[inst] < local.cpu_request_n[inst], false) } + mem_limit_low = { for inst in local.instance_keys : inst => try(local.k8s_mem_limit[inst] < local.memory_request_n[inst], false) } + cpu_limit = { for inst in local.instance_keys : inst => ( + ( + local.k8s_cpu_limit[inst] != null ? (local.cpu_limit_low[inst] ? local.cpu_request[inst] : lookup(local.k8s_limits[inst], "cpu", "")) : + local.reserved_cpu[inst] != null && try(local.reserved_cpu[inst] > local.param_cpu, false) ? local.cpu_request[inst] : data.coder_parameter.cpu.value + ) + ) } + memory_limit = { for inst in local.instance_keys : inst => ( + ( + local.k8s_mem_limit[inst] != null ? (local.mem_limit_low[inst] ? local.memory_request[inst] : lookup(local.k8s_limits[inst], "memory", "")) : + local.reserved_mem[inst] != null && try(local.reserved_mem[inst] > local.param_memory, false) ? local.memory_request[inst] : "${data.coder_parameter.memory.value}Gi" + ) + ) } + disk_size = { for inst in local.instance_keys : inst => local.required_disk[inst] != null && try(local.required_disk[inst] > local.param_disk, false) ? (local.k8s_storage_n[inst] != null ? local.k8s_storage_raw[inst] : tostring(floor(local.required_disk[inst]))) : "${data.coder_parameter.disk_size.value}Gi" } # ephemeral-storage: passed through (no template cap), limit never below # the request. - ephemeral_request = lookup(local.k8s_requests, "ephemeral-storage", null) - ephemeral_limit_low = try(local.k8s_limits_n["ephemeral-storage"] < local.k8s_requests_n["ephemeral-storage"], false) - ephemeral_limit = local.ephemeral_limit_low ? local.ephemeral_request : lookup(local.k8s_limits, "ephemeral-storage", null) + ephemeral_request = { for inst in local.instance_keys : inst => lookup(local.k8s_requests[inst], "ephemeral-storage", null) } + ephemeral_limit_low = { for inst in local.instance_keys : inst => try(local.k8s_limits_n[inst]["ephemeral-storage"] < local.k8s_requests_n[inst]["ephemeral-storage"], false) } + ephemeral_limit = { for inst in local.instance_keys : inst => local.ephemeral_limit_low[inst] ? local.ephemeral_request[inst] : lookup(local.k8s_limits[inst], "ephemeral-storage", null) } # Extended resources (nvidia.com/gpu, ...) and hugepages can't be # overcommitted: Kubernetes needs request == limit, and defaults the # request to the limit. So they're set as limits only - the limit if # given, else the request. - k8s_extended_names = distinct([for k in concat(keys(local.k8s_requests), keys(local.k8s_limits)) : k if !contains(["cpu", "memory", "ephemeral-storage"], k)]) - k8s_extended_limits = { for k in local.k8s_extended_names : k => lookup(local.k8s_limits, k, lookup(local.k8s_requests, k, null)) } - gpu_json = try(local.host_requirements.gpu_json, null) + k8s_extended_names = { for inst in local.instance_keys : inst => distinct([for k in concat(keys(local.k8s_requests[inst]), keys(local.k8s_limits[inst])) : k if !contains(["cpu", "memory", "ephemeral-storage"], k)]) } + k8s_extended_limits = { for inst in local.instance_keys : inst => { for k in local.k8s_extended_names[inst] : k => lookup(local.k8s_limits[inst], k, lookup(local.k8s_requests[inst], k, null)) } } + gpu_json = { for inst in local.instance_keys : inst => try(local.host_requirements[inst].gpu_json, null) } # hostRequirements.gpu: true or {cores, memory} -> one GPU; "optional" or # unset -> none. customizations.kubernetes' nvidia.com/gpu wins. - gpu_limit = local.gpu_json != null && local.gpu_json != "\"optional\"" && local.gpu_json != "false" ? { "nvidia.com/gpu" = "1" } : {} - pod_requests = { for k, v in { - "cpu" = local.cpu_request - "memory" = local.memory_request - "ephemeral-storage" = local.ephemeral_request - } : k => v if v != null } - pod_limits = { for k, v in merge({ - "cpu" = local.cpu_limit - "memory" = local.memory_limit - "ephemeral-storage" = local.ephemeral_limit - }, local.gpu_limit, local.k8s_extended_limits) : k => v if v != null } - - resource_warnings = concat( - local.k8s_invalid_resources, - local.k8s_resources_set && (local.hr_cpu != null || local.hr_mem != null) ? [ - "hostRequirements ${join(" and ", compact([local.hr_cpu != null ? "cpus ${local.hr_cpu}" : "", local.hr_mem != null ? "memory ${format("%.1f", local.hr_mem / local.gib)} GiB" : ""]))} (or runArgs --cpus/--memory) ignored - customizations.kubernetes.resources sets the workspace's resources" - ] : [], - local.k8s_storage_n != null && local.hr_disk != null ? ["hostRequirements.storage ${format("%.1f", local.hr_disk / local.gib)} GiB ignored - customizations.kubernetes.storage ${local.k8s_storage_raw} sets the volume's minimum size"] : [], - local.cpu_capped ? ["${local.k8s_resources_set ? "customizations.kubernetes.resources.requests.cpu ${lookup(local.k8s_requests, "cpu", "")}" : "hostRequirements.cpus ${local.required_cpu}"} is more than this template allows (max_cpu ${var.max_cpu}) - reserving ${var.max_cpu}"] : [], - local.mem_capped ? ["${local.k8s_resources_set ? "customizations.kubernetes.resources.requests.memory ${lookup(local.k8s_requests, "memory", "")}" : "hostRequirements.memory ${format("%.1f", local.required_mem / local.gib)} GiB"} is more than this template allows (max_memory ${var.max_memory} GiB) - reserving ${var.max_memory} GiB"] : [], - local.cpu_limit_low ? ["customizations.kubernetes.resources.limits.cpu ${lookup(local.k8s_limits, "cpu", "")} is below the request ${local.cpu_request} - the limit is ${local.cpu_request}"] : [], - local.mem_limit_low ? ["customizations.kubernetes.resources.limits.memory ${lookup(local.k8s_limits, "memory", "")} is below the request ${local.memory_request} - the limit is ${local.memory_request}"] : [], - local.ephemeral_limit_low ? ["customizations.kubernetes.resources.limits.ephemeral-storage ${lookup(local.k8s_limits, "ephemeral-storage", "")} is below the request ${local.ephemeral_request} - the limit is ${local.ephemeral_request}"] : [], - [for k in local.k8s_extended_names : "customizations.kubernetes.resources: ${k} request ${local.k8s_requests[k]} differs from its limit ${local.k8s_limits[k]} - Kubernetes needs them equal; using the limit" if contains(keys(local.k8s_requests), k) && contains(keys(local.k8s_limits), k) && try(local.k8s_requests_n[k] != local.k8s_limits_n[k], false)], - ) + gpu_limit = { for inst in local.instance_keys : inst => local.gpu_json[inst] != null && local.gpu_json[inst] != "\"optional\"" && local.gpu_json[inst] != "false" ? { "nvidia.com/gpu" = "1" } : {} } + pod_requests = { for inst in local.instance_keys : inst => ( + { for k, v in { + "cpu" = local.cpu_request[inst] + "memory" = local.memory_request[inst] + "ephemeral-storage" = local.ephemeral_request[inst] + } : k => v if v != null } + ) } + pod_limits = { for inst in local.instance_keys : inst => ( + { for k, v in merge({ + "cpu" = local.cpu_limit[inst] + "memory" = local.memory_limit[inst] + "ephemeral-storage" = local.ephemeral_limit[inst] + }, local.gpu_limit[inst], local.k8s_extended_limits[inst]) : k => v if v != null } + ) } + + resource_warnings = { for inst in local.instance_keys : inst => ( + concat( + local.k8s_invalid_resources[inst], + local.k8s_resources_set[inst] && (local.hr_cpu[inst] != null || local.hr_mem[inst] != null) ? [ + "hostRequirements ${join(" and ", compact([local.hr_cpu[inst] != null ? "cpus ${local.hr_cpu[inst]}" : "", local.hr_mem[inst] != null ? "memory ${format("%.1f", local.hr_mem[inst] / local.gib)} GiB" : ""]))} (or runArgs --cpus/--memory) ignored - customizations.kubernetes.resources sets the workspace's resources" + ] : [], + local.k8s_storage_n[inst] != null && local.hr_disk[inst] != null ? ["hostRequirements.storage ${format("%.1f", local.hr_disk[inst] / local.gib)} GiB ignored - customizations.kubernetes.storage ${local.k8s_storage_raw[inst]} sets the volume's minimum size"] : [], + local.cpu_capped[inst] ? ["${local.k8s_resources_set[inst] ? "customizations.kubernetes.resources.requests.cpu ${lookup(local.k8s_requests[inst], "cpu", "")}" : "hostRequirements.cpus ${local.required_cpu[inst]}"} is more than this template allows (max_cpu ${var.max_cpu}) - reserving ${var.max_cpu}"] : [], + local.mem_capped[inst] ? ["${local.k8s_resources_set[inst] ? "customizations.kubernetes.resources.requests.memory ${lookup(local.k8s_requests[inst], "memory", "")}" : "hostRequirements.memory ${format("%.1f", local.required_mem[inst] / local.gib)} GiB"} is more than this template allows (max_memory ${var.max_memory} GiB) - reserving ${var.max_memory} GiB"] : [], + local.cpu_limit_low[inst] ? ["customizations.kubernetes.resources.limits.cpu ${lookup(local.k8s_limits[inst], "cpu", "")} is below the request ${local.cpu_request[inst]} - the limit is ${local.cpu_request[inst]}"] : [], + local.mem_limit_low[inst] ? ["customizations.kubernetes.resources.limits.memory ${lookup(local.k8s_limits[inst], "memory", "")} is below the request ${local.memory_request[inst]} - the limit is ${local.memory_request[inst]}"] : [], + local.ephemeral_limit_low[inst] ? ["customizations.kubernetes.resources.limits.ephemeral-storage ${lookup(local.k8s_limits[inst], "ephemeral-storage", "")} is below the request ${local.ephemeral_request[inst]} - the limit is ${local.ephemeral_request[inst]}"] : [], + [for k in local.k8s_extended_names[inst] : "customizations.kubernetes.resources: ${k} request ${local.k8s_requests[inst][k]} differs from its limit ${local.k8s_limits[inst][k]} - Kubernetes needs them equal; using the limit" if contains(keys(local.k8s_requests[inst]), k) && contains(keys(local.k8s_limits[inst]), k) && try(local.k8s_requests_n[inst][k] != local.k8s_limits_n[inst][k], false)], + ) + ) } # Node placement, only with var.allow_node_placement: the repository's # nodeSelector under the template's own (which always wins), and its # tolerations. Tolerations Kubernetes would reject are dropped. template_node_selector = { "kubernetes.io/arch" = local.arch } - node_selector = merge({ for k, v in local.k8s_node_selector : k => v if var.allow_node_placement }, local.template_node_selector) - toleration_valid = [for t in local.k8s_tolerations : - contains(["Equal", "Exists"], t.operator == null ? "Equal" : t.operator) && - contains(["", "NoSchedule", "PreferNoSchedule", "NoExecute"], t.effect == null ? "" : t.effect) && - ((t.key == null ? "" : t.key) != "" || t.operator == "Exists") && - !(t.operator == "Exists" && (t.value == null ? "" : t.value) != "") && - (t.toleration_seconds == null || (t.effect == "NoExecute" && can(regex("^-?[0-9]+$", t.toleration_seconds)))) - ] - tolerations = [for i, t in local.k8s_tolerations : t if var.allow_node_placement && local.toleration_valid[i]] - placement_warnings = concat( - [for i, t in local.k8s_tolerations : "customizations.kubernetes.tolerations: ${jsonencode(t)} is not a valid toleration - ignored" if var.allow_node_placement && !local.toleration_valid[i]], - [for k, v in local.template_node_selector : "customizations.kubernetes.nodeSelector ${k}=${local.k8s_node_selector[k]} ignored - the template sets ${k}=${v}" if var.allow_node_placement && lookup(local.k8s_node_selector, k, v) != v], - !var.allow_node_placement && length(local.k8s_node_selector) + length(local.k8s_tolerations) > 0 ? [ - "customizations.kubernetes ${join(" and ", compact([length(local.k8s_node_selector) > 0 ? "nodeSelector" : "", length(local.k8s_tolerations) > 0 ? "tolerations" : ""]))} ignored, allow_node_placement is off (a template variable)" - ] : [], - ) + node_selector = { for inst in local.instance_keys : inst => merge({ for k, v in local.k8s_node_selector[inst] : k => v if var.allow_node_placement }, local.template_node_selector) } + toleration_valid = { for inst in local.instance_keys : inst => ( + [for t in local.k8s_tolerations[inst] : + contains(["Equal", "Exists"], t.operator == null ? "Equal" : t.operator) && + contains(["", "NoSchedule", "PreferNoSchedule", "NoExecute"], t.effect == null ? "" : t.effect) && + ((t.key == null ? "" : t.key) != "" || t.operator == "Exists") && + !(t.operator == "Exists" && (t.value == null ? "" : t.value) != "") && + (t.toleration_seconds == null || (t.effect == "NoExecute" && can(regex("^-?[0-9]+$", t.toleration_seconds)))) + ] + ) } + tolerations = { for inst in local.instance_keys : inst => [for i, t in local.k8s_tolerations[inst] : t if var.allow_node_placement && local.toleration_valid[inst][i]] } + placement_warnings = { for inst in local.instance_keys : inst => ( + concat( + [for i, t in local.k8s_tolerations[inst] : "customizations.kubernetes.tolerations: ${jsonencode(t)} is not a valid toleration - ignored" if var.allow_node_placement && !local.toleration_valid[inst][i]], + [for k, v in local.template_node_selector : "customizations.kubernetes.nodeSelector ${k}=${local.k8s_node_selector[inst][k]} ignored - the template sets ${k}=${v}" if var.allow_node_placement && lookup(local.k8s_node_selector[inst], k, v) != v], + !var.allow_node_placement && length(local.k8s_node_selector[inst]) + length(local.k8s_tolerations[inst]) > 0 ? [ + "customizations.kubernetes ${join(" and ", compact([length(local.k8s_node_selector[inst]) > 0 ? "nodeSelector" : "", length(local.k8s_tolerations[inst]) > 0 ? "tolerations" : ""]))} ignored, allow_node_placement is off (a template variable)" + ] : [], + ) + ) } # What the workspace got, for the agent metadata: request / limit, and # where they came from. - resources_source = join(" and ", compact([ - local.k8s_resources_set || local.k8s_storage_n != null ? "customizations.kubernetes" : "", - (!local.k8s_resources_set && (local.hr_cpu != null || local.hr_mem != null)) || (local.k8s_storage_n == null && local.hr_disk != null) ? "hostRequirements" : "", - ])) - resources_summary = join(" · ", concat( - [ - "CPU ${local.cpu_request} / ${local.cpu_limit}", - "memory ${local.memory_request} / ${local.memory_limit}", - "disk ${local.disk_size}", - ], - [for k in sort(distinct(concat(keys(local.pod_requests), keys(local.pod_limits)))) : "${k} ${lookup(local.pod_requests, k, lookup(local.pod_limits, k, ""))} / ${lookup(local.pod_limits, k, "none")}" if !contains(["cpu", "memory"], k)], - ["from ${local.resources_source != "" ? local.resources_source : "the parameters"}"], - )) + resources_source = { for inst in local.instance_keys : inst => ( + join(" and ", compact([ + local.k8s_resources_set[inst] || local.k8s_storage_n[inst] != null ? "customizations.kubernetes" : "", + (!local.k8s_resources_set[inst] && (local.hr_cpu[inst] != null || local.hr_mem[inst] != null)) || (local.k8s_storage_n[inst] == null && local.hr_disk[inst] != null) ? "hostRequirements" : "", + ])) + ) } + resources_summary = { for inst in local.instance_keys : inst => ( + join(" · ", concat( + [ + "CPU ${local.cpu_request[inst]} / ${local.cpu_limit[inst]}", + "memory ${local.memory_request[inst]} / ${local.memory_limit[inst]}", + "disk ${local.disk_size[inst]}", + ], + [for k in sort(distinct(concat(keys(local.pod_requests[inst]), keys(local.pod_limits[inst])))) : "${k} ${lookup(local.pod_requests[inst], k, lookup(local.pod_limits[inst], k, ""))} / ${lookup(local.pod_limits[inst], k, "none")}" if !contains(["cpu", "memory"], k)], + ["from ${local.resources_source[inst] != "" ? local.resources_source[inst] : "the parameters"}"], + )) + ) } # Pod Security "baseline" (the usual namespace level) only allows these # added capabilities; anything else (SYS_PTRACE, NET_ADMIN, ...) would get # the whole pod rejected, so it's only added with var.allow_privileged. baseline_capabilities = ["AUDIT_WRITE", "CHOWN", "DAC_OVERRIDE", "FOWNER", "FSETID", "KILL", "MKNOD", "NET_BIND_SERVICE", "SETFCAP", "SETGID", "SETPCAP", "SETUID", "SYS_CHROOT"] - requested_capabilities = try(local.runtime.cap_add, []) - capabilities = var.allow_privileged ? local.requested_capabilities : [for c in local.requested_capabilities : c if contains(local.baseline_capabilities, c)] - skipped_capabilities = var.allow_privileged ? [] : [for c in local.requested_capabilities : c if !contains(local.baseline_capabilities, c)] - all_warnings = concat(local.warnings, local.resource_warnings, local.placement_warnings, local.mount_warnings, length(local.skipped_capabilities) > 0 ? [ - "capAdd ${join(", ", local.skipped_capabilities)} not added - Pod Security baseline forbids it (template variable allow_privileged enables it)" - ] : [], local.dc != null && local.recorded_uid == null ? [ - "the image doesn't record its remote user's uid/gid (built by devcontainer-builder older than 0.3.0) - running as uid/gid 1000; bump the Rebuild parameter to rebuild it" - ] : []) + requested_capabilities = { for inst in local.instance_keys : inst => try(local.runtime[inst].cap_add, []) } + capabilities = { for inst in local.instance_keys : inst => var.allow_privileged ? local.requested_capabilities[inst] : [for c in local.requested_capabilities[inst] : c if contains(local.baseline_capabilities, c)] } + skipped_capabilities = { for inst in local.instance_keys : inst => var.allow_privileged ? [] : [for c in local.requested_capabilities[inst] : c if !contains(local.baseline_capabilities, c)] } + all_warnings = { for inst in local.instance_keys : inst => ( + concat(local.warnings[inst], local.resource_warnings[inst], local.placement_warnings[inst], local.mount_warnings[inst], length(local.skipped_capabilities[inst]) > 0 ? [ + "capAdd ${join(", ", local.skipped_capabilities[inst])} not added - Pod Security baseline forbids it (template variable allow_privileged enables it)" + ] : [], local.dc[inst] != null && local.recorded_uid[inst] == null ? [ + "the image doesn't record its remote user's uid/gid (built by devcontainer-builder older than 0.3.0) - running as uid/gid 1000; bump the Rebuild parameter to rebuild it" + ] : []) + ) } # Mount points that live on the PVC (the workspace folder itself, and any # mount target inside /workspaces, the home or the workspace folder): # seed-home creates them as the remote user first, or kubelet would # create them root-owned and unwritable (e.g. the workspace folder, when # a volume is mounted at /node_modules). - mount_point_dirs = compact([for t in concat([local.workspace_folder], [for m in local.mounts : m.target]) : - startswith(t, "/workspaces/") ? "workspaces/${trimprefix(t, "/workspaces/")}" : - startswith(t, "${local.home_dir}/") ? "home/${trimprefix(t, "${local.home_dir}/")}" : - local.workspace_folder_mounted && startswith(t, "${local.workspace_folder}/") ? "workspace-folder/${trimprefix(t, "${local.workspace_folder}/")}" : "" - ]) + mount_point_dirs = { for inst in local.instance_keys : inst => ( + compact([for t in concat([local.workspace_folder[inst]], [for m in local.mounts[inst] : m.target]) : + startswith(t, "/workspaces/") ? "workspaces/${trimprefix(t, "/workspaces/")}" : + startswith(t, "${local.home_dir[inst]}/") ? "home/${trimprefix(t, "${local.home_dir[inst]}/")}" : + local.workspace_folder_mounted[inst] && startswith(t, "${local.workspace_folder[inst]}/") ? "workspace-folder/${trimprefix(t, "${local.workspace_folder[inst]}/")}" : "" + ]) + ) } # localEnv variables without a default that the user must provide - # ignoring ones only used by mounts, which only matter for bind mounts # (dropped anyway: there's no host). # Only shell-safe names (the service only reports those, but they end up # in a shell script and a regex below, so it's checked here too). - required_variables = [for v in local.variables : v.name if v.kind == "localEnv" && v.default == null && can(regex("^[A-Za-z_][A-Za-z0-9_]*$", v.name)) && length([for u in v.used_in : u if !startswith(u, "mounts")]) > 0] + required_variables = { for inst in local.instance_keys : inst => [for v in local.variables[inst] : v.name if v.kind == "localEnv" && v.default == null && can(regex("^[A-Za-z_][A-Za-z0-9_]*$", v.name)) && length([for u in v.used_in : u if !startswith(u, "mounts")]) > 0] } # Names only - the checks below must never reference the parameter itself: # Terraform prints referenced values in a failed check's diagnostics, and # the parameter holds the user's tokens. - missing_variables = nonsensitive([for name in local.required_variables : nonsensitive(name) if !can(regex("(?m)^${name}=.+", data.coder_parameter.devcontainer_variables.value))]) + missing_variables = { for inst in local.instance_keys : inst => nonsensitive([for name in local.required_variables[inst] : nonsensitive(name) if !can(regex("(?m)^${name}=.+", data.coder_parameter.devcontainer_variables.value))]) } # Not secrets, but derived from the data source (whose registry # credentials input is sensitive) - unmarked so the build log shows them. - warning_lines = nonsensitive([for w in local.all_warnings : nonsensitive(w)]) + warning_lines = { for inst in local.instance_keys : inst => nonsensitive([for w in local.all_warnings[inst] : nonsensitive(w)]) } # Values written into the scripts below as single-quoted shell words # ('...' with each ' as '\''): they come from the user's parameters and # the repository's devcontainer.json, so they may contain anything. - sh_workspace_folder = "'${replace(local.workspace_folder, "'", "'\\''")}'" + sh_workspace_folder = { for inst in local.instance_keys : inst => "'${replace(local.workspace_folder[inst], "'", "'\\''")}'" } sh_repository = "'${replace(data.coder_parameter.repository.value, "'", "'\\''")}'" sh_branch = "'${replace(data.coder_parameter.branch.value, "'", "'\\''")}'" sh_own_extension = "'${replace(var.vscode_extension, "'", "'\\''")}'" # One extension ID per line. - sh_extensions = "'${replace(join("\n", local.vscode_extensions), "'", "'\\''")}'" + sh_extensions = { for inst in local.instance_keys : inst => "'${replace(join("\n", local.vscode_extensions[inst]), "'", "'\\''")}'" } # Where the lifecycle script records its exit status: on the pod's root # filesystem, which is fresh on every start (unlike the home). @@ -606,6 +677,62 @@ locals { # rebuilt image never reuses a tag a node may have cached (the pod pulls # IfNotPresent). Lowercase, OCI tag characters only, at most 128. image_tag = substr(replace(lower("ws-${data.coder_workspace.me.id}-${data.coder_parameter.rebuild.value}"), "/[^a-z0-9_.-]/", "-"), 0, 128) + + # What the per-instance resources read, as local.rt[each.key]. + rt = { for inst in local.instance_keys : inst => { + remote_user = local.remote_user[inst] + runtime = local.runtime[inst] + uid = local.uid[inst] + gid = local.gid[inst] + home_dir = local.home_dir[inst] + lifecycle_scripts = local.lifecycle_scripts[inst] + env_scripts = local.env_scripts[inst] + vscode_settings = local.vscode_settings[inst] + workspace_folder = local.workspace_folder[inst] + workspace_folder_mounted = local.workspace_folder_mounted[inst] + volume_mounts = local.volume_mounts[inst] + tmpfs_mounts = local.tmpfs_mounts[inst] + volume_dirs = local.volume_dirs[inst] + mount_point_dirs = local.mount_point_dirs[inst] + ports = local.ports[inst] + node_selector = local.node_selector[inst] + tolerations = local.tolerations[inst] + pod_requests = local.pod_requests[inst] + pod_limits = local.pod_limits[inst] + disk_size = local.disk_size[inst] + capabilities = local.capabilities[inst] + required_variables = local.required_variables[inst] + resources_summary = local.resources_summary[inst] + all_warnings = local.all_warnings[inst] + sh_workspace_folder = local.sh_workspace_folder[inst] + sh_extensions = local.sh_extensions[inst] + } } + + # For the checks below: every instance's, prefixed with its id unless + # it's `main`. + all_warning_lines = flatten([for inst in local.instance_keys : [ + for w in local.warning_lines[inst] : inst == "main" ? w : "${inst}: ${w}" + ]]) + all_missing_variables = distinct(flatten(values(local.missing_variables))) + + # coder_app.forwarded_port's slots: var.max_forwarded_ports per running + # instance, keyed "/". + forwarded_port_slots = { for pair in setproduct(keys(local.started_instances), range(var.max_forwarded_ports)) : "${pair[0]}/${pair[1]}" => { + instance = pair[0] + index = pair[1] + } } + + # The Deployment's labels, which are also its selector and its pods'. + deployment_labels = { for inst in local.instance_keys : inst => merge({ + "app.kubernetes.io/name" = "coder-workspace" + "app.kubernetes.io/instance" = "coder-workspace-${data.coder_workspace.me.id}${local.k8s_suffix[inst]}" + "app.kubernetes.io/part-of" = "coder" + "com.coder.resource" = "true" + "com.coder.workspace.id" = data.coder_workspace.me.id + "com.coder.workspace.name" = data.coder_workspace.me.name + "com.coder.user.id" = data.coder_workspace_owner.me.id + "com.coder.user.username" = data.coder_workspace_owner.me.name + }, local.k8s_instance_labels[inst]) } } # --- The actual build ---------------------------------------------------- @@ -667,16 +794,20 @@ resource "devcontainerbuilder_build" "workspace" { # The built image's Dev Container metadata - GET /devcontainer on # devcontainer-builder, read straight from the registry. data "devcontainerbuilder_devcontainer" "workspace" { - count = data.coder_workspace.me.start_count - registry = devcontainerbuilder_build.workspace.resolved_registry - name = devcontainerbuilder_build.workspace.resolved_name - tag = devcontainerbuilder_build.workspace.resolved_tag + for_each = local.started_instances + registry = each.value.registry + name = each.value.name + tag = each.value.tag platform = "linux/${local.arch}" } +# One agent per instance. Coder names an agent after its Terraform +# resource name, not the for_each key, so this stays coder_agent.main: the +# agent is "main", as it was before instances. resource "coder_agent" "main" { - os = "linux" - arch = local.arch + for_each = local.instances + os = "linux" + arch = local.arch # No `dir`: it's deprecated, and anything but $HOME breaks Coder Desktop # file sync - so terminals/SSH start in $HOME. code-server and # vscode_desktop open the repo. @@ -693,7 +824,7 @@ resource "coder_agent" "main" { key = "7_devcontainer_variables" script = <<-EOT missing="" - for name in ${join(" ", local.required_variables)}; do + for name in ${join(" ", local.rt[each.key].required_variables)}; do eval "value=\$${DEVCONTAINER_LOCALENV_$name-}" [ -n "$value" ] || missing="$missing $name" done @@ -706,7 +837,7 @@ resource "coder_agent" "main" { metadata { display_name = "Resources (reserved / limit)" key = "9_resources" - script = "echo '${replace(local.resources_summary, "'", "'\\''")}'" + script = "echo '${replace(local.rt[each.key].resources_summary, "'", "'\\''")}'" interval = 3600 timeout = 1 } @@ -714,7 +845,7 @@ resource "coder_agent" "main" { metadata { display_name = "Dev Container warnings" key = "8_devcontainer_warnings" - script = "echo '${length(local.all_warnings) == 0 ? "none" : "${length(local.all_warnings)} - see the build log"}'" + script = "echo '${length(local.rt[each.key].all_warnings) == 0 ? "none" : "${length(local.rt[each.key].all_warnings)} - see the build log"}'" interval = 3600 timeout = 1 } @@ -776,11 +907,11 @@ resource "coder_agent" "main" { # devcontainer_vscode below) with VS Code Desktop. Installed under the # persisted home. module "vscode_web" { - count = var.accept_vscode_license ? data.coder_workspace.me.start_count : 0 + for_each = { for inst, i in local.started_instances : inst => i if var.accept_vscode_license } source = "registry.coder.com/coder/vscode-web/coder" version = "1.6.2" - agent_id = coder_agent.main.id - folder = local.workspace_folder + agent_id = coder_agent.main[each.key].id + folder = local.rt[each.key].workspace_folder accept_license = true install_prefix = "$HOME/.cache/vscode-web" subdomain = var.subdomain_apps @@ -789,11 +920,11 @@ module "vscode_web" { # VS Code Desktop, opened on the cloned repo (in place of the agent's # built-in button - see coder_agent.main's display_apps). module "vscode_desktop" { - count = data.coder_workspace.me.start_count + for_each = local.started_instances source = "registry.coder.com/coder/vscode-desktop/coder" version = "1.3.0" - agent_id = coder_agent.main.id - folder = local.workspace_folder + agent_id = coder_agent.main[each.key].id + folder = local.rt[each.key].workspace_folder } # devcontainer.json's lifecycle commands, from the image's merged metadata: @@ -808,8 +939,8 @@ module "vscode_desktop" { # own filesystem, so never left over from an earlier start), which # postAttachCommand waits for. resource "coder_script" "devcontainer_lifecycle" { - count = data.coder_workspace.me.start_count - agent_id = coder_agent.main.id + for_each = local.started_instances + agent_id = coder_agent.main[each.key].id display_name = "Dev Container lifecycle" icon = "/icon/docker.svg" run_on_start = true @@ -820,7 +951,7 @@ resource "coder_script" "devcontainer_lifecycle" { status_file='${local.lifecycle_status_file}' rm -f "$status_file" trap 'rc=$?; echo "$rc" > "$status_file.tmp" && mv -f "$status_file.tmp" "$status_file"; exit "$rc"' EXIT - workspace_folder=${local.sh_workspace_folder} + workspace_folder=${local.rt[each.key].sh_workspace_folder} dir="$HOME/.cache/devcontainer-lifecycle" mkdir -p "$dir" @@ -904,11 +1035,11 @@ resource "coder_script" "devcontainer_lifecycle" { # initializeCommand runs on the host before the container exists in Dev # Containers; here it's simply the first hook. - run_hook initializeCommand '${base64encode(lookup(local.lifecycle_scripts, "initializeCommand", ""))}' || exit $? - run_hook onCreateCommand '${base64encode(lookup(local.lifecycle_scripts, "onCreateCommand", ""))}' || exit $? - run_hook updateContentCommand '${base64encode(lookup(local.lifecycle_scripts, "updateContentCommand", ""))}' || exit $? - run_hook postCreateCommand '${base64encode(lookup(local.lifecycle_scripts, "postCreateCommand", ""))}' || exit $? - run_hook postStartCommand '${base64encode(lookup(local.lifecycle_scripts, "postStartCommand", ""))}' || exit $? + run_hook initializeCommand '${base64encode(lookup(local.rt[each.key].lifecycle_scripts, "initializeCommand", ""))}' || exit $? + run_hook onCreateCommand '${base64encode(lookup(local.rt[each.key].lifecycle_scripts, "onCreateCommand", ""))}' || exit $? + run_hook updateContentCommand '${base64encode(lookup(local.rt[each.key].lifecycle_scripts, "updateContentCommand", ""))}' || exit $? + run_hook postCreateCommand '${base64encode(lookup(local.rt[each.key].lifecycle_scripts, "postCreateCommand", ""))}' || exit $? + run_hook postStartCommand '${base64encode(lookup(local.rt[each.key].lifecycle_scripts, "postStartCommand", ""))}' || exit $? EOT } @@ -917,10 +1048,11 @@ resource "coder_script" "devcontainer_lifecycle" { # login, after the lifecycle script above has succeeded (as in Dev # Containers, where it follows postStartCommand). Gives up after 30 minutes. resource "coder_script" "devcontainer_post_attach" { - # Always present (count can't depend on the image's metadata, which is - # unknown until the first build) - exits at once if no hook is set. - count = data.coder_workspace.me.start_count - agent_id = coder_agent.main.id + # Always present (whether it exists can't depend on the image's metadata, + # which is unknown until the first build) - exits at once if no hook is + # set. + for_each = local.started_instances + agent_id = coder_agent.main[each.key].id display_name = "Dev Container postAttachCommand" icon = "/icon/docker.svg" run_on_start = true @@ -928,9 +1060,9 @@ resource "coder_script" "devcontainer_post_attach" { script = <<-EOT #!/bin/sh set -u - hook='${base64encode(lookup(local.lifecycle_scripts, "postAttachCommand", ""))}' + hook='${base64encode(lookup(local.rt[each.key].lifecycle_scripts, "postAttachCommand", ""))}' [ -n "$hook" ] || exit 0 - workspace_folder=${local.sh_workspace_folder} + workspace_folder=${local.rt[each.key].sh_workspace_folder} status_file='${local.lifecycle_status_file}' echo "devcontainer: waiting for the Dev Container lifecycle script before postAttachCommand" waited=0 @@ -966,8 +1098,8 @@ resource "coder_script" "devcontainer_post_attach" { resource "coder_script" "devcontainer_vscode" { # Always present, for the same reason - exits at once if there's nothing # to install. - count = data.coder_workspace.me.start_count - agent_id = coder_agent.main.id + for_each = local.started_instances + agent_id = coder_agent.main[each.key].id display_name = "Dev Container VS Code extensions" icon = "/icon/code.svg" run_on_start = true @@ -975,8 +1107,8 @@ resource "coder_script" "devcontainer_vscode" { script = <<-EOT #!/bin/sh set -u - extensions=${local.sh_extensions} - settings_b64='${base64encode(local.vscode_settings)}' + extensions=${local.rt[each.key].sh_extensions} + settings_b64='${base64encode(local.rt[each.key].vscode_settings)}' own_extension=${local.sh_own_extension} data_dir="$HOME/.vscode-server" if [ -z "$extensions" ] && [ "$settings_b64" = "${base64encode("{}")}" ] && [ -z "$own_extension" ]; then @@ -1079,93 +1211,104 @@ resource "coder_script" "devcontainer_vscode" { # the agent metadata above only has room for a count. check "devcontainer_warnings" { assert { - condition = length(local.warning_lines) == 0 - error_message = "devcontainer.json settings this workspace doesn't honor:\n- ${join("\n- ", local.warning_lines)}" + condition = length(local.all_warning_lines) == 0 + error_message = "devcontainer.json settings this workspace doesn't honor:\n- ${join("\n- ", local.all_warning_lines)}" } } check "devcontainer_variables" { assert { - condition = length(local.missing_variables) == 0 - error_message = "devcontainer.json uses $${localEnv:...} variables without a value: ${join(", ", local.missing_variables)}. Add NAME=value lines to the workspace's \"Dev Container variables\" setting and restart." + condition = length(local.all_missing_variables) == 0 + error_message = "devcontainer.json uses $${localEnv:...} variables without a value: ${join(", ", local.all_missing_variables)}. Add NAME=value lines to the workspace's \"Dev Container variables\" setting and restart." } } # devcontainer.json's forwardPorts as dashboard apps. Coder needs a fixed -# number of apps at plan time, so there are var.max_forwarded_ports slots; -# the ones without a port are hidden. +# number of apps at plan time, so there are var.max_forwarded_ports slots +# per instance; the ones without a port are hidden. Slugs are unique per +# workspace, across agents: `main`'s are port-, any other instance's +# -port-. resource "coder_app" "forwarded_port" { - count = data.coder_workspace.me.start_count * var.max_forwarded_ports - agent_id = coder_agent.main.id - slug = "port-${count.index}" - display_name = count.index < length(local.ports) ? coalesce(try(local.ports[count.index].label, null), "Port ${try(local.ports[count.index].port, 0)}") : null - url = "${try(local.ports[count.index].protocol, "") == "https" ? "https" : "http"}://localhost:${try(local.ports[count.index].port, 0)}" - icon = count.index < length(local.ports) ? "/icon/widgets.svg" : null + for_each = local.forwarded_port_slots + agent_id = coder_agent.main[each.value.instance].id + slug = "${each.value.instance == "main" ? "" : "${each.value.instance}-"}port-${each.value.index}" + display_name = each.value.index < length(local.rt[each.value.instance].ports) ? coalesce(try(local.rt[each.value.instance].ports[each.value.index].label, null), "Port ${try(local.rt[each.value.instance].ports[each.value.index].port, 0)}") : null + url = "${try(local.rt[each.value.instance].ports[each.value.index].protocol, "") == "https" ? "https" : "http"}://localhost:${try(local.rt[each.value.instance].ports[each.value.index].port, 0)}" + icon = each.value.index < length(local.rt[each.value.instance].ports) ? "/icon/widgets.svg" : null subdomain = var.subdomain_apps share = "owner" - hidden = count.index >= length(local.ports) + hidden = each.value.index >= length(local.rt[each.value.instance].ports) } # For the rebuild prompt extension (see devcontainer_vscode): the commit # the image was built from, compared with origin/, and the Rebuild # value it increases. resource "coder_env" "devcontainer_image_commit" { - agent_id = coder_agent.main.id + for_each = local.instances + agent_id = coder_agent.main[each.key].id name = "DEVCONTAINER_IMAGE_COMMIT" value = devcontainerbuilder_build.workspace.commit != null ? devcontainerbuilder_build.workspace.commit : "" } resource "coder_env" "devcontainer_branch" { - agent_id = coder_agent.main.id + for_each = local.instances + agent_id = coder_agent.main[each.key].id name = "DEVCONTAINER_BRANCH" value = data.coder_parameter.branch.value } resource "coder_env" "devcontainer_rebuild" { - agent_id = coder_agent.main.id + for_each = local.instances + agent_id = coder_agent.main[each.key].id name = "DEVCONTAINER_REBUILD" value = data.coder_parameter.rebuild.value } # Commits made in the workspace are attributed to its owner. resource "coder_env" "git_author_name" { - agent_id = coder_agent.main.id + for_each = local.instances + agent_id = coder_agent.main[each.key].id name = "GIT_AUTHOR_NAME" value = coalesce(data.coder_workspace_owner.me.full_name, data.coder_workspace_owner.me.name) } resource "coder_env" "git_committer_name" { - agent_id = coder_agent.main.id + for_each = local.instances + agent_id = coder_agent.main[each.key].id name = "GIT_COMMITTER_NAME" value = coalesce(data.coder_workspace_owner.me.full_name, data.coder_workspace_owner.me.name) } resource "coder_env" "git_author_email" { - agent_id = coder_agent.main.id + for_each = local.instances + agent_id = coder_agent.main[each.key].id name = "GIT_AUTHOR_EMAIL" value = data.coder_workspace_owner.me.email } resource "coder_env" "git_committer_email" { - agent_id = coder_agent.main.id + for_each = local.instances + agent_id = coder_agent.main[each.key].id name = "GIT_COMMITTER_EMAIL" value = data.coder_workspace_owner.me.email } +# One PVC per instance, kept while the workspace is stopped. resource "kubernetes_persistent_volume_claim_v1" "data" { + for_each = local.instances metadata { - name = "coder-${data.coder_workspace.me.id}-data" + name = "coder-${data.coder_workspace.me.id}${local.k8s_suffix[each.key]}-data" namespace = var.namespace - labels = { + labels = merge({ "app.kubernetes.io/name" = "coder-pvc" - "app.kubernetes.io/instance" = "coder-pvc-${data.coder_workspace.me.id}" + "app.kubernetes.io/instance" = "coder-pvc-${data.coder_workspace.me.id}${local.k8s_suffix[each.key]}" "app.kubernetes.io/part-of" = "coder" "com.coder.resource" = "true" "com.coder.workspace.id" = data.coder_workspace.me.id "com.coder.workspace.name" = data.coder_workspace.me.name "com.coder.user.id" = data.coder_workspace_owner.me.id "com.coder.user.username" = data.coder_workspace_owner.me.name - } + }, local.k8s_instance_labels[each.key]) annotations = { "com.coder.user.email" = data.coder_workspace_owner.me.email } @@ -1175,7 +1318,7 @@ resource "kubernetes_persistent_volume_claim_v1" "data" { access_modes = ["ReadWriteOnce"] resources { requests = { - storage = local.disk_size + storage = local.rt[each.key].disk_size } } } @@ -1191,26 +1334,18 @@ resource "kubernetes_persistent_volume_claim_v1" "data" { } } +# One Deployment per running instance. resource "kubernetes_deployment_v1" "main" { - count = data.coder_workspace.me.start_count + for_each = local.started_instances depends_on = [ kubernetes_persistent_volume_claim_v1.data ] wait_for_rollout = false metadata { - name = "coder-${data.coder_workspace.me.id}" + name = "coder-${data.coder_workspace.me.id}${local.k8s_suffix[each.key]}" namespace = var.namespace - labels = { - "app.kubernetes.io/name" = "coder-workspace" - "app.kubernetes.io/instance" = "coder-workspace-${data.coder_workspace.me.id}" - "app.kubernetes.io/part-of" = "coder" - "com.coder.resource" = "true" - "com.coder.workspace.id" = data.coder_workspace.me.id - "com.coder.workspace.name" = data.coder_workspace.me.name - "com.coder.user.id" = data.coder_workspace_owner.me.id - "com.coder.user.username" = data.coder_workspace_owner.me.name - } + labels = local.deployment_labels[each.key] annotations = { "com.coder.user.email" = data.coder_workspace_owner.me.email } @@ -1219,16 +1354,7 @@ resource "kubernetes_deployment_v1" "main" { spec { replicas = 1 selector { - match_labels = { - "app.kubernetes.io/name" = "coder-workspace" - "app.kubernetes.io/instance" = "coder-workspace-${data.coder_workspace.me.id}" - "app.kubernetes.io/part-of" = "coder" - "com.coder.resource" = "true" - "com.coder.workspace.id" = data.coder_workspace.me.id - "com.coder.workspace.name" = data.coder_workspace.me.name - "com.coder.user.id" = data.coder_workspace_owner.me.id - "com.coder.user.username" = data.coder_workspace_owner.me.name - } + match_labels = local.deployment_labels[each.key] } strategy { type = "Recreate" @@ -1236,40 +1362,31 @@ resource "kubernetes_deployment_v1" "main" { template { metadata { - labels = { - "app.kubernetes.io/name" = "coder-workspace" - "app.kubernetes.io/instance" = "coder-workspace-${data.coder_workspace.me.id}" - "app.kubernetes.io/part-of" = "coder" - "com.coder.resource" = "true" - "com.coder.workspace.id" = data.coder_workspace.me.id - "com.coder.workspace.name" = data.coder_workspace.me.name - "com.coder.user.id" = data.coder_workspace_owner.me.id - "com.coder.user.username" = data.coder_workspace_owner.me.name - } + labels = local.deployment_labels[each.key] } spec { # Everything runs as the remote user; fsGroup makes the PVC # writable for it (OnRootMismatch: only re-owned when the volume # root doesn't match, not on every start). security_context { - run_as_user = local.uid - run_as_group = local.gid - fs_group = local.gid + run_as_user = local.rt[each.key].uid + run_as_group = local.rt[each.key].gid + fs_group = local.rt[each.key].gid fs_group_change_policy = "OnRootMismatch" } # devcontainer.json's `init` / --init: the pause container becomes # PID 1 and reaps zombies, like Docker's tini. - share_process_namespace = try(local.runtime.init, false) - hostname = try(local.runtime.hostname, null) + share_process_namespace = try(local.rt[each.key].runtime.init, false) + hostname = try(local.rt[each.key].runtime.hostname, null) # Only nodes of the agent's (and the image's) architecture, plus # customizations.kubernetes' nodeSelector with allow_node_placement. - node_selector = local.node_selector + node_selector = local.rt[each.key].node_selector # customizations.kubernetes' tolerations, with allow_node_placement. dynamic "toleration" { - for_each = local.tolerations + for_each = local.rt[each.key].tolerations content { key = toleration.value.key operator = toleration.value.operator @@ -1280,7 +1397,7 @@ resource "kubernetes_deployment_v1" "main" { } dynamic "host_aliases" { - for_each = try(local.runtime.host_aliases, []) + for_each = try(local.rt[each.key].runtime.host_aliases, []) content { ip = host_aliases.value.ip hostnames = host_aliases.value.hostnames @@ -1303,7 +1420,7 @@ resource "kubernetes_deployment_v1" "main" { # root-owned). init_container { name = "seed-home" - image = devcontainerbuilder_build.workspace.image + image = each.value.image image_pull_policy = "IfNotPresent" command = ["sh", "-c", <<-EOT set -eu @@ -1327,15 +1444,15 @@ resource "kubernetes_deployment_v1" "main" { ] env { name = "HOME_DIR" - value = local.home_dir + value = local.rt[each.key].home_dir } env { name = "VOLUME_DIRS" - value = join(" ", local.volume_dirs) + value = join(" ", local.rt[each.key].volume_dirs) } env { name = "MOUNT_POINT_DIRS" - value = join(" ", local.mount_point_dirs) + value = join(" ", local.rt[each.key].mount_point_dirs) } resources { requests = { @@ -1359,7 +1476,7 @@ resource "kubernetes_deployment_v1" "main" { # - a fixed/parameterized `image` variable becomes the real, # pushed image this workspace's own devcontainerbuilder_build # resource just built. - image = devcontainerbuilder_build.workspace.image + image = each.value.image image_pull_policy = "IfNotPresent" # Sets the Dev Container environment (workspace variables, the # user's Dev Container variables, then containerEnv and remoteEnv - @@ -1397,12 +1514,12 @@ resource "kubernetes_deployment_v1" "main" { EOT ] security_context { - privileged = var.allow_privileged && try(local.runtime.privileged, false) + privileged = var.allow_privileged && try(local.rt[each.key].runtime.privileged, false) capabilities { - add = local.capabilities + add = local.rt[each.key].capabilities } dynamic "seccomp_profile" { - for_each = var.allow_privileged && try(local.runtime.seccomp_unconfined, false) ? [1] : [] + for_each = var.allow_privileged && try(local.rt[each.key].runtime.seccomp_unconfined, false) ? [1] : [] content { type = "Unconfined" } @@ -1410,23 +1527,23 @@ resource "kubernetes_deployment_v1" "main" { } env { name = "CODER_AGENT_TOKEN" - value = coder_agent.main.token + value = coder_agent.main[each.key].token } env { name = "DC_INIT_SCRIPT" - value = coder_agent.main.init_script + value = coder_agent.main[each.key].init_script } env { name = "DC_REMOTE_USER" - value = local.remote_user + value = local.rt[each.key].remote_user } env { name = "DC_HOME" - value = local.home_dir + value = local.rt[each.key].home_dir } env { name = "DC_WORKSPACE_FOLDER" - value = local.workspace_folder + value = local.rt[each.key].workspace_folder } env { name = "DC_ID" @@ -1438,18 +1555,18 @@ resource "kubernetes_deployment_v1" "main" { } env { name = "DC_CONTAINER_ENV" - value = base64encode(lookup(local.env_scripts, "containerEnv", "")) + value = base64encode(lookup(local.rt[each.key].env_scripts, "containerEnv", "")) } env { name = "DC_REMOTE_ENV" - value = base64encode(lookup(local.env_scripts, "remoteEnv", "")) + value = base64encode(lookup(local.rt[each.key].env_scripts, "remoteEnv", "")) } resources { - requests = local.pod_requests - limits = local.pod_limits + requests = local.rt[each.key].pod_requests + limits = local.rt[each.key].pod_limits } volume_mount { - mount_path = local.home_dir + mount_path = local.rt[each.key].home_dir name = "data" sub_path = "home" read_only = false @@ -1464,14 +1581,14 @@ resource "kubernetes_deployment_v1" "main" { # gets its own directory on the PVC; otherwise this mount points at # an unused path. volume_mount { - mount_path = local.workspace_folder_mounted ? local.workspace_folder : "/mnt/.devcontainer-unused-workspace-folder" + mount_path = local.rt[each.key].workspace_folder_mounted ? local.rt[each.key].workspace_folder : "/mnt/.devcontainer-unused-workspace-folder" name = "data" sub_path = "workspace-folder" read_only = false } # devcontainer.json's volume mounts, persisted on the PVC. dynamic "volume_mount" { - for_each = local.volume_mounts + for_each = local.rt[each.key].volume_mounts content { mount_path = volume_mount.value.target name = "data" @@ -1480,7 +1597,7 @@ resource "kubernetes_deployment_v1" "main" { } } dynamic "volume_mount" { - for_each = local.tmpfs_mounts + for_each = local.rt[each.key].tmpfs_mounts content { mount_path = volume_mount.value.target name = "tmpfs-${volume_mount.value.index}" @@ -1495,7 +1612,7 @@ resource "kubernetes_deployment_v1" "main" { volume { name = "data" persistent_volume_claim { - claim_name = kubernetes_persistent_volume_claim_v1.data.metadata.0.name + claim_name = kubernetes_persistent_volume_claim_v1.data[each.key].metadata.0.name read_only = false } } @@ -1505,12 +1622,12 @@ resource "kubernetes_deployment_v1" "main" { name = "dshm" empty_dir { medium = "Memory" - size_limit = try(local.runtime.shm_size_bytes, null) != null ? tostring(local.runtime.shm_size_bytes) : "64Mi" + size_limit = try(local.rt[each.key].runtime.shm_size_bytes, null) != null ? tostring(local.rt[each.key].runtime.shm_size_bytes) : "64Mi" } } dynamic "volume" { - for_each = local.tmpfs_mounts + for_each = local.rt[each.key].tmpfs_mounts content { name = "tmpfs-${volume.value.index}" empty_dir { @@ -1541,3 +1658,135 @@ resource "kubernetes_deployment_v1" "main" { } } } + +# --- Upgrading workspaces from before instances ----------------------------- +# Before instances, these were single resources (or count = start_count). +# The moves keep an existing workspace's agent, PVC and Deployment as they +# are: its next build only renames them in the state. coder_app slots past +# the default max_forwarded_ports (10) aren't moved: they're recreated, +# which only re-registers the app in Coder. + +moved { + from = coder_agent.main + to = coder_agent.main["main"] +} + +moved { + from = module.vscode_web[0] + to = module.vscode_web["main"] +} + +moved { + from = module.vscode_desktop[0] + to = module.vscode_desktop["main"] +} + +moved { + from = coder_script.devcontainer_lifecycle[0] + to = coder_script.devcontainer_lifecycle["main"] +} + +moved { + from = coder_script.devcontainer_post_attach[0] + to = coder_script.devcontainer_post_attach["main"] +} + +moved { + from = coder_script.devcontainer_vscode[0] + to = coder_script.devcontainer_vscode["main"] +} + +moved { + from = coder_app.forwarded_port[0] + to = coder_app.forwarded_port["main/0"] +} + +moved { + from = coder_app.forwarded_port[1] + to = coder_app.forwarded_port["main/1"] +} + +moved { + from = coder_app.forwarded_port[2] + to = coder_app.forwarded_port["main/2"] +} + +moved { + from = coder_app.forwarded_port[3] + to = coder_app.forwarded_port["main/3"] +} + +moved { + from = coder_app.forwarded_port[4] + to = coder_app.forwarded_port["main/4"] +} + +moved { + from = coder_app.forwarded_port[5] + to = coder_app.forwarded_port["main/5"] +} + +moved { + from = coder_app.forwarded_port[6] + to = coder_app.forwarded_port["main/6"] +} + +moved { + from = coder_app.forwarded_port[7] + to = coder_app.forwarded_port["main/7"] +} + +moved { + from = coder_app.forwarded_port[8] + to = coder_app.forwarded_port["main/8"] +} + +moved { + from = coder_app.forwarded_port[9] + to = coder_app.forwarded_port["main/9"] +} + +moved { + from = coder_env.devcontainer_image_commit + to = coder_env.devcontainer_image_commit["main"] +} + +moved { + from = coder_env.devcontainer_branch + to = coder_env.devcontainer_branch["main"] +} + +moved { + from = coder_env.devcontainer_rebuild + to = coder_env.devcontainer_rebuild["main"] +} + +moved { + from = coder_env.git_author_name + to = coder_env.git_author_name["main"] +} + +moved { + from = coder_env.git_committer_name + to = coder_env.git_committer_name["main"] +} + +moved { + from = coder_env.git_author_email + to = coder_env.git_author_email["main"] +} + +moved { + from = coder_env.git_committer_email + to = coder_env.git_committer_email["main"] +} + +moved { + from = kubernetes_persistent_volume_claim_v1.data + to = kubernetes_persistent_volume_claim_v1.data["main"] +} + +moved { + from = kubernetes_deployment_v1.main[0] + to = kubernetes_deployment_v1.main["main"] +}