feat: enable cli-proxy in smoke-copilot workflow (#1820)

* feat: enable cli-proxy in smoke-copilot workflow

Add features.cli-proxy: true to smoke-copilot.md and recompile with
gh-aw dev build (bda91a78) that emits --difc-proxy-host and
--difc-proxy-ca-cert flags plus Start/Stop CLI proxy steps.

Post-processed with postprocess-smoke-workflows.ts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: handle quoted paths in postprocess install step regex

The new gh-aw compiler quotes the install_awf_binary.sh path:
  bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.25.17

The regex only matched unquoted paths, so the install step was not
replaced with npm ci / npm run build. This caused --build-local to
fail at runtime since the standalone bundle doesn't support it.

Add optional double-quote matching ("?) around the path in the regex.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: inject GH_TOKEN into cli-proxy container environment

The gh CLI inside the cli-proxy container needs a GitHub token to
authenticate API requests. Without it, gh commands hang waiting for
auth and time out after 30s.

The token is safe in the cli-proxy container: it's inside the firewall
perimeter, not accessible to the agent, and the DIFC proxy on the host
provides write-control via its guard policy.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: add iptables filter ACCEPT rule for cli-proxy

The cli-proxy container (172.30.0.50) had a NAT RETURN rule so traffic
wouldn't be DNAT'd to Squid, but was missing the corresponding filter
ACCEPT rule. The final 'iptables -A OUTPUT -p tcp -j DROP' rule silently
dropped all TCP connections to cli-proxy, causing 'curl exit 28' timeouts
in the agent's gh-cli-proxy-wrapper.sh.

This matches how api-proxy is handled: both NAT RETURN (line 173) and
filter ACCEPT (line 406).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat: add structured logging to cli-proxy

Add JSON-line access logging to server.js and connection logging to
tcp-tunnel.js for better observability in CI:

server.js:
- Writes structured JSON entries to /var/log/cli-proxy/access.log
  (volume already mounted and preserved by docker-manager.ts)
- Also emits to stderr for docker logs capture
- Logs: server_start (config summary), exec_start (args, cwd),
  exec_done (exit code, duration, output sizes), exec_denied,
  exec_error, and unhandled errors
- Includes truncated stderr preview on non-zero exit for debugging

tcp-tunnel.js:
- Logs new connections and disconnects with client address
- Includes client address in error messages for correlation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: ignore agent cwd in cli-proxy execFile

The agent wrapper sends its container workspace path (e.g.
/home/runner/work/gh-aw-firewall/gh-aw-firewall) as the cwd in
/exec requests. This path doesn't exist inside the cli-proxy
container, causing Node.js execFile to throw ENOENT — which
looks like 'gh' is missing but is actually a cwd resolution
failure.

The gh CLI doesn't need the agent's workspace path — it operates
on remote GitHub resources via --repo flags. Always use the
server's own cwd (/app) instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: create combined CA bundle for gh CLI TLS trust

The gh CLI is a Go binary that uses the system CA store (or
SSL_CERT_FILE), not NODE_EXTRA_CA_CERTS. The DIFC proxy's
self-signed CA cert was only trusted by Node.js, causing every
gh command to fail with 'x509: certificate signed by unknown
authority'.

Create a combined CA bundle (system CAs + DIFC proxy CA) at
startup and export SSL_CERT_FILE so the gh CLI trusts the proxy.
Also add SSL_CERT_FILE to the protected env keys to prevent
agent override.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: add CLI_PROXY_POLICY to DIFC proxy startup

The DIFC proxy (mcpg) requires a --policy flag to forward API
requests. Without it, it returns 503 'proxy enforcement not
configured' for all requests. The gh-aw compiler doesn't emit
CLI_PROXY_POLICY yet, so add it directly to the lock file.

Uses a permissive allow-only policy for smoke testing:
repos=all, min-integrity=none.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: lpcox

.github/workflows/smoke-copilot.lock.yml

@@ -34,6 +34,8 @@ safe-outputs:
     run-success: "📰 VERDICT: [{workflow_name}]({run_url}) has concluded. All systems operational. This is a developing story. 🎤"
     run-failure: "📰 DEVELOPING STORY: [{workflow_name}]({run_url}) reports {status}. Our correspondents are investigating the incident..."
 timeout-minutes: 5
+features:
+  cli-proxy: true
 strict: true
 steps:
   - name: Pre-compute smoke test data

.github/workflows/smoke-copilot.md

@@ -406,6 +406,11 @@ if [ -n "$AWF_API_PROXY_IP" ]; then
   iptables -A OUTPUT -p tcp -d "$AWF_API_PROXY_IP" -j ACCEPT
 fi
 
+# Allow traffic to CLI proxy sidecar (when enabled)
+if [ -n "$AWF_CLI_PROXY_IP" ]; then
+  iptables -A OUTPUT -p tcp -d "$AWF_CLI_PROXY_IP" -j ACCEPT
+fi
+
 # Log dangerous port access attempts for audit (rate-limited to avoid log flooding)
 # These ports are blocked by NAT RETURN + final DROP, but logging helps identify
 # what the agent tried to access

containers/agent/setup-iptables.sh

@@ -35,13 +35,22 @@ if [ ! -f /tmp/proxy-tls/ca.crt ]; then
 fi
 echo "[cli-proxy] TLS certificate available"
 
+# Build a combined CA bundle so the gh CLI (Go binary) trusts the DIFC proxy's
+# self-signed cert.  NODE_EXTRA_CA_CERTS only helps Node.js; Go programs use
+# the system store or SSL_CERT_FILE.
+COMBINED_CA="/tmp/proxy-tls/combined-ca.crt"
+cat /etc/ssl/certs/ca-certificates.crt /tmp/proxy-tls/ca.crt > "${COMBINED_CA}"
+echo "[cli-proxy] Combined CA bundle created at ${COMBINED_CA}"
+
 # Configure gh CLI to route through the DIFC proxy via the TCP tunnel
 # Uses localhost because the tunnel makes the DIFC proxy appear on localhost,
 # matching the self-signed cert's SAN.
 export GH_HOST="localhost:${DIFC_PORT}"
 export GH_REPO="${GH_REPO:-$GITHUB_REPOSITORY}"
-# The CA cert is guaranteed to exist at this point (we exit above if missing)
+# Node.js (server.js / tcp-tunnel.js) uses NODE_EXTRA_CA_CERTS;
+# gh CLI (Go) uses SSL_CERT_FILE pointing to the combined bundle.
 export NODE_EXTRA_CA_CERTS="/tmp/proxy-tls/ca.crt"
+export SSL_CERT_FILE="${COMBINED_CA}"
 
 echo "[cli-proxy] gh CLI configured to route through DIFC proxy at ${GH_HOST}"
 

containers/cli-proxy/entrypoint.sh

@@ -18,12 +18,42 @@
  */
 
 const http = require('http');
+const fs = require('fs');
+const path = require('path');
 const { execFile } = require('child_process');
 
 const CLI_PROXY_PORT = parseInt(process.env.AWF_CLI_PROXY_PORT || '11000', 10);
 const COMMAND_TIMEOUT_MS = parseInt(process.env.AWF_CLI_PROXY_TIMEOUT_MS || '30000', 10);
 const MAX_OUTPUT_BYTES = parseInt(process.env.AWF_CLI_PROXY_MAX_OUTPUT_BYTES || String(10 * 1024 * 1024), 10);
 
+// --- Structured logging to /var/log/cli-proxy/access.log ---
+
+const LOG_DIR = process.env.AWF_CLI_PROXY_LOG_DIR || '/var/log/cli-proxy';
+const LOG_FILE = path.join(LOG_DIR, 'access.log');
+
+let logStream = null;
+try {
+  if (fs.existsSync(LOG_DIR)) {
+    logStream = fs.createWriteStream(LOG_FILE, { flags: 'a' });
+  }
+} catch {
+  // Non-fatal: logging to file is best-effort
+}
+
+/**
+ * Write a structured JSON log entry to the access log file and stderr.
+ * Each line is a self-contained JSON object for easy parsing.
+ */
+function accessLog(entry) {
+  const record = { ts: new Date().toISOString(), ...entry };
+  const line = JSON.stringify(record);
+  if (logStream) {
+    logStream.write(line + '\n');
+  }
+  // Also emit to stderr so docker logs captures it
+  console.error(line);
+}
+
 /**
  * Meta-commands that are always denied.
  * These modify gh itself rather than GitHub resources.
@@ -169,13 +199,15 @@ function handleHealth(res) {
  * }
  */
 async function handleExec(req, res) {
+  const startTime = Date.now();
   let body;
   try {
     const raw = await readBody(req, res);
     // null means readBody already sent a 413 error response
     if (raw === null) return;
     body = JSON.parse(raw.toString('utf8'));
   } catch {
+    accessLog({ event: 'exec_error', error: 'Invalid JSON body' });
     return sendError(res, 400, 'Invalid JSON body');
   }
 
@@ -184,15 +216,18 @@ async function handleExec(req, res) {
   // Validate args
   const validation = validateArgs(args);
   if (!validation.valid) {
+    accessLog({ event: 'exec_denied', args, error: validation.error });
     return sendError(res, 403, validation.error);
   }
 
+  accessLog({ event: 'exec_start', args, cwd: cwd || null });
+
   // Build environment for the subprocess
   // Inherit server environment (includes GH_HOST, NODE_EXTRA_CA_CERTS, GH_REPO, etc.)
   const childEnv = Object.assign({}, process.env);
   if (extraEnv && typeof extraEnv === 'object') {
     // Only allow safe string env overrides; never allow overriding GH_HOST or GH_TOKEN
-    const PROTECTED_KEYS = new Set(['GH_HOST', 'GH_TOKEN', 'GITHUB_TOKEN', 'NODE_EXTRA_CA_CERTS']);
+    const PROTECTED_KEYS = new Set(['GH_HOST', 'GH_TOKEN', 'GITHUB_TOKEN', 'NODE_EXTRA_CA_CERTS', 'SSL_CERT_FILE']);
     for (const [key, value] of Object.entries(extraEnv)) {
       if (typeof key === 'string' && typeof value === 'string' && !PROTECTED_KEYS.has(key)) {
         childEnv[key] = value;
@@ -201,14 +236,16 @@ async function handleExec(req, res) {
   }
 
   // Execute gh directly (no shell — prevents injection attacks)
+  // Always use the server's own cwd — the agent sends its container workspace
+  // path which doesn't exist inside the cli-proxy container.
   let stdout = '';
   let stderr = '';
   let exitCode = 0;
 
   try {
     const result = await new Promise((resolve, reject) => {
       const child = execFile('gh', args, {
-        cwd: cwd || process.cwd(),
+        cwd: process.cwd(),
         env: childEnv,
         timeout: COMMAND_TIMEOUT_MS,
         maxBuffer: MAX_OUTPUT_BYTES,
@@ -251,6 +288,19 @@ async function handleExec(req, res) {
   }
 
   const responseBody = JSON.stringify({ stdout, stderr, exitCode });
+
+  const durationMs = Date.now() - startTime;
+  accessLog({
+    event: 'exec_done',
+    args,
+    exitCode,
+    durationMs,
+    stdoutBytes: stdout.length,
+    stderrBytes: stderr.length,
+    // Include truncated stderr for debugging failures (redact tokens)
+    ...(exitCode !== 0 && stderr ? { stderrPreview: stderr.slice(0, 500) } : {}),
+  });
+
   res.writeHead(200, {
     'Content-Type': 'application/json',
     'Content-Length': Buffer.byteLength(responseBody),
@@ -277,18 +327,27 @@ async function requestHandler(req, res) {
 if (require.main === module) {
   const server = http.createServer((req, res) => {
     requestHandler(req, res).catch(err => {
-      console.error('[cli-proxy] Unhandled request error:', err);
+      accessLog({ event: 'unhandled_error', error: err.message });
       if (!res.headersSent) {
         sendError(res, 500, 'Internal server error');
       }
     });
   });
 
   server.listen(CLI_PROXY_PORT, '0.0.0.0', () => {
+    accessLog({
+      event: 'server_start',
+      port: CLI_PROXY_PORT,
+      timeoutMs: COMMAND_TIMEOUT_MS,
+      ghHost: process.env.GH_HOST || '(not set)',
+      caCert: process.env.NODE_EXTRA_CA_CERTS || '(not set)',
+      hasGhToken: !!process.env.GH_TOKEN,
+    });
     console.log(`[cli-proxy] HTTP server listening on port ${CLI_PROXY_PORT}`);
   });
 
   server.on('error', err => {
+    accessLog({ event: 'server_error', error: err.message });
     console.error('[cli-proxy] Server error:', err);
     process.exit(1);
   });

containers/cli-proxy/server.js

@@ -39,11 +39,14 @@ if (isNaN(remotePort) || remotePort < 1 || remotePort > 65535) {
 }
 
 const server = net.createServer(client => {
+  const clientAddr = `${client.remoteAddress}:${client.remotePort}`;
+  console.error(`[tcp-tunnel] Connection from ${sanitizeForLog(clientAddr)}`);
   const upstream = net.connect(remotePort, remoteHost);
   client.pipe(upstream);
   upstream.pipe(client);
-  client.on('error', (err) => { console.error('[tcp-tunnel] Client error:', sanitizeForLog(err.message)); upstream.destroy(); });
-  upstream.on('error', (err) => { console.error('[tcp-tunnel] Upstream error:', sanitizeForLog(err.message)); client.destroy(); });
+  client.on('error', (err) => { console.error(`[tcp-tunnel] Client error (${sanitizeForLog(clientAddr)}): ${sanitizeForLog(err.message)}`); upstream.destroy(); });
+  upstream.on('error', (err) => { console.error(`[tcp-tunnel] Upstream error (${sanitizeForLog(clientAddr)}): ${sanitizeForLog(err.message)}`); client.destroy(); });
+  client.on('close', () => { console.error(`[tcp-tunnel] Connection closed: ${sanitizeForLog(clientAddr)}`); });
 });
 
 server.on('error', (err) => {

containers/cli-proxy/tcp-tunnel.js

@@ -54,8 +54,9 @@ const workflowPaths = [
 // Matches the install step with captured indentation:
 // - "Install awf binary" or "Install AWF binary" step at any indent level
 // - run command invoking install_awf_binary.sh with a version
+// - path may or may not be double-quoted (newer gh-aw compilers quote it)
 const installStepRegex =
-  /^(\s*)- name: Install [Aa][Ww][Ff] binary\n\1\s*run: bash (?:\/opt\/gh-aw|\$\{RUNNER_TEMP\}\/gh-aw)\/actions\/install_awf_binary\.sh v[0-9.]+\n/m;
+  /^(\s*)- name: Install [Aa][Ww][Ff] binary\n\1\s*run: bash "?(?:\/opt\/gh-aw|\$\{RUNNER_TEMP\}\/gh-aw)\/actions\/install_awf_binary\.sh"? v[0-9.]+\n/m;
 const installStepRegexGlobal = new RegExp(installStepRegex.source, 'gm');
 
 function buildLocalInstallSteps(indent: string): string {

scripts/ci/postprocess-smoke-workflows.ts

@@ -1713,6 +1713,12 @@ export function generateDockerCompose(
         AWF_DIFC_PROXY_PORT: difcProxyPort,
         // Pass GITHUB_REPOSITORY for GH_REPO default in entrypoint
         ...(process.env.GITHUB_REPOSITORY && { GITHUB_REPOSITORY: process.env.GITHUB_REPOSITORY }),
+        // The gh CLI inside the cli-proxy needs a GitHub token to authenticate API
+        // requests. The token is safe here: the cli-proxy container is inside the
+        // firewall perimeter and not accessible to the agent. The DIFC proxy on the
+        // host provides write-control via its guard policy.
+        ...(process.env.GH_TOKEN && { GH_TOKEN: process.env.GH_TOKEN }),
+        ...(process.env.GITHUB_TOKEN && !process.env.GH_TOKEN && { GH_TOKEN: process.env.GITHUB_TOKEN }),
         // Prevent curl/node from routing localhost or host.docker.internal through Squid
         NO_PROXY: `localhost,127.0.0.1,::1,host.docker.internal`,
         no_proxy: `localhost,127.0.0.1,::1,host.docker.internal`,