Alert on late purchase orders (stacked on #27; needs alerting PR and procurement PR #1)

README.md

@@ -98,7 +98,7 @@ assets are added:
 |---|---|
 | `HOMELAB` | health checks down, capacity policy critical |
 | `TEAM` | the above, plus backup promise broken, patch overdue and recovery plan not ready |
-| `ENTERPRISE` | the above (higher impact and urgency), plus vendor agreement about to end, licence over its limit and cloud resource gone missing |
+| `ENTERPRISE` | the above (higher impact and urgency), plus vendor agreement about to end, licence over its limit, cloud resource gone missing and purchase order late |
 
 It is only rules: capacity, backup and patch policies are about one asset and a number only its owner knows, so they cannot be guessed. Running it
 again keeps what is there and adds what is missing (moving a tenant from `HOMELAB` to `TEAM` adds the three new rules), and it never touches a rule that
@@ -141,7 +141,7 @@ With the stack up (either way):
 | Script | What it checks |
 |---|---|
 | `scripts/e2e/run-e2e.sh` (`run-e2e.ps1`) | every service's Postman suite (`docs/postman` in each repo), in dependency order, threading the `organisationId` created by the Party Reference Data Directory suite into the others |
-| `scripts/e2e/starter-pack-smoke.sh` (`starter-pack-smoke.ps1`) | the starter pack of alert rules through the gateway: an edition gets its rules (Homelab 2, Team 5, Enterprise 8), each taking the class of its source, a repeat creates nothing, moving up adds only what is missing, a paused rule stays paused, another tenant has none, a wrong call exits 2 |
+| `scripts/e2e/starter-pack-smoke.sh` (`starter-pack-smoke.ps1`) | the starter pack of alert rules through the gateway: an edition gets its rules (Homelab 2, Team 5, Enterprise 9), each taking the class of its source, a repeat creates nothing, moving up adds only what is missing, a paused rule stays paused, another tenant has none, a wrong call exits 2 |
 | `scripts/e2e/netdata-connector-smoke.sh` (`netdata-connector-smoke.ps1`) | the Netdata connector against a test double of an agent (`mock-netdata.mjs`, started by the script): the node reaches the discovery queue once, a chart becomes a capacity sample (dimensions added up, a factor), a failing chart or an unreachable agent fails only what it must, the token is sent and never printed |
 | `scripts/e2e/dr-smoke.sh` (`dr-smoke.ps1`) | the DR registry through the gateway (DR registry ADR-030..033): a recovery plan judged when read, a walk through the runbook that counts as a pass but measures nothing, a failover and a full recovery that measure the recovery time and point, twenty reports of one exercise make one, a later pass clears a failure and a late report moves nothing, a backup-restore plan that cannot promise to lose nothing, the runbook as an identifier, a race between two people, staff only, every mutation recorded on the ledger |
 | `scripts/e2e/cloud-inventory-smoke.sh` (`cloud-inventory-smoke.ps1`) | the cloud inventory through the gateway (cloud inventory ADR-030..034): an AWS or Azure resource whose provider id is its identity, the same one again restated, a changed report that replaces the snapshot, twenty reports of one new resource that make one, fixed lists and four tags with nothing a tool says repeated, judged present or gone when read, a terminated resource that comes back, the filters and the summary, staff only, recorded on the ledger |

docker-compose.yml

@@ -373,6 +373,7 @@ services:
       LICENSE_REGISTRY_SERVICE_URL: http://it-license-registry:8080
       DR_REGISTRY_SERVICE_URL: http://it-dr-registry:8080
       CLOUD_INVENTORY_SERVICE_URL: http://it-cloud-inventory:8080
+      PROCUREMENT_SERVICE_URL: http://it-procurement:8080
       # Webhook notices (alerting ADR-036): the rules name these variables, never the address. They point at the test double (profile health-test),
       # which is reachable over plain http only because its host is listed here; a real deployment sets https addresses and leaves the list empty.
       THINKLAB_ALERT_HOOK_OPS: http://mock-target:9200/hook/ops

scripts/e2e/promises-alerting-smoke.ps1

@@ -16,13 +16,16 @@
     (7) a licence over its limit opens one alert and one incident (ADR-039), and freed seats resolve it;
     (8) a recovery plan whose exercise failed opens one alert and one incident (ADR-041), and a later pass resolves it;
     (9) a cloud resource the tool has not seen for days opens one alert and one incident (ADR-042), seeing it again resolves it, and a terminated one opens nothing
-        (the smoke moves the moment the tool last saw the resource back in the store with docker exec on the MongoDB container, MONGO_CONTAINER, the way two days passing would).
+        (the smoke moves the moment the tool last saw the resource back in the store with docker exec on the MongoDB container, MONGO_CONTAINER, the way two days passing would);
+    (10) a placed purchase order past its expected day opens one alert and one incident (ADR-043), receiving it completely or cancelling it resolves the alert (the smoke moves
+        the expected day back in the store, and approves the order on the REAL approval service with the policy in .e2e/procurement-policy.json).
 
     pwsh -File .\promises-alerting-smoke.ps1
 #>
 param(
     [string]$GatewayUrl = 'http://localhost:8088',
-    [string]$MongoContainer = 'thinklab-mongo-1'
+    [string]$MongoContainer = 'thinklab-mongo-1',
+    [string]$PolicyFile = (Join-Path $PSScriptRoot '..\..\.e2e\procurement-policy.json')
 )
 
 $ErrorActionPreference = 'Stop'
@@ -53,7 +56,7 @@ function Assert-Equal {
 function Ahead([double]$Minutes) { (Get-Date).ToUniversalTime().AddMinutes($Minutes).ToString("yyyy-MM-dd'T'HH:mm:ss'Z'") }
 
 $alr = "$GatewayUrl/it-alerting/v1"; $pch = "$GatewayUrl/it-patch-registry/v1"; $cap = "$GatewayUrl/it-capacity-registry/v1"
-$bkp = "$GatewayUrl/it-backup-registry/v1"; $vnd = "$GatewayUrl/it-vendor-registry/v1"; $lic = "$GatewayUrl/it-license-registry/v1"; $dr = "$GatewayUrl/it-dr-registry/v1"; $invReg = "$GatewayUrl/it-cloud-inventory/v1"; $inc = "$GatewayUrl/it-incident-management/v1"
+$bkp = "$GatewayUrl/it-backup-registry/v1"; $vnd = "$GatewayUrl/it-vendor-registry/v1"; $lic = "$GatewayUrl/it-license-registry/v1"; $dr = "$GatewayUrl/it-dr-registry/v1"; $invReg = "$GatewayUrl/it-cloud-inventory/v1"; $po = "$GatewayUrl/it-procurement/v1"; $inc = "$GatewayUrl/it-incident-management/v1"
 $tenantId = [guid]::NewGuid().ToString()
 $user = [guid]::NewGuid().ToString(); $filedFor = [guid]::NewGuid().ToString()
 $patchAsset = [guid]::NewGuid().ToString(); $diskAsset = [guid]::NewGuid().ToString(); $backupAsset = [guid]::NewGuid().ToString()
@@ -198,6 +201,43 @@ Assert-Equal 'a resource the tool said is terminated is GONE' (Invoke-Api GET "$
 [void](Evaluate)
 Assert-Equal 'a terminated resource opens nothing, however long ago it was seen' @((Invoke-Api GET "$alr/retrieve?checkId=$gone" $staff).Body).Count 0
 
+# 10. A purchase order that is late (ADR-043). An order cannot be drafted with an expected day in the past, so the smoke moves the expected day of a placed order
+# back in the store (docker exec, as for section 9); the approval goes through the REAL approval service with the purchase-order policy.
+if (-not (Test-Path $PolicyFile)) { throw "Missing $PolicyFile - run start-local-stack.ps1 (or provision-change-management.sh) first." }
+$approver = @((Get-Content $PolicyFile -Raw | ConvertFrom-Json).approverIds)[0]
+function Order-Body2([string]$Reference) { @{ reference = $Reference; vendorId = [guid]::NewGuid().ToString(); requesterId = $filedFor; currency = 'BRL'; expectedOn = (Get-Date).AddDays(5).ToString('yyyy-MM-dd'); lines = @(@{ category = 'HARDWARE'; item = 'LAP-14'; quantity = 10; unitPrice = 1500.00 }) } }
+function Placed-Order([string]$Reference) {
+    $id = (Invoke-Api POST "$po/initiate" $staff (Order-Body2 $Reference)).Body.id
+    [void](Invoke-Api PUT "$po/$id/control/submit" $staff)
+    [void](Invoke-Api PUT "$po/$id/approval/capture" $staff @{ approverId = $approver; outcome = 'APPROVE' })
+    [void](Invoke-Api PUT "$po/$id/control/place" $staff)
+    $id
+}
+function Postpone([string]$Id) { $day = (Get-Date).AddDays(-3).ToString('yyyy-MM-dd'); docker exec $MongoContainer mongosh --quiet --eval "db.getSiblingDB('thinklab_it_procurement_db').purchase_orders.updateOne({_id: UUID('$Id')}, {`$set: {expectedOn: '$day'}})" | Out-Null }
+$order = Placed-Order 'PO-LATE-1'
+Assert-Equal 'a placed order is not late while its expected day is ahead' "$((Invoke-Api GET "$po/$order/retrieve" $staff).Body.status)/$((Invoke-Api GET "$po/$order/retrieve" $staff).Body.late)" 'ORDERED/False'
+Assert-Equal 'a rule for the procurement registry is set (201)' (Invoke-Api POST "$alr/rule/initiate" $staff (Rule-Body 'Purchase orders' 'PROCUREMENT')).Status 201
+[void](Evaluate)
+Assert-Equal 'an order that is not late opens nothing' @((Invoke-Api GET "$alr/retrieve?checkId=$order" $staff).Body).Count 0
+Postpone $order
+Assert-Equal 'three days past the expected day: the registry says it is late' (Invoke-Api GET "$po/$order/retrieve" $staff).Body.late $true
+[void](Evaluate)
+Assert-Equal 'the order gets one alert and one incident' (Wait-For 20 { Linked $order }) $true
+$orderAlert = AlertOf $order
+Assert-Equal 'the alert names the order and the fixed code PURCHASE_ORDER_LATE, and carries no vendor or item' "$($orderAlert.checkName)/$($orderAlert.lastError)/$((($orderAlert | ConvertTo-Json -Depth 6) -like '*LAP-14*'))" 'PO-LATE-1/PURCHASE_ORDER_LATE/False'
+[void](Evaluate)
+Assert-Equal 'more evaluations open nothing more for the order' @((Invoke-Api GET "$alr/retrieve?checkId=$order" $staff).Body).Count 1
+Assert-Equal 'the goods are received completely (201)' (Invoke-Api POST "$po/$order/receipt/initiate" $staff @{ lineNo = 1; deliveryRef = 'DN-LATE'; quantity = 10 }).Status 201
+[void](Evaluate)
+Assert-Equal 'receiving everything resolves the alert' (Wait-For 20 { Status-Is $order 'RESOLVED' }) $true
+$cancelled = Placed-Order 'PO-LATE-2'
+Postpone $cancelled
+[void](Evaluate)
+Assert-Equal 'a second late order gets its alert' (Wait-For 20 { Linked $cancelled }) $true
+Assert-Equal 'it is cancelled (204)' (Invoke-Api PUT "$po/$cancelled/control/cancel" $staff).Status 204
+[void](Evaluate)
+Assert-Equal 'cancelling the order resolves the alert: nothing is owed' (Wait-For 20 { Status-Is $cancelled 'RESOLVED' }) $true
+
 $results | Format-Table -AutoSize | Out-String | Write-Output
 $failed = @($results | Where-Object { $_.Result -eq 'FAIL' }).Count
 Write-Output ("Promises alerting smoke: {0} checks, {1} failed" -f $results.Count, $failed)

scripts/e2e/promises-alerting-smoke.sh

@@ -18,6 +18,8 @@ lic="$gateway/it-license-registry/v1"
 dr="$gateway/it-dr-registry/v1"
 inv_reg="$gateway/it-cloud-inventory/v1"
 mongo_container="${MONGO_CONTAINER:-thinklab-mongo-1}"
+po="$gateway/it-procurement/v1"
+policy_file="${POLICY_FILE:-$(dirname "$0")/../../.e2e/procurement-policy.json}"
 inc="$gateway/it-incident-management/v1"
 checks=0
 failed=0
@@ -189,5 +191,42 @@ check_equal 'a resource the tool said is terminated is GONE' GONE "$(stb GET "$i
 evaluate > /dev/null
 check_equal 'a terminated resource opens nothing, however long ago it was seen' 0 "$(stb GET "$alr/retrieve?checkId=$gone" | count)"
 
+# 10. A purchase order that is late (ADR-043). An order cannot be drafted with an expected day in the past, so the smoke moves the expected day of a placed order
+# back in the store (docker exec, as for section 9); the approval goes through the REAL approval service with the purchase-order policy.
+[[ -f "$policy_file" ]] || { echo "Missing $policy_file - run provision-change-management.sh first." >&2; exit 1; }
+approver=$(python3 -c "import json,sys; print(json.load(open(sys.argv[1]))['approverIds'][0])" "$policy_file")
+order_body() { printf '{"reference":"%s","vendorId":"%s","requesterId":"%s","currency":"BRL","expectedOn":"%s","lines":[{"category":"HARDWARE","item":"LAP-14","quantity":10,"unitPrice":1500.00}]}' "$1" "$(uuid)" "$filed_for" "$(python3 -c "import datetime; print((datetime.date.today() + datetime.timedelta(days=5)).isoformat())")"; }
+placed_order() {
+  local id; id=$(stb POST "$po/initiate" "$(order_body "$1")" | json "['id']")
+  st PUT "$po/$id/control/submit" > /dev/null
+  api_status "$(api PUT "$po/$id/approval/capture" "{\"approverId\":\"$approver\",\"outcome\":\"APPROVE\"}" "${staff[@]}")" > /dev/null
+  st PUT "$po/$id/control/place" > /dev/null
+  echo "$id"
+}
+postpone() { docker exec "$mongo_container" mongosh --quiet --eval "db.getSiblingDB('thinklab_it_procurement_db').purchase_orders.updateOne({_id: UUID('$1')}, {\$set: {expectedOn: '$(python3 -c "import datetime; print((datetime.date.today() - datetime.timedelta(days=3)).isoformat())")'}})" > /dev/null; }
+order=$(placed_order PO-LATE-1)
+check_equal 'a placed order is not late while its expected day is ahead' 'ORDERED/False' "$(stb GET "$po/$order/retrieve" | json "['status']")/$(stb GET "$po/$order/retrieve" | json "['late']")"
+check_equal 'a rule for the procurement registry is set (201)' 201 "$(st POST "$alr/rule/initiate" "$(rule_json 'Purchase orders' PROCUREMENT)")"
+evaluate > /dev/null
+check_equal 'an order that is not late opens nothing' 0 "$(stb GET "$alr/retrieve?checkId=$order" | count)"
+postpone "$order"
+check_equal 'three days past the expected day: the registry says it is late' True "$(stb GET "$po/$order/retrieve" | json "['late']")"
+evaluate > /dev/null
+check_equal 'the order gets one alert and one incident' True "$(truth wait_for 20 linked "$order")"
+check_equal 'the alert names the order and the fixed code PURCHASE_ORDER_LATE, and carries no vendor or item' 'PO-LATE-1/PURCHASE_ORDER_LATE/False' \
+  "$(alert_of "$order" "['checkName']")/$(alert_of "$order" "['lastError']")/$(stb GET "$alr/retrieve?checkId=$order" | python3 -c "import sys; t=sys.stdin.read(); print('LAP-14' in t)")"
+evaluate > /dev/null
+check_equal 'more evaluations open nothing more for the order' 1 "$(stb GET "$alr/retrieve?checkId=$order" | count)"
+check_equal 'the goods are received completely (201)' 201 "$(api_status "$(api POST "$po/$order/receipt/initiate" '{"lineNo":1,"deliveryRef":"DN-LATE","quantity":10}' "${staff[@]}")")"
+evaluate > /dev/null
+check_equal 'receiving everything resolves the alert' True "$(truth wait_for 20 status_is "$order" RESOLVED)"
+cancelled=$(placed_order PO-LATE-2)
+postpone "$cancelled"
+evaluate > /dev/null
+check_equal 'a second late order gets its alert' True "$(truth wait_for 20 linked "$cancelled")"
+check_equal 'it is cancelled (204)' 204 "$(st PUT "$po/$cancelled/control/cancel")"
+evaluate > /dev/null
+check_equal 'cancelling the order resolves the alert: nothing is owed' True "$(truth wait_for 20 status_is "$cancelled" RESOLVED)"
+
 echo "Promises alerting smoke: $checks checks, $failed failed"
 [[ $failed -eq 0 ]]

scripts/e2e/starter-pack-smoke.ps1

@@ -3,7 +3,7 @@
   Live proof of the starter pack of alert rules (scripts/stack/seed-starter-pack.ps1) through the platform gateway (PowerShell port of starter-pack-smoke.sh).
 
 .DESCRIPTION
-  An edition gets its rules (Homelab 2, Team 5, Enterprise 8), each watching every policy of its source and taking the class of that source; running it
+  An edition gets its rules (Homelab 2, Team 5, Enterprise 9), each watching every policy of its source and taking the class of that source; running it
   again creates nothing; moving up an edition adds only what is missing; a rule a person paused stays paused; another tenant has none of it; an unknown
   edition or a missing argument is refused (the PowerShell script refuses them when it binds its parameters, so the exit code is 1, not the 2 of the bash one).
 
@@ -74,8 +74,8 @@ Assert-Equal 'still five rules' (Rules).Count 5
 
 # 4. Enterprise.
 $run = Pack 'ENTERPRISE'
-Assert-Equal 'Enterprise adds the vendor, licence and cloud resource rules' "$($run.Code)/$($run.Last)" '0/ENTERPRISE: 3 created, 5 already there, 0 failed'
-Assert-Equal 'eight rules, all eight sources' (Sources) 'BACKUP,CAPACITY,CLOUD_INVENTORY,DR,HEALTH_CHECK,LICENSE,PATCH,VENDOR'
+Assert-Equal 'Enterprise adds the vendor, licence, cloud resource and purchase order rules' "$($run.Code)/$($run.Last)" '0/ENTERPRISE: 4 created, 5 already there, 0 failed'
+Assert-Equal 'nine rules, all nine sources' (Sources) 'BACKUP,CAPACITY,CLOUD_INVENTORY,DR,HEALTH_CHECK,LICENSE,PATCH,PROCUREMENT,VENDOR'
 Assert-Equal 'the contract rules are CONTRACT' ((@(Rules | Where-Object { $_.source -in 'VENDOR', 'LICENSE' } | ForEach-Object { $_.alertClass })) -join ',') 'CONTRACT,CONTRACT'
 Assert-Equal 'the cloud resource rule is AVAILABILITY' ((@(Rules | Where-Object { $_.source -eq 'CLOUD_INVENTORY' } | ForEach-Object { $_.alertClass })) -join ',') 'AVAILABILITY'
 

scripts/e2e/starter-pack-smoke.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 # Live proof of the starter pack of alert rules (scripts/stack/seed-starter-pack.sh) through the platform gateway: an edition gets its rules (Homelab 2,
-# Team 5, Enterprise 8), each watching every policy of its source and taking the class of that source; running it again creates nothing; moving up an
+# Team 5, Enterprise 9), each watching every policy of its source and taking the class of that source; running it again creates nothing; moving up an
 # edition adds only what is missing; a rule a person paused stays paused; another tenant has none of it; an unknown edition or a missing argument
 # exits 2.
 set -uo pipefail
@@ -67,8 +67,8 @@ check_equal 'still five rules' 5 "$(rules | count)"
 
 # 4. Enterprise.
 out=$("$pack" ENTERPRISE "$tenant_id" "$requester" 2>&1); status=$?
-check_equal 'Enterprise adds the vendor, licence and cloud resource rules' '0/ENTERPRISE: 3 created, 5 already there, 0 failed' "$status/$(last_line "$out")"
-check_equal 'eight rules, all eight sources' 'BACKUP,CAPACITY,CLOUD_INVENTORY,DR,HEALTH_CHECK,LICENSE,PATCH,VENDOR' "$(sources)"
+check_equal 'Enterprise adds the vendor, licence, cloud resource and purchase order rules' '0/ENTERPRISE: 4 created, 5 already there, 0 failed' "$status/$(last_line "$out")"
+check_equal 'nine rules, all nine sources' 'BACKUP,CAPACITY,CLOUD_INVENTORY,DR,HEALTH_CHECK,LICENSE,PATCH,PROCUREMENT,VENDOR' "$(sources)"
 check_equal 'the contract rules are CONTRACT' 'CONTRACT,CONTRACT' \
   "$(rules | python3 -c "import json,sys; print(','.join(r['alertClass'] for r in json.load(sys.stdin) if r.get('source') in ('VENDOR','LICENSE')))")"
 check_equal 'the cloud resource rule is AVAILABILITY' AVAILABILITY "$(rules | python3 -c "import json,sys; print(','.join(r['alertClass'] for r in json.load(sys.stdin) if r.get('source') == 'CLOUD_INVENTORY'))")"

scripts/stack/seed-starter-pack.ps1

@@ -34,6 +34,7 @@ $enterprise = @(
     'Starter: a vendor agreement is about to end|VENDOR|MEDIUM|LOW|0',
     'Starter: a licence is over its limit|LICENSE|MEDIUM|MEDIUM|0'
     'Starter: a cloud resource went missing|CLOUD_INVENTORY|MEDIUM|MEDIUM|0'
+    'Starter: a purchase order is late|PROCUREMENT|MEDIUM|LOW|0'
 )
 $pack = switch ($Edition) { 'HOMELAB' { $homelab } 'TEAM' { $team } 'ENTERPRISE' { $enterprise } }
 

scripts/stack/seed-starter-pack.sh

@@ -36,6 +36,7 @@ enterprise=(
   "Starter: a vendor agreement is about to end|VENDOR|MEDIUM|LOW|0"
   "Starter: a licence is over its limit|LICENSE|MEDIUM|MEDIUM|0"
   "Starter: a cloud resource went missing|CLOUD_INVENTORY|MEDIUM|MEDIUM|0"
+  "Starter: a purchase order is late|PROCUREMENT|MEDIUM|LOW|0"
 )
 case "$edition" in
   HOMELAB) pack=("${homelab[@]}") ;;