ci: fix distributed smoke-test OOM (bigger machine, smaller heap, retries)

[joseanesONMS]

Problem

The distributed smoke tests fail with ContainerLaunchException / container-shutdown timeouts that look like an infra flake but are OOM. Each distributed test stands up a full stack — opennms/horizon (~3GB) + opennms/sentinel ×2 (~4GB) + Kafka + Zookeeper + Postgres + Grafana/Helm + Selenium/VNC (~1.2GB) — on top of a 3.2GB Maven heap. That's ~14GB against the default medium machine's 7.5GB, so the OOM killer aborts container startup (Killed in the logs).

The tell: IntegratedCorrelationTest (one lightweight OpenNMS container) passes every run, while every multi-container distributed test fails.

Fix

• smoke-test-executor: resource_class: large (4 vCPU / 15GB)
• MAVEN_OPTS: -Xmx3200m → -Xmx1500m — the test JVM only orchestrates containers, so hand the RAM to the containers
• surefire rerunFailingTestsCount=2 to absorb residual transient container flakiness without masking a reproducible failure

It also lets branches with -smoke in the name run the full smoke suite (previously develop/release/tags only) — which is how this branch validates the fix against the whole distributed suite before merge.

Cost

large uses more CircleCI credits per smoke run than medium. Approved as the tradeoff for getting the distributed suite reliably green.

Validation

This branch (…-smoke) triggers smoke-test-full, so its own run exercises all distributed tests under the new resources. Will confirm green before requesting merge.

[joseanesONMS]

@cgorantla this is ready for review — it fixes the long-standing distributed smoke-test failures.

Validated on this branch's full-smoke run (#8924): the fix takes the distributed suite from 4 failures → 1.

Test	before	after
DistributedRedundant (OOM)	❌	✅
DistributedStandalone[cluster]	❌	✅
DistributedStandalone[dbscan]	❌	✅
DistributedUDL	❌	✅
DistributedStandalone[deeplearning]	❌	❌

Two root causes fixed:

1. OOM — the distributed stack (~14GB) was overflowing the default 7.5GB machine → resource_class: large (15GB) + Maven heap 3200m→1500m + rerunFailingTestsCount=2.
2. OIA version skew — smoke-test-full has been red since ~March because alec-driver-main couldn't resolve the OIA health package in the standalone sentinel: ALEC targets OIA 2.0.0 but the images were pinned at 35.0.3 (OIA 1.6.1). Bumped opennms/sentinel + opennms/horizon to 36.0.2.

The one remaining red — DistributedStandalone[deeplearning] — is the slow TensorFlow engine timing out on model load. That engine (and this test param) is removed by #166 (ALEC-305), so once both land, the full suite is green. Details in my analysis above.

[cgorantla]

LGTM!