Wire up hive tests on CI by SamAg19 · Pull Request #118 · grandinetech/lean

SamAg19 · 2026-07-26T10:41:29Z

fixes #117

Checklist

Self-check your solution before submitting:

Verify make test-hive works, on a clean repo checkout, without additional setup (apart from system installation, like go toolchain);
Verify that running make test-hive second time doesn't rebuild hive itself;
Verify that running make test-hive HIVE_VERSION=<some_other_commit_hash> rebuilds hive properly and re-runs tests;
Verify that modifying code in grandine (you can just insert some print in ./lean_client/src/main.rs`) rebuilds docker image and re-runs hive tests;
Verify that your script runs all hive tests, available for grandine lean client;
Verify your CI works (not necessarily passes, just runs tests) on your own fork;
Verify that make clean cleans up hive directory;
Don't forget to update README.md - include section what needs to be installed (probably only go toolchain) for running hive tests, and how to run them.

Additions beyond the checklist:

Per-suite summary - each suite's pass/total count + names of failed test cases, printed to the job logs and rendered as a GitHub Step Summary panel.
Infra-vs-test failure semantics - a completed run with failing tests is reported but does not fail the job; only a genuine build/run failure (no results produced) fails it. A red check means the pipeline broke, not that a spec test failed.
~~Label-gated CI — runs only on PRs with the run-hive label (plus manual workflow_dispatch), not on every push, since the full suite is slow.~~

Example run (summary panel on the Summary page): https://github.com/SamAg19/lean/actions/runs/30262177584?pr=1; the committed workflow runs the full suite. Example PR where i tested out everything: SamAg19#1

SamAg19 · 2026-07-27T11:39:03Z

@ArtiomTr @bomanaps The PR is ready for review

ArtiomTr

Very good, although small fixes still needed

ArtiomTr · 2026-07-29T08:28:36Z

+# Runs on manual dispatch, and on pull requests only when they carry the
+# `run-hive` label (added on label, and re-run on subsequent pushes) so the slow
+# suite doesn't run on every PR push.


probably fine to run this on every PR, because if it isn't running automatically, then probably no one will run these

resolved 92e8014

ArtiomTr · 2026-07-29T08:32:39Z

+CLIENT_SOURCES := $(shell find . \( -name '*.rs' -o -name 'Cargo.toml' \) \
+	-not -path './target/*' -not -path './hive/*' -not -path './spec/*' \
+	-not -path './bin/*') Cargo.lock


Claude said:

One thing worth knowing: -not -path filters results but doesn't prune traversal, so find still walks all of ./target/, which can be slow on a large build dir. -path './target/*' -prune -o ... -print avoids that.

Don't know if that's true, but probably still worth looking - target directory can indeed be large and slow down source file discovery.

resolved 92e8014

ArtiomTr · 2026-07-29T08:33:05Z

+### Hive
+# ethereum/hive ref to build. Unset (default) => latest master, resolved at
+# clone time. Override with `make test-hive HIVE_VERSION=<commit>` to pin.
+HIVE_VERSION ?=
+HIVE_REF := $(or $(HIVE_VERSION),master)


Probably would be better to put variables on top of the file, next to all other variables - this way it becomes a bit more obvious how we can parametrize our make runs.

also, probably no need to have 2 variables here - both HIVE_VERSION and HIVE_REF? If you want to set default version, then you can just do:

HIVE_VERSION ?= master

?= already means "assign if unset".

resolved 92e8014

ArtiomTr · 2026-07-29T08:34:29Z

+		--client-file simulators/lean/clients/devnet5.yaml \
+		--client grandine_lean_devnet5 \
+		--docker.output \
+		--docker.nocache="hive/clients/grandine_lean_devnet5" \
+		--results-root ./workspace/logs; \


Not exactly sure, but I think we should be able to parametrize the devnet5 part

resolved 92e8014

SamAg19

@ArtiomTr I have resolved the comments so you can re-review the PR once again. Example run post the changes suggested by you: https://github.com/SamAg19/lean/actions/runs/30647017513?pr=1

ArtiomTr

looks good 👍

Example run post the changes suggested by you: https://github.com/SamAg19/lean/actions/runs/30647017513?pr=1

Why isn't it failed? It reports 98 failed tests, but the status of job is still "success"?

ArtiomTr · 2026-08-10T13:24:14Z

+# Runs on manual dispatch and on every pull request. The suite is slow, but
+# running it automatically ensures it actually runs (a label-gated job tends to
+# never get triggered).


no need to keep this as a comment :)

ArtiomTr · 2026-08-10T13:32:18Z

Also, this currently consists of ~12 commits - maybe you can squash them into one?

SamAg19 · 2026-08-14T08:11:47Z

looks good 👍

Example run post the changes suggested by you: https://github.com/SamAg19/lean/actions/runs/30647017513?pr=1

Why isn't it failed? It reports 98 failed tests, but the status of job is still "success"?

@ArtiomTr The criteria for success and failure in the CI is that whether hive ran to completion or not. If it ran to completion, then a result summary has been produced which we can go and inspect to see which tests are failing. Only in the case of infrastructure failure such as building/setup failures is when the CI will fail

ArtiomTr · 2026-08-14T08:15:03Z

Only in the case of infrastructure failure such as building/setup failures is when the CI will fail

That doesn't sound right, better to fail if any test fails - otherwise no one cares about it

SamAg19 · 2026-08-14T08:17:21Z

Only in the case of infrastructure failure such as building/setup failures is when the CI will fail

That doesn't sound right, better to fail if any test fails - otherwise no one cares about it

Forgot to mention one other thing is that this is a similar to zeam's setup as well hence i went ahead this route. I can revert back to failing incase a hive test failure if you think thats the best way to go ahead

ArtiomTr · 2026-08-14T08:18:52Z

Yeah, I do believe failing is better, in case tests fail - not sure why zeam chose other approach

SamAg19 · 2026-08-14T11:03:11Z

Yeah, I do believe failing is better, in case tests fail - not sure why zeam chose other approach

@ArtiomTr Fixed 93db8b2
Summary: https://github.com/SamAg19/lean/actions/runs/31784273113?pr=2
Run: https://github.com/SamAg19/lean/actions/runs/31784273113/job/94716590095?pr=2

SamAg19 marked this pull request as draft July 27, 2026 06:39

SamAg19 marked this pull request as ready for review July 27, 2026 11:36

ArtiomTr reviewed Jul 29, 2026

View reviewed changes

SamAg19 commented Jul 31, 2026

View reviewed changes

SamAg19 requested a review from ArtiomTr July 31, 2026 16:31

ArtiomTr reviewed Aug 10, 2026

View reviewed changes

Wire up hive tests on CI

1e0ce77

SamAg19 force-pushed the ci-hive branch from 22afde9 to 1e0ce77 Compare August 14, 2026 08:18

Fail test-hive when any hive test fails

93db8b2

SamAg19 requested a review from ArtiomTr August 17, 2026 10:00

ArtiomTr merged commit 632e66a into grandinetech:devnet-5 Aug 20, 2026
2 of 4 checks passed

Uh oh!

Conversation

SamAg19 commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Additions beyond the checklist:

Uh oh!

SamAg19 commented Jul 27, 2026

Uh oh!

ArtiomTr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SamAg19 left a comment

Choose a reason for hiding this comment

Uh oh!

ArtiomTr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArtiomTr commented Aug 10, 2026

Uh oh!

SamAg19 commented Aug 14, 2026

Uh oh!

ArtiomTr commented Aug 14, 2026

Uh oh!

SamAg19 commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ArtiomTr commented Aug 14, 2026

Uh oh!

SamAg19 commented Aug 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SamAg19 commented Jul 26, 2026 •

edited

Loading

SamAg19 commented Aug 14, 2026 •

edited

Loading