Remote server tls authentication by asher-pem-arm · Pull Request #1885 · labgrid-project/labgrid

asher-pem-arm · 2026-06-03T17:02:49Z

This MR adds support for gRPC SSL/TLS server authentication to labgrid-coordinator, labgrid-exporter and labgrid-client.

Enabling a secure channel on all three components is done by adding the --secure argument. Paths to the certificate and key (labgrid-coordinator-only) can be specified with the --cert and --key arguments.

Testing has been added to verify that, given a secure-enabled labgrid-coordinator, both labgrid-client and labgrid-exporter can connect successfully and perform an operation.

Note on use of `labgrid-client` and `labgrid-exporter` without an explicit `--cert` specified

On Linux, this currently reads the Debian/Ubuntu CA bundle at:
/etc/ssl/certs/ca-certificates.crt

On macOS, this reads certificates from the system Keychain using:
security find-certificate -a -p

If the coordinator certificate is not trusted by the host, or the platform's system roots cannot be loaded, pass the coordinator certificate explicitly with --cert.

Signed-off-by: Asher Pemberton <asher.pemberton@arm.com> Reviewed-by: Asher Pemberton <asher.pemberton@arm.com> # gatekeeper Co-authored-by: Luke Beardsmore <luke.beardsmore2@arm.com>

codecov · 2026-06-03T18:35:14Z

Codecov Report

❌ Patch coverage is 52.68817% with 44 lines in your changes missing coverage. Please review.
✅ Project coverage is 46.8%. Comparing base (915b591) to head (c6d1da1).
⚠️ Report is 10 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
labgrid/remote/common.py	35.8%	25 Missing ⚠️
labgrid/remote/coordinator.py	57.8%	8 Missing ⚠️
labgrid/remote/exporter.py	0.0%	7 Missing ⚠️
labgrid/remote/client.py	50.0%	4 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #1885     +/-   ##
========================================
+ Coverage    46.0%   46.8%   +0.8%     
========================================
  Files         180     180             
  Lines       14464   14555     +91     
========================================
+ Hits         6654    6819    +165     
+ Misses       7810    7736     -74

Flag	Coverage Δ
3.10	`46.8% <52.6%> (+0.8%)`	⬆️
3.11	`46.8% <52.6%> (+0.8%)`	⬆️
3.12	`?`
3.13	`46.8% <52.6%> (+0.8%)`	⬆️
3.14	`46.8% <52.6%> (+0.8%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Emantor

Otherwise looks good

Emantor · 2026-06-09T09:05:44Z

+
+.. code-block:: bash
+
+    $ labgrid-client --secure [--cert PATH] places


Feels very much like bikeshedding to write this, but should we rename --secure to --tls instead? This makes it clear that we are using TLS for gRPC secure channels instead of a generic --secure which does not carry much meaning.

gastmaier

Looks good! Some minor comments

gastmaier · 2026-06-09T11:35:59Z

+Refer to the ``labgrid-coordinator`` man page for details.
+
+When you are connecting with ``labgrid-client`` or ``labgrid-exporter`` to a
+``labgrid-coordinator``that has secure gRPC channels enabled you need to pass


Suggested change

``labgrid-coordinator``that has secure gRPC channels enabled you need to pass

``labgrid-coordinator`` that has secure gRPC channels enabled you need to pass

gastmaier · 2026-06-09T11:46:06Z

+--secure
+    enable TLS gRPC channel
+--cert
+    path to TLS certificate (in PEM format)
+--key
+    path to TLS key (in PEM format)


On the coordinator side, the tls at the reverse proxy is also an option, right?
At servers, I would rather have nginx handling the certificates.

If so, could you add an alternative, such as

Or use a reverse proxy to add TLS, for example with ``nginx``: .. code-block:: nginx server { listen 20407 ssl http2; server_name labgrid.example.com; ssl_certificate /etc/ssl/labgrid-coordinator.crt; ssl_certificate_key /etc/ssl/labgrid-coordinator.key; location / { grpc_pass grpc://127.0.0.1:20408; } }

https://blog.nginx.org/blog/nginx-1-13-10-grpc

gastmaier · 2026-06-09T12:35:52Z

+    return None
+
+
+def _fetch_root_certificates_linux():


On why this is needed.

the certificate precedence is (src/core/credentials/transport/tls/ssl_utils.cc @ ComputePemRootCerts)

GRPC_DEFAULT_SSL_ROOTS_FILE_PATH UseSystemRootsOverLanguageCallback() ssl_roots_override_cb <= never reached LoadSystemRootCerts() installed_roots_path fallback

the cb is set at src/python/grpcio/grpc/_cython/cygrpc.pyx
added at github.com/grpc/grpc/commit/fa6cad701c7993aa6e5746824931efbfca84ca24

The only options are indeed GRPC_DEFAULT_SSL_ROOTS_FILE_PATH or as the argument value.

the grpc defaults for linux are (https://github.com/grpc/grpc/blob/master/src/core/credentials/transport/tls/load_system_roots_supported.cc#L48-L62)

"/etc/ssl/certs/ca-certificates.crt", "/etc/pki/tls/certs/ca-bundle.crt", "/etc/ssl/ca-bundle.pem", "/etc/pki/tls/cacert.pem", "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"

I wouldn't have one of the default here, but just document that it will default to the bundled roots.pem if not explicitly set through the env or as the argument value.

Hi @gastmaier I think the important Python bit here is that grpcio installs a roots override callback which loads its bundled roots.pem, so on Python the normal default is not the Linux system trust store. That means if we do not pass root_certificates, --tls may ignore CAs installed on the host.

So I think the correct ordering for Labgrid should be:

explicit --cert

system trust store loaded by Labgrid

gRPC/Python default only as fallback if we cannot load system roots

I'll update the getting_started docs to represent this

--tls may ignore CAs installed on the host.
Due to how the gprc source code is structured, it will ignore CAs installed on the host (never read) if roots.pem is present.
and roots.pem is always present with the pip package install:

find . -name 'roots.pem' ./lib/python3.14/site-packages/grpc/_cython/_credentials/roots.pem

Hacking rm ./lib/python3.14/site-packages/grpc/_cython/_credentials/roots.pem as a post-install step for grpcio is not an option.

gastmaier · 2026-06-09T12:37:34Z

+the ``--secure`` (and ``--cert`` if the certificate is not trusted by the host
+machine) option.
+Refer to the ``labgrid-client`` and ``labgrid-exporter`` man pages for details.
+


explain roots.pem and GRPC_DEFAULT_SSL_ROOTS_FILE_PATH precedence here, please

Signed-off-by: Asher Pemberton <asher.pemberton@arm.com> Reviewed-by: Asher Pemberton <asher.pemberton@arm.com> # gatekeeper

asher-pem-arm · 2026-06-10T09:34:00Z

@Emantor I have added 3 commits for the comments in the PR, if you've happy with these I'll squash all 3 into the first commit to keep the PR tidy

gastmaier

Sounds good!

Users have the option to pass explicitly or read gprc doc for the env alternative GRPC_DEFAULT_SSL_ROOTS_FILE_PATH.

gastmaier · 2026-06-10T10:07:29Z

+    return None
+
+
+def _fetch_root_certificates_linux():


--tls may ignore CAs installed on the host.
Due to how the gprc source code is structured, it will ignore CAs installed on the host (never read) if roots.pem is present.
and roots.pem is always present with the pip package install:

find . -name 'roots.pem' ./lib/python3.14/site-packages/grpc/_cython/_credentials/roots.pem

Hacking rm ./lib/python3.14/site-packages/grpc/_cython/_credentials/roots.pem as a post-install step for grpcio is not an option.

Emantor · 2026-06-10T12:25:34Z

@Emantor I have added 3 commits for the comments in the PR, if you've happy with these I'll squash all 3 into the first commit to keep the PR tidy

Looks good, please squash.

asher-pem-arm and others added 2 commits June 3, 2026 17:45

Add support for gRPC TLS server authentication

b1d507c

Signed-off-by: Asher Pemberton <asher.pemberton@arm.com> Reviewed-by: Asher Pemberton <asher.pemberton@arm.com> # gatekeeper Co-authored-by: Luke Beardsmore <luke.beardsmore2@arm.com>

Manually load system root certificates on macOS/Linux

474dfb9

Signed-off-by: Asher Pemberton <asher.pemberton@arm.com> Reviewed-by: Asher Pemberton <asher.pemberton@arm.com> # gatekeeper Co-authored-by: Luke Beardsmore <luke.beardsmore2@arm.com>

Emantor requested review from Bastian-Krause, Emantor and jluebbe and removed request for Emantor and jluebbe June 3, 2026 18:29

Emantor previously approved these changes Jun 9, 2026

View reviewed changes

gastmaier reviewed Jun 9, 2026

View reviewed changes

asher-pem-arm added 3 commits June 9, 2026 15:46

rename secure to tls

85e45c7

Signed-off-by: Asher Pemberton <asher.pemberton@arm.com> Reviewed-by: Asher Pemberton <asher.pemberton@arm.com> # gatekeeper

resource/remote: support TLS coordinator sessions for RemotePlace

a9a4120

Signed-off-by: Asher Pemberton <asher.pemberton@arm.com> Reviewed-by: Asher Pemberton <asher.pemberton@arm.com> # gatekeeper

improve labgrid tls docs

c6d1da1

Signed-off-by: Asher Pemberton <asher.pemberton@arm.com> Reviewed-by: Asher Pemberton <asher.pemberton@arm.com> # gatekeeper

asher-pem-arm dismissed Emantor’s stale review via c6d1da1 June 10, 2026 09:32

gastmaier approved these changes Jun 10, 2026

View reviewed changes

Emantor requested a review from jluebbe June 10, 2026 12:25


		.. code-block:: bash

		$ labgrid-client --secure [--cert PATH] places

	``labgrid-coordinator``that has secure gRPC channels enabled you need to pass
	``labgrid-coordinator`` that has secure gRPC channels enabled you need to pass

Uh oh!

Conversation

asher-pem-arm commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Note on use of labgrid-client and labgrid-exporter without an explicit --cert specified

Uh oh!

codecov Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Emantor left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gastmaier left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gastmaier Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asher-pem-arm commented Jun 10, 2026

Uh oh!

gastmaier left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Emantor commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

asher-pem-arm commented Jun 3, 2026 •

edited

Loading

Note on use of `labgrid-client` and `labgrid-exporter` without an explicit `--cert` specified

codecov Bot commented Jun 3, 2026 •

edited

Loading

gastmaier Jun 9, 2026 •

edited

Loading