Management ip address resolution by EduFrazao · Pull Request #13603 · apache/cloudstack

EduFrazao · 2026-07-14T01:31:05Z

Description

This PR solves a situation when the management network interface does not have a defined IP address.
When this interface is a bridge, the management ip can be placed directly in the bridge (more common) and sometimes in virtual interfaces linked to it.
On this scenarios, the cloudstack agent iterates over all interfaces and select the first one with a valid ip address.
This brings unpredictable behavior, like choosing wrong interface address, and publishing this address to management servers, causing problems with migrations, guest consoles for example.

I've added two new ways to search for the correct management address:

First and more "secure" is to explicitly define the management ip address on agent.properties. The agent will find the interface bound to this address and configure it as private interface.
If configuring this address is not desirable, agent will get the management server addresses, and try to establish a socket (without tcp overhead) with it. If it suceeds, will possible to collect the source ip address used to reach management servers. I belive that this is a more assertive way to find out the host management address.
If the above methods don't work, the actual behavior is used without changes.

Setup a Full L3 network design, with management defined as a traffic label over a "virtual physical address".
Create a bridge and setup a vxlan interface with this bridge as master interface.
Define a veth interface and define the bridge as master for this interface too.
Derfine in this veth interface, the IP address that will be used to reach the management servers.
Theres no guarantee that this will be the IP address reported to the management server.

Fixes: #13519

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

I have two hipervisors in production with this changes. Whitout it, I can't make my cluster to work well, cause I can't do migrations and open guest instances terminal
To test it I used both the explicit configuration and route based detection for several days, marking servers for maintenance, rebooting hypervisors and management servers.

How did you try to break this feature and the system with this change?

Configuring an invalid Ip address as management address. The setting was ignored as expected, because no interface was found with it. System fallback to another methods.

boring-cyborg · 2026-07-14T01:31:09Z

Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:

In case of a new feature add useful documentation (raise doc PR at https://github.com/apache/cloudstack-documentation)
Be patient and persistent. It might take some time to get a review or get the final approval from the committers.
Pay attention to the quality of your code, ensure tests are passing and your PR doesn't have conflicts.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Issues, Mailing list and Slack.
Be sure to read the CloudStack Coding Conventions.
Apache CloudStack is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@cloudstack.apache.org (https://cloudstack.apache.org/mailing-lists.html)
Slack: https://apachecloudstack.slack.com/

DaanHoogland

thanks @EduFrazao , clgtm.

DaanHoogland · 2026-07-20T09:39:35Z

also @EduFrazao , I marked it for 24 as you based the PR off main. please rebase if you need it on 22.2 or somewhere else.

codecov · 2026-07-20T10:09:45Z

Codecov Report

❌ Patch coverage is 3.44828% with 56 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.69%. Comparing base (0339f31) to head (d2c5f0f).
⚠️ Report is 1 commits behind head on 4.22.

Files with missing lines	Patch %	Lines
...in/java/com/cloud/resource/ServerResourceBase.java	1.81%	54 Missing ⚠️
...ervisor/kvm/resource/LibvirtComputingResource.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.22   #13603      +/-   ##
============================================
- Coverage     17.69%   17.69%   -0.01%     
+ Complexity    15833    15831       -2     
============================================
  Files          5925     5925              
  Lines        533534   533596      +62     
  Branches      65273    65286      +13     
============================================
- Hits          94421    94411      -10     
- Misses       428434   428508      +74     
+ Partials      10679    10677       -2

Flag	Coverage Δ
uitests	`3.69% <ø> (ø)`
unittests	`18.77% <3.44%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

EduFrazao · 2026-07-22T17:15:07Z

also @EduFrazao , I marked it for 24 as you based the PR off main. please rebase if you need it on 22.2 or somewhere else.

Thank you very mutch.
I will do a rebase, I belive that this is a minor change, maybe it suitable for a bugfix release. Thank you for pointing this.

- By explicit definition of the management ip address on agent.properties - By using OS routing table to determine the source address used to connect to any of avaliable management servers. - Fallback to current methods.

DaanHoogland · 2026-07-24T10:15:04Z

@DaanHoogland rebase done! Let me know if there is something more to do!

PS: I think that my rebase was not made on the correct branch right?

looks like it is still based on main. let me know if you want it on 4.22 or 4.20 and if you need help with that. (quick preview: something like git rebase --onto <other branch> <the commit before your first one> should do the trick)

EduFrazao · 2026-07-27T23:38:32Z

@DaanHoogland rebase fixed..
Sorry for the delay!

DaanHoogland · 2026-07-28T07:05:29Z

@blueorangutan package

blueorangutan · 2026-07-28T07:06:04Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-07-28T08:06:38Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18691

blueorangutan · 2026-07-29T23:33:30Z

[SF] Trillian test result (tid-16648)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 53430 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13603-t16648-kvm-ol8.zip
Smoke tests completed. 148 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_ssl_offloading_isolated_network	`Failure`	706.31	test_ssl_offloading.py

Copilot

Pull request overview

This PR aims to make KVM agent “private/management” IP selection deterministic when the configured management NIC has no IP (e.g., IP is on a bridge member/veth), preventing the agent from reporting an incorrect private IP to management servers.

Changes:

Pass additional agent.properties values into KVM resource configuration (management server host list and an optional explicit private IP).
Add two new private-NIC discovery strategies in ServerResourceBase: explicit private IP mapping to a NIC, and route-based source-IP discovery via UDP socket.
Introduce a new agent property private.network.address to optionally pin the management/private IP.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java	Adds agent.properties-derived config values into the resource `params` map used by base configuration.
core/src/main/java/com/cloud/resource/ServerResourceBase.java	Adds explicit-IP and route-lookup based private NIC discovery before the existing iteration fallback.
agent/src/main/java/com/cloud/agent/properties/AgentProperties.java	Adds a new agent property definition for explicitly setting the private NIC IP address.

Suppressed comments (2)

core/src/main/java/com/cloud/resource/ServerResourceBase.java:182

Avoid catching Throwable here; it can hide serious JVM errors and drops the stack trace. Catch Exception and pass the throwable to the logger so failures in route lookup can be diagnosed.

            } catch (Throwable e) {
                // Logging only, if this method was unable to find a valid interface, iteration will be tested
                logger.debug(String.format("Unable to use routing table to determine private management interface: [%s]", e.getMessage()));
            }

core/src/main/java/com/cloud/resource/ServerResourceBase.java:87

New private-NIC resolution behaviors (private.network.address override and route-lookup discovery) are introduced here but there are no accompanying unit tests (ServerResourceBaseTest currently covers only defineResourceNetworkInterfaces/isValidNic/iteration-based discovery). Adding tests for: (1) selecting NIC by configured IP; (2) falling back to route lookup when the configured NIC has no usable address; and (3) falling back to simple iteration when route lookup fails, would help prevent regressions.

        if (privateNic == null) {
            checkForPrivateInterfaceDefinedByIp(params);
        }
        if (privateNic == null) {
            tryToAutoDiscoverResourcePrivateNetworkInterfaceByRouteLookup(params);

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

        defineResourceNetworkInterfaces(params);

+        if (privateNic == null) {
+            checkForPrivateInterfaceDefinedByIp(params);
+        }
+        if (privateNic == null) {
+            tryToAutoDiscoverResourcePrivateNetworkInterfaceByRouteLookup(params);
+        }
        if (privateNic == null) {
            tryToAutoDiscoverResourcePrivateNetworkInterface();
        }


+        final String ifAddr = (String) params.get("private.network.address");
+        if (ifAddr != null) {
+            logger.debug(String.format("Trying to use private address to resolve interface: [%s]", ifAddr));


+                } else {
+                    logger.info(String.format("Unable to found private NIC with defined ip [%s]", ifAddr));
+                }
+            } catch (Throwable e) {
+                 // Logging only, if this method was unable to find a valid interface, iteration will be tested
+                logger.info(String.format("Unable to use private address to get the management interface: [%s]", e.getMessage()));
+            }


+     * Private NIC device address. If this property is commented, it will be autodetected on service startup.<br>
+     * Data type: String.<br>
+     * Default value: <code>cloudbr1</code>


+        params.put(AgentProperties.HOST.getName(), AgentPropertiesFileHandler.getPropertyValue(AgentProperties.HOST));
+        params.put(AgentProperties.PRIVATE_NETWORK_DEVICE_ADDRESS.getName(), AgentPropertiesFileHandler.getPropertyValue(AgentProperties.PRIVATE_NETWORK_DEVICE_ADDRESS));


boring-cyborg Bot added component:agent component:kvm labels Jul 14, 2026

DaanHoogland reviewed Jul 20, 2026

View reviewed changes

Comment thread core/src/main/java/com/cloud/resource/ServerResourceBase.java Outdated

DaanHoogland added this to the 4.24.0 milestone Jul 20, 2026

EduFrazao added 2 commits July 22, 2026 21:41

New way to resolve private interface implemented:

daf9bd0

- By explicit definition of the management ip address on agent.properties - By using OS routing table to determine the source address used to connect to any of avaliable management servers. - Fallback to current methods.

Fix method access modifier

9e1f3be

EduFrazao force-pushed the management_ip_addr_resolution branch from 0a2c926 to 9e1f3be Compare July 23, 2026 00:44

EduFrazao changed the base branch from main to 4.22 July 23, 2026 00:49

DaanHoogland removed the component:backup label Jul 24, 2026

DaanHoogland changed the base branch from 4.22 to main July 24, 2026 10:12

EduFrazao force-pushed the management_ip_addr_resolution branch from 813ea6e to 9e1f3be Compare July 27, 2026 23:37

EduFrazao changed the base branch from main to 4.22 July 27, 2026 23:43

Fix typo

d2c5f0f

Copilot AI reviewed Aug 10, 2026

View reviewed changes

Navigation Menu

Uh oh!

Management ip address resolution - #13603

Management ip address resolution#13603
EduFrazao wants to merge 3 commits into
apache:4.22from
EduFrazao:management_ip_addr_resolution

EduFrazao commented Jul 14, 2026

Uh oh!

boring-cyborg Bot commented Jul 14, 2026

Uh oh!

DaanHoogland left a comment

Uh oh!

Uh oh!

DaanHoogland commented Jul 20, 2026

Uh oh!

codecov Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

EduFrazao commented Jul 22, 2026

Uh oh!

DaanHoogland commented Jul 24, 2026

Uh oh!

EduFrazao commented Jul 27, 2026

Uh oh!

DaanHoogland commented Jul 28, 2026

Uh oh!

blueorangutan commented Jul 28, 2026

Uh oh!

blueorangutan commented Jul 28, 2026

Uh oh!

blueorangutan commented Jul 29, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		params.put(AgentProperties.HOST.getName(), AgentPropertiesFileHandler.getPropertyValue(AgentProperties.HOST));
		params.put(AgentProperties.PRIVATE_NETWORK_DEVICE_ADDRESS.getName(), AgentPropertiesFileHandler.getPropertyValue(AgentProperties.PRIVATE_NETWORK_DEVICE_ADDRESS));

Uh oh!

Conversation

EduFrazao commented Jul 14, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

boring-cyborg Bot commented Jul 14, 2026

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DaanHoogland commented Jul 20, 2026

Uh oh!

codecov Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

EduFrazao commented Jul 22, 2026

Uh oh!

DaanHoogland commented Jul 24, 2026

Uh oh!

EduFrazao commented Jul 27, 2026

Uh oh!

DaanHoogland commented Jul 28, 2026

Uh oh!

blueorangutan commented Jul 28, 2026

Uh oh!

blueorangutan commented Jul 28, 2026

Uh oh!

blueorangutan commented Jul 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov Bot commented Jul 20, 2026 •

edited

Loading