Repository navigation
V8 CI failing on x64 #4462
Description
Activity
Might be related to https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/8333927.
Might be related to https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/8333927.
We use
export 'VPYTHON_BYPASS=manually managed python not supported by chrome operations'but the above CL has switched
fetchandninjafrom vpython to the hemetic sealed Python from depot-tools,We can also try removing the depot_tools override from https://github.com/nodejs/node/blob/fd6682c8713e0f8c348ffb427ad48ce20378b150/tools/v8/fetch_deps.py#L27-L28
We can also try removing the depot_tools override from https://github.com/nodejs/node/blob/fd6682c8713e0f8c348ffb427ad48ce20378b150/tools/v8/fetch_deps.py#L27-L28
I tried removing and separately also rolling forward the override, but those have not fixed the issue.
I suspect the issue is that because we use
VPYTHON_BYPASSwe don't pull down the hermetic sealed Python from depot-tools, but https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/8333927 assumes it is there. That CL broke upstream V8 builds on ppc64(le) and s390x which @miladfarca fixed in https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/8367685 but that doesn't affect the generalVPYTHON_BYPASScase and has no effect on x64.I took a copy of the job and removed (commented out)
export 'VPYTHON_BYPASS=manually managed python not supported by chrome operations'but this hasn't fixed the CI either (including with the override rolled forward) (and it breaks ppc64/s390x as expected as it cannot find binaries for those).
https://ci.nodejs.org/job/richardlau-node-test-commit-v8-linux/781/nodes=benchmark-ubuntu2404-intel-64,v8test=v8test/consoleInteresting datapoint, https://ci.nodejs.org/job/node-test-commit-v8-linux/7371/ for the 22.23.3 proposal has passed (i.e. Node.js 22 does not appear to be affected).
So to recap,
- https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/8333927 landed upstream in depot_tools which bypasses vpython and assumes existence of Google's hermetic Python.
- Our V8 CI has never used Google's hermetic Python and is using
VPYTHON_BYPASS="manually managed python not supported by chrome operations"which tells vpython to fall back to system Python. However the above CL bypasses vpython so that workaround no longer affects the updated tooling. - For Linux on ppc64le/s390x, @miladfarca submitted https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/8367685 to patch depot_tools
python-bin/python3wrapper but only on those architectures (since Google does not host binaries for those architectures). So we don't see the problem there as it has been worked around upstream. - The node-test-commit-v8-linux job pulls down three(!) versions of depot_tools:
depot_tools(under the workspace directory) at the start of the jobdeps/v8/depot_toolsbyEnsureDepotTools()intools/v8/node_common.py. This is always a fresh clone as wegit cleanbefore building.deps/v8/third_party/depot_toolsby a combination ofdeps/v8/DEPSandtools/v8/fetch_deps.py.
- The check and error is coming from the copy of depot_tools in
deps/v8/depot_tools.
So I think we have two options:
- Try to use the hermetic Python from Google. We have never used it and are not pulling it down in the job/Node.js scripts. I don't know what is missing to pull it down and use it on x64.
- Try to patch depot_tools ourselves so that we can continue to use external Python. Short term maybe we can carry that patch in Node.js but I'd prefer if we could upstream something.
I've opened a draft nodejs/node#66070 that patches the
python-bin/python3wrapper indeps/v8/depot_tools:diff --git a/tools/v8/node_common.py b/tools/v8/node_common.py index f873065c1df..ae40d91f0f9 100755 --- a/tools/v8/node_common.py +++ b/tools/v8/node_common.py @@ -6,6 +6,7 @@ # for py2/py3 compatibility from __future__ import print_function +import fileinput import os import pipes import shutil @@ -40,6 +41,16 @@ def EnsureDepotTools(v8_path, fetch_if_not_exist): depot_tools = _Get(v8_path) assert depot_tools is not None print("Using depot tools in %s" % depot_tools) + # Patch python3 wrapper to not depend on Google's hermetic Python. + for line in fileinput.input(files=(os.path.join(depot_tools, "python-bin", "python3")),inplace=True): + sys.stdout.write(line) + if "DEPOT_TOOLS=$(dirname \"$0\")/.." in line: + sys.stdout.write(""" +if [[ $VPYTHON_BYPASS == \"manually managed python not supported by chrome operations\" ]] +then + exec python3 \"$@\" +fi +""") return depot_tools def UninitGit(v8_path):
I'm wondering if this patch to the wrapper might be accepted upstream in depot_tools (cc @miladfarca / @nodejs/v8-update )?
I've previously tested a much simpler change that extended https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/8367685 to also add
x86_64which worked but presumably would not be accepted upstream.diff --git a/tools/v8/node_common.py b/tools/v8/node_common.py index f873065c1df..c0e567441cb 100755 --- a/tools/v8/node_common.py +++ b/tools/v8/node_common.py @@ -6,6 +6,7 @@ # for py2/py3 compatibility from __future__ import print_function +import fileinput import os import pipes import shutil @@ -40,6 +41,9 @@ def EnsureDepotTools(v8_path, fetch_if_not_exist): depot_tools = _Get(v8_path) assert depot_tools is not None print("Using depot tools in %s" % depot_tools) + for line in fileinput.input(files=(os.path.join(depot_tools, "python-bin", "python3")),inplace=True): + line = line.replace("powerpc64le", "powerpc64le | x86_64") + sys.stdout.write(line) return depot_tools def UninitGit(v8_path):
One question I do not have the answer for at the moment is why the v22.x/v24.x builds continue to work but the v26.x/main ones (including v26.0.0) do not . All of those should be pulling the same depot_tools in
deps/v8/depot_toolsbecause we do not specify branches/commit shas intools/v8/node_common.py.Well there's a new complication.
From @aduh95,
FYI I've tried nodejs/node#66070 with nodejs/node#65891 in https://ci.nodejs.org/job/node-test-commit-v8-linux/7380/nodes=benchmark-ubuntu2404-intel-64,v8test=v8test/, but it shows
Error: Command 'git -c color.ui=never checkout --quiet --end-of-options 25c29f04c9127e1ca09e6c1181f74850aa7f118b' returned non-zero exit status 1 in ./v8/third_party/libpfm4 error: pathspec '--end-of-options' did not match any file(s) known to gitWould that indicate an incompatible version of git? --end-of-options was added in https://gitlab.com/git-scm/git/-/blob/HEAD/Documentation/RelNotes/2.44.0.adoc
AFAICT use of
--end-of-optionshas come from https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/8401307. On our V8 CI this is causing the Ubuntu 24.04 x64 build to fail but the RHEL 8 ppc64le and s390x builds still work, which appear to be due to the version of git installed.From the CI I ran for nodejs/node#66070, which was before depot_tools adopted
--end-of-options,17:55:55 git update is recommended. 17:55:55 Installed git version is 2.43.0; 17:55:55 depot_tools recommends version 2.46.0 or later. 17:55:55 Disable this warning by setting the GCLIENT_SUPPRESS_GIT_VERSION_WARNING 17:55:55 environment variable to 1.- https://ci.nodejs.org/job/node-test-commit-v8-linux/7377/nodes=rhel8-s390x,v8test=v8test/consoleFull (git version is the same on RHEL 8 ppc64le)
17:58:00 git update is recommended. 17:58:00 Installed git version is 2.43.7; 17:58:00 depot_tools recommends version 2.46.0 or later. 17:58:00 Disable this warning by setting the GCLIENT_SUPPRESS_GIT_VERSION_WARNING 17:58:00 environment variable to 1.so the version of git on Ubuntu 24.04 is slightly older than the one in RHEL 8, but both are less than the depot_tools recommended version. We may need to track separately updating the machines we run the V8 CI on (both Ubuntu 26.04 and RHEL 9 have later versions of git than the recommended git 2.46.0), but for now I'm going to experiment with how we're checking out depot_tools in the job.
I think we can eliminate the checkout of depot_tools in
depot_tools(in the workspace root). My plan is to try:- Have the job clone into
deps/v8/depot_toolswhich is where the second clone (from the scripts intools/v8) would be. - Checkout the version of depot_tools according to:
a. Any override intools/v8/fetch_deps.py(see deps: V8: overridedepot_toolsversion node#62344). (Incidentally I think we can remove the override on main/v26.x now.)
b.deps/v8/DEPS
EnsureDepotTools()fromtools/v8/node_common.pywould find the existing gclient.py indeps/v8/depot_toolsand reuse that without doing a fresh clone.Added this to the job:
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git deps/v8/depot_tools # find the commit sha from any override, or from V8's DEPS. DEPOT_TOOLS_VERSION=$(awk '/depot_tools\.git/ { split($NF, arr, "@"); print arr[2] }' tools/v8/fetch_deps.py | tr -d [[:punct:]]) if [ -z "${DEPOT_TOOLS_VERSION}" ]; then DEPOT_TOOLS_VERSION=$(awk '/depot_tools\.git/ { print $NF }' deps/v8/DEPS | tr -d [[:punct:]]) fi git -C deps/v8/depot_tools checkout ${DEPOT_TOOLS_VERSION}Test runs:
a. Any override in
tools/v8/fetch_deps.py(see deps: V8: overridedepot_toolsversion node#62344). (Incidentally I think we can remove the override on main/v26.x now.)Opened nodejs/node#66110 to remove the override. While it's not directly affecting the problems in this issue, it might cause complications for future V8 updates if we're overriding (i.e. pinning) to older versions of depot_tools.
Created a CL to add an env variable for bypassing hermetic python: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/8434299
Once it land you can doDEPOT_TOOLS_BOOTSTRAP_PYTHON3=0 fetch v8.Reacted by Richard LauCl has now landed.
- added a commit that references this issue
on Sep 26, 2026 I've set
export DEPOT_TOOLS_BOOTSTRAP_PYTHON3=0in the V8 job.
I've tested in https://ci.nodejs.org/job/richardlau-node-test-commit-v8-linux/788/nodes=benchmark-ubuntu2404-intel-64,v8test=v8test/console without the change from #4462 (comment) (so we use the HEAD of depot_tools).
I have left the change from #4462 (comment) in node-test-commit-v8-linux (in addition to setting the environment variable) to avoid any other future surprises (at least until we test later versions of V8).
Consistently failing since https://ci.nodejs.org/job/node-test-commit-v8-linux/nodes=benchmark-ubuntu2404-intel-64,v8test=v8test/7350/console yesterday morning.
e.g. https://ci.nodejs.org/job/node-test-commit-v8-linux/nodes=benchmark-ubuntu2404-intel-64,v8test=v8test/7353/console