CVE scanning enhancement by stutiibm · Pull Request #8437 · ppc64le/build-scripts

stutiibm · 2026-07-19T19:27:20Z

Checklist

Have you checked and followed all the points mention in the CONTRIBUTING.MD
Have you validated script on UBI 9 container
Did you run the script(s) on fresh container with set -e option enabled and observe success ?
Did you have Legal approvals for patch files ?

Adds generalized_wheel_scanner.py — a single-script CVE scanner that runs inside the build container after auditwheel repair to detect vulnerabilities in bundled .so files that standard scanners cannot identify (metadata stripped, hash-renamed filenames). Scanner is invoked automatically from create_wheel_wrapper.sh after auditwheel repair completes. Failure is non-blocking — build continues regardless. Scanning strategy (three lanes + phase 1): Phase 1 — grype scan directly on extracted wheel directory. Results deduplicated against Lane 1 and Lane 2 before final report. Lane 1 — Walks METADATA/egg-info inside the wheel to find all Python runtime dependencies, then queries grype for CVEs. Lane 2 — Strips auditwheel hash from each bundled .so filename, matches it against a live RPM inventory (rpm -q --filesbypkg -a) to identify the source package, then queries grype for CVEs. Lane 3 — Parses the build script (git clone + git checkout lines) to extract source-built C/C++ library names and versions. Queries NVD CPE API to resolve a unique identifier, then NVD CVE API for findings. All .so files not claimed by Lane 2 are attributed to Lane 3 libs. Other details: - grype self-installs at runtime (3 retries, 10s delay); non-blocking if it fails - Shell variable substitution pre-pass handles $VAR / ${VAR:-default} in build scripts - Layered NVD CPE search: raw repo name first, stripped name as fallback - Language-binding CPE variants (-python, -java, etc.) filtered out - Output: single unified JSON report per wheel saved to disk for COS upload Fields in final_report: total_unique_cve — globally distinct CVE IDs across all lanes total_cve — per-library occurrence count cpe_resolved — true = unique NVD CPE found; false = flagged, no CVE query

…per.sh Adds the generalized_wheel_scanner.py invocation block to create_wheel_wrapper.sh. The scanner is called after auditwheel repair completes and before SHA generation, passing the repaired wheel and the original build script as arguments. The call is non-blocking — if the scanner fails or the script is not found, a warning is printed and the build continues normally. Scanner invocation: if [ -f "generalized_wheel_scanner.py" ]; then python generalized_wheel_scanner.py "${wheel_final}" "${BUILD_SCRIPT_PATH}" fi Arguments passed: $1 — repaired wheel file (manylinux-tagged .whl) $2 — original build script path (used by Lane 3 to parse source-built libs) Output: <wheel_stem>_cve_report.json written to the build working directory, picked up and uploaded to COS by the workflow upload step.

Adds an "Upload CVE report to COS" step to all five wheel build jobs (py310, py311, py312, py313, py314) in the build workflow. The step runs after the wheel artifact upload and pushes the <wheel_stem>_cve_report.json file produced by generalized_wheel_scanner.py to the same COS bucket and path as the existing build logs: ose-power-toolci-bucket-production/<PACKAGE_NAME>/<VERSION>/<wheel_stem>_cve_report.json Example for numpy 2.2.5 (py312): ose-power-toolci-bucket-production/numpy/2.2.5/ numpy-2.2.5-cp312-cp312-manylinux_2_34_ppc64le_cve_report.json Design decisions: - Reuses existing upload_file.sh — no new upload infrastructure needed - Non-blocking — missing report only prints a warning, build continues - One report per wheel per Python version, all landing under the same PACKAGE_NAME/VERSION prefix, no extra collector job required - No changes to create_wheel_wrapper.sh or any scanner script

Added CVE report renaming logic after wheel post-processing.

cdeepali · 2026-07-24T12:54:54Z

+        }
+
+    # Grype install URL
+    _GRYPE_INSTALL_URL = (


Add a step in workflow to install grype, instead of installing it in the python script?

Addressed. Removed _ensure_grype() method and _GRYPE_INSTALL_URL entirely from the Python script. Grype is now installed once in a dedicated install_scan_tools workflow job using wget with checksum verification, cached as a scan-tools-cache artifact, and downloaded by each wheel_build_pyXXX job. The Python script now simply checks shutil.which('grype') and skips grype-dependent lanes gracefully if not found.

cdeepali · 2026-07-24T13:57:01Z

+        try:
+            # Step 1: one call — all package names + versions
+            meta_result = subprocess.run(
+                ['rpm', '-qa', '--queryformat', '%{NAME}|%{VERSION}-%{RELEASE}\n'],


Can you try with, it will give package name, version and files all in one go.

rpm -qa --queryformat 'PKG:%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n[%{FILENAMES}\n]'

Addressed. Replaced the two-call approach with a single rpm -qa --queryformat 'PKG:%{NAME}|%{VERSION}-%{RELEASE}\n[%{FILENAMES}\n]' call as suggested. Lines starting with PKG: are parsed as package headers; subsequent lines are file paths belonging to that package.

cdeepali · 2026-07-24T14:38:40Z

+            print("  Detected RPM-based system")
+            inventory['packages'] = self._get_rpm_packages()
+        # Check if DEB-based system
+        elif shutil.which('dpkg'):


Is there a need to prepare inventory for DEB based system, considering that the build happens on RHEL container?

Addressed. Removed the elif shutil.which('dpkg') branch from _create_system_inventory() and deleted _get_dpkg_packages() entirely. The builds run on RHEL containers so only RPM-based inventory is needed.

cdeepali · 2026-07-24T14:39:54Z

+        try:
+            # Get all packages
+            result = subprocess.run(
+                ['dpkg-query', '-W', '-f=${Package}|${Version}\n'],


On Debian systems, dpkg stores the list of files for every installed package in plain text files under /var/lib/dpkg/info/.list. Reading these files directly in Python is better compared to executing the dpkg-query and then dpkg for each package.

Addressed as part of Comment 3 — _get_dpkg_packages() has been removed entirely since DEB inventory is not needed for RHEL-based builds.

cdeepali · 2026-07-24T14:49:56Z

+                                'basename': Path(file_path).name
+                            })
+                        except:
+                            pass


Can use except Exception: here instead of passing silently.

Addressed. The bare except: pass was inside _get_dpkg_packages() which has been removed entirely (Comment 3). All remaining exception handlers in the file already use except Exception as e:.

cdeepali · 2026-07-24T15:47:46Z

+            #   libgfortran-2758a8fd.so.5.0.0    -> libgfortran
+            #   libopenblasp-r0-a8f83a82.3.32.so -> libopenblasp-r0
+            #   libabsl_base-eb206faa.so.2401.0.0 -> libabsl_base
+            base_name = re.sub(r'-[a-f0-9]{8,}', '', bundled_name)  # Remove hex hash


Can this be replaced with single regex?

Addressed. Replaced three chained re.sub() calls with a single regex: re.sub(r'(-[a-f0-9]{8,})?(.\d[^/])?.so.$', '', bundled_name) which strips the auditwheel hash, version suffix, and .so extension in one pass.

cdeepali · 2026-07-24T15:55:57Z

+        site_dirs = []
+        try:
+            result = subprocess.run(
+                ['find', '/', '-name', 'site-packages', '-type', 'd',


Can the following be used instead?

import site dirs = site.getsitepackages() + [site.getusersitepackages()]

Addressed. Replaced the find / subprocess in _find_site_packages() with the suggested approach: site.getsitepackages() + [site.getusersitepackages()] using Python's built-in site module. Added import site at the top of the file.

cdeepali · 2026-07-24T16:05:18Z

+        queue = [wheel_pkg_name]
+
+        while queue:
+            pkg = queue.pop(0)


use deque instead.

Addressed. Added from collections import deque import, changed queue = [wheel_pkg_name] to queue: deque = deque([wheel_pkg_name]), and queue.pop(0) to queue.popleft() for O(1) dequeue instead of O(n) list shift.

cdeepali · 2026-07-24T16:05:31Z

+        queue = [wheel_pkg_name]
+
+        while queue:
+            pkg = queue.pop(0)


use deque instead.

Addressed. Added from collections import deque import, changed queue = [wheel_pkg_name] to queue: deque = deque([wheel_pkg_name]), and queue.pop(0) to queue.popleft() for O(1) dequeue instead of O(n) list shift.

cdeepali · 2026-07-24T16:09:10Z

+                if result.returncode == 0 and shutil.which('grype'):
+                    print("  grype installed successfully.")
+                    return True
+                print(f"  Attempt {attempt} failed: {result.stderr.strip()[:200]}")


Use constant instead of magic number 200.

Addressed. Added module-level constant MAX_STDERR_LOG_CHARS = 200 and replaced all [:200] slices throughout the file with [:MAX_STDERR_LOG_CHARS].

cdeepali · 2026-07-24T16:12:44Z

+            with urllib.request.urlopen(req, timeout=30) as resp:
+                return json.loads(resp.read())
+        except Exception as e:
+            print(f"  Warning: NVD API call failed ({url}): {e}")


Check that this does not print the Key here.

Addressed. Confirmed and documented: the API key is read from os.environ.get("NVD_API_KEY", "") and passed only via req.add_header("apiKey", api_key) as an HTTP request header. It is never interpolated into any print() or log statement. Added an explicit comment in the code confirming this.

Added install_scan_tools job to install grype and scancode-toolkit, and updated wheel build jobs to include scanning steps.

Removed grype installation logic and updated checks for availability. Adjusted package inventory creation to only check for RPM-based systems.

stutiibm added 7 commits July 20, 2026 00:37

Update create_wheel_wrapper.sh

f1750dc

Fix exit command formatting in create_wheel_wrapper.sh

272782a

Refactor CVE scanner path handling in script

a8059a7

Implement CVE report renaming after wheel processing

66b3262

Added CVE report renaming logic after wheel post-processing.

stutiibm marked this pull request as ready for review July 23, 2026 13:37

cdeepali reviewed Jul 24, 2026

View reviewed changes

stutiibm added 3 commits July 27, 2026 20:40

Enhance CI workflow with scanning tools installation

0d8e0eb

Added install_scan_tools job to install grype and scancode-toolkit, and updated wheel build jobs to include scanning steps.

Refactor grype handling and system inventory creation

658f12d

Removed grype installation logic and updated checks for availability. Adjusted package inventory creation to only check for RPM-based systems.

Enable shared library support for Python builds

02543bb

Uh oh!

Conversation

stutiibm commented Jul 19, 2026

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stutiibm Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

stutiibm Jul 27, 2026 •

edited

Loading