Skip to content

Revived #2415: [CI] add CodeQL query to enforce OpenSSL return code handling (#1867) - #2634

Open
owaldron wants to merge 12 commits into
open-quantum-safe:mainfrom
owaldron:feat/revived-codeql-openssl-return-check
Open

owaldron wants to merge 12 commits into
open-quantum-safe:mainfrom
owaldron:feat/revived-codeql-openssl-return-check

Conversation

@owaldron

@owaldron owaldron commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

This PR revives #2415 by addressing the main concerns cited at its closure. This PR adds a codeql.yml workflow to enable custom CodeQL queries. This new workflow runs in parallel to GitHub's default OpenQL queries (which is enabled by repo code scan settings). The openssl-return-check query is also added to verify OpenSSL status codes are being checked, resolving #1867. This required adding missing OQS_OPENSSL_GUARD where appropriate. Other issues like #1868 can use the provided codeql.yml framework to implement solutions too.

Inspired by this comment by @baentsch, instead of uploading an artifact, this workflow will print errors to the job log, annotate the PR, and fail when errors were encountered. A demo of what this looks like can be found here: (demo PR, failing check, inline annotation).

Additional enhancements to the codeql.yml workflow include:

  • Runs only liboqs's custom queries, since the standard suite already runs via default setup.
  • Add CodeQL unit tests for each query which run in a seperate job.
  • Analyse a full default build with OpenSSL used for AES, SHA2 and SHA3.
  • Flag only EVP_* calls whose return value is discarded, since there are other safe ways beyon OQS_OPENSSL_GUARD to check this (eg. if (EVP_DigestInit_ex(...) != 1) {...}).
  • In pr.yml, depend on check-push-result rather than basic-checks and drop security-events: write, so the job can be run in forks.

  • Does this PR change the input/output behaviour of a cryptographic algorithm (i.e., does it change known answer test values)? (If so, a version bump will be required from x.y.z to x.(y+1).0.)
  • Does this PR change the list of algorithms available -- either adding, removing, or renaming? Does this PR otherwise change an API? (If so, PRs in fully supported downstream projects dependent on these, i.e., oqs-provider will also need to be ready for review and merge by the time this is merged. Also, make sure to update the list of algorithms in the continuous benchmarking files: .github/workflows/kem-bench.yml and sig-bench.yml)

Generative AI Disclosure: The original PR had no disclosed use of generative AI. Generative AI was used to draft code and write tests for the ammendments. All generated code was reviewed and modified when appropriate before inclusion. Generative AI was also used to generate the demo PRs.

Vishnu2707 and others added 12 commits October 16, 2026 12:42
Adds a custom CodeQL query (.github/codeql/openssl-return-check.ql)
that detects calls to OpenSSL EVP_* functions whose return value is
not guarded by the OQS_OPENSSL_GUARD macro. Integrates the query into
a new CodeQL workflow (.github/workflows/codeql.yml) that runs on
every PR and push to main.

The query was originally written by Trail of Bits during their audit
of liboqs and reported in issue open-quantum-safe#1867.

Closes open-quantum-safe#1867
Signed-off-by: Vishnu Ajith <27vishnu07@gmail.com>
Addresses Scorecard Token-Permissions finding: security-events write
permission scoped to codeql job only, not workflow level.
Signed-off-by: Vishnu Ajith <27vishnu07@gmail.com>
Replace inline queries: path with a codeql-config.yml config file.
The CodeQL init action requires local queries to be referenced via
a config file rather than a direct relative path in the queries field.
Signed-off-by: Vishnu Ajith <27vishnu07@gmail.com>
The CodeQL cpp extractor pre-finalize script crashes with exit code 134
when the trapCaches/cpp/tarballs directory does not exist on the runner.
Disabling TRAP caching avoids the crash.
Signed-off-by: Vishnu Ajith <27vishnu07@gmail.com>
Replace config-file approach with inline queries field combining
security-and-quality suite with the custom EVP return check query,
as suggested by @baentsch. Remove codeql-config.yml.
Signed-off-by: Vishnu Ajith <27vishnu07@gmail.com>
Signed-off-by: Vishnu Ajith <27vishnu07@gmail.com>
Adding upload: false to the analyze step prevents the SARIF conflict
with GitHub's default CodeQL setup which is enabled on this repo.
The query still runs and catches issues but does not upload results.
Signed-off-by: Vishnu Ajith <27vishnu07@gmail.com>
Add bad example at .github/codeql/test/openssl-return-check-bad.c
to confirm the query catches unchecked EVP_* calls. Document
verification steps in CONTRIBUTING.md.

Signed-off-by: Vishnu Ajith <27vishnu07@gmail.com>
Signed-off-by: Vishnu Ajith <27vishnu07@gmail.com>
Address the review concerns from open-quantum-safe#2415:
- Do not upload results to code scanning, which rejects CodeQL results
  from other workflows while default setup is enabled. Instead, print
  each result as an error annotation and fail the job if there are any
- Run only liboqs's own queries, since the standard suite already runs
  via default setup.
- Add CodeQL unit tests for each query which run in a seperate job
- Analyse a full default build with OpenSSL used for AES, SHA2 and SHA3
- Flag only EVP_* calls whose return value is discarded, since there are
  other safe ways beyond OQS_OPENSSL_GUARD to check this.
- In pr.yml, depend on check-push-result rather than basic-checks and
  drop `security-events: write`, so the job can be run in forks.

The workflow currently reports 119 unchecked EVP_* calls in src/common.
These will be fixed separately.

Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Owen Waldron <owen.waldron@uwaterloo.ca>
Assited-by: Claude:claude-opus-5.5
Signed-off-by: Owen Waldron <owen.waldron@uwaterloo.ca>
These were detected using the codeql workflow.

Signed-off-by: Owen Waldron <owen.waldron@uwaterloo.ca>
@coveralls

Copy link
Copy Markdown

Coverage Status

coverage: 80.307% (-4.2%) from 84.466% — owaldron:feat/revived-codeql-openssl-return-check into open-quantum-safe:main

@baentsch baentsch left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm&claude. Previously un-protected EVP calls now protected & test added to check the logic. Good to merge, I'd say. Thanks for the improvement and implementing the suggestion @owaldron !

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants