Enforce FIPS-approved cryptography across hashing, TLS, tokens and the CLI

[beanuwave]

Description

Category: Enhancement, Bug fix

Completes FIPS support on top of the mode added in #6394. This is where the plugin actually becomes FIPS-capable.

Key changes

Keystores / TLS

• PemKeyReader rewritten onto BouncyCastle (PEMParser / JcaPEMKeyConverter / PKCS8 decryptor) instead of raw JCE; adds BCFKS and store-type auto-detection.
• SSLConfigConstants: both defaults become FIPS-conditional - default store type is forced to BCFKS in FIPS, and ALLOWED_SSL_PROTOCOLS drops TLSv1.1 in FIPS.
• The BC FIPS provider is declared, not instantiated. main self-registered it at plugin load (OpenSearchSecuritySSLPlugin.tryAddSecurityProvider() -> Security.addProvider(new BouncyCastleFipsProvider())); that method is removed. Providers now come solely from the active java.security file (JCA lazy-loads them), so FIPS vs non-FIPS is a launch-time provider swap (BCJSSE vs SunJSSE) with no code branch - the security files are a core/distribution concern. Consequence: BCFKS stores on a non-FIPS node (e.g. OBO tokens: load the encryption key from a keystore and harden the crypto #6397's OBO Scenario C) now need BCFIPS registered as a provider on the non-FIPS node; the jars ship with core, but only core's lazy PemUtils initializer registers it. The plugin's tests always ship their own java_test.security with BCFIPS, so they do not catch a gap here - the release depends on core carrying that change (see Core / distribution follow-ups).

Auth hardening (found mid-audit)

• HTTPSpnegoAuthenticator: no longer mutates global System.setProperty debug flags; stops logging the acceptor principal; proper LoginContext.logout() and decoded-header zeroing in finally.
• InternalAuthenticationBackend + PasswordHasher.getDummyHash(): the not-found timing path now uses the configured hasher, closing a user-enumeration side-channel under PBKDF2.
• Password-length floor. PBKDF2 keys are derived from the password itself, and BC FIPS rejects key material under 112 bits (< 14 ASCII chars) at hashing time. PasswordValidator.FIPS_MIN_PASSWORD_LENGTH (14) anchors both ends: FIPS raises an unset restapi.password_min_length to 14, and startup rejects a lower explicit value - otherwise the REST API accepts passwords the hasher then refuses. validateFipsMode collects every violation and reports them together, and additionally verifies the BC provider is genuinely in approved-only mode.
• Randomness from core. Randomness.createSecure() replaces new SecureRandom() for api-tokens and user passwords (OBO encryption already uses it since OBO tokens: load the encryption key from a keystore and harden the crypto #6397) - it resolves to the approved SP 800-90A^[4] DRBG in FIPS. UserService generates 20-27 chars in FIPS (>=119 bits over the 62-char alphabet), 8-15 otherwise, with char[] zeroed in finally.

CLI

• SecurityAdmin accepts BCFKS/PKCS11, with a PKCS#11 PIN prompt. buildPkcs11SslContext routes PKCS#11 keys through SunJSSE client-side, the same reasoning as the server TLS layer in Model key and trust store configurations as records and support PKCS#11 #6395.
• Launcher scripts source core's opensearch-env, so the CLI runs with the bundled JDK, the Java version check and - under OPENSEARCH_FIPS_MODE=true - the FIPS java.security and approved_only flag. They keep Copy scripts from tools to bin/ directory in assembly and make scripts agnostic to directory #6023's installation-independent home discovery and return to the caller's working directory, so documented relative -cd/-cacert paths keep resolving.

Test support

• The remaining 30 *FipsTests / *FipsIT variants that gradle/fips.gradle selects (the first two arrive with OBO tokens: load the encryption key from a keystore and harden the crypto #6397).
• FipsHashAdapter rewrites static BCrypt fixtures and their short demo passwords to PBKDF2, padded past the 14-char floor (a no-op outside FIPS).
• A few timing-sensitive integration tests scale down under FIPS, where PBKDF2 logins and BCTLS handshakes are markedly slower.
• HTTP/3 is refused in FIPS mode because the bundled BoringSSL is not built from the FIPS-validated branch - a build-level constraint, not a categorical ban. A FIPS-certified BoringSSL substituted at the OS level re-enables it.

Reviewer call-outs

1. securityadmin now runs through core's launcher environment - this targets the in-distribution path. The standalone bundle has not been self-launching since upstream Copy scripts from tools to bin/ directory in assembly and make scripts agnostic to directory #6023 removed the deps/-relative classpath, so this PR does not change that; it is also not wired into assemble (since Remove standalone admin tools zip file #1628, 2022).
2. SAML is scoped out. Both SAML stacks in use (OneLogin java-saml and OpenSAML/Shibboleth) are not FIPS-compliant, so those tests do not run in FIPS mode.
3. Operational: keystore passwords must be >=14 chars (BCFIPS), as must user passwords (the 112-bit PBKDF2 floor, now enforced).
4. The provider list is a complete override; SunJGSS is deliberately retained for Kerberos/SPNEGO.

Testing

./gradlew test integrationTest
OPENSEARCH_FIPS_MODE=true ./gradlew test integrationTest

In FIPS mode the build swaps in the FIPS java.security (BCFIPS-only providers), enables -Dorg.bouncycastle.fips.approved_only=true, and points the JVM at the BCFKS truststore. FIPS-incompatible baselines (BCrypt, Argon2, SAML, SSLv3, JKS/PKCS12, weak/short passwords) are replaced by their *FipsTests/*FipsIT variants, following core's fork-the-suite convention.

For a running cluster, select the FIPS-approved hasher in opensearch.yml:

plugins.security.password.hashing.algorithm: pbkdf2

The demo hashes in config/opensearch-security/internal_users.yml are BCrypt and will not verify under PBKDF2 - regenerate each test account's hash (e.g. with tools/hash.sh) before applying the security config.

The manual walkthroughs below are the FIPS runs of the non-FIPS walkthroughs in #6395, #6396 and #6397 - only the FIPS deltas are spelled out; shared steps reference those PRs. Run the node with OPENSEARCH_FIPS_MODE=true (the launcher loads fips_java.security -> BCJSSE).

Manual test: securityadmin.sh with BCFKS + PKCS#11 keystores (SoftHSM), FIPS

Extends #6395's node-side PKCS#11 test: the trust store becomes BCFKS, SunPKCS11 is registered in fips_java.security, and securityadmin authenticates with a PKCS#11 client key (new in this PR). All paths are relative to $OPENSEARCH_HOME.

# 1. Set up a test cluster.
cd $OPENSEARCH_HOME
sh bin/opensearch-keystore create --password
sh plugins/opensearch-security/tools/install_demo_configuration.sh -y -i -s

# 2. Init token - the --pin becomes the keystore/truststore password.
softhsm2-util --init-token \
  --free \
  --label opensearch \
  --so-pin 4321 \
  --pin 1234

# 3. Register the provider in config/fips_java.security:
      security.provider.<n>=SunPKCS11 /path/to/config/softhsm-pkcs11.cfg
#    with softhsm-pkcs11.cfg in the same dir:
      name = SoftHSM
      library = /usr/lib/softhsm/libsofthsm2.so
      slotListIndex = 0

# 4. Import node + admin keys WITH chains (PKCS#12 -> token).
#    Repeat the keytool step for kirk (swap -name / -srcalias / -destalias).
openssl pkcs12 -export \
  -inkey config/esnode-key.pem \
  -in config/esnode.pem \
  -certfile config/root-ca.pem \
  -name esnode-cert \
  -out /tmp/esnode.p12 \
  -passout pass:1234

jdk/bin/keytool \
  -importkeystore \
  -srckeystore /tmp/esnode.p12 \
  -srcstoretype PKCS12 \
  -srcstorepass 1234 \
  -srcalias esnode-cert \
  -destkeystore NONE \
  -deststoretype PKCS11 \
  -deststorepass 1234 \
  -destalias esnode-cert \
  -addprovider SunPKCS11 \
  -providerarg config/softhsm-pkcs11.cfg

# 5. Trust anchor -> BCFKS file.
jdk/bin/keytool -importcert -noprompt \
  -alias root-ca \
  -file config/root-ca.pem \
  -keystore config/root-ca.bcfks \
  -storetype BCFKS \
  -storepass changeit \
  -providerClass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider \
  -providerPath lib/bc-fips-2.1.2.jar

# 6. In opensearch.yml: comment the demo *.pem*_filepath lines, keep admin_dn: CN=kirk,...,
#    and add for both transport and http:

# --- Node identity from the PKCS#11 token ---
plugins.security.ssl.transport.keystore_type: PKCS11
plugins.security.ssl.transport.keystore_alias: esnode-cert
plugins.security.ssl.transport.keystore_password: "1234"      # SoftHSM PIN
plugins.security.ssl.http.keystore_type: PKCS11
plugins.security.ssl.http.keystore_alias: esnode-cert
plugins.security.ssl.http.keystore_password: "1234"

# --- Trust anchor from a BCFKS file ---
plugins.security.ssl.transport.truststore_type: BCFKS
plugins.security.ssl.transport.truststore_filepath: root-ca.bcfks
plugins.security.ssl.transport.truststore_password: "changeit"
plugins.security.ssl.http.truststore_type: BCFKS
plugins.security.ssl.http.truststore_filepath: root-ca.bcfks
plugins.security.ssl.http.truststore_password: "changeit"

# 7. Run the cluster and apply security config with the PKCS#11 admin key.
#    Pass = 'Connected as "CN=kirk,..."' followed by 'Done with success'.
OPENSEARCH_FIPS_MODE=true sh plugins/opensearch-security/tools/securityadmin.sh \
  -cd config/opensearch-security/ \
  -icl \
  -nhnv \
  -cacert config/root-ca.pem \
  -kst PKCS11 \
  -kspass 1234 \
  -ksalias kirk

Manual test: LDAP authentication over LDAPS, FIPS

Re-run #6396's LDAPS walkthrough (baseline, Scenario 1 hostname verification, Scenario 2 mTLS) for both backends with the node started under OPENSEARCH_FIPS_MODE=true. Config, certs and expected outcomes are identical; what differs:

	non-FIPS (#6396)	FIPS (this PR)
TLS provider	SunJSSE	BCJSSE
default enabled_ssl_protocols	[TLSv1.3, TLSv1.2, TLSv1.1]	[TLSv1.3, TLSv1.2]

FIPS agent-build gotchas (both are Core / distribution follow-ups): (1) core's base security.policy grants bc-fips/bcpkix-fips but not bctls-fips, so every LDAPS bind is denied under the agent; (2) the JUL->log4j bridge isn't active for BC FIPS JSSE at handshake time, so each bind's INFO traces leak to stderr as [WARN][stderr] spam (noise, not a failure).

Provider-specific log lines under BCJSSE (the ldaptive / plugin lines are unchanged):

1c  CertificateException: No subject alternative name found matching domain name ldap-wrong.example.com  # layer 2
        at org.bouncycastle.jsse.provider.ProvX509TrustManager.checkEndpointID(...)
2b  received fatal(2) certificate_required(116) alert       # client cert withheld
2c  CertPathBuilderException: No issuer certificate for certificate in certification path found.
    TlsFatalAlert: certificate_unknown(46)  ->  Authentication finally failed

Additional FIPS check (protocol floor, default ldap backend with trace logging on): the node logs enabled ssl/tls protocols for ldaps [TLSv1.3, TLSv1.2], where the non-FIPS run in #6396 logs [TLSv1.3, TLSv1.2, TLSv1.1]. Only the default narrows - an explicit enabled_ssl_protocols is passed through unfiltered.

Manual test: OnBehalfOf (OBO) token, FIPS

Re-run #6397's OBO walkthrough with the node under OPENSEARCH_FIPS_MODE=true. Deltas:

• Scenario A (inline keys) - works; encryption_key must now decode to >= 32 bytes (the FIPS IKM floor).
• Scenario B (PKCS#12) - rejected in FIPS mode (no FIPS-approved PKCS#12 in BC FIPS); OBO is declined and the misconfiguration is logged once.
• Scenario C (BCFKS) - works; the FIPS keystore for secret keys.

FIPS-only negative check: Scenario A with a too-short encryption_key (e.g. ZW5jcnlwdGlvbktleQ==, 13 bytes) is declined with Configured encryption_key decodes to 13 bytes of key material, but FIPS mode requires at least 32 bytes. - the same key #6397 shows is accepted in non-FIPS.

Manual test: OnBehalfOf (OBO) tokens across a rolling upgrade (3.8 -> 3.9), FIPS

Re-run #6397's rolling-upgrade walkthrough (phases 0-3, same stop / start / verify commands) on a cluster that runs on BC FIPS from the start, 3.8 nodes included - BC FIPS permits the ECB primitive, so a pre-3.9 FIPS cluster did issue AES/ECB tokens. Both cores switch through OPENSEARCH_FIPS_MODE=true alone (bin/opensearch-env loads config/fips_java.security and sets -Dorg.bouncycastle.fips.approved_only=true).

Setup deltas: start from a fresh cluster - the non-FIPS security index holds BCrypt hashes, and 3.9 refuses FIPS mode unless the hasher is PBKDF2. Before the first FIPS start, on every node: plugins.security.password.hashing.algorithm: pbkdf2 in opensearch.yml and a PBKDF2 admin hash in internal_users.yml; a >= 14-char password on the node keystore, set with opensearch-keystore passwd outside FIPS (with an empty password BC FIPS cannot open it: password must be at least 112 bits); and a JVM-wide BCFKS trust store via opensearch-fips-demo-installer generated -n -p <password> (core refuses to start without -Djavax.net.ssl.trustStoreType=BCFKS). Then recreate role and obouser.

Every start of #6397's walkthrough then runs with both variables exported:

export OPENSEARCH_FIPS_MODE=true
export KEYSTORE_PASSWORD='<node-keystore-password>'

# after the first start, on every node:
grep -c 'running in FIPS-140-3 mode' $BASE/state/node{1,2,3}/logs/obo-rolling.log

What differs from #6397:

Direction	Expected	Why
3.8 issues, 3.8 verifies (phase 0)	200 + role; format: legacy	pre-3.9 behaviour on BC FIPS
3.8 issues, 3.9 verifies	200 + role, and a WARN instead of the INFO, once per 3.9 node	such tokens exist, so they are read during the window
3.9 issues, 3.8 verifies	401 - phases 1 and 2 therefore show 7/9	under FIPS the legacy format is never written, and 3.8 cannot read AES-GCM
3.9 issues, 3.9 verifies	200 + role; format check on a 3.9 node: AES-GCM from phase 1 on	FIPS overrides the write gate, whatever the node attributes say
tail token (3.8 node 3) after phase 3	200 + role on every node	same read window as without FIPS

1+  (3.9 node log, once)  WARN  Decrypted an on-behalf-of token in the legacy AES/ECB format, which is not an approved
                                mode for confidentiality. Such tokens were issued before the upgrade and are accepted
                                only until every node is upgraded and the tokens predating the upgrade have expired.
                                Nothing is written in that format.
401 (3.8 node log)              No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'

The 401 is not a role problem: 3.8's decrypt() throws, its authenticator logs that only at DEBUG (Invalid or expired JWT token.) and returns no credentials, so the request falls through to HTTP Basic. That is the one-directional gap of #6397's call-out - for the length of the upgrade every token a 3.9 node issues is rejected by every 3.8 node. Either route token issuance to not-yet-upgraded nodes for that period, or accept 401 until the client lands on an upgraded node or asks for a new token.

Check List

• New functionality includes testing
• New functionality has been documented
• New Roles/Permissions have a corresponding security dashboards plugin PR
• API changes companion pull request created
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit ff7a6f9.

⛔ Hard block: Issues at High severity or above will block this PR from merging.

'Diff too large, requires skip by maintainers after manual review'

---

Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.

---

⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.