Repository navigation
Conversation
if async_wait() can race with complete() across threads, the yield_waiter's handler_state needs to be protected by a mutex. add an async_wait() overload for unique_lock that behaves like condition_variable::wait(): the lock is released immediately before suspending, and reacquired immediately before calling its completion handler Signed-off-by: Casey Bodley <cbodley@redhat.com> (cherry picked from commit 604fad5)
also adds an empty() function so it's easier to specify its precondition Signed-off-by: Casey Bodley <cbodley@redhat.com> (cherry picked from commit 165f2ec)
modeled after std::call_once() to guarantee that racing callers wait for the initial caller to finish. the main differences here are * support for coroutine callers to suspend instead of blocking while waiting for the initial caller, and * the wrapped function must return a value, which is cached and returned to all callers Signed-off-by: Casey Bodley <cbodley@redhat.com> (cherry picked from commit 3cd3c11)
A cache data structure for values that need to be retrieved form outside systems (e.g Key Management Systems). Features: - Thread safe, optimized for concurrent lookups and cache hits - Entry TTL expiration - Cache replacement strategy tuned to "web" workloads (SIEVE) - Performance Counters on hit, miss, expire, size, capacity, clears Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com> On-behalf-of: SAP marcel.lauhoff@sap.com (cherry picked from commit 9618827) Conflicts: src/test/common/test_web_cache.cc: - tentacle uses Formatter.h instead of JSONFormatter.h
Add Google benchmark [0] based micro benchmarks for Cache/LRU implementations in the Ceph code base. [0] https://github.com/google/benchmark Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com> On-behalf-of: SAP marcel.lauhoff@sap.com (cherry picked from commit 8354a97)
Benchmark: - Linux Kernel Key Retention Service (kernel keystore) [0] - memfd_secret(2) - plain memory Tests: - Random reads - (keystore) Write, Read, Remove [0] https://docs.kernel.org/security/keys/core.html Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com> On-behalf-of: SAP marcel.lauhoff@sap.com (cherry picked from commit a4b8d7d)
Add RAII wrapper around the Linux Key Retention Service add_key(2), keyctl_read(3), keyctl_invalidate(3) Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com> On-behalf-of: SAP marcel.lauhoff@sap.com (cherry picked from commit 0307119)
To allow RGW threads to share possession over process keyring keys the keyring must be created before a child thread adds keys. Since we only use the process keyring for KMS cache secrets, only initialize the keyring if it is enabled on startup. Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com> On-behalf-of: SAP marcel.lauhoff@sap.com (cherry picked from commit 0000467)
Add caching section to the RGW Encryption docs. Add cache settings to the RGW configuration reference. Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com> On-behalf-of: SAP marcel.lauhoff@sap.com (cherry picked from commit a88f029)
Goal: Support multiple backends and faking / mocking for testing. Add abstract classes Keyring (factory) and KeyringSecret. Add "Unsupported" implementation for non-Linux platforms. Add a get_best factory function that currently returns the LinuxKeyring impl on Linux or Unsupported elsewhere. Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com> On-behalf-of: SAP marcel.lauhoff@sap.com (cherry picked from commit 35f2bb4)
Add SSE Key Management System secrets cache to RGW. It is common to have secrets shared by many if not all objects in a bucket. Without RGW-side caching every PUT/GET will cause a request to an external KSM. This not only adds load to the KSM, but also slows down read and writes. Combine WebCache, ceph::async::call_once and LinuxKeyringSecret into KMSCache. WebCache stores async::once_result to wrap results of a KMS secret fetch to mitigate cache stampedes (concurrent cache requests to the same key coalesce into one). The retrieved secrets are stored in the Linux kernel key retention service (LinuxKeyringSecret) for safe keeping and retrial by subsequent requests. KMSCache adds a TTL reaper and life cycle. Cache values and error handling: The cache stores positive fetch results, permanent errors (e.g key does not exists) and transient errors (e.g fetch timeout). Each with a different TTL. Unit tests to cover cached / uncached KMS retrieve and runtime cache disable via config. Add perf counter `kms_fetch_lat` to track KMS fetch request latency and error counters to track permanent, transient and key store errors. Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com> Fixes: https://tracker.ceph.com/issues/68524 On-behalf-of: SAP marcel.lauhoff@sap.com (cherry picked from commit b6186fc) Conflicts: src/test/rgw/test_rgw_kms_cache.cc - tentacle does not have c826e15; use previous get_tavg_ns() return tuple order src/rgw/rgw_crypt.cc - 20.2.4 does not support SSE AES GCM yet
Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com> On-behalf-of: SAP marcel.lauhoff@sap.com (cherry picked from commit fb5bbc7)
KMS backends Barbican, Vault KV, and KMIP have a static key per key_id. However, with Vault Transit, each object has a unique DEK wrapped by the transit key. Keying th cache with key_id in Transit mode results in only the first DEK to be cached for all subsequent objects. Fix this by appending a hash of the wrapped DEK to the cache key. Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com> On-behalf-of: SAP marcel.lauhoff@sap.com (cherry picked from commit 4d7c556)
Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com> On-behalf-of: SAP marcel.lauhoff@sap.com (cherry picked from commit 78f2ec3)
Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com> On-behalf-of: SAP marcel.lauhoff@sap.com (cherry picked from commit 898cd5b)
sieve_expire_erase_unmutexed did not update the sieve hand passed as advertised. Make it return the updated hand and use that to update the global _sieve_hand in expire_erase Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com> On-behalf-of: SAP marcel.lauhoff@sap.com (cherry picked from commit 3518aaf)
Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com> On-behalf-of: SAP marcel.lauhoff@sap.com (cherry picked from commit f96f7a6)
Reset reaper state to monostate in the async and threaded case. Fixes a possible use after free in the async reaper case. Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com> On-behalf-of: SAP marcel.lauhoff@sap.com (cherry picked from commit e104372)
1. Don't delete the KMS cache before draining/joining the frontend coroutine threads. They may still depend on the KMS cache. 2. Stop the TTL reaper early to get it off the coroutine pool. Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com> On-behalf-of: SAP marcel.lauhoff@sap.com (cherry picked from commit 62ba03f)
The testing backend uses a 'keysel' attribute to derive a per object key from the KEK in the config. A single key_id with distinct keysel has different keys and need to be cached as such. Add the keysel to the cache key id to handle these collisions. Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com> On-behalf-of: SAP marcel.lauhoff@sap.com (cherry picked from commit 12de72b)
Signed-off-by: Marcel Lauhoff <marcel.lauhoff@clyso.com>
s0nea
force-pushed
the
backport/20.2.4/rgw-kms-cache
branch
from
November 5, 2026 08:26
1bb06da to
e226369
Compare
|
Hi @irq0 just a heads-up: I rebased your PR to add the fix for the unit tests. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes cobaltcore-dev/cloud-storage#562