diff --git a/doc/radosgw/config-ref.rst b/doc/radosgw/config-ref.rst index 756af32858b1..e21be19db1c6 100644 --- a/doc/radosgw/config-ref.rst +++ b/doc/radosgw/config-ref.rst @@ -269,6 +269,16 @@ SSE-S3 Settings .. confval:: rgw_crypt_sse_s3_vault_ssl_clientcert .. confval:: rgw_crypt_sse_s3_vault_ssl_clientkey +KMS Secrets Cache Settings +========================== + +.. confval:: rgw_crypt_s3_kms_cache_enabled +.. confval:: rgw_crypt_s3_kms_cache_max_size +.. confval:: rgw_crypt_s3_kms_cache_positive_ttl +.. confval:: rgw_crypt_s3_kms_cache_transient_error_ttl +.. confval:: rgw_crypt_s3_kms_cache_negative_ttl + +.. _Encryption: ../encryption QoS settings ------------ diff --git a/doc/radosgw/encryption.rst b/doc/radosgw/encryption.rst index ef7aa9471a26..9b214cb34a51 100644 --- a/doc/radosgw/encryption.rst +++ b/doc/radosgw/encryption.rst @@ -81,7 +81,53 @@ The configuration expects a base64-encoded 256 bit key. For example:: file is not a secure method for storing encryption keys. Keys that are accidentally exposed in this way should be considered compromised. - +Caching +======= + +The caching feature for Key Management Service (KMS) secrets greatly +improves the performance of server-side encryption and lessens the +load on the KMS. + +Secrets are stored using the `Linux Kernel Key Retention Service`_ in +the RGW processes' process keyring. This is subject to a global quota +and must be set in accordance with the configured cache size. +Depending on whether RGW runs as root, these quotas can be managed by adjusting: +- ``/proc/sys/kernel/keys/root_maxkeys`` and ``/proc/sys/kernel/keys/root_maxbytes`` +- ``/proc/sys/kernel/keys/maxkeys`` and ``/proc/sys/kernel/keys/maxbytes`` + +Exceeding a quota will disable the cache, fail the request with an +internal error, and log a failure message. + +Three different Cache Time-to-Live (TTL) values can be set: +- **Positive TTL**: How long a successfully retrieved secret remains + in the cache. +- **Negative TTL**: How long to remember that a key does not exist, + preventing unnecessary requests to the KMS. +- **Transient Error TTL**: How long to cache failures due to temporary + issues like KMS timeouts. + +Metrics +--------- + +The cache exports metrics under the ``kms-cache`` collection. +- ``hit``: Hit counter +- ``miss``: Miss counter +- ``expired``: Number of TTL expired entries +- ``size``: Current cache size +- ``capacity``: Cache maximum size +- ``clear``: Number of cache clears. Resets ``size``, ``hit``, + ``miss``, ``expired`` + +In addition the ``rgw`` collection has: +- ``kms_fetch_lat``: Average KMS fetch latency. Also includes a + successful request counter. Each event results in a positive cache + entry. +- ``kms_error_transient``: Transient KMS fetch error counter. Each + event results in a transient error cache entry. +- ``kms_error_permanent``: Permanent KMS fetch error counter. Each + event results in a negative cache cache entry. + +.. _Linux Kernel Key Retention Service: https://www.kernel.org/doc/html/latest/security/keys/core.html .. _Amazon SSE-C: https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html .. _Amazon SSE-KMS: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html .. _Amazon SSE-S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingServerSideEncryption.html diff --git a/release-management/backports/01a1799e-f898-75cc-9444-e275adb11100.md b/release-management/backports/01a1799e-f898-75cc-9444-e275adb11100.md new file mode 100644 index 000000000000..da7c66c4ab0e --- /dev/null +++ b/release-management/backports/01a1799e-f898-75cc-9444-e275adb11100.md @@ -0,0 +1,95 @@ +--- +id: 01a1799e-f898-75cc-9444-e275adb11100 +title: KMS Cache +component: rgw +category: feature +provenance: + type: upstream_pr + upstream_prs: + - https://github.com/ceph/ceph/pull/61256 + upstream_pr_state: merged-main + upstream_commits: + - 12de72b3f34f53357e5a918b55f4a5d20c200761 + upstream_trackers: + - https://tracker.ceph.com/issues/68524 +justification: + type: customer + refs: + - cobaltcore-dev/cloud-storage#562 + - cobaltcore-dev/cloud-storage#125 +risk: + blast: data-loss + conflict: substantive + coverage: partial +--- + +## Justification (public) + +Adds a secure in-memory cache for SSE-KMS encryption keys retrieved +from Barbican (or other KMS backends), eliminating redundant +round-trips on repeated access to the same encrypted objects. Uses a +new SIEVE-based eviction cache (WebCache) and stores decrypted secrets +in the Linux kernel keyring. Please refer to the Ceph documentation on +how to configure this feature. + +## Justification (internal) + +Customer roadmap feature. Upstream PR is merged into main. + +## Risk notes + +- The cache keeps each key for a TTL (time to live, how long an entry + stays valid). If an operator deletes or rotates a key in Barbican, + radosgw uses the cached key until the TTL ends. +- The new async function `async::call_once` and the WebCache are new + library code. This release is the first to ship them. +- For production workloads, you must tune the cache size and the + system keyring quota. Refer to the Ceph documentation. + +### Known bugs + +The two known bugs have low risk. Neither bug changes stored data or +the result of a request that succeeds. Upstream PR +https://github.com/ceph/ceph/pull/73032 fixes both bugs. The PR is +under review, and this backport does not include it. + +#### Slow shutdown (https://tracker.ceph.com/issues/81968) + +radosgw shutdown can hang for up to 2 minutes. The TTL reaper is a +background task that removes expired cache entries. At shutdown, +radosgw sends a stop signal to the reaper. If the signal arrives +between the end of the reaper timer and the next reaper step, the +reaper does not see it. That window is very short. By default, the +timer ends once every 10 seconds (the smallest cache TTL). + +If the reaper misses the stop signal, the `rgw_exit_timeout_secs` alarm +(default 2 minutes) or the systemd stop timeout ends the process. The +result is a rare slow restart. The valgrind leak report in the tracker +comes from the stuck reaper at exit. Memory does not grow while +radosgw runs. + +#### No cache in a daemonized radosgw (https://tracker.ceph.com/issues/82031) + +The KMS cache does not work in a daemonized radosgw. A daemonized +radosgw starts without `-f` and moves to the background with `fork()`. +This mode is uncommon. Standard deployment tools (e.g. Rook, cephadm) +start radosgw in the foreground. + +If a thread reads a secret that a different thread cached, the read +fails. That request returns `ERR_INTERNAL_ERROR`, and the cache turns +itself off. After that, radosgw does one KMS lookup per request, as +before this feature. + +To prevent this bug, start radosgw with `-f`. If you cannot use `-f`, +set `rgw_crypt_s3_kms_cache_enabled = false`. + +## Behavior change + +Repeated SSE-KMS access no longer hits the KMS backend on every +request; decrypted secrets are cached in-memory and in the Linux +kernel keyring. KMS-cache is ON out of the box. + +## Upgrade notes + +Feature flag: rgw_crypt_s3_kms_cache_enabled. Set false to disable +caching entirely and revert to per-request KMS lookups. diff --git a/src/common/CMakeLists.txt b/src/common/CMakeLists.txt index ae34d330813f..64483d0e6ffa 100644 --- a/src/common/CMakeLists.txt +++ b/src/common/CMakeLists.txt @@ -255,4 +255,9 @@ if(HAVE_KEYUTILS) set(parse_secret_srcs secret.c) add_library(parse_secret_objs OBJECT ${parse_secret_srcs}) + + set(keyring_srcs + keyring.cc) + add_library(keyring STATIC ${keyring_srcs}) + target_link_libraries(keyring keyutils::keyutils) endif() diff --git a/src/common/async/call_once.h b/src/common/async/call_once.h new file mode 100644 index 000000000000..8c48a2c3c6a8 --- /dev/null +++ b/src/common/async/call_once.h @@ -0,0 +1,113 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab ft=cpp + +/* + * Ceph - scalable distributed file system + * + * Copyright contributors to the Ceph project + * + * This is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License version 2.1, as published by the Free Software + * Foundation. See file COPYING. + * + */ + +#pragma once + +#include +#include +#include +#include + +#include "include/function2.hpp" + +#include "yield_context.h" +#include "detail/call_once.h" + +namespace ceph::async { + +/// Stores the result of call_once(), if any. +/// +/// The cached Result type must be semiregular (copyable and default- +/// constructible). +template +class once_result { + public: + /// Call f() once and cache its return value. + /// + /// If a call has already completed, return the cached result. If a call + /// is already in progress, wait for its completion and return the result. + /// If a yield context is provided in y, its coroutine will be suspended + /// while waiting. f() may also suspend/resume the same coroutine. + /// + /// f() must return a Result when called with no arguments. If it throws, + /// that exception is rethrown by all calls. + /// + /// This function is thread-safe. + template + friend Result call_once(once_result& self, optional_yield y, Callable&& f) + requires std::convertible_to, Result> + { + auto lock = std::unique_lock{self.mutex}; + return std::visit( + fu2::overload( + [&] (const Init&) { return self.do_init(lock, f); }, + [&] (Wait& w) { return self.do_wait(lock, y, w); }, + [] (const Complete& c) { return do_complete(c); }), + self.state); + } + + private: + std::mutex mutex; + + // state machine: [Init] -> [Wait] -> [Complete] + using Init = std::monostate; + using Wait = detail::call_once::WaitState; + struct Complete { + Result result; + std::exception_ptr eptr; + }; + std::variant state; + + Result do_init(std::unique_lock& lock, auto&& f) + { + // transition to Wait state + auto& wait = state.template emplace(); + lock.unlock(); + + Complete complete; + try { + complete.result = f(); + } catch (const std::exception&) { + complete.eptr = std::current_exception(); + } + + lock.lock(); + // wake any waiters + wait.wake_all(); + + // transition to Complete state and return the result + return do_complete(state.template emplace(std::move(complete))); + } + + Result do_wait(std::unique_lock& lock, + optional_yield y, Wait& wait) + { + // wait for the response to an outstanding request + wait.wait(lock, y); + + // state must be Complete after wakeup + return do_complete(std::get(state)); + } + + static Result do_complete(const Complete& complete) + { + if (complete.eptr) { // rethrow cached exception + std::rethrow_exception(complete.eptr); + } + return complete.result; // return cached result + } +}; + +} // namespace ceph::async diff --git a/src/common/async/detail/call_once.h b/src/common/async/detail/call_once.h new file mode 100644 index 000000000000..c7d676457934 --- /dev/null +++ b/src/common/async/detail/call_once.h @@ -0,0 +1,129 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab ft=cpp + +/* + * Ceph - scalable distributed file system + * + * Copyright contributors to the Ceph project + * + * This is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License version 2.1, as published by the Free Software + * Foundation. See file COPYING. + * + */ + +#pragma once + +#include +#include + +#include + +#include "common/async/service.h" +#include "common/async/yield_context.h" +#include "common/async/yield_waiter.h" + +namespace ceph::async::detail::call_once { + +class AsyncWaiter : public boost::intrusive::list_base_hook<> { + yield_waiter impl; + public: + void wait(std::unique_lock& lock, + boost::asio::yield_context yield) + { + impl.async_wait(lock, yield); + } + void wake() + { + // async_wait() may complete inline via boost::asio::dispatch(), but would + // immediately try to reacquire the lock we're already holding. defer the + // completion until after we've returned and dropped our lock + boost::asio::defer(impl.get_executor(), [this] { + impl.complete(boost::system::error_code{}); + }); + } + void destroy() + { + impl.shutdown(); + } +}; + +class SyncWaiter : public boost::intrusive::list_base_hook<> { + std::condition_variable cond; + bool done = false; + public: + void wait(std::unique_lock& lock) + { + cond.wait(lock, [this] { return done; }); + } + void wake() + { + done = true; + cond.notify_one(); + } +}; + +struct WaitState : service_list_base_hook { + service* svc = nullptr; + boost::intrusive::list sync_waiters; + boost::intrusive::list async_waiters; + + ~WaitState() + { + if (svc) { + svc->remove(*this); + } + } + + void wait(std::unique_lock& lock, optional_yield y) + { + if (y) { + auto yield = y.get_yield_context(); + if (!svc) { + // on first async wait, register for service_shutdown() notifications + svc = &boost::asio::use_service>( + boost::asio::query(yield.get_executor(), + boost::asio::execution::context)); + svc->add(*this); + } + + AsyncWaiter waiter; + async_waiters.push_back(waiter); + + waiter.wait(lock, yield); + } else { + SyncWaiter waiter; + sync_waiters.push_back(waiter); + + waiter.wait(lock); + } + } + + void wake_all() + { + while (!sync_waiters.empty()) { + auto& waiter = sync_waiters.front(); + sync_waiters.pop_front(); + waiter.wake(); + } + while (!async_waiters.empty()) { + auto& waiter = async_waiters.front(); + async_waiters.pop_front(); + waiter.wake(); + } + } + + // on shutdown, destroy async waiters that didn't complete. this breaks any + // ownership cycles between once_result and the completion handlers + void service_shutdown() + { + while (!async_waiters.empty()) { + auto& waiter = async_waiters.front(); + async_waiters.pop_front(); + waiter.destroy(); + } + } +}; + +} // namespace ceph::async::detail::call_once diff --git a/src/common/async/yield_waiter.h b/src/common/async/yield_waiter.h index 9c14d9bafe4b..667dbbbb2521 100644 --- a/src/common/async/yield_waiter.h +++ b/src/common/async/yield_waiter.h @@ -16,15 +16,40 @@ #pragma once #include +#include #include #include #include +#include #include #include #include namespace ceph::async { +namespace detail { + +// handler wrapper that reacquires a lock immediately before completion +template +struct lock_handler { + Handler handler; + BasicLockable* lock = nullptr; + + template + void operator()(Args&& ...args) { + if (lock) { + lock->lock(); + } + std::move(handler)(std::forward(args)...); + } +}; + +// deduction guide required by windows? +template +lock_handler(Handler&&, BasicLockable*) -> lock_handler; + +} // namespace detail + /// Captures a yield_context handler for deferred completion or cancellation. template class yield_waiter { @@ -41,6 +66,18 @@ class yield_waiter { /// Returns true if there's a handler awaiting completion. operator bool() const { return state.has_value(); } + /// Returns false if there's a handler awaiting completion. + bool empty() const { return !state.has_value(); } + + /// Returns the captured completion handler's executor. + /// Precondition: !empty() + using executor_type = boost::asio::any_io_executor; + executor_type get_executor() const + { + return boost::asio::get_associated_executor( + state->handler, state->work.get_executor()); + } + /// Suspends the given yield_context until the captured handler is invoked /// via complete() or cancel(). template @@ -52,7 +89,25 @@ class yield_waiter { if (slot.is_connected()) { slot.template emplace(this); } - state.emplace(std::move(h)); + constexpr std::unique_lock* lock = nullptr; + state.emplace(std::move(h), lock); + }, token); + } + + /// Suspends the given yield_context until the captured handler is invoked + /// via complete() or cancel(). The given lock is released immediately before + /// it suspends and reacquired immediately after it resumes. + template + auto async_wait(std::unique_lock& lock, CompletionToken&& token) + { + return boost::asio::async_initiate( + [this, &lock] (handler_type h) { + auto slot = get_associated_cancellation_slot(h); + if (slot.is_connected()) { + slot.template emplace(this); + } + state.emplace(std::move(h), &lock); + lock.unlock(); // unlock before suspend }, token); } @@ -61,8 +116,10 @@ class yield_waiter { { auto s = std::move(*state); state.reset(); - auto h = boost::asio::append(std::move(s.handler), ec, std::move(value)); - boost::asio::dispatch(std::move(h)); + boost::asio::dispatch( + boost::asio::append( + detail::lock_handler{std::move(s.handler), s.lock}, + ec, std::move(value))); } /// Destroy the completion handler. @@ -80,10 +137,10 @@ class yield_waiter { struct handler_state { handler_type handler; work_guard work; + std::unique_lock* lock = nullptr; - explicit handler_state(handler_type&& h) - : handler(std::move(h)), - work(make_work_guard(handler)) + handler_state(handler_type&& h, std::unique_lock* lock) + : handler(std::move(h)), work(make_work_guard(handler)), lock(lock) {} }; std::optional state; @@ -123,6 +180,18 @@ class yield_waiter { /// Returns true if there's a handler awaiting completion. operator bool() const { return state.has_value(); } + /// Returns false if there's a handler awaiting completion. + bool empty() const { return !state.has_value(); } + + /// Returns the captured completion handler's executor. + /// Precondition: !empty() + using executor_type = boost::asio::any_io_executor; + executor_type get_executor() const + { + return boost::asio::get_associated_executor( + state->handler, state->work.get_executor()); + } + /// Suspends the given yield_context until the captured handler is invoked /// via complete() or cancel(). template @@ -134,7 +203,25 @@ class yield_waiter { if (slot.is_connected()) { slot.template emplace(this); } - state.emplace(std::move(h)); + constexpr std::unique_lock* lock = nullptr; + state.emplace(std::move(h), lock); + }, token); + } + + /// Suspends the given yield_context until the captured handler is invoked + /// via complete() or cancel(). The given lock is released immediately before + /// it suspends and reacquired immediately after it resumes. + template + auto async_wait(std::unique_lock& lock, CompletionToken&& token) + { + return boost::asio::async_initiate( + [this, &lock] (handler_type h) { + auto slot = get_associated_cancellation_slot(h); + if (slot.is_connected()) { + slot.template emplace(this); + } + state.emplace(std::move(h), &lock); + lock.unlock(); // unlock before suspend }, token); } @@ -143,7 +230,9 @@ class yield_waiter { { auto s = std::move(*state); state.reset(); - boost::asio::dispatch(boost::asio::append(std::move(s.handler), ec)); + boost::asio::dispatch( + boost::asio::append( + detail::lock_handler{std::move(s.handler), s.lock}, ec)); } /// Destroy the completion handler. @@ -161,10 +250,10 @@ class yield_waiter { struct handler_state { handler_type handler; work_guard work; + std::unique_lock* lock = nullptr; - explicit handler_state(handler_type&& h) - : handler(std::move(h)), - work(make_work_guard(handler)) + handler_state(handler_type&& h, std::unique_lock* lock) + : handler(std::move(h)), work(make_work_guard(handler)), lock(lock) {} }; std::optional state; @@ -189,3 +278,23 @@ class yield_waiter { }; } // namespace ceph::async + +namespace boost::asio { + +// forward the handler's associated executor, allocator, cancellation slot, etc +template