tentacle: kms cache

doc/radosgw/config-ref.rst

@@ -269,6 +269,16 @@ SSE-S3 Settings
 .. confval:: rgw_crypt_sse_s3_vault_ssl_clientcert
 .. confval:: rgw_crypt_sse_s3_vault_ssl_clientkey
 
+KMS Secrets Cache Settings
+==========================
+
+.. confval:: rgw_crypt_s3_kms_cache_enabled
+.. confval:: rgw_crypt_s3_kms_cache_max_size
+.. confval:: rgw_crypt_s3_kms_cache_positive_ttl
+.. confval:: rgw_crypt_s3_kms_cache_transient_error_ttl
+.. confval:: rgw_crypt_s3_kms_cache_negative_ttl
+
+.. _Encryption: ../encryption
 
 QoS settings
 ------------

doc/radosgw/encryption.rst

@@ -81,7 +81,53 @@ The configuration expects a base64-encoded 256 bit key. For example::
    file is not a secure method for storing encryption keys. Keys that are
    accidentally exposed in this way should be considered compromised.
 
-
+Caching
+=======
+
+The caching feature for Key Management Service (KMS) secrets greatly
+improves the performance of server-side encryption and lessens the
+load on the KMS.
+
+Secrets are stored using the `Linux Kernel Key Retention Service`_ in
+the RGW processes' process keyring. This is subject to a global quota
+and must be set in accordance with the configured cache size.
+Depending on whether RGW runs as root, these quotas can be managed by adjusting:
+- ``/proc/sys/kernel/keys/root_maxkeys`` and ``/proc/sys/kernel/keys/root_maxbytes``
+- ``/proc/sys/kernel/keys/maxkeys`` and ``/proc/sys/kernel/keys/maxbytes``
+
+Exceeding a quota will disable the cache, fail the request with an
+internal error, and log a failure message.
+
+Three different Cache Time-to-Live (TTL) values can be set:
+- **Positive TTL**: How long a successfully retrieved secret remains
+  in the cache.
+- **Negative TTL**: How long to remember that a key does not exist,
+  preventing unnecessary requests to the KMS.
+- **Transient Error TTL**: How long to cache failures due to temporary
+  issues like KMS timeouts.
+
+Metrics
+---------
+
+The cache exports metrics under the ``kms-cache`` collection.
+- ``hit``: Hit counter
+- ``miss``:  Miss counter
+- ``expired``: Number of TTL expired entries
+- ``size``: Current cache size
+- ``capacity``: Cache maximum size
+- ``clear``: Number of cache clears. Resets ``size``, ``hit``,
+  ``miss``, ``expired``
+
+In addition the ``rgw`` collection has:
+- ``kms_fetch_lat``: Average KMS fetch latency. Also includes a
+  successful request counter. Each event results in a positive cache
+  entry.
+- ``kms_error_transient``: Transient KMS fetch error counter. Each
+  event results in a transient error cache entry.
+- ``kms_error_permanent``: Permanent KMS fetch error counter. Each
+  event results in a negative cache cache entry.
+
+.. _Linux Kernel Key Retention Service:  https://www.kernel.org/doc/html/latest/security/keys/core.html
 .. _Amazon SSE-C: https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
 .. _Amazon SSE-KMS: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
 .. _Amazon SSE-S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingServerSideEncryption.html

release-management/backports/01a1799e-f898-75cc-9444-e275adb11100.md

@@ -0,0 +1,95 @@
+---
+id: 01a1799e-f898-75cc-9444-e275adb11100
+title: KMS Cache
+component: rgw
+category: feature
+provenance:
+  type: upstream_pr
+  upstream_prs:
+  - https://github.com/ceph/ceph/pull/61256
+  upstream_pr_state: merged-main
+  upstream_commits:
+  - 12de72b3f34f53357e5a918b55f4a5d20c200761
+  upstream_trackers:
+  - https://tracker.ceph.com/issues/68524
+justification:
+  type: customer
+  refs:
+  - cobaltcore-dev/cloud-storage#562
+  - cobaltcore-dev/cloud-storage#125
+risk:
+  blast: data-loss
+  conflict: substantive
+  coverage: partial
+---
+
+## Justification (public)
+
+Adds a secure in-memory cache for SSE-KMS encryption keys retrieved
+from Barbican (or other KMS backends), eliminating redundant
+round-trips on repeated access to the same encrypted objects. Uses a
+new SIEVE-based eviction cache (WebCache) and stores decrypted secrets
+in the Linux kernel keyring. Please refer to the Ceph documentation on
+how to configure this feature.
+
+## Justification (internal)
+
+Customer roadmap feature. Upstream PR is merged into main.
+
+## Risk notes
+
+- The cache keeps each key for a TTL (time to live, how long an entry
+  stays valid). If an operator deletes or rotates a key in Barbican,
+  radosgw uses the cached key until the TTL ends.
+- The new async function `async::call_once` and the WebCache are new
+  library code. This release is the first to ship them.
+- For production workloads, you must tune the cache size and the
+  system keyring quota. Refer to the Ceph documentation.
+
+### Known bugs
+
+The two known bugs have low risk. Neither bug changes stored data or
+the result of a request that succeeds. Upstream PR
+https://github.com/ceph/ceph/pull/73032 fixes both bugs. The PR is
+under review, and this backport does not include it.
+
+#### Slow shutdown (https://tracker.ceph.com/issues/81968)
+
+radosgw shutdown can hang for up to 2 minutes. The TTL reaper is a
+background task that removes expired cache entries. At shutdown,
+radosgw sends a stop signal to the reaper. If the signal arrives
+between the end of the reaper timer and the next reaper step, the
+reaper does not see it. That window is very short. By default, the
+timer ends once every 10 seconds (the smallest cache TTL).
+
+If the reaper misses the stop signal, the `rgw_exit_timeout_secs` alarm
+(default 2 minutes) or the systemd stop timeout ends the process. The
+result is a rare slow restart. The valgrind leak report in the tracker
+comes from the stuck reaper at exit. Memory does not grow while
+radosgw runs.
+
+#### No cache in a daemonized radosgw (https://tracker.ceph.com/issues/82031)
+
+The KMS cache does not work in a daemonized radosgw. A daemonized
+radosgw starts without `-f` and moves to the background with `fork()`.
+This mode is uncommon. Standard deployment tools (e.g. Rook, cephadm)
+start radosgw in the foreground.
+
+If a thread reads a secret that a different thread cached, the read
+fails. That request returns `ERR_INTERNAL_ERROR`, and the cache turns
+itself off. After that, radosgw does one KMS lookup per request, as
+before this feature.
+
+To prevent this bug, start radosgw with `-f`. If you cannot use `-f`,
+set `rgw_crypt_s3_kms_cache_enabled = false`.
+
+## Behavior change
+
+Repeated SSE-KMS access no longer hits the KMS backend on every
+request; decrypted secrets are cached in-memory and in the Linux
+kernel keyring. KMS-cache is ON out of the box.
+
+## Upgrade notes
+
+Feature flag: rgw_crypt_s3_kms_cache_enabled. Set false to disable
+caching entirely and revert to per-request KMS lookups.

src/common/CMakeLists.txt

@@ -255,4 +255,9 @@ if(HAVE_KEYUTILS)
   set(parse_secret_srcs
     secret.c)
   add_library(parse_secret_objs OBJECT ${parse_secret_srcs})
+
+  set(keyring_srcs
+    keyring.cc)
+  add_library(keyring STATIC ${keyring_srcs})
+  target_link_libraries(keyring keyutils::keyutils)
 endif()

src/common/async/call_once.h

@@ -0,0 +1,113 @@
+// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*-
+// vim: ts=8 sw=2 smarttab ft=cpp
+
+/*
+ * Ceph - scalable distributed file system
+ *
+ * Copyright contributors to the Ceph project
+ *
+ * This is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License version 2.1, as published by the Free Software
+ * Foundation. See file COPYING.
+ *
+ */
+
+#pragma once
+
+#include <concepts>
+#include <exception>
+#include <mutex>
+#include <variant>
+
+#include "include/function2.hpp"
+
+#include "yield_context.h"
+#include "detail/call_once.h"
+
+namespace ceph::async {
+
+/// Stores the result of call_once(), if any.
+///
+/// The cached Result type must be semiregular (copyable and default-
+/// constructible).
+template <std::semiregular Result>
+class once_result {
+ public:
+  /// Call f() once and cache its return value.
+  ///
+  /// If a call has already completed, return the cached result. If a call
+  /// is already in progress, wait for its completion and return the result.
+  /// If a yield context is provided in y, its coroutine will be suspended
+  /// while waiting. f() may also suspend/resume the same coroutine.
+  ///
+  /// f() must return a Result when called with no arguments. If it throws,
+  /// that exception is rethrown by all calls.
+  ///
+  /// This function is thread-safe.
+  template <typename Callable>
+  friend Result call_once(once_result& self, optional_yield y, Callable&& f)
+      requires std::convertible_to<std::invoke_result_t<Callable>, Result>
+  {
+    auto lock = std::unique_lock{self.mutex};
+    return std::visit(
+        fu2::overload(
+            [&] (const Init&) { return self.do_init(lock, f); },
+            [&] (Wait& w) { return self.do_wait(lock, y, w); },
+            [] (const Complete& c) { return do_complete(c); }),
+        self.state);
+  }
+
+ private:
+  std::mutex mutex;
+
+  // state machine: [Init] -> [Wait] -> [Complete]
+  using Init = std::monostate;
+  using Wait = detail::call_once::WaitState;
+  struct Complete {
+    Result result;
+    std::exception_ptr eptr;
+  };
+  std::variant<Init, Wait, Complete> state;
+
+  Result do_init(std::unique_lock<std::mutex>& lock, auto&& f)
+  {
+    // transition to Wait state
+    auto& wait = state.template emplace<Wait>();
+    lock.unlock();
+
+    Complete complete;
+    try {
+      complete.result = f();
+    } catch (const std::exception&) {
+      complete.eptr = std::current_exception();
+    }
+
+    lock.lock();
+    // wake any waiters
+    wait.wake_all();
+
+    // transition to Complete state and return the result
+    return do_complete(state.template emplace<Complete>(std::move(complete)));
+  }
+
+  Result do_wait(std::unique_lock<std::mutex>& lock,
+                 optional_yield y, Wait& wait)
+  {
+    // wait for the response to an outstanding request
+    wait.wait(lock, y);
+
+    // state must be Complete after wakeup
+    return do_complete(std::get<Complete>(state));
+  }
+
+  static Result do_complete(const Complete& complete)
+  {
+    if (complete.eptr) { // rethrow cached exception
+      std::rethrow_exception(complete.eptr);
+    }
+    return complete.result; // return cached result
+  }
+};
+
+} // namespace ceph::async