diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 42482045bf..5b2b96241b 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -358,6 +358,7 @@ include::cli:partial$cbcli/nav.adoc[] **** xref:rest-api:rest-cluster-removenode.adoc[Removing Nodes from Clusters] *** xref:rest-api:rest-rebalance-overview.adoc[Rebalance] + **** xref:rest-api:file-based-data-rebalance.adoc[] **** xref:rest-api:rest-retrieve-cluster-rebalance-reason-codes.adoc[Getting Rebalance Reason Codes] **** xref:rest-api:rest-cluster-rebalance.adoc[Rebalancing the Cluster] **** xref:rest-api:rest-get-rebalance-progress.adoc[Getting Rebalance Progress] diff --git a/modules/introduction/pages/whats-new.adoc b/modules/introduction/pages/whats-new.adoc index 817883fe6a..0f8d501e99 100644 --- a/modules/introduction/pages/whats-new.adoc +++ b/modules/introduction/pages/whats-new.adoc @@ -1,16 +1,17 @@ -= What's New in Version 8.1 += What's New in Version {totoro} :description: Couchbase is the modern database for enterprise applications. :page-aliases: security:security-watsnew +:totoro: codename Totoro release :page-toclevels: 2 [abstract] {description} + -Couchbase Server 8.1 combines the strengths of relational databases with the flexibility, performance, and scale of Couchbase. +Couchbase Server {totoro} combines the strengths of relational databases with the flexibility, performance, and scale of Couchbase. For information about platform support changes, deprecation notifications, and fixed and known issues, see the xref:release-notes:relnotes.adoc[Release Notes]. [#new-features-81] -== New Features and Enhancements in 8.1.0 +== New Features and Enhancements in {totoro} This release introduces the following new features. diff --git a/modules/introduction/partials/new-features-81.adoc b/modules/introduction/partials/new-features-81.adoc index 58b3a29b1f..29762d2697 100644 --- a/modules/introduction/partials/new-features-81.adoc +++ b/modules/introduction/partials/new-features-81.adoc @@ -1,7 +1,7 @@ [#section-new-feature-810-platform-support] === Platform Support -Couchbase Server 8.1 adds support for the following operating systems: +Couchbase Server {totoro} adds support for the following operating systems: * TBD @@ -13,24 +13,49 @@ TBD === Data Service Changes -Couchbase Server 8.1 introduces several new features for the Data Service. +Couchbase Server {totoro} introduces several new features for the Data Service. -TBD +==== File-Based Rebalance (FBR) +[.edition]#{enterprise}# + +Couchbase Server Enterprise Edition {totoro} introduces File-Based Rebalance (FBR) for the Data Service. +FBR accelerates cluster rebalance by copying vBucket storage files between nodes rather than streaming individual documents using Database Change Protocol (DCP) replication. +It eliminates the serialization and pipeline overhead of DCP backfill for large, disk-resident datasets. + +The following changes apply to the Data Service rebalance behavior in {totoro}: + +* *Enabled by default*: FBR is enabled by default for Couchbase Server Enterprise Edition, both for self-managed deployments and Couchbase Capella. +If you choose to, you can turn it off globally using the new `dataServiceFileBasedRebalanceEnabled` setting. +You can also override the global setting for specific buckets using the new `dataServiceRebalanceType` setting when creating or editing buckets. + +* *Automatic rebalance type selection*: Couchbase Server automatically determines if it can use FBR to move a vBucket. +If it cannot, it automatically falls back to copying the vBucket using DCP. + +* *Separate vBucket move concurrency for FBR*: A new setting, `dataServiceFileBasedRebalanceMovesPerNode`, controls the maximum number of concurrent file-based vBucket moves per node. +This is independent of the existing `rebalanceMovesPerNode` setting, which applies to DCP rebalance. + +* *Data transfer rate throttle*: The new `snapshot_download_throttle_bytes` setting lets you limit FBR's data transfer rate between nodes. +Setting this value too high (or setting it to 0, which makes those transfers unlimited) can increase the latency of KV operations. + +NOTE: FBR is a Couchbase Server Enterprise Edition feature. +Couchbase Server Community Edition continues to use DCP-based rebalance for all vBucket moves. + +See xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance] for more information. === Non-Data Services -Couchbase Server 8.1 release has key non-Data Services enhancements. +Couchbase Server {totoro} release has key non-Data Services enhancements. === Couchbase Cluster -Couchbase Server 8.1 release has added cluster enhancements and diagnostic capabilities. +Couchbase Server {totoro} release has added cluster enhancements and diagnostic capabilities. [#unstable-metric] ==== New Metric to Detect Unstable Nodes -Couchbase Server 8.1.0 introduces a new metric, `cm_node_unreachable_total`, to help you monitor for unstable nodes in your cluster. +Couchbase Server {totoro} introduces a new metric, `cm_node_unreachable_total`, to help you monitor for unstable nodes in your cluster. An unstable node periodically becomes unavailable but recovers before the auto failover timeout expires. This metric counts the number of times a node has been unable to reach another node in the cluster. By monitoring it, you can identify nodes that are having issues before they become unavailable for long enough to be automatically failed over. @@ -39,25 +64,25 @@ See xref:learn:clusters-and-availability/unstable-nodes.adoc[] for more informat === XDCR -Couchbase Server 8.1 release has key cross datacenter replication (XDCR) enhancements and diagnostic capabilities. +Couchbase Server {totoro} release has key cross datacenter replication (XDCR) enhancements and diagnostic capabilities. TBD. === Security and Authentication -Couchbase Server 8.1 release has key security and authentication enhancements. +Couchbase Server {totoro} release has key security and authentication enhancements. TBD. === Query Service -Couchbase Server 8.1 release adds these Query Service features. +Couchbase Server {totoro} release adds these Query Service features. TBD. === Search Service -Couchbase Server 8.1 introduces several new features for the xref:search:search.adoc[Search Service]. +Couchbase Server {totoro} introduces several new features for the xref:search:search.adoc[Search Service]. TBD. diff --git a/modules/learn/assets/images/clusters-and-availability/rebalance-stages-fbr-1.svg b/modules/learn/assets/images/clusters-and-availability/rebalance-stages-fbr-1.svg new file mode 100644 index 0000000000..306d966e26 --- /dev/null +++ b/modules/learn/assets/images/clusters-and-availability/rebalance-stages-fbr-1.svg @@ -0,0 +1,527 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 1. Backfill + + + + + + + + 3. Book-keeping + + + + + + + + + + + + + + + + + + + + + + + + + + + Copy the vBucket file snapshots from the origin to the destination node. + + + + + + Copies the items in the source cache via DCP.Transfer is memory-to-memory without writing to disk. + + + + + + Perform various ancillary tasks. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Persistence Time + + + Persistence Time + + + + + + + + Backfill Time + + + Backfill Time + + + + + + + + Move Time + + + Move Time + + + + + + + + Start + + + Start + + + + + + + + Finish + + + Finish + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Persistence Time + + + Persistence Time + + + + + + + + Task Persistence + + + + + + + + + + + + + + + + + + + + + 2. Cache Transfer + + + + \ No newline at end of file diff --git a/modules/learn/assets/images/clusters-and-availability/rebalance-stages-fbr-2.svg b/modules/learn/assets/images/clusters-and-availability/rebalance-stages-fbr-2.svg new file mode 100644 index 0000000000..b15d805bad --- /dev/null +++ b/modules/learn/assets/images/clusters-and-availability/rebalance-stages-fbr-2.svg @@ -0,0 +1,661 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 1. Backfill + + + + + + + + 3. Book-keeping + + + + + + + + + + + + + + + + + + + + + + + + + + + Move the active vBucket data from the node of origin to the memory of the destination node. + + + + + + Copies the items in the source cache via DCP.Transfer is memory-to-memory without writing to disk. + + + + + + Perform various ancillary tasks. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Persistence Time + + + Persistence Time + + + + + + + + Backfill Time + + + Backfill Time + + + + + + + + Move Time + + + Move Time + + + + + + + + 4. Active Takeover + + + + + + + + + + + + + Assign control to the new active vBucket, on the destination node. + + + + + + + + 5. Book-keeping + + + + + + + + + + + + + Perform various ancillary tasks. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Takeover Time + + + Takeover Time + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Persistence Time + + + Persistence Time + + + + + + + + Task Persistence + + + + + + + + + + + + + + + + + + + + + Start + + + Start + + + + + + + + Finish + + + Finish + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 2. Cache Transfer + + + + \ No newline at end of file diff --git a/modules/learn/assets/source/cbArchDiagrams/clustersAndAvailability/nodes/rebalance/rebalanceStagesFBR.odg b/modules/learn/assets/source/cbArchDiagrams/clustersAndAvailability/nodes/rebalance/rebalanceStagesFBR.odg new file mode 100644 index 0000000000..1ed739c3aa Binary files /dev/null and b/modules/learn/assets/source/cbArchDiagrams/clustersAndAvailability/nodes/rebalance/rebalanceStagesFBR.odg differ diff --git a/modules/learn/pages/clusters-and-availability/rebalance.adoc b/modules/learn/pages/clusters-and-availability/rebalance.adoc index 798acba9d7..a8f53e5e7e 100644 --- a/modules/learn/pages/clusters-and-availability/rebalance.adoc +++ b/modules/learn/pages/clusters-and-availability/rebalance.adoc @@ -1,5 +1,6 @@ = Rebalance -:description: pass:q[_Rebalance_ redistributes data, indexes, event processing, and query processing among available nodes.] +:description: pass:q[Rebalance redistributes data, indexes, event processing, and query processing among available nodes.] +:totoro: codename Totoro release :page-aliases: clustersetup:rebalance :page-toclevels: 3 @@ -9,31 +10,32 @@ [#understanding-rebalance] == Understanding Rebalance -When one or more nodes have been _brought into_ a cluster (either by xref:learn:clusters-and-availability/nodes.adoc#node-addition[adding] or xref:learn:clusters-and-availability/nodes.adoc#node-joining[joining]), or have been _taken out_ of a cluster (either through xref:learn:clusters-and-availability/removal.adoc[Removal] or xref:learn:clusters-and-availability/failover.adoc[Failover]), _rebalance_ redistributes data, indexes, event processing, and query processing among available nodes. -The _cluster map_ is correspondingly updated and distributed to clients. -The process occurs while the cluster continues to service requests for data. +When you add one or more nodes to a cluster by xref:learn:clusters-and-availability/nodes.adoc#node-addition[adding] or xref:learn:clusters-and-availability/nodes.adoc#node-joining[joining], or remove nodes by xref:learn:clusters-and-availability/removal.adoc[removal] or xref:learn:clusters-and-availability/failover.adoc[failover], you must rebalance the cluster. +Rebalance redistributes data, indexes, event processing, and query processing among available nodes. +Rebalance also updates the cluster map and distributes it to clients. +This process occurs while the cluster continues to service requests for data. -See xref:learn:clusters-and-availability/cluster-manager.adoc[Cluster Manager], for information on the cluster map. +See xref:learn:clusters-and-availability/cluster-manager.adoc[Cluster Manager], for information about the cluster map. See xref:manage:manage-nodes/node-management-overview.adoc[Manage Nodes and Clusters], for practical examples of using rebalance. [#rebalance-bucket-rank] == Bucket Rank -In Couchbase Server Version 7.6 and later, each bucket on the cluster (Couchbase or Ephemeral) can be assigned a _rank_. +In Couchbase Server Version 7.6 and later, each bucket on the cluster (Couchbase or Ephemeral) can be assigned a rank. The value is an integer from `0` (the default) to `1000`, inclusive. -Buckets with higher ranks are handled by the rebalance process _before_ buckets with lower ranks. -For example, if a cluster hosts four buckets, which are named _A_, _B_, _C_, and _D_; and bucket _A_ is explicitly assigned a rank of `10`, while buckets _B_, _C_, and _D_ are left with the default rank of `0`; when rebalance occurs, the vBuckets for bucket _A_ are addressed first; then, vBuckets for the other buckets are addressed, with the Cluster Manager making determinations as to the appropriate handling-order for those other buckets. +Buckets with higher ranks are handled by the rebalance process before buckets with lower ranks. +For example, if a cluster hosts four buckets, which are named A, B, C, and D; and bucket A is explicitly assigned a rank of `10`, while buckets B, C, and D are left with the default rank of `0`; when rebalance occurs, the vBuckets for bucket A are addressed first; then, vBuckets for the other buckets are addressed, with the Cluster Manager making determinations as to the appropriate handling-order for those other buckets. This assignment of `rank` allows a cluster's most mission-critical data to be rebalanced with top priority. -Bucket _rank_ can be established with either the CLI (see xref:cli:cbcli/couchbase-cli-bucket-create.adoc[bucket-create] and xref:cli:cbcli/couchbase-cli-bucket-edit.adoc[bucket-edit]) or the REST API (see xref:rest-api:rest-bucket-create.adoc[Creating and Editing Buckets]). +Bucket rank can be established with either the CLI (see xref:cli:cbcli/couchbase-cli-bucket-create.adoc[bucket-create] and xref:cli:cbcli/couchbase-cli-bucket-edit.adoc[bucket-edit]) or the REST API (see xref:rest-api:rest-bucket-create.adoc[Creating and Editing Buckets]). [#rebalance-stages] == Rebalance Stages -Each rebalance proceeds in sequential _stages_. +Each rebalance proceeds in sequential stages. Each stage corresponds to a Couchbase Service, deployed on the cluster. -Therefore, if all services have been deployed, there are _seven_ stages in all -- one each for the _Data_, _Query_, _Index_, _Search_, _Eventing_, _Backup_, and _Analytics_ services. +Therefore, if all services have been deployed, there are seven stages in all -- one each for the Data, Query, Index, Search, Eventing, Backup, and Analytics services. When all stages have been completed, the rebalance process itself is complete. [#rebalancing-the-data-service] @@ -55,8 +57,8 @@ See xref:learn:clusters-and-availability/intra-cluster-replication.adoc[Intra-Cl [#data-service-rebalance-phases] === Data-Service Rebalance Phases -During the Data Service rebalance stage, vBuckets are moved in _phases_. -The phases -- which differ, depending on whether the vBucket is an _active_ or a _replica_ vBucket -- are described below. +During the Data Service rebalance stage, vBuckets are moved in phases. +The phases -- which differ, depending on whether the vBucket is an active or a replica vBucket -- are described below. [#rebalance-phases-for-replica-vbuckets] ==== Rebalance Phases for Replica vBuckets @@ -65,17 +67,22 @@ The phases through which rebalance moves a replica vBucket are shown by the foll image::clusters-and-availability/replicaVbucketMove.png[,640,align=left] -The move has two principal phases. Phase 1 is _Backfill_. Phase 2 is _Book-keeping_. +The move has two principal phases. +Phase 1 is Backfill. +Phase 2 is Book-keeping. -Phase 1, _Backfill_, itself consists of two subphases. +Phase 1, Backfill, itself consists of two subphases. The first subphase comprises the movement of the replica vBucket data from its node of origin to the memory of the destination node. The second subphase comprises the writing of the replica vBucket data from the memory to the disk of the destination node. -The time required for this second subphase, which only applies to Couchbase Buckets, is termed _Persistence Time_. -The time required for the entire _Backfill_ process, including _Persistence Time_, is termed _Backfill Time_. +The time required for this second subphase, which only applies to Couchbase Buckets, is termed Persistence Time. +The time required for the entire Backfill process, including Persistence Time, is termed Backfill Time. -Phase 2, _Book-keeping_, comprises various ancillary tasks required for move-completion. +Phase 2, Book-keeping, comprises various ancillary tasks required for move-completion. -The total time required for the move is calculated by adding _Backfill Time_ to the time required for Phase 2, _Book-keeping_; and is termed _Move Time_. +The total time required for the move is calculated by adding Backfill Time to the time required for Phase 2, Book-keeping; and is termed Move Time. + +NOTE: File-based rebalance (FBR) changes these phases. +See <<#file-based-rebalance>> for more information. [#rebalance-phases-for-active-vbuckets] ==== Rebalance Phases for Active vBuckets @@ -84,39 +91,175 @@ The phases in which rebalance moves an active vBucket are shown by the following image::clusters-and-availability/activeVbucketMove.png[,780,align=left] -The move has four principal phases. -Phase 1, _Backfill_, and Phase 2, _Book-keeping_, are identical to those required for replica vBuckets; except that the _Book-keeping_ phase includes additional _Persistence Time_. +The move has 4 principal phases. +Phase 1, Backfill, and Phase 2, Book-keeping, are identical to those required for replica vBuckets; except that the Book-keeping phase includes additional Persistence Time. + +Phase 3, Active Takeover, comprises the operations required to establish the relocated vBucket as the new active copy. +The time required for Phase 3 is termed Takeover Time. + +Phase 4, Book-keeping, comprises a final set of ancillary tasks, required for move-completion. + +The total time for the move is termed Move Time. + +NOTE: File-based rebalance (FBR) also changes the phases for active vBuckets. +See <<#file-based-rebalance>> for more information. + +[#file-based-rebalance] +=== File-Based Rebalance (FBR) + +File-based rebalance (FBR) for the Data Service copies the underlying Couchstore or Magma storage files directly from the source node to the destination node during the backfill phase of a vBucket move. +Copying the files reduces CPU overhead on both nodes and decouples rebalance time from item count, making rebalance time proportional to data size on disk rather than document count. + +FBR only works on Couchstore or Magma buckets. +It does not work on ephemeral buckets. + +FBR is enabled by default. +You can choose to turn it off globally or on a bucket-by-bucket basis. +For more about configuring FBR using the REST API, see xref:manage:manage-settings/general-settings.adoc#fbr-settings[Configure File-Based Rebalance via REST]. + +NOTE: FBR is a Couchbase Server Enterprise Edition feature. +Couchbase Server Community Edition only uses DCP-based rebalance for all vBucket moves. + +[#fbr-vs-dcp] +==== DCP Rebalance vs File-Based Rebalance + +Prior to Couchbase Server {totoro}, the Data Service always rebalanced using Data Change Protocol (DCP) backfills. +A DCP backfill reads each document from the vBucket on the source node. +It then transmits it via the network using the DCP streaming protocol to the destination node. +The destination node then writes the document into the target vBucket. +This approach is reliable, but its overhead is proportional to the number of documents in the dataset. +The DCP backfill has to deserialize, transmit, and re-serialize each document. + +Instead of performing a DCP backfill, FBR moves eligible vBuckets by copying the storage files directly. +Directly copying the vBucket files reduces CPU usage and improves network throughput by avoiding deserializing and reserializing each document. + +FBR adds an additional cache transfer phase beyond DCP's 2-step process for replica vBucket transfers: + +image::clusters-and-availability/rebalance-stages-fbr-1.svg[align=left] + +FBR also adds the cache transfer phase for active vBucket transfers: + +image::clusters-and-availability/rebalance-stages-fbr-2.svg[align=left] + +See <<#data-service-rebalance-phases>> for more information about rebalance phases. + +The following table summarizes the differences between DCP and FBR. + +[width="100%",cols="25%,36%,39%",options="header"] +|=== +|Aspect |DCP Rebalance |File-Based Rebalance (FBR) + +|Transfer mechanism +|Stream documents through DCP pipeline +|Copy storage files directly over the network + +|Time scales with +|Number of items in the dataset +|Size of data on disk -Phase 3, _Active Takeover_, comprises the operations required to establish the relocated vBucket as the new active copy. -The time required for Phase 3 is termed _Takeover Time_. +|CPU overhead +|Higher — serialization on source, deserialization on destination +|Lower — file copy with no document processing -Phase 4, _Book-keeping_, comprises a final set of ancillary tasks, required for move-completion. +|Best suited for +|Small datasets, storage migration, ephemeral buckets +|Large disk-resident (DGM) datasets, swap rebalance, rebalance-in -The total time for the move is termed _Move Time_. +|Enterprise Edition only +|No — available in all editions +|Yes — EE only + +|Default from {totoro} onward +|Fallback when FBR is not applicable +|Default for all eligible vBucket moves +|=== + +[#fbr-backfill-takeover] +==== FBR Backfill and Takeover Phases + +FBR applies only to the Backfill phase of a vBucket move, as described in xref:learn:clusters-and-availability/rebalance.adoc#data-service-rebalance-phases[Data Service Rebalance Phases]. +The Active Takeover phase always uses DCP, regardless of whether FBR was used for backfill. + +[#fbr-automatic-selection] +==== Automatic Rebalance Method Selection + +The server automatically uses FBR (when enabled) to backfill vBuckets when it can. +If it cannot directly copy the vBucket files, Couchbase Server falls back to using DCP to move the data. +Situations where Couchbase Server cannot directly copy the vBucket files include: + +Storage engine migration between Couchstore and Magma:: +Migrating the storage format requires a full data reload, which is only possible through DCP. + +Eviction policy changes:: +Changing a bucket's eviction policy requires data to be reprocessed during rebalance, which requires DCP. + +Ephemeral buckets:: +Ephemeral buckets store data entirely in memory and have no persistent storage files for FBR to copy. + +[#fbr-performance] +==== Performance + +The primary goal of FBR is to improve rebalance speed for large datasets. +Rebalance time scales proportionally with the amount of data on disk and is independent of item count. +Throughput depends on the available network bandwidth, disk IOPS, and CPU resources on the participating nodes. + +NOTE: Workloads with lower resident ratios (the ratio of data stored on disk vs. in memory) show the greatest benefit from FBR. [#limiting-concurrent-vbucket-moves] === Limiting Concurrent vBucket Moves -Since vBucket moves are highly resource-intensive, Couchbase Server allows the concurrency of such moves to be _limited_: a setting is provided that determines the maximum number of concurrent vBucket moves permitted on any node. -The minimum value for the setting is `1`, the maximum `64`, the default `4`. +Because copying vBuckets via DCP or FBR is resource-intensive, Couchbase Server lets you limit the number of concurrent moves for both types of backfills. +DCP and FBR rebalances have their own concurrency settings. +These settings limit the maximum number of simultaneous vBucket moves allowed on a node (as either the source or destination) at a time. +They apply to both active and replica vBuckets. + +For example, suppose a node is the source for 2 moves that are in the DCP backfill phase. +Also, it's the target for 2 additional DCP backfill moves. +With the default setting of `4`, any additional backfills involving the node (either as a source or destination) have to wait until 1 of its current moves completes its backfill phase. + +Both settings default to `4` and have a minimum value of `1`. +You can set DCP concurrency to a maximum of `64` and FBR to a maximum of `1024`. + +They only have an effect on rebalances in the backfill phase. +See <<#data-service-rebalance-phases>> and <<#fbr-backfill-takeover>> for more about rebalance phases. + +You can change the concurrency settings using the xref:manage:manage-settings/general-settings.adoc#rebalance-settings[Couchbase Web Console], the xref:manage:manage-settings/general-settings.adoc#rebalance-settings-via-cli[Couchbase CLI], or the xref:manage:manage-settings/general-settings.adoc#rest_rebalance_concurrent_moves[REST API]. + +Increasing the vBucket move concurrency can improve rebalance performance. +However, higher concurrency uses more CPU, memory, disk, and network bandwidth resources. +Conversely, a lower setting can degrade rebalance performance, while reducing the resources used by rebalance. + +Other factors can affect rebalance performance. +Therefore, changing these parameters does not always result in a faster rebalance. +Also consider that higher concurrency settings can degrade the performance of other database processes such the Data Service. + +[#fbr-throttle] +==== FBR Backfill Network Throttle + +FBR copies vBucket snapshots via the network during the rebalance backfill phase. +Directly copying vBucket files is efficient, but it can consume more network bandwidth than DCP backfills. +Depending on available network bandwidth and the size of the vBuckets, these transfers can add latency to other database activities such as key-value retrievals. + +If you see additional latency during rebalances, you can monitor the amount of bandwidth FBR file transfers are consuming using the following metrics: + +* xref:metrics-reference:data-service-metrics.adoc#kv_ep_snapshot_read_bytes[`kv_ep_snapshot_read_bytes`]: the number of bytes a node has read from other node's files for FBR. +* xref:metrics-reference:data-service-metrics.adoc#kv_ep_dcp_cache_transfer_read_bytes[`kv_ep_dcp_cache_transfer_read_bytes`]: the number of bytes a node has read from other nodes caches. +This metric measures the transfers made by other features in addition to FBR. -A _move_ counts toward this restriction only when in the _backfill_ phase, as described above, in xref:learn:clusters-and-availability/rebalance.adoc#data-service-rebalance-phases[Data Service Rebalance Phases]. -The move may be of either an _active_ or a _replica_ vBucket. -A node's participation in the move may be as either a source or a target. +See xref:manage:monitor/set-up-prometheus-for-monitoring.adoc[] for more information about monitoring metrics. -For example, if a node is at a given time the source for two moves in backfill phase, and is the target for two additional moves in backfill phase, and the setting stands at `4`, the node may participate in the backfill phase of no additional moves, until at least one of its current moves has completed its backfill phase. +If you find that the FBR transfers are consuming too much bandwidth, you can limit the bandwidth FBR can use to transfer vBuckets. -The setting may be established by means of the xref:manage:manage-settings/general-settings.adoc#rebalance-settings[Couchbase Web Console], the xref:manage:manage-settings/general-settings.adoc#rebalance-settings-via-cli[Couchbase CLI], or the xref:manage:manage-settings/general-settings.adoc#rebalance-settings-via-rest[REST API]. +You can set the limit using the *File-Based Rebalance Throttle Rate* setting in the Couchbase Server Web Console's *Settings* page. +See xref:manage:manage-settings/general-settings.adoc#rebalance-settings[Rebalance Settings] for more information. -A higher setting may improve rebalance performance, at the cost of higher resource consumption; in terms of CPU, memory, disk, and bandwidth. -Conversely, a lower setting may degrade rebalance performance, while freeing up such resources. -Note, however, that rebalance performance can be affected by many additional factors; and that in consequence, changing this parameter may not always have the expected effects. -Note also that a higher setting, due to its additional consumption of resources, may degrade the performance of other systems, including the Data Service. +To change this setting via the REST API, alter the xref:rest-api:rest-reader-writer-thread-config.adoc#fbr-throttle[`snapshot_download_throttle_bytes`] setting in the `/pools/default/settings/memcached/global` REST API endpoint. +See xref:manage:manage-settings/general-settings.adoc#fbr-throttle-rest-api[Set FBR Bandwidth Throttle with the REST API] for more information. [#rebalance-reporting] === Accessing Rebalance Reports -Couchbase Server creates a _report_ on every rebalance that occurs. +Couchbase Server creates a report on every rebalance that occurs. The report contains a JSON document, which can be inspected in any browser or editor. The document provides summaries of the concluded rebalance activity, as well as details for each of the vBuckets affected: in consequence, the report may be of considerable length. @@ -126,8 +269,8 @@ On conclusion of a rebalance, its report can be accessed in any of the following * By means of the REST API, as described in xref:rest-api:rest-get-cluster-tasks.adoc[Getting Cluster Tasks]. -* By accessing the directory `/opt/couchbase/var/lib/couchbase/logs/rebalance` on _any_ of the cluster nodes. -A rebalance report is maintained here for (up to) the last _five_ rebalances performed. +* By accessing the directory `/opt/couchbase/var/lib/couchbase/logs/rebalance` on any of the cluster nodes. +A rebalance report is maintained here for (up to) the last five rebalances performed. Each report is provided as a `*.json` file, whose name indicates the time at which the report was run -- for example, `rebalance_report_2020-03-17T11:10:17Z.json`. A complete account of the report-content is provided in the xref:rebalance-reference:rebalance-reference.adoc[Rebalance Reference]. @@ -139,7 +282,7 @@ Rebalance affects each service differently. The following sections describe how rebalance affects non-Data Services. [#rebalancing-the-index-service] -== Index Service +=== Index Service The Index Service maintains a cluster-wide set of index definitions and metadata, which allows the redistribution of indexes and index replicas during a rebalance. @@ -156,14 +299,14 @@ For more information about the rebalance operation on Index Service, see xref:le The Search Service automatically partitions its indexes across all Search nodes in the cluster, ensuring optimal distribution, following rebalance. -To achieve this, in versions of Couchbase Server prior to 7.1, by default, partitions needing to be newly created were entirely _built_, on their newly assigned nodes. -In 7.1 and later versions, by default, new partitions are instead created by the _transfer_ of partition files from old nodes to new nodes: this significantly enhances performance. -This is an Enterprise-only feature, which requires all Search Service nodes _either_ to be running 7.1 or later; _or_ to be running 7.0.2, with the feature explicitly switched on. +To achieve this, in versions of Couchbase Server prior to 7.1, by default, partitions needing to be newly created were entirely built, on their newly assigned nodes. +In 7.1 and later versions, by default, new partitions are instead created by the transfer of partition files from old nodes to new nodes: this significantly enhances performance. +This is an Enterprise-only feature, which requires all Search Service nodes either to be running 7.1 or later; or to be running 7.0.2, with the feature explicitly switched on. Community Edition clusters that are upgraded to Enterprise Edition 7.1 and later versions thus gain this feature in its default setting. Community Edition clusters that are upgraded to Enterprise Edition 7.0.2 can have this feature switched on, subsequent to upgrade. -During file transfer, should an unresolvable error occur, file transfer is automatically abandoned, and _partition build_ is used instead. +During file transfer, should an unresolvable error occur, file transfer is automatically abandoned, and partition build is used instead. The file-transfer feature can be enabled and disabled by means of the REST API. See xref:fts-rest-manage:index.adoc[Search Manager Options]. @@ -190,26 +333,26 @@ If needed, you can retry these requests on another Query node that is still in t [#rebalancing-the-eventing-service] === Eventing Service -When an Eventing Service node has been added or removed, rebalance causes the mutation (_vBucket_ processing ownership) and timer event processing workload to be redistributed among available Eventing Service nodes. +When an Eventing Service node has been added or removed, rebalance causes the mutation (vBucket processing ownership) and timer event processing workload to be redistributed among available Eventing Service nodes. The Eventing Service continues to process mutations both during and after rebalance. Checkpoint information ensures that no mutations are lost. [#rebalancing-the-analytics-service] === Analytics Service -The Analytics Service uses _shadow data_, which is a copy of all or some of the data maintained by the Data Service. +The Analytics Service uses shadow data, which is a copy of all or some of the data maintained by the Data Service. By default, the shadow data is not replicated; however, it may be partitioned across all cluster nodes that run the Analytics Service. Starting with Couchbase Server 7.1, the shadow data and its partitions may be replicated up to 3 times. Each replica resides on an Analytics node: a given Analytics node can host a replica partition, or the active partition on which replicas are based. -If there are _no_ Analytics replicas, and an Analytics node fails over, the Analytics Service stops working cluster-wide: ingestion of shadow data stops and no Analytics operations can be run. +If there are no Analytics replicas, and an Analytics node fails over, the Analytics Service stops working cluster-wide: ingestion of shadow data stops and no Analytics operations can be run. In this case: * If the Analytics node is recovered, the Analytics Service is resumed and ingestion of shadow data resumes from the point before the node failed over. * If the Analytics node is removed, the Analytics Service becomes active again after rebalance, but ingestion of shadow data must begin again from scratch. -If there _are_ Analytics replicas, and an Analytics node fails over, the Analytics Service continues to work: one of the replicas is promoted to serve the shadow data that was stored on the failed over node. +If there are Analytics replicas, and an Analytics node fails over, the Analytics Service continues to work: one of the replicas is promoted to serve the shadow data that was stored on the failed over node. The Analytics Service only needs to rebuild any shadow data that isn't already ingested from the Data Service, depending on the state of the promoted replica. In this case: @@ -218,7 +361,7 @@ In this case: * If the Analytics node is removed, the shadow data is redistributed among the remaining Analytics nodes in the cluster. If no Analytics Service node has been removed or replaced, shadow data is not affected by rebalance. -In consequence of rebalance, the Analytics Service receives an updated _cluster map_, and continues to work with the modified vBucket-topology. +In consequence of rebalance, the Analytics Service receives an updated cluster map, and continues to work with the modified vBucket-topology. [#rebalancing-the-backup-service] === Backup Service @@ -227,12 +370,12 @@ A rebalance causes the scheduler for the Backup Service to stop running. This means that no new backup tasks are triggered until the rebalance has concluded; at which point, the scheduler restarts, and reconstructs the task schedule. Then, the triggering of Backup Service tasks is resumed. -Note that a rebalance has the effect of _restarting_ the Backup Service whenever the service has previously been stopped, due to loss of its _leader_: for information, see xref:learn:services-and-indexes/services/backup-service.adoc#backup-service-architecture[Backup-Service Architecture]. +Note that a rebalance has the effect of restarting the Backup Service whenever the service has previously been stopped, due to loss of its leader: for information, see xref:learn:services-and-indexes/services/backup-service.adoc#backup-service-architecture[Backup-Service Architecture]. [#rebalance-failure-handling] == Rebalance Failure-Handling -Rebalance failures can optionally be responded to automatically, with up to 3 _retries_. +Rebalance failures can optionally be responded to automatically, with up to 3 retries. The number of seconds required to elapse between retries can also be configured. For information on configuration options, see xref:manage:manage-settings/general-settings.adoc[General Settings]. For information on failure-notifications, and options for cancelling rebalance-retries, see xref:manage:manage-nodes/add-node-and-rebalance.adoc#automated-rebalance-failure-handling[Automated Rebalance Failure Handling]. diff --git a/modules/manage/assets/images/manage-settings/rebalance-settings.png b/modules/manage/assets/images/manage-settings/rebalance-settings.png index 3e6efaaf6e..2ccd2f8f2f 100644 Binary files a/modules/manage/assets/images/manage-settings/rebalance-settings.png and b/modules/manage/assets/images/manage-settings/rebalance-settings.png differ diff --git a/modules/manage/pages/manage-nodes/add-node-and-rebalance.adoc b/modules/manage/pages/manage-nodes/add-node-and-rebalance.adoc index d323773ea5..1f03e04aa0 100644 --- a/modules/manage/pages/manage-nodes/add-node-and-rebalance.adoc +++ b/modules/manage/pages/manage-nodes/add-node-and-rebalance.adoc @@ -182,10 +182,13 @@ The screen now appears as follows: [#servers-screen-with-node-added-after-rebalance] image::manage-nodes/twoNodeClusterAfterRebalance.png[,800,align=middle] + -Note that the figure in the *Items* column for node `10.142.181.101` is `31.5 K/31.6 K`, which indicates that 31.5 K items are stored on the node in _active_ vBuckets, and 31.6 K in _replica_ vBuckets. +Note that the figure in the *Items* column for node `10.142.181.101` is `31.5 K/31.6 K`, which indicates that 31.5 K items are stored on the node in active vBuckets, and 31.6 K in replica vBuckets. The figure for `10.142.181.102` indicates the converse. Therefore, replication has successfully distributed the contents of `travel-sample` across both nodes, providing a single replica vBucket for each active vBucket. +NOTE: By default, Couchbase Server Enterprise Edition automatically uses File-Based Rebalance (FBR) to move data for eligible vBuckets during node addition. +For information, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)]. + [#node-information-within-the-ui] ==== Node Information Within the UI diff --git a/modules/manage/pages/manage-settings/general-settings.adoc b/modules/manage/pages/manage-settings/general-settings.adoc index cd5b2601f4..5e6e48e594 100644 --- a/modules/manage/pages/manage-settings/general-settings.adoc +++ b/modules/manage/pages/manage-settings/general-settings.adoc @@ -174,24 +174,54 @@ See xref:learn:data/durability.adoc[Durability], for information. [#rebalance-settings] === Rebalance Settings -_Rebalance_ redistributes data, indexes, event processing, and query processing among available nodes. +Rebalance redistributes data, indexes, event processing, and query processing among available nodes. For an overview, see xref:learn:clusters-and-availability/rebalance.adoc[Rebalance]. -Fully open, the panel appears as follows: +You can expand the *Rebalance Settings* section to see all of the rebalance settings: -image::manage-settings/rebalance-settings.png["The Rebalance Settings panel",548,align=center] +image::manage-settings/rebalance-settings.png["The Rebalance Settings panel",align=center] -The *Retry rebalance* option allows rebalance to be _retried_, in cases where it has failed. -Check the checkbox, to enable. -The specifiable, _maximum number of retries_ must be in the range of 1 to 3, inclusive. -The specifiable, _maximum number of seconds_ must be in the range of 5 to 3600, inclusive. +The settings in the section are: -Note that this option should _not_ be enabled if the cluster is managed by _Couchbase Autonomous Operator_, or if custom scripts are already being used to trigger rebalance. -Note also that no administrative tasks should be attempted when rebalance-retries are pending. -However, pending rebalance-retries can be cancelled: see xref:manage:manage-nodes/add-node-and-rebalance.adoc#automated-rebalance-failure-handling[Automated Rebalance-Failure Handling], for information. +Retry rebalance:: +Controls whether Couchbase Server retries failed rebalances. +This setting is off by default. +Select *Retry rebalance* to enable it. +You can choose to have Couchbase Server retry a rebalance between 1 and 3 times. +You also choose how long it waits between rebalance attempts. +The minimum is 5 seconds and the maximum is 1 hour (3600 seconds). + ++ +Do not enable this option when Couchbase Autonomous Operator manages your cluster or if you're using custom scripts to trigger rebalances. + ++ +NOTE: Do not perform administrative tasks when rebalance-retries are pending. +If necessary, you can cancel pending rebalance retries. +See xref:manage:manage-nodes/add-node-and-rebalance.adoc#automated-rebalance-failure-handling[Automated Rebalance-Failure Handling] for more information. + +[#fbr-throttle] +File-Based Rebalance Throttle Rate:: +Limits the rate of data transfer for file-based rebalance (FBR) on a per node basis. +This limit applies to the total of incoming and outgoing data per second. +The default value of `0` lets the node use unlimited bandwidth when transferring data during rebalance backfills. +See xref:learn:clusters-and-availability/rebalance.adoc#fbr-throttle[FBR Backfill Network Throttle] for more information. + + +Max moves per node during rebalance:: +Sets the maximum number of concurrent vBucket moves permitted on each node during rebalance backfills. +The *For DCP-based rebalance* field sets the limit for Data Change Protocol (DCP) backfills. +This is the only backfill type supported by Couchbase Server Community Edition. +The range for this setting is from `1` to `64`. +The default is `4`. + ++ +Couchbase Server Enterprise Edition clusters have an additional field named *For file-based rebalance*. +It controls the number of vBucket files a node can copy concurrently when performing an FBR. +The range for this setting is from `1` to `64`. +The default is `4`. +For information about FBR, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)]. -The *Max moves per node during rebalance* option establishes the maximum number of concurrent vBucket moves permitted on every individual node. -The minimum value for the parameter is `1`, the maximum `64`, the default `4`. -For information, see xref:learn:clusters-and-availability/rebalance.adoc#limiting-concurrent-vbucket-moves[Limiting Concurrent vBucket Moves]. ++ +For information about these settings, see xref:learn:clusters-and-availability/rebalance.adoc#limiting-concurrent-vbucket-moves[Limiting Concurrent vBucket Moves]. [#data-settings] === Data Settings @@ -871,33 +901,38 @@ For additional information, refer to xref:n1ql-rest-settings:index.adoc[]. [#rebalance-settings-via-rest] === Rebalance Settings via REST -By means of the REST API, both _rebalance retries_ and _maximum concurrent moves per node_ can be configured. +You can change rebalance settings such as rebalance retries and maximum concurrent moves per node using REST API calls. ==== Rebalance Retries via REST -To obtain the cluster's current settings for _rebalance retries_ by means of the REST API, use the `GET /settings/retryRebalance` HTTP method and URI, as follows: +To obtain the cluster's current settings for rebalance retries using the REST API, use the `GET /settings/retryRebalance` HTTP method and URI, as follows: [source,shell] ---- -curl -X GET -u Administrator:password \ -http://10.143.192.101:8091/settings/retryRebalance +curl -v -X GET -u $USERNAME:$PASSWORD \ + http://node1.example.com:8091/settings/retryRebalance | jq ---- If successful, the command returns the following object: [source,json] ---- -{"enabled":true,"afterTimePeriod":200,"maxAttempts":3} +{ + "enabled": true, + "afterTimePeriod": 300, + "maxAttempts": 3 +} + ---- -This output shows that rebalance retry is enabled, with `200` seconds required to elapse before a retry is attempted, and a maximum of `3` retries possible. +This output shows that rebalance retry is enabled, with `300` seconds required to elapse before a retry is attempted, and a maximum of `3` retries possible. To change the rebalance settings, use the `POST` method with the same URI, specifying appropriate values: [source,shell] ---- -curl -X POST -u Administrator:password \ -http://10.143.192.101:8091/settings/retryRebalance \ +curl -v -X POST -u $USERNAME:$PASSWORD \ +http://node1.example.com:8091/settings/retryRebalance \ -d enabled=false \ -d afterTimePeriod=100 \ -d maxAttempts=2 @@ -910,45 +945,151 @@ If successful, the command returns the following object: {"enabled":false,"afterTimePeriod":100,"maxAttempts":2} ---- -This verifies that rebalance retry has been disabled, the required period between retries changed to `100` seconds, and the maximum number of retries changed to `2`. +This verifies that rebalance retry has been turned off, the required period between retries changed to `100` seconds, and the maximum number of retries changed to `2`. + +For more information about getting and setting the rebalance retry status, see xref:rest-api:rest-configure-rebalance-retry.adoc[Configure Rebalance Retries], xref:rest-api:rest-get-rebalance-retry.adoc[Get Rebalance-Retry Status], and xref:rest-api:rest-cancel-rebalance-retry.adoc[Cancel Rebalance Retries]. + +[#fbr-settings] +==== Configure File-Based Rebalance via REST + +The Data Service defaults to using file-based rebalance (FBR) when possible to transfer vBucket data between nodes. +See xref:learn:clusters-and-availability:rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)] for more information. + +You can use the REST API to view the status of FBR in the cluster by making a `GET` request to the `/internalSettings` endpoint to view the `dataServiceFileBasedRebalanceEnabled` setting. +The following example uses `jq` to filter out all other values: + +[source,shell] +---- +curl -X GET -u $USERNAME:$PASSWORD \ + http://node1.example.com:8091/internalSettings \ + | jq '{dataServiceFileBasedRebalanceEnabled}' +---- + +The result of running this command returns a JSON object: + +[source,json] +---- +{ + "dataServiceFileBasedRebalanceEnabled": true +} +---- + +To turn off FBR, make a `POST` request to set `dataServiceFileBasedRebalanceEnabled` to `false`: + +[source,bash] +---- +curl -X POST -u $USERNAME:$PASSWORD \ + http://node1.example.com:8091/internalSettings \ + -d dataServiceFileBasedRebalanceEnabled=false +---- -For more information on getting and setting the rebalance retry status, see xref:rest-api:rest-configure-rebalance-retry.adoc[Configure Rebalance Retries], xref:rest-api:rest-get-rebalance-retry.adoc[Get Rebalance-Retry Status], and xref:rest-api:rest-cancel-rebalance-retry.adoc[Cancel Rebalance Retries]. +You can also override the FBR use on a per-bucket basis using the bucket-level xref:rest-api:rest-bucket-create.adoc#dataservicerebalancetype[`dataServiceRebalanceType`] setting. +The following example sets the `travel-sample` bucket to use DCP rebalance instead of FBR. +[source,bash] +``` +curl -X POST http://node1.example.com:8091/pools/default/buckets/travel-sample \ + -u $USERNAME:$PASSWORD + -d dataServiceRebalanceType=preferDcp +``` + +[#rest_rebalance_concurrent_moves] ==== Maximum Concurrent vBucket Moves via REST -To inspect the current maximum number of concurrent vBucket moves permitted for every node, use the `GET /settings/rebalance` HTTP method and URI, with the `rebalanceMovesPerNode` parameter, as follows: +Couchbase Server has 2 settings to control the number of vBucket moves a node can take part in concurrently: 1 for data change protocol (DCP) backfills and another for file-based rebalance (FBR) backfills. + +See xref:docs-server/modules/learn/pages/clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)] for more information about these backfill modes. + +To find the current settings, make a `GET` request to the `/settings/rebalance` REST API endpoint: [source,shell] ---- -curl -v -X GET http://10.143.201.101:8091/settings/rebalance \ --u Administrator:password +curl -v -X GET -u $USERNAME:$PASSWORD \ + http://node1.example.com:8091/settings/rebalance | jq ---- -This returns an object, confirming the current setting as being `4` (which is the default value): +This request returns an object containing the current settings: [source,json] ---- -{"rebalanceMovesPerNode":4} +{ + "rebalanceMovesPerNode": 4, + "dataServiceFileBasedRebalanceMovesPerNode": 4 +} + ---- -To _set_ a new value for the parameter use the `POST` method with the same URI, and with the `rebalanceMovesPerNode` parameter. -Note that the minimum value is `1`, and the maximum `64`. +`rebalanceMovesPerNode` is the concurrency limit for DCP backfills and `dataServiceFileBasedRebalanceMovesPerNode` is the limit for FBR. +Both are set to the default value of `4`. + +To set a new values, send a `POST` request to the same endpoint setting the parameters to a new values. +The valid range for these settings are from `1` to `64`. [source,shell] ---- -curl -v -X POST http://10.143.201.101:8091/settings/rebalance \ --u Administrator:password \ --d rebalanceMovesPerNode=10 +curl -X POST http://node1.example.com:8091/settings/rebalance \ + -u $USERNAME:$PASSWORD \ + -d rebalanceMovesPerNode=8 \ + -d dataServiceFileBasedRebalanceMovesPerNode=6 \ + | jq ---- If successful, the call returns an object confirming the new setting: [source,json] ---- -{"rebalanceMovesPerNode":10} +{ + "rebalanceMovesPerNode": 8, + "dataServiceFileBasedRebalanceMovesPerNode": 6 +} +---- + +For more information about limiting concurrent backfills, see xref:rest-api:rest-limit-rebalance-moves.adoc[Limiting Concurrent vBucket Moves]. + +[#fbr-throttle-rest-api] +==== Set FBR Bandwidth Throttle with the REST API + +By default, Couchbase Server does not limit the network bandwidth FBR uses when copying vBucket data between nodes. +These unlimited transfers can increase latency for other Couchbase Server features, such as key-value operations. +You can limit FBR's network use by setting the `snapshot_download_throttle_bytes` option using the REST API. +See +See xref:learn:clusters-and-availability/rebalance.adoc#fbr-throttle[FBR Backfill Network Throttle] for more information about limiting FBR's network use. + +To view any existing FBR network throttle setting, make a `GET` request to the `pools/default/settings/memcached/global` REST API endpoint: + +[source,bash] +---- +curl -X GET -u $USERNAME:$PASSWORD \ + http://node1.example.com:8091/pools/default/settings/memcached/global | jq +---- + +If there are no overrides of any storage thread allocations (including `snapshot_download_throttle_bytes`), the command returns an empty JSON object. + +[source,json] +---- +{} +---- + +To set a limit, make a `POST` request to the endpoint, setting `snapshot_download_throttle_bytes` to the maximum number of bytes per second FBR should use to transfer files. +This example limits FBR transfers to 600MiB/s per node. + +[source,bash] +---- +curl -X POST -u $USERNAME:$PASSWORD \ + http://node1.example.com:8091/pools/default/settings/memcached/global \ + -d snapshot_download_throttle_bytes=629145600 | jq +---- + +If successful, the result returns the current overrides to the settings: + +[source,json] +---- +{ + "snapshot_download_throttle_bytes": 629145600 +} ---- -For more information, see the REST reference page xref:rest-api:rest-limit-rebalance-moves.adoc[Limiting Concurrent vBucket Moves]. +See xref:rest-api:rest-reader-writer-thread-config.adoc[] for more information about overriding reader and writer thread settings. [#xdcr-process-setting-via-rest] === XDCR Process Setting via REST diff --git a/modules/metrics-reference/attachments/kv_metrics_metadata.json b/modules/metrics-reference/attachments/kv_metrics_metadata.json index c8089a023d..5521975c4d 100644 --- a/modules/metrics-reference/attachments/kv_metrics_metadata.json +++ b/modules/metrics-reference/attachments/kv_metrics_metadata.json @@ -2,6 +2,7 @@ "kv_audit_dropped_events": { "added": "7.0.0", "help": "The number of audit events dropped due to errors while trying to insert them to the audit trail", + "long_description": "Provides the number of audit events that were failed and never written to the audit trail. When audit logging is enabled, Couchbase logs security related operations such as authentication attempts, bucket operations, and key modifications for compliance purposes. The audit daemon maintains an internal event queue, when that queue reaches its capacity of 50,000 pending events, new events are dropped and this counter is incremented to prevent unbounded memory growth under high load. A non-zero value means audit records have been permanently lost.", "stability": "committed", "type": "counter" }, @@ -20,6 +21,7 @@ "kv_auth_errors": { "added": "7.0.0", "help": "The number of failed authentication requests", + "long_description": "Provides the number of failed client authentication attempt at the memcached layer. Failure causes include invalid credentials (wrong username or password), an unsupported or malformed SASL mechanism, bad authentication request parameters, SASL-level protocol errors, and unavailability of an external authentication provider.", "stability": "committed", "type": "gauge" }, @@ -417,7 +419,8 @@ }, "kv_ep_access_scanner_enabled": { "added": "7.0.0", - "help": "True if access scanner task is enabled", + "config_param": true, + "help": "True if access scanner task is enabled. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, @@ -442,37 +445,43 @@ }, "kv_ep_allow_sanitize_value_in_deletion": { "added": "7.0.0", - "help": "Let EPE delete/prepare/del_with_meta prune any invalid body in the payload instead of failing", + "config_param": true, + "help": "Let EPE delete/prepare/del_with_meta prune any invalid body in the payload instead of failing. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, "kv_ep_alog_block_size": { "added": "7.0.0", - "help": "Logging block size.", + "config_param": true, + "help": "Logging block size. Configuration parameter with a default value of 4096.", "stability": "volatile", "type": "gauge" }, "kv_ep_alog_max_stored_items": { "added": "7.0.0", - "help": "The maximum number of items the Access Scanner will hold in memory before commiting them to disk", + "config_param": true, + "help": "The maximum number of items the Access Scanner will hold in memory before commiting them to disk. Configuration parameter with a default value of 1024.", "stability": "volatile", "type": "gauge" }, "kv_ep_alog_resident_ratio_threshold": { "added": "7.0.0", - "help": "Resident ratio percentage above which we do not generate access log", + "config_param": true, + "help": "Resident ratio percentage above which we do not generate access log. Configuration parameter with a default value of 95.", "stability": "volatile", "type": "gauge" }, "kv_ep_alog_sleep_time": { "added": "7.0.0", - "help": "Number of minutes between each sweep for the access log", + "config_param": true, + "help": "Number of minutes between each sweep for the access log. Configuration parameter with a default value of 1440.", "stability": "volatile", "type": "gauge" }, "kv_ep_alog_task_time": { "added": "7.0.0", - "help": "Hour in GMT time when access scanner task is scheduled to run", + "config_param": true, + "help": "Hour in GMT time when access scanner task is scheduled to run. Configuration parameter with a default value of 2.", "stability": "volatile", "type": "gauge" }, @@ -492,7 +501,8 @@ }, "kv_ep_backfill_mem_threshold": { "added": "7.0.0", - "help": "Memory usage threshold (percentage of bucket quota) after which backfill will be snoozed.", + "config_param": true, + "help": "Memory usage threshold (percentage of bucket quota) after which backfill will be snoozed. Configuration parameter with a default value of 96.", "stability": "volatile", "type": "gauge" }, @@ -504,25 +514,29 @@ }, "kv_ep_bfilter_enabled": { "added": "7.0.0", - "help": "Enable or disable the bloom filter", + "config_param": true, + "help": "Enable or disable the bloom filter. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, "kv_ep_bfilter_fp_prob": { "added": "7.0.0", - "help": "Bloomfilter: Allowed probability for false positives", + "config_param": true, + "help": "Bloomfilter: Allowed probability for false positives. Configuration parameter with a default value of 0.01.", "stability": "volatile", "type": "gauge" }, "kv_ep_bfilter_key_count": { "added": "7.0.0", - "help": "Bloomfilter: Estimated key count per vbucket", + "config_param": true, + "help": "Bloomfilter: Estimated key count per vbucket. Configuration parameter with a default value of 10000.", "stability": "volatile", "type": "gauge" }, "kv_ep_bfilter_residency_threshold": { "added": "7.0.0", - "help": "If resident ratio (during full eviction) were found less than this threshold, compaction will include all items into bloomfilter", + "config_param": true, + "help": "If resident ratio (during full eviction) were found less than this threshold, compaction will include all items into bloomfilter. Configuration parameter with a default value of 0.1.", "stability": "volatile", "type": "gauge" }, @@ -644,14 +658,17 @@ "type": "counter" }, "kv_ep_bucket_quota_change_task_poll_interval": { - "added": "7.0.0", - "help": "Time in seconds between the BucketQuotaChangeTask polling memory usage to attempt to reduce the bucket quota", + "added": "7.6.0", + "config_param": true, + "help": "Time in seconds between the BucketQuotaChangeTask polling memory usage to attempt to reduce the bucket quota. Configuration parameter with a default value of 30.", "stability": "volatile", "type": "gauge" }, "kv_ep_cache_size": { "added": "7.0.0", - "help": "Memory quota (in bytes) for this bucket.", + "config_param": true, + "help": "Memory quota (in bytes) for this bucket. Configuration parameter with a default value of (100 * 1024 * 1024).", + "long_description": "Provides the absolute maximum amount of memory, measured in bytes, allocated to a specific bucket on a single node. It represents the hard memory quota configured by the administrator, serving as the strict baseline for calculating critical thresholds like high and low watermarks. This value is critical for capacity planning.", "stability": "volatile", "type": "gauge" }, @@ -670,14 +687,16 @@ "unit": "bytes" }, "kv_ep_checkpoint_destruction_tasks": { - "added": "7.0.0", - "help": "Number of tasks responsible for destroying closed unreferenced checkpoints.", + "added": "7.1.0", + "config_param": true, + "help": "Number of tasks responsible for destroying closed unreferenced checkpoints. Configuration parameter with a default value of 2.", "stability": "volatile", "type": "gauge" }, "kv_ep_checkpoint_max_size": { - "added": "7.0.0", - "help": "Max size (in bytes) of a single checkpoint. '0' for EPEngine auto-setup.", + "added": "7.1.0", + "config_param": true, + "help": "Max size (in bytes) of a single checkpoint. '0' for EPEngine auto-setup. Configuration parameter with a default value of 0.", "stability": "volatile", "type": "gauge" }, @@ -703,14 +722,17 @@ "unit": "bytes" }, "kv_ep_checkpoint_memory_ratio": { - "added": "7.0.0", - "help": "Max ratio of the bucket quota that can be allocated in checkpoints. The system enters a TempOOM phase if hit.", + "added": "7.1.0", + "config_param": true, + "help": "Max ratio of the bucket quota that can be allocated in checkpoints. The system enters a TempOOM phase if hit. Configuration parameter with a default value of 0.2.", + "long_description": "Provides the maximum percentage of the bucket's memory quota dedicated to checkpoints. When this ratio is hit, the system enters a temporary TempOOM phase and immediately triggers memory recovery to minimize the duration of TempOOM state. A checkpoint is a sequenced queue of mutations that acts as the single source of truth for both persistence and replication within a bucket. Persistence and replication consume from this queue independently, each progressing at its own pace without coordinating with the other. The checkpoint queue allow disk flusher and DCP consumers to read from the same data structure to ensure consistency. This value helps ensure efficient memory utilization.", "stability": "volatile", "type": "gauge" }, "kv_ep_checkpoint_memory_recovery_lower_mark": { - "added": "7.0.0", - "help": "Fraction of the checkpoint quota (as computed by checkpoint_memory_ratio) that represents the target of checkpoint memory recovery. Memory recovery yields when reached.", + "added": "7.1.0", + "config_param": true, + "help": "Fraction of the checkpoint quota (as computed by checkpoint_memory_ratio) that represents the target of checkpoint memory recovery. Memory recovery yields when reached. Configuration parameter with a default value of 0.6.", "stability": "volatile", "type": "gauge" }, @@ -721,8 +743,9 @@ "type": "gauge" }, "kv_ep_checkpoint_memory_recovery_upper_mark": { - "added": "7.0.0", - "help": "Fraction of the checkpoint quota (as computed by checkpoint_memory_ratio) that triggers attempt of memory releasing from checkpoint.", + "added": "7.1.0", + "config_param": true, + "help": "Fraction of the checkpoint quota (as computed by checkpoint_memory_ratio) that triggers attempt of memory releasing from checkpoint. Configuration parameter with a default value of 0.9.", "stability": "volatile", "type": "gauge" }, @@ -733,14 +756,16 @@ "type": "gauge" }, "kv_ep_checkpoint_remover_task_count": { - "added": "7.0.0", - "help": "Number of concurrent tasks performing ItemExpel and CursorDrop/CheckpointRemoval", + "added": "7.1.0", + "config_param": true, + "help": "Number of concurrent tasks performing ItemExpel and CursorDrop/CheckpointRemoval. Configuration parameter with a default value of 1.", "stability": "volatile", "type": "gauge" }, "kv_ep_chk_expel_enabled": { "added": "7.0.0", - "help": "Enable the ability to expel (remove from memory) items from a checkpoint. An item can be expelled if all cursors in the checkpoint have iterated past the item.", + "config_param": true, + "help": "Enable the ability to expel (remove from memory) items from a checkpoint. An item can be expelled if all cursors in the checkpoint have iterated past the item. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, @@ -758,13 +783,15 @@ }, "kv_ep_collections_drop_compaction_delay": { "added": "7.0.0", - "help": "How many milliseconds before compaction runs following the drop of a collection", + "config_param": true, + "help": "How many milliseconds before compaction runs following the drop of a collection. Configuration parameter with a default value of 5000.", "stability": "volatile", "type": "gauge" }, "kv_ep_collections_enabled": { "added": "7.0.0", - "help": "Enable the collections functionality, enabling the storage of collection metadata", + "config_param": true, + "help": "Enable the collections functionality, enabling the storage of collection metadata. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, @@ -795,44 +822,51 @@ "type": "gauge" }, "kv_ep_compaction_expire_from_start": { - "added": "7.0.0", - "help": "Should compaction expire items that were logically deleted at the start of the compaction (true) or at the point in time at which they were visited (false)?", + "added": "7.1.0", + "config_param": true, + "help": "Should compaction expire items that were logically deleted at the start of the compaction (true) or at the point in time at which they were visited (false)?. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, "kv_ep_compaction_expiry_fetch_inline": { - "added": "7.0.0", - "help": "If compaction requires a bgfetch before attempting expiry to ensure it does not expire an older version of the document, true: fetch it in the compaction thread. false: queue a bgfetch for the bgfetcher task to complete", + "added": "7.1.4-MP2", + "config_param": true, + "help": "If compaction requires a bgfetch before attempting expiry to ensure it does not expire an older version of the document, true: fetch it in the compaction thread. false: queue a bgfetch for the bgfetcher task to complete. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, "kv_ep_compaction_failed": { "added": "7.1.0", "help": "Counter of how many times compaction has failed, e.g. a system call error caused compaction to fail", + "long_description": "Provides the number of time the compaction process stats but forced to abort or failed. Common causes include out of disk space, permission issues, disk I/O bottlenecks, and system crashes or restarts. A rising value indicates a serious problem: disk usage grows indefinitely as old document versions are never purged, fragmented files degrade read performance, and eventually the Data Service will stop accepting writes when the disk reaches capacity.", "stability": "committed", "type": "gauge" }, "kv_ep_compaction_max_concurrent_ratio": { - "added": "7.0.0", - "help": "Maximum number of CompactVBucketTask tasks which can run concurrently, as a fraction of the possible Writer task concurrency. Note that a minimum of 1, and a maximum of N-1 CompactVBucketTasks will be run (where N is the possible Writer task concurrency), to ensure both forward progress for Compaction and Flushing.", + "added": "7.1.0", + "config_param": true, + "help": "Maximum number of CompactVBucketTask tasks which can run concurrently, as a fraction of the possible Writer task concurrency. Note that a minimum of 1, and a maximum of N-1 CompactVBucketTasks will be run (where N is the possible Writer task concurrency), to ensure both forward progress for Compaction and Flushing. Configuration parameter with a default value of 0.5.", "stability": "volatile", "type": "gauge" }, "kv_ep_concurrent_pagers": { - "added": "7.0.0", - "help": "Number of eviction pager tasks to create when memory usage is high", + "added": "7.1.0", + "config_param": true, + "help": "Number of eviction pager tasks to create when memory usage is high. Configuration parameter with a default value of 2.", "stability": "volatile", "type": "gauge" }, "kv_ep_connection_cleanup_interval": { - "added": "7.0.0", - "help": "How often connection manager task should release dead connections (in seconds).", + "added": "7.6.0", + "config_param": true, + "help": "How often connection manager task should release dead connections (in seconds). Configuration parameter with a default value of 1.", "stability": "volatile", "type": "gauge" }, "kv_ep_connection_manager_interval": { "added": "7.0.0", - "help": "How often connection manager task should be run (in seconds).", + "config_param": true, + "help": "How often connection manager task should be run (in seconds). Configuration parameter with a default value of 0.1.", "stability": "volatile", "type": "gauge" }, @@ -850,44 +884,51 @@ "unit": "seconds" }, "kv_ep_continuous_backup_enabled": { - "added": "7.0.0", - "help": "True if continouous backup is enabled.", + "added": "8.0.0", + "config_param": true, + "help": "True if continouous backup is enabled. Configuration parameter with a default value of false.", "stability": "volatile", "type": "gauge" }, "kv_ep_continuous_backup_interval": { - "added": "7.0.0", - "help": "The continouous backup interval (in seconds).", + "added": "8.0.0", + "config_param": true, + "help": "The continouous backup interval (in seconds). Configuration parameter with a default value of 120.", "stability": "volatile", "type": "gauge" }, "kv_ep_couchstore_midpoint_rollback_optimisation": { - "added": "7.0.0", - "help": "Should we have to rollback more than half of the seqnos seen by this vBucket we will instead rollback to 0 and re-stream from the active if set to true", + "added": "7.1.0", + "config_param": true, + "help": "Should we have to rollback more than half of the seqnos seen by this vBucket we will instead rollback to 0 and re-stream from the active if set to true. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, "kv_ep_couchstore_mprotect": { "added": "7.0.0", - "help": "Enable couchstore to mprotect the iobuffer", + "config_param": true, + "help": "Enable couchstore to mprotect the iobuffer. Configuration parameter with a default value of false.", "stability": "volatile", "type": "gauge" }, "kv_ep_couchstore_tracing": { "added": "7.0.0", - "help": "Enable couchstore tracing", + "config_param": true, + "help": "Enable couchstore tracing. Configuration parameter with a default value of false.", "stability": "volatile", "type": "gauge" }, "kv_ep_couchstore_write_validation": { "added": "7.0.0", - "help": "Validate couchstore writes ", + "config_param": true, + "help": "Validate couchstore writes . Configuration parameter with a default value of false.", "stability": "volatile", "type": "gauge" }, "kv_ep_cross_bucket_ht_quota_sharing": { - "added": "7.0.0", - "help": "Allow this Bucket's HashTable quota to be shared with other Buckets which have this setting enabled.", + "added": "7.6.0", + "config_param": true, + "help": "Allow this Bucket's HashTable quota to be shared with other Buckets which have this setting enabled. Configuration parameter with a default value of false.", "stability": "volatile", "type": "gauge" }, @@ -905,7 +946,8 @@ }, "kv_ep_data_traffic_enabled": { "added": "7.0.0", - "help": "True if we want to enable data traffic after warmup is complete", + "config_param": true, + "help": "True if we want to enable data traffic after warmup is complete. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, @@ -943,6 +985,12 @@ "type": "gauge", "unit": "seconds" }, + "kv_ep_db_max_format_version": { + "added": "8.1.0", + "help": "Maximum disk format version supported by the bucket's storage engine", + "stability": "committed", + "type": "gauge" + }, "kv_ep_db_prepare_size_bytes": { "added": "7.0.0", "help": "Total size of SyncWrite prepares in db files", @@ -956,63 +1004,108 @@ "stability": "committed", "type": "gauge" }, + "kv_ep_dcp_active_stream_inline_checkpoint_item_limit": { + "added": "8.1.0", + "config_param": true, + "help": "When the number of items pending for an ActiveStream's cursor is less than or equal to this value, the stream will run checkpoint item extraction inline rather than scheduling the ActiveStreamCheckpointProcessorTask. This avoids task-scheduling latency for small numbers of items. Set to 0 to always schedule the task. Configuration parameter with a default value of 50.", + "stability": "volatile", + "type": "gauge" + }, + "kv_ep_dcp_backfill_antilag_interval": { + "added": "8.1.0", + "config_param": true, + "help": "Send SeqnoAdvanced on the n-th filtered item. Disabled if set to 0. Configuration parameter with a default value of 10000.", + "stability": "volatile", + "type": "gauge" + }, "kv_ep_dcp_backfill_byte_drain_ratio": { - "added": "7.0.0", - "help": "What ratio of the dcp_backfill_byte_limit must be drained for un-pausing a paused backfill", + "added": "7.6.2", + "config_param": true, + "help": "What ratio of the dcp_backfill_byte_limit must be drained for un-pausing a paused backfill. Configuration parameter with a default value of 0.25.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_backfill_byte_limit": { "added": "7.0.0", - "help": "Max bytes a connection can backfill into memory before backfill is paused", + "config_param": true, + "help": "Max bytes a connection can backfill into memory before backfill is paused. Configuration parameter with a default value of 20 * 1024 * 1024.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_backfill_idle_disk_threshold": { - "added": "7.0.0", - "help": "The percentage of disk usage at which DCP backfills would be ended when no progress is made for dcp_backfill_idle_limit_seconds", + "added": "8.0.0", + "config_param": true, + "help": "The percentage of disk usage at which DCP backfills would be ended when no progress is made for dcp_backfill_idle_limit_seconds. Configuration parameter with a default value of 90.0.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_backfill_idle_limit_seconds": { - "added": "7.0.0", - "help": "How long (in seconds) a DCP Backfill can be held open with no progress being made. When this limit is exceeded the stream is force ended (reason Slow) releasing the resources being held by the stream. The default value is 2x of dcp_idle_timeout", + "added": "8.0.0", + "config_param": true, + "help": "How long (in seconds) a DCP Backfill can be held open with no progress being made. When this limit is exceeded the stream is force ended (reason Slow) releasing the resources being held by the stream. The default value is 2x of dcp_idle_timeout. Configuration parameter with a default value of 720.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_backfill_idle_protection_enabled": { - "added": "7.0.0", - "help": "When true, DCP backfills will be checked for progress. Any backfill which makes no progress for dcp_backfill_idle_limit_seconds will be subject to further checks. If the directory referenced by dbname is over dcp_backfill_idle_disk_threshold percent used and cancelling the backfill will free disk space, the scan cancels and the associated DCP stream will end with reason slow.", + "added": "8.0.0", + "config_param": true, + "help": "When true, DCP backfills will be checked for progress. Any backfill which makes no progress for dcp_backfill_idle_limit_seconds will be subject to further checks. If the directory referenced by dbname is over dcp_backfill_idle_disk_threshold percent used and cancelling the backfill will free disk space, the scan cancels and the associated DCP stream will end with reason slow. This is ignored for emphemeral buckets. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_backfill_in_progress_per_connection_limit": { - "added": "7.0.0", - "help": "The maximum number of backfills each connection can have in-progress (i.e. KVStore snapshot open and reading data from)", + "added": "7.2.1", + "config_param": true, + "help": "The maximum number of backfills each connection can have in-progress (i.e. KVStore snapshot open and reading data from). Configuration parameter with a default value of 64.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_backfill_run_duration_limit": { - "added": "7.0.0", - "help": "Maximum time (in ms) backfill task will run before yielding.", + "added": "8.0.0", + "config_param": true, + "help": "Maximum time (in ms) backfill task will run before yielding. Configuration parameter with a default value of 300.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_cache_transfer_enabled": { - "added": "7.0.0", - "help": "Does the producer support a cache transfer?", + "added": "8.1.0", + "config_param": true, + "help": "Does the producer support a cache transfer?. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_cache_transfer_high_memory_backoff_duration": { - "added": "7.0.0", - "help": "The duration in floating point seconds to snooze the CacheTransferTask when memory pressure is present and the task should back-off to avoid making memory usage worse (and wait for memory to reduce).", + "added": "8.1.0", + "config_param": true, + "help": "The duration in floating point seconds to snooze the CacheTransferTask when memory pressure is present and the task should back-off to avoid making memory usage worse (and wait for memory to reduce). Configuration parameter with a default value of 0.01.", + "stability": "volatile", + "type": "gauge" + }, + "kv_ep_dcp_cache_transfer_ht_presize": { + "added": "8.1.0", + "config_param": true, + "help": "When true, a vbucket created from a snapshot (FBR/Fusion) that is eligible for a DCP cache transfer has its HashTable sized up-front from the snapshot's item count, avoiding repeated resizing while the transfer populates the table. Configuration parameter with a default value of true.", + "stability": "volatile", + "type": "gauge" + }, + "kv_ep_dcp_cache_transfer_max_batch_bytes": { + "added": "8.1.0", + "config_param": true, + "help": "Maximum batch size in bytes when batching cache transfer items. A batch is limited by this or dcp_cache_transfer_max_batch_items, whichever is reached first. At least 1 item will included in a batch, so a value of 0 is always 1 item per batch. Configuration parameter with a default value of 1440.", + "stability": "volatile", + "type": "gauge" + }, + "kv_ep_dcp_cache_transfer_max_batch_items": { + "added": "8.1.0", + "config_param": true, + "help": "Maximum number of items when batching cache transfer items. A batch is limited by this or dcp_cache_transfer_max_batch_bytes, whichever is reached first. At least 1 item will included in a batch, so a value of 0 is always 1 item per batch. Configuration parameter with a default value of 20.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_cache_transfer_one_visit_per_step": { - "added": "7.0.0", - "help": "Set to false for simpler unit-testing. When true CacheTransferTask to visit the hash table many times per step, when false one item per task step", + "added": "8.1.0", + "config_param": true, + "help": "Set to false for simpler unit-testing. When true CacheTransferTask to visit the hash table many times per step, when false one item per task step. Configuration parameter with a default value of false.", "stability": "volatile", "type": "gauge" }, @@ -1024,200 +1117,240 @@ "unit": "bytes" }, "kv_ep_dcp_cache_transfer_visit_duration_ms": { - "added": "7.0.0", - "help": "The upper bound in milliseconds for the CacheTransferTask hash table visitor", + "added": "8.1.0", + "config_param": true, + "help": "The upper bound in milliseconds for the CacheTransferTask hash table visitor. Configuration parameter with a default value of 25.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_checkpoint_dequeue_limit": { - "added": "7.0.0", - "help": "The limit given to CheckpointManager::getNextItemsForDcp by ActiveStream", + "added": "8.0.0", + "config_param": true, + "help": "The limit given to CheckpointManager::getNextItemsForDcp by ActiveStream. Configuration parameter with a default value of std::numeric_limits::max().", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_consumer_buffer_ratio": { - "added": "7.0.0", - "help": "Ratio of the BucketQuota that can be allocated by all DCP consumers for buffered messages", + "added": "7.6.0", + "config_param": true, + "help": "Ratio of the BucketQuota that can be allocated by all DCP consumers for buffered messages. Configuration parameter with a default value of 0.05.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_consumer_flow_control_ack_ratio": { - "added": "7.0.0", - "help": "Ratio of freed bytes in the DCP Consumer buffer that triggers a BufferAck message to the Producer", + "added": "7.6.0", + "config_param": true, + "help": "Ratio of freed bytes in the DCP Consumer buffer that triggers a BufferAck message to the Producer. Configuration parameter with a default value of 0.2.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_consumer_flow_control_ack_seconds": { - "added": "7.0.0", - "help": "Max seconds after which a Consumer acks all the remaining freed bytes, regardless of whether dcp_consumer_flow_control_ack_ratio has kicked-in or not", + "added": "7.6.0", + "config_param": true, + "help": "Max seconds after which a Consumer acks all the remaining freed bytes, regardless of whether dcp_consumer_flow_control_ack_ratio has kicked-in or not. Configuration parameter with a default value of 5.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_consumer_flow_control_enabled": { - "added": "7.0.0", - "help": "Whether DCP Consumer on this node enable flow control", + "added": "7.6.0", + "config_param": true, + "help": "Whether DCP Consumer on this node enable flow control. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_enable_noop": { "added": "7.0.0", - "help": "Whether DCP Consumer connections should attempt to negotiate no-ops with the Producer", + "config_param": true, + "help": "Whether DCP Consumer connections should attempt to negotiate no-ops with the Producer. Configuration parameter with a default value of true.", + "stability": "volatile", + "type": "gauge" + }, + "kv_ep_dcp_flow_control_buffer_size": { + "added": "8.1.0", + "config_param": true, + "help": "If non-zero, overrides the computed per-consumer DCP flow control buffer size with this absolute value (bytes), bypassing the dcp_consumer_buffer_ratio/BucketQuota calculation. A value of 0 disables the override and restores the ratio-based sizing. Configuration parameter with a default value of 0.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_idle_timeout": { "added": "7.0.0", - "help": "The maximum number of seconds between dcp messages before a connection is disconnected", + "config_param": true, + "help": "The maximum number of seconds between dcp messages before a connection is disconnected. Configuration parameter with a default value of 360.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_min_compression_ratio": { "added": "7.0.0", - "help": "Compression ratio to be achieved above which producer will ship documents as is", + "config_param": true, + "help": "Compression ratio to be achieved above which producer will ship documents as is. Configuration parameter with a default value of 0.85.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_noop_mandatory_for_v5_features": { "added": "7.0.0", - "help": "Forces clients to enable noop for v5 features", + "config_param": true, + "help": "Forces clients to enable noop for v5 features. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_noop_tx_interval": { "added": "7.0.0", - "help": "The time interval in seconds between noop messages being sent to the consumer", + "config_param": true, + "help": "The time interval in seconds between noop messages being sent to the consumer. Configuration parameter with a default value of 0.1.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_oso_backfill_large_value_ratio": { - "added": "7.0.0", - "help": "When considering out-of-seqno order (OSO) DCP backfill for a collection with 'large' mean value size, OSO will only be selected if the backfilling collection item count is less than this fraction of the vBucket item count. Whether the collection has 'small' or 'large' items depends on the value of 'dcp_oso_backfill_mean_item_size_threshold'.", + "added": "7.2.1", + "config_param": true, + "help": "When considering out-of-seqno order (OSO) DCP backfill for a collection with 'large' mean value size, OSO will only be selected if the backfilling collection item count is less than this fraction of the vBucket item count. Whether the collection has 'small' or 'large' items depends on the value of 'dcp_oso_backfill_mean_item_size_threshold'. Configuration parameter with a default value of 0.04.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_oso_backfill_small_item_size_threshold": { - "added": "7.0.0", - "help": "When considering out-of-seqno order (OSO) DCP backfill for a collection, should the items in the collection be considered 'small' or 'large' and subsequently should backfill use 'dcp_oso_backfill_small_value_collection_ratio' or 'dcp_oso_backfill_large_value_collection_ratio' when deciding ot use OSO or not? Collections whoss mean on-disk value size is less than this parameter are considered 'small', otherwise they are considered 'large'", + "added": "7.2.1", + "config_param": true, + "help": "When considering out-of-seqno order (OSO) DCP backfill for a collection, should the items in the collection be considered 'small' or 'large' and subsequently should backfill use 'dcp_oso_backfill_small_value_collection_ratio' or 'dcp_oso_backfill_large_value_collection_ratio' when deciding ot use OSO or not? Collections whoss mean on-disk value size is less than this parameter are considered 'small', otherwise they are considered 'large'. Configuration parameter with a default value of 256.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_oso_backfill_small_value_ratio": { - "added": "7.0.0", - "help": "When considering out-of-seqno order (OSO) DCP backfill for a collection with 'small' mean value size, OSO will only be selected if the backfilling collection item count is less than this fraction of the vBucket item count. Whether the collection has 'small' or 'large' items depends on the value of 'dcp_oso_backfill_mean_item_size_threshold'.", + "added": "7.2.1", + "config_param": true, + "help": "When considering out-of-seqno order (OSO) DCP backfill for a collection with 'small' mean value size, OSO will only be selected if the backfilling collection item count is less than this fraction of the vBucket item count. Whether the collection has 'small' or 'large' items depends on the value of 'dcp_oso_backfill_mean_item_size_threshold'. Configuration parameter with a default value of 0.005.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_oso_max_collections_per_backfill": { - "added": "7.0.0", - "help": "This is the maximum number of collections that a DCP stream can be filtering to be eligible for OSO", + "added": "7.6.0", + "config_param": true, + "help": "This is the maximum number of collections that a DCP stream can be filtering to be eligible for OSO. Configuration parameter with a default value of 1.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_producer_catch_exceptions": { - "added": "7.0.0", - "help": "If true, ActiveStream will catch exceptions during item processing and close the stream's related connection (and thus all streams for that connection). If false, exception will be re-thrown.", + "added": "7.2.8", + "config_param": true, + "help": "If true, ActiveStream will catch exceptions during item processing and close the stream's related connection (and thus all streams for that connection). If false, exception will be re-thrown. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_producer_processor_run_duration_us": { - "added": "7.0.0", - "help": "The approximate maximum runtime in microseconds for ActiveStreamCheckpointProcessorTask", + "added": "7.2.5-MP1", + "config_param": true, + "help": "The approximate maximum runtime in microseconds for ActiveStreamCheckpointProcessorTask. Configuration parameter with a default value of 100000.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_producer_snapshot_marker_yield_limit": { "added": "7.0.0", - "help": "Not in use - replaced by dcp_producer_processor_run_duration_us", + "config_param": true, + "help": "Not in use - replaced by dcp_producer_processor_run_duration_us. Configuration parameter with a default value of 0.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_scan_byte_limit": { "added": "7.0.0", - "help": "Max bytes that can be read in a single backfill scan before yielding", + "config_param": true, + "help": "Max bytes that can be read in a single backfill scan before yielding. Configuration parameter with a default value of 4 * 1024 * 1024.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_scan_item_limit": { "added": "7.0.0", - "help": "Max items that can be read in a single backfill scan before yielding", + "config_param": true, + "help": "Max items that can be read in a single backfill scan before yielding. Configuration parameter with a default value of 4096.", "stability": "volatile", "type": "gauge" }, "kv_ep_dcp_takeover_max_time": { "added": "7.0.0", - "help": "Max amount of time for takeover send (in seconds) after which front end ops would return ETMPFAIL", + "config_param": true, + "help": "Max amount of time for takeover send (in seconds) after which front end ops would return ETMPFAIL. Configuration parameter with a default value of 60.", "stability": "volatile", "type": "gauge" }, "kv_ep_defragmenter_age_threshold": { "added": "7.0.0", - "help": "How old (measured in number of DefragmenterVisitor passes) must a document be to be considered for defragmentation.", + "config_param": true, + "help": "How old (measured in number of DefragmenterVisitor passes) must a document be to be considered for defragmentation. Configuration parameter with a default value of 1.", "stability": "volatile", "type": "gauge" }, "kv_ep_defragmenter_auto_lower_threshold": { - "added": "7.0.0", - "help": "When mode is not static and scored fragmentation is above this value, a sleep time between defragmenter_auto_min_sleep and defragmenter_auto_max_sleep will be used", + "added": "7.0.2", + "config_param": true, + "help": "When mode is not static and scored fragmentation is above this value, a sleep time between defragmenter_auto_min_sleep and defragmenter_auto_max_sleep will be used. Configuration parameter with a default value of 0.07.", "stability": "volatile", "type": "gauge" }, "kv_ep_defragmenter_auto_max_sleep": { - "added": "7.0.0", - "help": "The maximum sleep that the auto controller can set", + "added": "7.0.2", + "config_param": true, + "help": "The maximum sleep that the auto controller can set. Configuration parameter with a default value of 10.0.", "stability": "volatile", "type": "gauge" }, "kv_ep_defragmenter_auto_min_sleep": { - "added": "7.0.0", - "help": "The minimum sleep that the auto controller can set", + "added": "7.0.2", + "config_param": true, + "help": "The minimum sleep that the auto controller can set. Configuration parameter with a default value of 0.6.", "stability": "volatile", "type": "gauge" }, "kv_ep_defragmenter_auto_pid_d": { - "added": "7.0.0", - "help": "The d term for the PID controller", + "added": "7.0.2", + "config_param": true, + "help": "The d term for the PID controller. Configuration parameter with a default value of 0.0.", "stability": "volatile", "type": "gauge" }, "kv_ep_defragmenter_auto_pid_dt": { - "added": "7.0.0", - "help": "The dt (interval) term for the PID controller. Value represents milliseconds", + "added": "7.0.2", + "config_param": true, + "help": "The dt (interval) term for the PID controller. Value represents milliseconds. Configuration parameter with a default value of 30000.", "stability": "volatile", "type": "gauge" }, "kv_ep_defragmenter_auto_pid_i": { - "added": "7.0.0", - "help": "The i term for the PID controller", + "added": "7.0.2", + "config_param": true, + "help": "The i term for the PID controller. Configuration parameter with a default value of 0.0000197.", "stability": "volatile", "type": "gauge" }, "kv_ep_defragmenter_auto_pid_p": { - "added": "7.0.0", - "help": "The p term for the PID controller", + "added": "7.0.2", + "config_param": true, + "help": "The p term for the PID controller. Configuration parameter with a default value of 52.", "stability": "volatile", "type": "gauge" }, "kv_ep_defragmenter_auto_upper_threshold": { - "added": "7.0.0", - "help": "When mode is auto_linear and scored fragmentation is above this value, the defragmenter will use defragmenter_auto_min_sleep", + "added": "7.0.2", + "config_param": true, + "help": "When mode is auto_linear and scored fragmentation is above this value, the defragmenter will use defragmenter_auto_min_sleep. Configuration parameter with a default value of 0.25.", "stability": "volatile", "type": "gauge" }, "kv_ep_defragmenter_chunk_duration": { "added": "7.0.0", - "help": "Maximum time (in ms) defragmentation task will run for before being paused (and resumed at the next defragmenter_interval).", + "config_param": true, + "help": "Maximum time (in ms) defragmentation task will run for before being paused (and resumed at the next defragmenter_interval). Configuration parameter with a default value of 20.", "stability": "volatile", "type": "gauge" }, "kv_ep_defragmenter_enabled": { "added": "7.0.0", - "help": "True if defragmenter task is enabled", + "config_param": true, + "help": "True if defragmenter task is enabled. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, "kv_ep_defragmenter_interval": { "added": "7.0.0", - "help": "How often defragmenter task should be run (in seconds).", + "config_param": true, + "help": "How often defragmenter task should be run (in seconds). Configuration parameter with a default value of 10.0.", "stability": "volatile", "type": "gauge" }, @@ -1242,7 +1375,8 @@ }, "kv_ep_defragmenter_stored_value_age_threshold": { "added": "7.0.0", - "help": "How old (measured in number of DefragmenterVisitor passes) must a StoredValue be to be considered for defragmentation.", + "config_param": true, + "help": "How old (measured in number of DefragmenterVisitor passes) must a StoredValue be to be considered for defragmentation. Configuration parameter with a default value of 1.", "stability": "volatile", "type": "gauge" }, @@ -1290,56 +1424,65 @@ "type": "gauge" }, "kv_ep_ephemeral_mem_recovery_enabled": { - "added": "7.0.0", - "help": "Whether ephemeral memory recovery task is enabled. If disabled, ItemPager will carry out memory recovery for AutoDelete buckets.", + "added": "8.0.0", + "config_param": true, + "help": "Whether ephemeral memory recovery task is enabled. If disabled, ItemPager will carry out memory recovery for AutoDelete buckets. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, "kv_ep_ephemeral_mem_recovery_sleep_time": { - "added": "7.0.0", - "help": "Duration in milliseconds the EphemeralMemRecovery task will sleep between periodic executions.", + "added": "8.0.0", + "config_param": true, + "help": "Duration in milliseconds the EphemeralMemRecovery task will sleep between periodic executions. Configuration parameter with a default value of 5000.", "stability": "volatile", "type": "gauge" }, "kv_ep_ephemeral_metadata_mark_stale_chunk_duration": { "added": "7.0.0", - "help": "Maximum time (in ms) ephemeral hash table cleaner task will run for before being paused (and resumed at the next ephemeral_metadata_purge_interval).", + "config_param": true, + "help": "Maximum time (in ms) ephemeral hash table cleaner task will run for before being paused (and resumed at the next ephemeral_metadata_purge_interval). Configuration parameter with a default value of 20.", "stability": "volatile", "type": "gauge" }, "kv_ep_ephemeral_metadata_purge_age": { "added": "7.0.0", - "help": "Age in seconds after which Ephemeral metadata is purged entirely from memory. Purging disabled if set to -1.", + "config_param": true, + "help": "Age in seconds after which Ephemeral metadata is purged entirely from memory. Purging disabled if set to -1. Configuration parameter with a default value of 60.", "stability": "volatile", "type": "gauge" }, "kv_ep_ephemeral_metadata_purge_interval": { "added": "7.0.0", - "help": "Time in seconds between automatic, periodic runs of the Ephemeral metadata purge task. Periodic purging disabled if set to 0.", + "config_param": true, + "help": "Time in seconds between automatic, periodic runs of the Ephemeral metadata purge task. Periodic purging disabled if set to 0. Configuration parameter with a default value of 60.", "stability": "volatile", "type": "gauge" }, "kv_ep_ephemeral_metadata_purge_stale_chunk_duration": { "added": "7.0.0", - "help": "Maximum time (in ms) ephemeral stale metadata purge task will run for before being paused (and resumed at the next ephemeral_metadata_purge_interval).", + "config_param": true, + "help": "Maximum time (in ms) ephemeral stale metadata purge task will run for before being paused (and resumed at the next ephemeral_metadata_purge_interval). Configuration parameter with a default value of 20.", "stability": "volatile", "type": "gauge" }, "kv_ep_exp_pager_enabled": { "added": "7.0.0", - "help": "True if expiry pager task is enabled", + "config_param": true, + "help": "True if expiry pager task is enabled. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, "kv_ep_exp_pager_initial_run_time": { "added": "7.0.0", - "help": "Hour in GMT time when expiry pager can be scheduled for initial run", + "config_param": true, + "help": "Hour in GMT time when expiry pager can be scheduled for initial run. Configuration parameter with a default value of -1.", "stability": "volatile", "type": "gauge" }, "kv_ep_exp_pager_stime": { "added": "7.0.0", - "help": "Number of seconds between expiry pager runs.", + "config_param": true, + "help": "Number of seconds between expiry pager runs. Configuration parameter with a default value of 600.", "stability": "volatile", "type": "gauge" }, @@ -1362,32 +1505,37 @@ "type": "gauge" }, "kv_ep_expiry_pager_concurrency": { - "added": "7.0.0", - "help": "Number of tasks which are created to scan for and delete expired items", + "added": "7.1.0", + "config_param": true, + "help": "Number of tasks which are created to scan for and delete expired items. Configuration parameter with a default value of 2.", "stability": "volatile", "type": "gauge" }, "kv_ep_expiry_visitor_expire_after_visit_duration_ms": { - "added": "7.0.0", - "help": "The time limit in milliseconds for processing expired items after visiting hash tables. After finding expired items during hash table traversal, the expiry pager will process them until this duration is reached before yielding.", + "added": "8.0.0", + "config_param": true, + "help": "The time limit in milliseconds for processing expired items after visiting hash tables. After finding expired items during hash table traversal, the expiry pager will process them until this duration is reached before yielding. Configuration parameter with a default value of 25.", "stability": "volatile", "type": "gauge" }, "kv_ep_expiry_visitor_items_only_duration_ms": { - "added": "7.0.0", - "help": "The time limit in milliseconds for processing items from the expired items list at the start of an expiry pager run. If the expired items list is not empty when the pager starts, it will only process items from this list for up to this duration before yielding.", + "added": "8.0.0", + "config_param": true, + "help": "The time limit in milliseconds for processing items from the expired items list at the start of an expiry pager run. If the expired items list is not empty when the pager starts, it will only process items from this list for up to this duration before yielding. Configuration parameter with a default value of 50.", "stability": "volatile", "type": "gauge" }, "kv_ep_failpartialwarmup": { "added": "7.0.0", - "help": "If true then do not allow traffic to be enabled to the bucket if warmup didn't complete successfully", + "config_param": true, + "help": "If true then do not allow traffic to be enabled to the bucket if warmup didn't complete successfully. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, "kv_ep_flush_batch_max_bytes": { - "added": "7.0.0", - "help": "Max size (in bytes) of a single flush-batch passed to the KVStore for persistence.", + "added": "7.6.1", + "config_param": true, + "help": "Max size (in bytes) of a single flush-batch passed to the KVStore for persistence. Configuration parameter with a default value of 2147483648.", "stability": "volatile", "type": "gauge" }, @@ -1406,19 +1554,22 @@ }, "kv_ep_flusher_total_batch_limit": { "added": "7.0.0", - "help": "Number of items that all flushers can be currently flushing. Each flusher has flusher_total_batch_limit / num_writer_threads individual batch size. Individual batches may be larger than this value, as we cannot split Memory checkpoints across multiple commits.", + "config_param": true, + "help": "Number of items that all flushers can be currently flushing. Each flusher has flusher_total_batch_limit / num_writer_threads individual batch size. Individual batches may be larger than this value, as we cannot split Memory checkpoints across multiple commits. Configuration parameter with a default value of 4000000.", "stability": "volatile", "type": "gauge" }, "kv_ep_freq_counter_increment_factor": { - "added": "7.0.0", - "help": "The increment factor of the ProbabilisticCounter being used for the frequency counter. The default value of 0.012 is set such that it allows an 8-bit ProbabilisticCounter to mimic a uint16 counter. See the comment on the ProbabilisticCounter class for more information.", + "added": "7.6.0", + "config_param": true, + "help": "The increment factor of the ProbabilisticCounter being used for the frequency counter. The default value of 0.012 is set such that it allows an 8-bit ProbabilisticCounter to mimic a uint16 counter. See the comment on the ProbabilisticCounter class for more information. Configuration parameter with a default value of 0.012.", "stability": "volatile", "type": "gauge" }, "kv_ep_fsync_after_every_n_bytes_written": { "added": "7.0.0", - "help": "Perform a file sync() operation after every N bytes written. Disabled if set to 0.", + "config_param": true, + "help": "Perform a file sync() operation after every N bytes written. Disabled if set to 0. Configuration parameter with a default value of 1048576.", "stability": "volatile", "type": "gauge" }, @@ -1597,6 +1748,13 @@ "type": "counter", "unit": "count" }, + "kv_ep_fusion_pending_upload_bytes_bytes": { + "added": "8.1.0", + "help": "Total amount of data not yet synced to FusionLogStore.", + "stability": "committed", + "type": "gauge", + "unit": "bytes" + }, "kv_ep_fusion_sync_attempts": { "added": "8.1.0", "help": "Total number of sync attempts to Fusion including both successful and failed attempts.", @@ -1641,37 +1799,43 @@ }, "kv_ep_getl_default_timeout": { "added": "7.0.0", - "help": "The default timeout for a getl lock in (s)", + "config_param": true, + "help": "The default timeout for a getl lock in (s). Configuration parameter with a default value of 15.", "stability": "volatile", "type": "gauge" }, "kv_ep_getl_max_timeout": { "added": "7.0.0", - "help": "The maximum timeout for a getl lock in (s)", + "config_param": true, + "help": "The maximum timeout for a getl lock in (s). Configuration parameter with a default value of 30.", "stability": "volatile", "type": "gauge" }, "kv_ep_history_retention_bytes": { - "added": "7.0.0", - "help": "Max bytes of history a bucket should aim to retain on disk.", + "added": "7.2.0", + "config_param": true, + "help": "Max bytes of history a bucket should aim to retain on disk. Configuration parameter with a default value of 0.", "stability": "volatile", "type": "gauge" }, "kv_ep_history_retention_seconds": { - "added": "7.0.0", - "help": "Seconds of history the bucket should aim to retain on disk.", + "added": "7.2.0", + "config_param": true, + "help": "Seconds of history the bucket should aim to retain on disk. Configuration parameter with a default value of 0.", "stability": "volatile", "type": "gauge" }, "kv_ep_hlc_drift_ahead_threshold_us": { "added": "7.0.0", - "help": "The μs threshold of drift at which we will increment a vbucket's ahead counter.", + "config_param": true, + "help": "The μs threshold of drift at which we will increment a vbucket's ahead counter. Configuration parameter with a default value of 5000000.", "stability": "volatile", "type": "gauge" }, "kv_ep_hlc_drift_behind_threshold_us": { "added": "7.0.0", - "help": "The μs threshold of drift at which we will increment a vbucket's behind counter.", + "config_param": true, + "help": "The μs threshold of drift at which we will increment a vbucket's behind counter. Configuration parameter with a default value of 5000000.", "stability": "volatile", "type": "gauge" }, @@ -1689,8 +1853,9 @@ "unit": "seconds" }, "kv_ep_hlc_max_future_threshold_us": { - "added": "7.0.0", - "help": "The acceptable μs threshold of drift at which we accept a new cas value.", + "added": "8.0.0", + "config_param": true, + "help": "The acceptable μs threshold of drift at which we accept a new cas value. Configuration parameter with a default value of 3900000000.", "stability": "volatile", "type": "gauge" }, @@ -1703,31 +1868,36 @@ }, "kv_ep_ht_locks": { "added": "7.0.0", - "help": "", + "config_param": true, + "help": " Configuration parameter with a default value of 47.", "stability": "volatile", "type": "gauge" }, "kv_ep_ht_resize_interval": { "added": "7.0.0", - "help": "Interval in seconds to wait between HashtableResizerTask executions.", + "config_param": true, + "help": "Interval in seconds to wait between HashtableResizerTask executions. Configuration parameter with a default value of 1.", "stability": "volatile", "type": "gauge" }, "kv_ep_ht_size": { "added": "7.0.0", - "help": "The initial and minimum number of slots in HashTable objects.", + "config_param": true, + "help": "The initial and minimum number of slots in HashTable objects. Configuration parameter with a default value of 47.", "stability": "volatile", "type": "gauge" }, "kv_ep_ht_size_decrease_delay": { - "added": "7.0.0", - "help": "Delay in seconds before decreasing the size of a HashTable following a previous resize.", + "added": "8.0.0", + "config_param": true, + "help": "Delay in seconds before decreasing the size of a HashTable following a previous resize. Configuration parameter with a default value of 60.", "stability": "volatile", "type": "gauge" }, "kv_ep_ht_temp_items_allowed_percent": { - "added": "7.0.0", - "help": "", + "added": "7.6.10", + "config_param": true, + "help": " Configuration parameter with a default value of 10.", "stability": "volatile", "type": "gauge" }, @@ -1786,13 +1956,15 @@ }, "kv_ep_item_compressor_chunk_duration": { "added": "7.0.0", - "help": "Maximum time (in ms) item compression task will run for before being paused (and resumed at the next item_compressor_interval).", + "config_param": true, + "help": "Maximum time (in ms) item compression task will run for before being paused (and resumed at the next item_compressor_interval). Configuration parameter with a default value of 20.", "stability": "volatile", "type": "gauge" }, "kv_ep_item_compressor_interval": { "added": "7.0.0", - "help": "How often the item compressor task should run (in milliseconds)", + "config_param": true, + "help": "How often the item compressor task should run (in milliseconds). Configuration parameter with a default value of 250.", "stability": "volatile", "type": "gauge" }, @@ -1810,25 +1982,29 @@ }, "kv_ep_item_eviction_age_percentage": { "added": "7.0.0", - "help": "The age percentage used when determining the age threshold in the learning_age_and_mfu eviction policy.", + "config_param": true, + "help": "The age percentage used when determining the age threshold in the learning_age_and_mfu eviction policy. Configuration parameter with a default value of 30.", "stability": "volatile", "type": "gauge" }, "kv_ep_item_eviction_freq_counter_age_threshold": { "added": "7.0.0", - "help": "", + "config_param": true, + "help": " Configuration parameter with a default value of 1.", "stability": "volatile", "type": "gauge" }, "kv_ep_item_eviction_initial_mfu_percentile": { - "added": "7.0.0", - "help": "Percentile of existing item MFU distribution to use to determine the MFU to give to new items (0 would insert new items with MFU equal to that of the coldest item present; 100 to that of hottest item present) for the upfront_mfu_only eviction strategy.", + "added": "7.6.0", + "config_param": true, + "help": "Percentile of existing item MFU distribution to use to determine the MFU to give to new items (0 would insert new items with MFU equal to that of the coldest item present; 100 to that of hottest item present) for the upfront_mfu_only eviction strategy. Configuration parameter with a default value of 25.", "stability": "volatile", "type": "gauge" }, "kv_ep_item_eviction_initial_mfu_update_interval": { - "added": "7.0.0", - "help": "Time between updates of the initial MFU given to new items (seconds)", + "added": "7.6.0", + "config_param": true, + "help": "Time between updates of the initial MFU given to new items (seconds). Configuration parameter with a default value of 1.", "stability": "volatile", "type": "gauge" }, @@ -1841,18 +2017,21 @@ "kv_ep_item_flush_failed": { "added": "7.0.0", "help": "Number of times an item failed to flush due to storage errors", + "long_description": "Provides the number of items that failed to be persisted to disk due to storage errors, typically disk I/O failures. The metric increments when a mutation (set/add/replace) cannot be written to disk, a deletion cannot be committed to disk, or the underlying storage engine returns an error during flush operations. Persistent failures indicate a serious storage problem that risks data loss and potential service disruption.", "stability": "committed", "type": "gauge" }, "kv_ep_item_freq_decayer_chunk_duration": { "added": "7.0.0", - "help": "Maximum time (in ms) itemFreqDecayer task will run for before being paused.", + "config_param": true, + "help": "Maximum time (in ms) itemFreqDecayer task will run for before being paused. Configuration parameter with a default value of 20.", "stability": "volatile", "type": "gauge" }, "kv_ep_item_freq_decayer_percent": { "added": "7.0.0", - "help": "The percent that the frequency counter of a document is decayed when visited by item_freq_decayer.", + "config_param": true, + "help": "The percent that the frequency counter of a document is decayed when visited by item_freq_decayer. Configuration parameter with a default value of 50.", "stability": "volatile", "type": "gauge" }, @@ -1921,16 +2100,42 @@ }, "kv_ep_magma_bloom_filter_accuracy": { "added": "7.0.0", - "help": "Magma maintains a bloom filter per sstable in the LSMTree. The bloom filters are used to reduce IO in case of non-existent This config sets the accuracy of the bloom filters ie. (1 - accuracy) = false positive rate", + "config_param": true, + "help": "Magma maintains a bloom filter per sstable in the LSMTree. The bloom filters are used to reduce IO in case of non-existent This config sets the accuracy of the bloom filters ie. (1 - accuracy) = false positive rate. Configuration parameter with a default value of 0.99.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_bloom_filter_accuracy_for_bottom_level": { "added": "7.0.0", - "help": "The bloom filters at the bottom level are used to avoid IO in case of non-existent keys. Also most of the data resides in the bottom level. This config allows for lowering of bloom filter accuracy of sstables residing in the lowermost level of the LSMTree.", + "config_param": true, + "help": "The bloom filters at the bottom level are used to avoid IO in case of non-existent keys. Also most of the data resides in the bottom level. This config allows for lowering of bloom filter accuracy of sstables residing in the lowermost level of the LSMTree. Configuration parameter with a default value of 0.95.", "stability": "volatile", "type": "gauge" }, + "kv_ep_magma_bloom_filter_cache_hits": { + "added": "8.1.0", + "help": "Number of bloom filter lookups served from the in-memory bloom filter cache", + "stability": "committed", + "type": "counter" + }, + "kv_ep_magma_bloom_filter_cache_misses": { + "added": "8.1.0", + "help": "Number of bloom filter lookups that required a disk read", + "stability": "committed", + "type": "counter" + }, + "kv_ep_magma_bloom_filter_lookups_hit": { + "added": "8.1.0", + "help": "Number of bloom filter lookups where the filter correctly indicated the key did not exist", + "stability": "committed", + "type": "counter" + }, + "kv_ep_magma_bloom_filter_lookups_miss": { + "added": "8.1.0", + "help": "Number of bloom filter lookups where the filter returned a false positive", + "stability": "committed", + "type": "counter" + }, "kv_ep_magma_bloom_filter_mem_used_bytes": { "added": "7.1.0", "help": "Bloom filter memory usage in all versions of the LSM Trees", @@ -1982,13 +2187,22 @@ }, "kv_ep_magma_checkpoint_interval": { "added": "7.0.0", - "help": "Frequency of checkpoint interval; in seconds. A checkpoint provides a rollback point to which the data store can rollback to in the event of a failure.", + "config_param": true, + "help": "Frequency of checkpoint interval; in seconds. A checkpoint provides a rollback point to which the data store can rollback to in the event of a failure. Configuration parameter with a default value of 120.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_checkpoint_threshold": { "added": "7.0.0", - "help": "Threshold of data written before a checkpoint is created; threshold is based on a fraction of the total data size. Checkpoints require data to be retained in order to provide rollback capability. If the amount of data written during a checkpoint interval is large, we need to do more frequent checkpoints to reduce space amplification.", + "config_param": true, + "help": "Threshold of data written before a checkpoint is created; threshold is based on a fraction of the total data size. Checkpoints require data to be retained in order to provide rollback capability. If the amount of data written during a checkpoint interval is large, we need to do more frequent checkpoints to reduce space amplification. Configuration parameter with a default value of 0.02.", + "stability": "volatile", + "type": "gauge" + }, + "kv_ep_magma_collection_high_seqno_refresh_interval": { + "added": "8.1.0", + "config_param": true, + "help": "Interval (in seconds) at which the cached per-collection persisted high seqnos are refreshed during a magma implicit compaction; 0 disables the periodic refresh. Configuration parameter with a default value of 60.", "stability": "volatile", "type": "gauge" }, @@ -1998,6 +2212,48 @@ "stability": "committed", "type": "gauge" }, + "kv_ep_magma_contbk_num_backup_hardlinks": { + "added": "8.1.0", + "help": "Number of file hardlinks that have occurred as a part of backup", + "stability": "committed", + "type": "counter", + "unit": "count" + }, + "kv_ep_magma_contbk_num_backups": { + "added": "8.1.0", + "help": "Number of Backups that have occurred", + "stability": "committed", + "type": "counter", + "unit": "count" + }, + "kv_ep_magma_contbk_num_bytes_written_bytes": { + "added": "8.1.0", + "help": "Number of bytes written as a part of the backup", + "stability": "committed", + "type": "gauge", + "unit": "bytes" + }, + "kv_ep_magma_contbk_num_failed_backups": { + "added": "8.1.0", + "help": "Number of Backups that have failed", + "stability": "committed", + "type": "counter", + "unit": "count" + }, + "kv_ep_magma_contbk_num_table_bytes_backedup_bytes": { + "added": "8.1.0", + "help": "Number of Table Bytes that have been backed up", + "stability": "committed", + "type": "gauge", + "unit": "bytes" + }, + "kv_ep_magma_contbk_num_write_ios": { + "added": "8.1.0", + "help": "Number of Write IOs that have occurred as a part of backup", + "stability": "committed", + "type": "counter", + "unit": "count" + }, "kv_ep_magma_corruption_errors": { "added": "8.1.0", "help": "Number of corruption errors detected by Magma", @@ -2032,13 +2288,15 @@ }, "kv_ep_magma_delete_frag_ratio": { "added": "7.0.0", - "help": "Magma compaction always removes duplicate keys but not all sstables are visited during compaction.. This is the minimum fragmentation ratio threshold for when a compaction will be triggerred.", + "config_param": true, + "help": "Magma compaction always removes duplicate keys but not all sstables are visited during compaction.. This is the minimum fragmentation ratio threshold for when a compaction will be triggerred. Configuration parameter with a default value of 0.50.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_delete_memtable_writecache": { "added": "7.0.0", - "help": "Magma uses a lazy update model to maintain the sequence index. It maintains a list of deleted seq #s that were deleted from the key Index.", + "config_param": true, + "help": "Magma uses a lazy update model to maintain the sequence index. It maintains a list of deleted seq #s that were deleted from the key Index. Configuration parameter with a default value of 8192.", "stability": "volatile", "type": "gauge" }, @@ -2050,61 +2308,71 @@ }, "kv_ep_magma_enable_block_cache": { "added": "7.0.0", - "help": "The block cache is an LRU policy driven cache that is used to maintain index blocks for the sstable's btrees.", + "config_param": true, + "help": "The block cache is an LRU policy driven cache that is used to maintain index blocks for the sstable's btrees. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_enable_data_block_autotuning": { - "added": "7.0.0", - "help": "Enable auto-tuning of data block size based on compression ratio.", + "added": "8.1.0", + "config_param": true, + "help": "Enable auto-tuning of data block size based on compression ratio. Configuration parameter with a default value of false.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_enable_direct_io": { "added": "7.0.0", - "help": "Using direct IO tells magma to bypass the file system cache when writing or reading sstables.", + "config_param": true, + "help": "Using direct IO tells magma to bypass the file system cache when writing or reading sstables. Configuration parameter with a default value of false.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_enable_group_commit": { - "added": "7.0.0", - "help": "Group Commit allows transactions in magma to be grouped together to reduce the number of WAL fsyncs. When a transaction is ready to fsync, if there are new transactions waiting to start, we stall the transaction waiting to fsync until there are no more transactions waiting to start for a given magma instance.", + "added": "7.1.0", + "config_param": true, + "help": "Group Commit allows transactions in magma to be grouped together to reduce the number of WAL fsyncs. When a transaction is ready to fsync, if there are new transactions waiting to start, we stall the transaction waiting to fsync until there are no more transactions waiting to start for a given magma instance. Configuration parameter with a default value of false.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_enable_index_block_autotuning": { - "added": "7.0.0", - "help": "Enable auto-tuning of index block size based on compression ratio.", + "added": "8.1.0", + "config_param": true, + "help": "Enable auto-tuning of index block size based on compression ratio. Configuration parameter with a default value of false.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_enable_memory_optimized_writes": { - "added": "7.0.0", - "help": "When enabled, if copying a write batch into memtable results in exceeding the write cache quota, Magma avoids the copy and instead flushes the batch to disk on the writer thread itself. This tradeoffs an increase in write latency for reduced memory consumption and obeys quota limits. If copying a batch keeps us under the quota, Magma will to continue to copy and do the flush in background.", + "added": "7.6.0", + "config_param": true, + "help": "When enabled, if copying a write batch into memtable results in exceeding the write cache quota, Magma avoids the copy and instead flushes the batch to disk on the writer thread itself. This tradeoffs an increase in write latency for reduced memory consumption and obeys quota limits. If copying a batch keeps us under the quota, Magma will to continue to copy and do the flush in background. Configuration parameter with a default value of false.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_enable_upsert": { "added": "7.0.0", - "help": "When true, the kv_engine will utilize Magma's upsert capabiltiy but accurate document counts for the data store or collections can not be maintained.", + "config_param": true, + "help": "When true, the kv_engine will utilize Magma's upsert capabiltiy but accurate document counts for the data store or collections can not be maintained. Configuration parameter with a default value of false.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_enable_wal": { - "added": "7.0.0", - "help": "WAL ensures Magma's atomicity, durability. Disabling it is useful in performance analysis.", + "added": "7.1.0", + "config_param": true, + "help": "WAL ensures Magma's atomicity, durability. Disabling it is useful in performance analysis. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_expiry_frag_threshold": { "added": "7.0.0", - "help": "All compactions perform expiry but not all sstables are visited by compaction. Magma maintains an expiry histogram across the kvstore to help determine which range of sstables need to have compaction run on them because there are a significant number of expired items. The frag threshold is the number of expired keys vs keys in the data store.", + "config_param": true, + "help": "All compactions perform expiry but not all sstables are visited by compaction. Magma maintains an expiry histogram across the kvstore to help determine which range of sstables need to have compaction run on them because there are a significant number of expired items. The frag threshold is the number of expired keys vs keys in the data store. Configuration parameter with a default value of 0.25.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_expiry_purger_interval": { "added": "7.0.0", - "help": "Magma maintains statistics about expired documents to run compaction based on magma_expiry_frag_threshold. This config determines the the expiry purger polling interval in seconds to trigger compaction on eligible sstables", + "config_param": true, + "help": "Magma maintains statistics about expired documents to run compaction based on magma_expiry_frag_threshold. This config determines the the expiry purger polling interval in seconds to trigger compaction on eligible sstables. Configuration parameter with a default value of 3600.", "stability": "volatile", "type": "gauge" }, @@ -2122,7 +2390,8 @@ }, "kv_ep_magma_fragmentation_percentage": { "added": "7.0.0", - "help": "The percentage of fragmentation a magma bucket aims to maintain. A 100 value will disable sequence tree compactions by setting the desired fragmentation percentage to 100%. Smaller compactions of the key and local indexes will still run.", + "config_param": true, + "help": "The percentage of fragmentation a magma bucket aims to maintain. A 100 value will disable sequence tree compactions by setting the desired fragmentation percentage to 100%. Smaller compactions of the key and local indexes will still run. Configuration parameter with a default value of 50.", "stability": "volatile", "type": "gauge" }, @@ -2134,38 +2403,44 @@ "unit": "ratio" }, "kv_ep_magma_fusion_logstore_fragmentation_threshold": { - "added": "7.0.0", - "help": "The threshold at which the fusion log store will perform garbage collection. This is a ratio between 0.0 and 1.0.", + "added": "8.0.0", + "config_param": true, + "help": "The threshold at which the fusion log store will perform garbage collection. This is a ratio between 0.0 and 1.0. Configuration parameter with a default value of 0.5.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_fusion_max_log_cleaning_size_ratio": { - "added": "7.0.0", - "help": "Minimum log segment size after which logs will be split. The ratio is a percentage of used log data size.", + "added": "8.1.0", + "config_param": true, + "help": "Minimum log segment size after which logs will be split. The ratio is a percentage of used log data size. Configuration parameter with a default value of 0.05.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_fusion_max_log_size": { - "added": "7.0.0", - "help": "Upper cap for the effective log size, in bytes.", + "added": "8.1.0", + "config_param": true, + "help": "Upper cap for the effective log size, in bytes. Configuration parameter with a default value of (1024 * 1024 * 1024).", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_fusion_max_num_log_files": { - "added": "7.0.0", - "help": "Base cap for maximum number of log files.", + "added": "8.1.0", + "config_param": true, + "help": "Base cap for maximum number of log files. Configuration parameter with a default value of 100.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_fusion_max_upload_interval": { - "added": "7.0.0", - "help": "Maximum interval after which an upload will be permitted irrespective of upload thresholds, in seconds. This will be set to the smaller of the current value and 25% of persistent_metadata_purge_age.", + "added": "8.1.0", + "config_param": true, + "help": "Maximum interval after which an upload will be permitted irrespective of upload thresholds, in seconds. This will be set to the smaller of the current value and 25% of persistent_metadata_purge_age. Configuration parameter with a default value of 64800.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_fusion_upload_interval": { - "added": "7.0.0", - "help": "The interval between kvstore syncs to fusion, in seconds.", + "added": "8.0.0", + "config_param": true, + "help": "The interval between kvstore syncs to fusion, in seconds. Configuration parameter with a default value of (60 * 10).", "stability": "volatile", "type": "gauge" }, @@ -2176,20 +2451,23 @@ "type": "gauge" }, "kv_ep_magma_group_commit_max_sync_wait_duration_ms": { - "added": "7.0.0", - "help": "When a transaction is about to stall because there are pending transactions waiting to start, if there already are transactions waiting and the oldest transaction has been waiting for magma_group_commit_max_sync_wait_duration ms or more, the current transaction will perform the fsync. When group commit is enabled and both magma_group_commit_max_sync_wait_duration and magma_group_commit_max_transaction_count are set to 0, transactions will stall until there are no more transactions waiting to start. Unit is milliseconds.", + "added": "7.1.0", + "config_param": true, + "help": "When a transaction is about to stall because there are pending transactions waiting to start, if there already are transactions waiting and the oldest transaction has been waiting for magma_group_commit_max_sync_wait_duration ms or more, the current transaction will perform the fsync. When group commit is enabled and both magma_group_commit_max_sync_wait_duration and magma_group_commit_max_transaction_count are set to 0, transactions will stall until there are no more transactions waiting to start. Unit is milliseconds. Configuration parameter with a default value of 20.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_group_commit_max_transaction_count": { - "added": "7.0.0", - "help": "When a transaction is about to stall because there are pending transactions waiting to start, if there already are magma_group_commit_max_transaction_count including the current transaction waiting, the current transaction will perform the fsync. When group commit is enabled and both magma_group_commit_max_sync_wait_duration and magma_group_commit_max_transaction_count are set to 0, transactions will stall until there are no more transactions waiting to start.", + "added": "7.1.0", + "config_param": true, + "help": "When a transaction is about to stall because there are pending transactions waiting to start, if there already are magma_group_commit_max_transaction_count including the current transaction waiting, the current transaction will perform the fsync. When group commit is enabled and both magma_group_commit_max_sync_wait_duration and magma_group_commit_max_transaction_count are set to 0, transactions will stall until there are no more transactions waiting to start. Configuration parameter with a default value of 20.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_heartbeat_interval": { "added": "7.0.0", - "help": "Frequency of heartbeat interval; in seconds. A heartbeat task is scheduled to provide cleanup and maintenance when magma is idle.", + "config_param": true, + "help": "Frequency of heartbeat interval; in seconds. A heartbeat task is scheduled to provide cleanup and maintenance when magma is idle. Configuration parameter with a default value of 10.", "stability": "volatile", "type": "gauge" }, @@ -2237,7 +2515,8 @@ }, "kv_ep_magma_initial_wal_buffer_size": { "added": "7.0.0", - "help": "The WAL buffer is used to stage items to the write ahead log along with control information like begin and end transaction. This parameter refers to the initial WAL buffer size. The WAL buffer will adjust its size up to a maximum of 4MB or down to a minimum of 64KB depending on the transaction batch size with consideration for other magma components which consume memory such as the block cache, bloom filters, write cache and meta data overhead.", + "config_param": true, + "help": "The WAL buffer is used to stage items to the write ahead log along with control information like begin and end transaction. This parameter refers to the initial WAL buffer size. The WAL buffer will adjust its size up to a maximum of 4MB or down to a minimum of 64KB depending on the transaction batch size with consideration for other magma components which consume memory such as the block cache, bloom filters, write cache and meta data overhead. Configuration parameter with a default value of 65536.", "stability": "volatile", "type": "gauge" }, @@ -2248,14 +2527,16 @@ "type": "gauge" }, "kv_ep_magma_key_tree_data_block_size": { - "added": "7.0.0", - "help": "Magma uses SSTables for storage. SSTables are made up of different types of blocks. Data blocks contain the bulk of the data and contain the key and metadata for each of the items in the block. Larger block sizes can decrease storage space by better block compression but they require more memory, cpu and io bandwidth to read and write them.", + "added": "7.2.0", + "config_param": true, + "help": "Magma uses SSTables for storage. SSTables are made up of different types of blocks. Data blocks contain the bulk of the data and contain the key and metadata for each of the items in the block. Larger block sizes can decrease storage space by better block compression but they require more memory, cpu and io bandwidth to read and write them. Configuration parameter with a default value of 4096.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_key_tree_index_block_size": { - "added": "7.0.0", - "help": "Magma uses SSTables for storage. SSTables are made up of different types of blocks. Index blocks contain keys that help traverse the SSTable to locate the data item. Larger block sizes can decrease storage space by better block compression but they require more memory, cpu and io bandwidth to read and write them.", + "added": "7.2.0", + "config_param": true, + "help": "Magma uses SSTables for storage. SSTables are made up of different types of blocks. Index blocks contain keys that help traverse the SSTable to locate the data item. Larger block sizes can decrease storage space by better block compression but they require more memory, cpu and io bandwidth to read and write them. Configuration parameter with a default value of 4096.", "stability": "volatile", "type": "gauge" }, @@ -2292,39 +2573,59 @@ "type": "gauge", "unit": "bytes" }, + "kv_ep_magma_max_base_level_size": { + "added": "8.1.0", + "config_param": true, + "help": "Upper bound on the dynamic L0/L1 target size. The target scales with the bottom level; values above this are clamped. Configuration parameter with a default value of 1073741824.", + "stability": "volatile", + "type": "gauge" + }, "kv_ep_magma_max_checkpoints": { "added": "7.0.0", - "help": "Maximum # of checkpoints retained for rollback.", + "config_param": true, + "help": "Maximum # of checkpoints retained for rollback. Configuration parameter with a default value of 5.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_max_level0_ttl": { - "added": "7.0.0", - "help": "Maximum time (in seconds) that data is kept in level 0 before it is merged.", + "added": "8.1.0", + "config_param": true, + "help": "Maximum time (in seconds) that data is kept in level 0 before it is merged. Configuration parameter with a default value of 600.", + "stability": "volatile", + "type": "gauge" + }, + "kv_ep_magma_max_num_level0_tables": { + "added": "8.1.0", + "config_param": true, + "help": "Maximum number of L0 tables in the magma LSM trees. Configuration parameter with a default value of 16.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_max_recovery_bytes": { "added": "7.0.0", - "help": "Maximum amount of data that is replayed from the WAL during magma recovery. When this threshold is reached magma, creates a temporary checkpoint to recover at. This is per kvstore and in bytes.", + "config_param": true, + "help": "Maximum amount of data that is replayed from the WAL during magma recovery. When this threshold is reached magma, creates a temporary checkpoint to recover at. This is per kvstore and in bytes. Configuration parameter with a default value of 67108864.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_max_write_cache": { "added": "7.0.0", - "help": "Magma uses a common skiplist to buffer all items at the shard level called the write cache. The write cache contains items from all the kvstores that are part of the shard and when it is flushed, each kvstore will receive a few items each. Regardless of how much memory might be available, this would be the maximum amount that could be allocated.", + "config_param": true, + "help": "Magma uses a common skiplist to buffer all items at the shard level called the write cache. The write cache contains items from all the kvstores that are part of the shard and when it is flushed, each kvstore will receive a few items each. Regardless of how much memory might be available, this would be the maximum amount that could be allocated. Configuration parameter with a default value of 134217728.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_mem_quota_low_watermark_ratio": { - "added": "7.0.0", - "help": "Fraction of memory quota used by magma as it's low water mark. Magma uses this low watermark to size it's write cache and block cache. This sizing includes bloom filters memory usage but bloom filter eviction is based on the memory quota", + "added": "7.1.0", + "config_param": true, + "help": "Fraction of memory quota used by magma as it's low water mark. Magma uses this low watermark to size it's write cache and block cache. This sizing includes bloom filters memory usage but bloom filter eviction is based on the memory quota. Configuration parameter with a default value of 0.2.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_mem_quota_ratio": { "added": "7.0.0", - "help": "Magma total memory ratio of the Bucket Quota across all shards and Magma limit's it's memory usage to this value.", + "config_param": true, + "help": "Magma total memory ratio of the Bucket Quota across all shards and Magma limit's it's memory usage to this value. Configuration parameter with a default value of 0.5.", "stability": "volatile", "type": "gauge" }, @@ -2335,14 +2636,16 @@ "type": "gauge" }, "kv_ep_magma_min_checkpoint_interval": { - "added": "7.0.0", - "help": "Minimum interval between two checkpoints; in seconds. Prevents excessive creation of checkpoints.", + "added": "7.1.0", + "config_param": true, + "help": "Minimum interval between two checkpoints; in seconds. Prevents excessive creation of checkpoints. Configuration parameter with a default value of 1.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_min_value_block_size_threshold": { - "added": "7.0.0", - "help": "Magma creates value blocks for values larger than this size. Value blocks only contain a single KV item and their reads/writes are optimised for lesser memory consumption as it avoids many value copies. For example, magma block compression is turned off for them as compression requires an output buffer as large as the input buffer. This is fine since for such large docs, per document Snappy compression already should give good enough space savings. This setting should be >= SeqIndex data block size or else it won't take effect.", + "added": "7.6.0", + "config_param": true, + "help": "Magma creates value blocks for values larger than this size. Value blocks only contain a single KV item and their reads/writes are optimised for lesser memory consumption as it avoids many value copies. For example, magma block compression is turned off for them as compression requires an output buffer as large as the input buffer. This is fine since for such large docs, per document Snappy compression already should give good enough space savings. This setting should be >= SeqIndex data block size or else it won't take effect. Configuration parameter with a default value of (64 * 1024).", "stability": "volatile", "type": "gauge" }, @@ -2353,8 +2656,9 @@ "type": "gauge" }, "kv_ep_magma_per_document_compression_enabled": { - "added": "7.0.0", - "help": "Apply Snappy compression to each document when persisted (magma only)", + "added": "7.6.0", + "config_param": true, + "help": "Apply Snappy compression to each document when persisted (magma only). Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, @@ -2406,6 +2710,18 @@ "stability": "committed", "type": "gauge" }, + "kv_ep_magma_readio_get": { + "added": "8.1.0", + "help": "Number of read IOs performed on the get path", + "stability": "committed", + "type": "counter" + }, + "kv_ep_magma_readio_set": { + "added": "8.1.0", + "help": "Number of read IOs performed on the set path", + "stability": "committed", + "type": "counter" + }, "kv_ep_magma_readioamp_ratio": { "added": "7.1.0", "help": "Number of read IOs performed by GetDocs divided by the number of GetDocs", @@ -2414,14 +2730,16 @@ "unit": "ratio" }, "kv_ep_magma_seq_tree_data_block_size": { - "added": "7.0.0", - "help": "Magma uses SSTables for storage. SSTables are made up of different types of blocks. Data blocks contain the bulk of the data and contain the key, metadata and value for each of the items in the block. Larger block sizes can decrease storage space by better block compression but they require more memory, cpu and io bandwidth to read and write them. If this is less than magma_min_value_block_size_threshold, Magma will internally auto configure the value block size to be as large as this.", + "added": "7.2.0", + "config_param": true, + "help": "Magma uses SSTables for storage. SSTables are made up of different types of blocks. Data blocks contain the bulk of the data and contain the key, metadata and value for each of the items in the block. Larger block sizes can decrease storage space by better block compression but they require more memory, cpu and io bandwidth to read and write them. If this is less than magma_min_value_block_size_threshold, Magma will internally auto configure the value block size to be as large as this. Configuration parameter with a default value of 4096.", "stability": "volatile", "type": "gauge" }, "kv_ep_magma_seq_tree_index_block_size": { - "added": "7.0.0", - "help": "Magma uses SSTables for storage. SSTables are made up of different types of blocks. Index blocks contain keys that help traverse the SSTable to locate the data item. Larger block sizes can decrease storage space by better block compression but they require more memory, cpu and io bandwidth to read and write them.", + "added": "7.2.0", + "config_param": true, + "help": "Magma uses SSTables for storage. SSTables are made up of different types of blocks. Index blocks contain keys that help traverse the SSTable to locate the data item. Larger block sizes can decrease storage space by better block compression but they require more memory, cpu and io bandwidth to read and write them. Configuration parameter with a default value of 4096.", "stability": "volatile", "type": "gauge" }, @@ -2464,8 +2782,9 @@ "type": "gauge" }, "kv_ep_magma_sync_every_batch": { - "added": "7.0.0", - "help": "Couchstore generates a commit point at the end of every batch of items. During normal operation, Magma checkpoints are taken at every magma_checkpoint_interval. Many of the tests require more frequent checkpoints so this configuration parameter makes sure every batch generates a checkpoint. Each checkpoint generated in this way is a \"Sync\" checkpoint and isn't going to be useful for rollback as it only the latest checkpoint is a \"Sync\" checkpoiont. A \"Rollback\" checkpoint will be made instead if we set magma_checkpoint_interval to 0. The \"Rollback\" checkpoints are stored in the checkpoint queue as potential rollback points. Should be used for testing only!", + "added": "7.1.0", + "config_param": true, + "help": "Couchstore generates a commit point at the end of every batch of items. During normal operation, Magma checkpoints are taken at every magma_checkpoint_interval. Many of the tests require more frequent checkpoints so this configuration parameter makes sure every batch generates a checkpoint. Each checkpoint generated in this way is a \"Sync\" checkpoint and isn't going to be useful for rollback as it only the latest checkpoint is a \"Sync\" checkpoiont. A \"Rollback\" checkpoint will be made instead if we set magma_checkpoint_interval to 0. The \"Rollback\" checkpoints are stored in the checkpoint queue as potential rollback points. Should be used for testing only!. Configuration parameter with a default value of false.", "stability": "volatile", "type": "gauge" }, @@ -2571,7 +2890,8 @@ }, "kv_ep_magma_write_cache_ratio": { "added": "7.0.0", - "help": "Memory is maintained across 3 magma components; Bloom filters, Block cache and Write cache. The least important of these is the write cache. If there is insufficent memory for the write cache, the write cache will grow to the size of the batch and then be immediately flushed and freed. If there is available memory, the write cache is limited to 20% of the available memory (after bloom filter and block cache get their memory up to magma_max_write_cache (128MB). Bloom filters are the most important and are never paged out. Bloom filter memory can cause magma to go above the memory quota. To allevaite this, the bottom layer where the majority of bloom filter memory is, won't use bloom filters when OptimizeBloomFilterForMisses is on (which it is by default). The block cache grows each time the index sizes change. But its growth is bounded by the available memory or what's left over after the bloom filter memory is subtracted.", + "config_param": true, + "help": "Memory is maintained across 3 magma components; Bloom filters, Block cache and Write cache. The least important of these is the write cache. If there is insufficent memory for the write cache, the write cache will grow to the size of the batch and then be immediately flushed and freed. If there is available memory, the write cache is limited to 20% of the available memory (after bloom filter and block cache get their memory up to magma_max_write_cache (128MB). Bloom filters are the most important and are never paged out. Bloom filter memory can cause magma to go above the memory quota. To allevaite this, the bottom layer where the majority of bloom filter memory is, won't use bloom filters when OptimizeBloomFilterForMisses is on (which it is by default). The block cache grows each time the index sizes change. But its growth is bounded by the available memory or what's left over after the bloom filter memory is subtracted. Configuration parameter with a default value of 0.2.", "stability": "volatile", "type": "gauge" }, @@ -2583,73 +2903,86 @@ }, "kv_ep_max_checkpoints": { "added": "7.0.0", - "help": "The expected max number of checkpoints in each VBucket on a balanced system. Note: That is not a hard limit on the single vbucket. That is used (together with checkpoint_memory_ratio) for computing checkpoint_max_size, which triggers checkpoint creation.", + "config_param": true, + "help": "The expected max number of checkpoints in each VBucket on a balanced system. Note: That is not a hard limit on the single vbucket. That is used (together with checkpoint_memory_ratio) for computing checkpoint_max_size, which triggers checkpoint creation. Configuration parameter with a default value of 10.", "stability": "volatile", "type": "gauge" }, "kv_ep_max_failover_entries": { "added": "7.0.0", - "help": "maximum number of failover log entries", + "config_param": true, + "help": "maximum number of failover log entries. Configuration parameter with a default value of 25.", "stability": "volatile", "type": "gauge" }, "kv_ep_max_item_privileged_bytes": { "added": "7.0.0", - "help": "Maximum number of bytes allowed for 'privileged' (system) data for an item in addition to the max_item_size bytes", + "config_param": true, + "help": "Maximum number of bytes allowed for 'privileged' (system) data for an item in addition to the max_item_size bytes. Configuration parameter with a default value of (1024 * 1024).", "stability": "volatile", "type": "gauge" }, "kv_ep_max_item_size": { "added": "7.0.0", - "help": "Maximum number of bytes allowed for an item", + "config_param": true, + "help": "Maximum number of bytes allowed for an item. Configuration parameter with a default value of (20 * 1024 * 1024).", "stability": "volatile", "type": "gauge" }, "kv_ep_max_num_bgfetchers": { "added": "7.0.0", - "help": "Maximum number of bg fetcher objects (the number of concurrent bg fetch tasks we can run). 0 = auto-configure which means we use the same number as the number of reader threads (num_reader_threads).", + "config_param": true, + "help": "Maximum number of bg fetcher objects (the number of concurrent bg fetch tasks we can run). 0 = auto-configure which means we use the same number as the number of reader threads (num_reader_threads). Configuration parameter with a default value of 0.", "stability": "volatile", "type": "gauge" }, "kv_ep_max_num_flushers": { - "added": "7.0.0", - "help": "Maximum number of flusher objects (the number of concurrent flusher tasks we can run). 0 = auto-configure which means we use the same number as the number of shards (max_num_shards - for historic reasons). See also num_writer_threads.", + "added": "7.1.0", + "config_param": true, + "help": "Maximum number of flusher objects (the number of concurrent flusher tasks we can run). 0 = auto-configure which means we use the same number as the number of shards (max_num_shards - for historic reasons). See also num_writer_threads. Configuration parameter with a default value of 0.", "stability": "volatile", "type": "gauge" }, "kv_ep_max_num_shards": { "added": "7.0.0", - "help": "Maximum mumber of shards (0 = auto-configure)", + "config_param": true, + "help": "Maximum mumber of shards (0 = auto-configure). Configuration parameter with a default value of 0.", "stability": "volatile", "type": "gauge" }, "kv_ep_max_num_workers": { "added": "7.0.0", - "help": "Bucket Priority relative to other buckets", + "config_param": true, + "help": "Bucket Priority relative to other buckets. Configuration parameter with a default value of 4.", "stability": "volatile", "type": "gauge" }, "kv_ep_max_size": { "added": "7.0.0", - "help": "Memory quota (in bytes) for this bucket.", + "config_param": true, + "help": "Memory quota (in bytes) for this bucket. Configuration parameter with a default value of (100 * 1024 * 1024).", + "long_description": "Provides the absolute maximum amount of memory, measured in bytes, allocated to a specific bucket on a single node. It represents the hard memory quota configured by the administrator, serving as the strict baseline for calculating critical thresholds like high and low watermarks. This value is critical for capacity planning.", "stability": "volatile", "type": "gauge" }, "kv_ep_max_threads": { "added": "7.0.0", - "help": "Maximum number of threads of any single class (0 = automatically select based on core count)", + "config_param": true, + "help": "Maximum number of threads of any single class (0 = automatically select based on core count). Configuration parameter with a default value of 0.", "stability": "volatile", "type": "gauge" }, "kv_ep_max_ttl": { "added": "7.0.0", - "help": "A maximum TTL (in seconds) that will apply to all new documents, documents set with no TTL will be given this value. A value of 0 means this is disabled", + "config_param": true, + "help": "A maximum TTL (in seconds) that will apply to all new documents, documents set with no TTL will be given this value. A value of 0 means this is disabled. Configuration parameter with a default value of 0.", "stability": "volatile", "type": "gauge" }, "kv_ep_max_vbuckets": { "added": "7.0.0", - "help": "Maximum number of vbuckets expected", + "config_param": true, + "help": "Maximum number of vbuckets expected. Configuration parameter with a default value of 1024.", "stability": "volatile", "type": "gauge" }, @@ -2674,8 +3007,10 @@ "type": "gauge" }, "kv_ep_mem_high_wat_percent": { - "added": "7.0.0", - "help": "Ratio of the Bucket Quota at which to place the high watermark. This is the maximum desired memory usage. This value should be lower than mutation_mem_ratio to avoid no memory errors.", + "added": "8.0.0", + "config_param": true, + "help": "Ratio of the Bucket Quota at which to place the high watermark. This is the maximum desired memory usage. This value should be lower than mutation_mem_ratio to avoid no memory errors. Configuration parameter with a default value of 0.85.", + "long_description": "Provides the specific percentage of the total bucket memory quota that acts as the high watermark threshold. By default, the high watermark is set at 85% of the bucket's allocated memory. It represents the critical trigger point where the engine proactively initiates background memory management routines to evict data from RAM to disk until memory usage drops below the low watermark. This value is vital for identifying heavy memory pressure.", "stability": "volatile", "type": "gauge" }, @@ -2686,8 +3021,10 @@ "type": "gauge" }, "kv_ep_mem_low_wat_percent": { - "added": "7.0.0", - "help": "Ratio of the Bucket Quota at which to place the low watermark. This is the point to which to reduce the memory usage of the bucket after hitting the high watermark.", + "added": "8.0.0", + "config_param": true, + "help": "Ratio of the Bucket Quota at which to place the low watermark. This is the point to which to reduce the memory usage of the bucket after hitting the high watermark. Configuration parameter with a default value of 0.75.", + "long_description": "Provides the specific percentage of the maximum bucket memory quota designated as the low watermark. By default it is set at 10% lower than the high watermark. It represents the target recovery threshold for the item pager, indicating the point at which continuous eviction from memory to disk ceases after a high watermark breach. This value ensures a healthy buffer of free memory.", "stability": "volatile", "type": "gauge" }, @@ -2699,7 +3036,8 @@ }, "kv_ep_mem_used_merge_threshold_percent": { "added": "7.0.0", - "help": "What percent of max_data size should we allow the estimated total memory to lag by (EPStats::getEstimatedTotalMemoryUsed)", + "config_param": true, + "help": "What percent of max_data size should we allow the estimated total memory to lag by (EPStats::getEstimatedTotalMemoryUsed). Configuration parameter with a default value of 0.5.", "stability": "volatile", "type": "gauge" }, @@ -2719,31 +3057,44 @@ }, "kv_ep_min_compression_ratio": { "added": "7.0.0", - "help": "specifies a minimum compression ratio below which storing the document will be stored as uncompressed.", + "config_param": true, + "help": "specifies a minimum compression ratio below which storing the document will be stored as uncompressed. Configuration parameter with a default value of 1.2.", + "stability": "volatile", + "type": "gauge" + }, + "kv_ep_monitor_task_interval": { + "added": "8.1.0", + "config_param": true, + "help": "Schedule interval (in seconds) for the MonitorTask. Configuration parameter with a default value of 1.", "stability": "volatile", "type": "gauge" }, "kv_ep_mutation_mem_ratio": { - "added": "7.0.0", - "help": "Ratio of the Bucket Quota that can be used before mutations return tmpOOMs", + "added": "7.6.0", + "config_param": true, + "help": "Ratio of the Bucket Quota that can be used before mutations return tmpOOMs. Configuration parameter with a default value of 0.93.", + "long_description": "Provides the precise percentage of the maximum bucket memory quota beyond which the engine will begin to reject frontend mutations with temporary OOM errors. It is set at a default of 93%. It represents a protective throttling threshold, indicating the system is nearing complete memory exhaustion and must temporarily block incoming writes. It is a dynamic value that can be modified. This value protects the node from allocating beyond the assigned bucket quota. It prevent crashes in the cases like high memory fragmentation where memcached memory usage is higher than logical allocation.", "stability": "volatile", "type": "gauge" }, "kv_ep_nexus_concurrent_flush_compaction_enabled": { - "added": "7.0.0", - "help": "Should NexusKVStore enable concurrent flushing and compaction?", + "added": "7.1.0", + "config_param": true, + "help": "Should NexusKVStore enable concurrent flushing and compaction?. Configuration parameter with a default value of true.", "stability": "internal", "type": "gauge" }, "kv_ep_nexus_implicit_compaction_enabled": { - "added": "7.0.0", - "help": "Should NexusKVStore enable implicit compaction?", + "added": "7.1.0", + "config_param": true, + "help": "Should NexusKVStore enable implicit compaction?. Configuration parameter with a default value of true.", "stability": "internal", "type": "gauge" }, "kv_ep_not_locked_returns_tmpfail": { - "added": "7.0.0", - "help": "Controls which error code should be returned when attempting to unlock an item that is not locked. When value is true, the legacy temporary_failure is used instead of not_locked.", + "added": "7.6.5-MP2", + "config_param": true, + "help": "Controls which error code should be returned when attempting to unlock an item that is not locked. When value is true, the legacy temporary_failure is used instead of not_locked. Configuration parameter with a default value of false.", "stability": "volatile", "type": "gauge" }, @@ -2840,12 +3191,14 @@ "kv_ep_num_workers": { "added": "7.0.0", "help": "Global number of shared worker threads", + "long_description": "Provides the global number of shared worker threads. It represents the processing concurrency level, indicating how many parallel operations—such as reads, writes, and background maintenance tasks the system can simultaneously handle. It is calculated by aggregating thread counts across the reader, writer, auxiliary I/O, non-I/O, slow-I/O and quick-I/O pools. This value is available in Prometheus, cbstats, and mcstat.", "stability": "committed", "type": "gauge" }, "kv_ep_oom_errors": { "added": "7.0.0", "help": "Number of times unrecoverable OOMs happened while processing operations", + "long_description": "Provides number of unrecoverable out-of-memory errors operations that fail because the bucket has exceeded its total memory quota. The metric increments when memory usage exceeds the bucket quota, operations cannot proceed even after memory reclamation attempts, and the system is in a hard OOM state with no space available. A non-zero and rising value indicates the bucket quota is insufficient for the current workload and immediate intervention is required.", "stability": "committed", "type": "gauge" }, @@ -2856,15 +3209,31 @@ "type": "gauge", "unit": "bytes" }, + "kv_ep_pageable_mem_high_watermark_bytes": { + "added": "8.1.0", + "help": "Pageable memory high watermark (used in paging management)", + "stability": "committed", + "type": "gauge", + "unit": "bytes" + }, + "kv_ep_pageable_mem_low_watermark_bytes": { + "added": "8.1.0", + "help": "Pageable memory low watermark (used in paging management)", + "stability": "committed", + "type": "gauge", + "unit": "bytes" + }, "kv_ep_pager_sleep_time_ms": { "added": "7.0.0", - "help": "How long in milliseconds the ItemPager will sleep for when not being requested to run", + "config_param": true, + "help": "How long in milliseconds the ItemPager will sleep for when not being requested to run. Configuration parameter with a default value of 5000.", "stability": "volatile", "type": "gauge" }, "kv_ep_paging_visitor_pause_check_count": { - "added": "7.0.0", - "help": "Expected number of times the PagingVisitor will check the pause condition per vBucket", + "added": "7.6.2", + "config_param": true, + "help": "Expected number of times the PagingVisitor will check the pause condition per vBucket. Configuration parameter with a default value of 50.", "stability": "volatile", "type": "gauge" }, @@ -2914,19 +3283,23 @@ }, "kv_ep_persistent_metadata_purge_age": { "added": "7.0.0", - "help": "Age in seconds after which tombstones may be purged. Defaults to 3 days. Max of 60 days. If this is dynamically changed for a magma bucket then magma may not trigger compactions when it should, this can be avoided by running a full manual compaction after changing this parameter.", + "config_param": true, + "help": "Age in seconds after which tombstones may be purged. Defaults to 3 days. Max of 60 days. If this is dynamically changed for a magma bucket then magma may not trigger compactions when it should, this can be avoided by running a full manual compaction after changing this parameter. Configuration parameter with a default value of 259200.", + "long_description": "Provides the configured time interval after which the persistent metadata of deleted items, known as tombstones, is permanently purged from disk storage. It represents the active policy for tombstone expiration, balancing robust replication consistency against disk space reclamation. This value is crucial for tuning storage overhead.", "stability": "volatile", "type": "gauge" }, "kv_ep_primary_warmup_min_items_threshold": { - "added": "7.0.0", - "help": "Mark primary warm-up as complete when the number of values loaded by warm-up reaches this percentage of the available values. This is checked after evaluating primry_warmup_min_memory_threshold.", + "added": "8.0.0", + "config_param": true, + "help": "Mark primary warm-up as complete when the number of values loaded by warm-up reaches this percentage of the available values. This is checked after evaluating primry_warmup_min_memory_threshold. Configuration parameter with a default value of 0.", "stability": "volatile", "type": "gauge" }, "kv_ep_primary_warmup_min_memory_threshold": { - "added": "7.0.0", - "help": "Mark primary warm-up as complete when bucket memory usage reaches this percentage of max_data_size. This is checked before evaluating primry_warmup_min_items_threshold.", + "added": "8.0.0", + "config_param": true, + "help": "Mark primary warm-up as complete when bucket memory usage reaches this percentage of max_data_size. This is checked before evaluating primry_warmup_min_items_threshold. Configuration parameter with a default value of 0.", "stability": "volatile", "type": "gauge" }, @@ -2937,32 +3310,37 @@ "type": "gauge" }, "kv_ep_range_scan_kv_store_scan_ratio": { - "added": "7.0.0", - "help": "The ratio for calculating how many RangeScans can exist, a ratio of total KVStore scans.", + "added": "7.6.0", + "config_param": true, + "help": "The ratio for calculating how many RangeScans can exist, a ratio of total KVStore scans. Configuration parameter with a default value of 0.8.", "stability": "volatile", "type": "gauge" }, "kv_ep_range_scan_max_continue_tasks": { - "added": "7.0.0", - "help": "The maximum number of range scan tasks that can exist concurrently. Setting to 0 results in num_auxio_threads - 1 tasks", + "added": "7.6.0", + "config_param": true, + "help": "The maximum number of range scan tasks that can exist concurrently. Setting to 0 results in num_auxio_threads - 1 tasks. Configuration parameter with a default value of 0.", "stability": "volatile", "type": "gauge" }, "kv_ep_range_scan_max_lifetime": { - "added": "7.0.0", - "help": "The maximum lifetime in seconds for a range-scan. Scans that don't complete before this limit are cancelled", + "added": "7.6.0", + "config_param": true, + "help": "The maximum lifetime in seconds for a range-scan. Scans that don't complete before this limit are cancelled. Configuration parameter with a default value of 180.", "stability": "volatile", "type": "gauge" }, "kv_ep_range_scan_read_buffer_send_size": { - "added": "7.0.0", - "help": "The size of a buffer used to store data read during the I/O phase of a range-scan-continue. Once the buffer size is >= to this value the data is sent to the connection", + "added": "7.6.0", + "config_param": true, + "help": "The size of a buffer used to store data read during the I/O phase of a range-scan-continue. Once the buffer size is >= to this value the data is sent to the connection. Configuration parameter with a default value of 8192.", "stability": "volatile", "type": "gauge" }, "kv_ep_retain_erroneous_tombstones": { "added": "7.0.0", - "help": "whether erroneous tombstones need to be retain during compaction. Erroneous tombstones are those that have invalid meta data in it. For example, a delete time of 0.", + "config_param": true, + "help": "whether erroneous tombstones need to be retain during compaction. Erroneous tombstones are those that have invalid meta data in it. For example, a delete time of 0. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, @@ -2979,20 +3357,23 @@ "type": "counter" }, "kv_ep_secondary_warmup_min_items_threshold": { - "added": "7.0.0", - "help": "Stop secondary warm-up when the number of values loaded by warm-up reaches this percentage of the available values. This is checked after evaluating secondary_warmup_min_memory_threshold.", + "added": "8.0.0", + "config_param": true, + "help": "Stop secondary warm-up when the number of values loaded by warm-up reaches this percentage of the available values. This is checked after evaluating secondary_warmup_min_memory_threshold. Configuration parameter with a default value of 100.", "stability": "volatile", "type": "gauge" }, "kv_ep_secondary_warmup_min_memory_threshold": { - "added": "7.0.0", - "help": "Stop secondary warm-up when bucket memory usage reaches this percentage of max_data_size. This is checked before evaluating secondary_warmup_min_items_threshold.", + "added": "8.0.0", + "config_param": true, + "help": "Stop secondary warm-up when bucket memory usage reaches this percentage of max_data_size. This is checked before evaluating secondary_warmup_min_items_threshold. Configuration parameter with a default value of 100.", "stability": "volatile", "type": "gauge" }, "kv_ep_seqno_persistence_timeout": { - "added": "7.0.0", - "help": "Timeout in seconds after which a pending SeqnoPersistence operation is temp-failed", + "added": "7.1.0", + "config_param": true, + "help": "Timeout in seconds after which a pending SeqnoPersistence operation is temp-failed. Configuration parameter with a default value of 30.", "stability": "volatile", "type": "gauge" }, @@ -3044,7 +3425,8 @@ }, "kv_ep_sync_writes_max_allowed_replicas": { "added": "7.0.0", - "help": "The maximum number of supported replicas for SyncWrites. Attempts to issue SyncWrites against a topology with more replicas than this setting will fail with DurabilityImpossible.", + "config_param": true, + "help": "The maximum number of supported replicas for SyncWrites. Attempts to issue SyncWrites against a topology with more replicas than this setting will fail with DurabilityImpossible. Configuration parameter with a default value of 2.", "stability": "volatile", "type": "gauge" }, @@ -3164,38 +3546,44 @@ "unit": "seconds" }, "kv_ep_vbucket_mapping_sanity_checking": { - "added": "7.0.0", - "help": "Are vBucket mappings (key -> vBucket) checked by the server? This is a sanity checking mode which crc32 hashes the key to ensure that the client is supplying the expected vBucket for each key.", + "added": "7.1.0", + "config_param": true, + "help": "Are vBucket mappings (key -> vBucket) checked by the server? This is a sanity checking mode which crc32 hashes the key to ensure that the client is supplying the expected vBucket for each key. Configuration parameter with a default value of false.", "stability": "volatile", "type": "gauge" }, "kv_ep_warmup": { "added": "7.0.0", - "help": "Is Warmup of existing data enabled", + "config_param": true, + "help": "Is Warmup of existing data enabled. Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, "kv_ep_warmup_accesslog_load_batch_size": { - "added": "7.0.0", - "help": "AccessLog loading operates in batches of this size (dictates the max number of keys issued to the KVStore::getMulti function)", + "added": "8.0.0", + "config_param": true, + "help": "AccessLog loading operates in batches of this size (dictates the max number of keys issued to the KVStore::getMulti function). Configuration parameter with a default value of 10.", "stability": "volatile", "type": "gauge" }, "kv_ep_warmup_accesslog_load_duration": { - "added": "7.0.0", - "help": "The duration (in ms) after which warmup's LoadAccessLog phase will yield and re-schedule; allowing other tasks on the same thread pool to run.", + "added": "8.0.0", + "config_param": true, + "help": "The duration (in ms) after which warmup's LoadAccessLog phase will yield and re-schedule; allowing other tasks on the same thread pool to run. Configuration parameter with a default value of 100.", "stability": "volatile", "type": "gauge" }, "kv_ep_warmup_backfill_scan_chunk_duration": { - "added": "7.0.0", - "help": "The duration (in ms) after which warmup's backfill scans will yield and re-schedule; allowing other tasks on the same threads to run.", + "added": "7.0.5", + "config_param": true, + "help": "The duration (in ms) after which warmup's backfill scans will yield and re-schedule; allowing other tasks on the same threads to run. Configuration parameter with a default value of 100.", "stability": "volatile", "type": "gauge" }, "kv_ep_warmup_backfill_task_shard_ratio": { - "added": "7.0.0", - "help": "Ratio controlling how many tasks that will be created in the KeyDump, LoadingKVPairs and LoadingData phases. This is a ratio using the number of shards as the denominator and least 1 task will be created per phase. A value of 0.0 will create tasks equal to the lower of #shards or #reader-threads, a value of 1.0 restores the orginal behaviour, 1 task per shard.", + "added": "8.0.0", + "config_param": true, + "help": "Ratio controlling how many tasks that will be created in the KeyDump, LoadingKVPairs and LoadingData phases. This is a ratio using the number of shards as the denominator and least 1 task will be created per phase. A value of 0.0 will create tasks equal to the lower of #shards or #reader-threads, a value of 1.0 restores the orginal behaviour, 1 task per shard. Configuration parameter with a default value of 0.0.", "stability": "volatile", "type": "gauge" }, @@ -3249,24 +3637,19 @@ "type": "counter" }, "kv_ep_workload_monitor_enabled": { - "added": "7.0.0", - "help": "", + "added": "7.2.8", + "config_param": true, + "help": " Configuration parameter with a default value of false.", "stability": "volatile", "type": "gauge" }, "kv_ep_xattr_enabled": { "added": "7.0.0", - "help": "", + "config_param": true, + "help": " Configuration parameter with a default value of true.", "stability": "volatile", "type": "gauge" }, - "kv_ephemeral_vb_checkpoint_memory_overhead_bytes": { - "added": "7.6.0", - "help": "Total memory overhead of all checkpoints", - "stability": "internal", - "type": "gauge", - "unit": "bytes" - }, "kv_ephemeral_vb_ht_memory_bytes": { "added": "7.6.0", "help": "Total memory used by HashTable items", @@ -3281,16 +3664,23 @@ "type": "counter", "unit": "bytes" }, + "kv_fusion_deferred_upload_bytes_bytes": { + "added": "8.1.0", + "help": "The amount of data Fusion knows is pending sync but has decided to defer in order to reduce the number of uploads to FusionLogStore", + "stability": "committed", + "type": "gauge", + "unit": "bytes" + }, "kv_fusion_max_pending_upload_bytes_bytes": { "added": "8.1.0", - "help": "The maximum number of pending upload bytes to be synced across all volumes.", + "help": "Configuration value for the maximum amount of pending upload bytes across all volumes.", "stability": "committed", "type": "gauge", "unit": "bytes" }, "kv_fusion_max_pending_upload_bytes_lwm_ratio": { "added": "8.1.0", - "help": "The proportion of max_pending_upload_bytes beyond which syncs for volumes with highest pending bytes are only allowed.", + "help": "Configuration value for the proportion of max_pending_upload_bytes beyond which syncs for volumes with highest pending bytes are only allowed.", "stability": "committed", "type": "gauge" }, @@ -3315,13 +3705,6 @@ "type": "gauge", "unit": "bytes" }, - "kv_fusion_pending_upload_bytes_bytes": { - "added": "8.1.0", - "help": "The total pending upload bytes across all fusion volumes.", - "stability": "committed", - "type": "gauge", - "unit": "bytes" - }, "kv_fusion_sync_rate_limit_bytes": { "added": "8.0.0", "help": "The rate limit for fusion sync uploads, in bytes per second.", @@ -3410,13 +3793,13 @@ "type": "gauge" }, "kv_magma_num_compactor_threads": { - "added": "8.1.0", + "added": "7.6.11", "help": "The number of magma compactor threads.", "stability": "committed", "type": "gauge" }, "kv_magma_num_flusher_threads": { - "added": "8.1.0", + "added": "7.6.11", "help": "The number of magma flusher threads.", "stability": "committed", "type": "gauge" @@ -3456,6 +3839,7 @@ "kv_mem_used_bytes": { "added": "7.0.0", "help": "Engine's total memory usage", + "long_description": "Provides the most comprehensive view of bucket memory consumption. It is used for quota enforcement, eviction decisions, and monitoring memory pressure. It is calculated using jemalloc's ArenaMalloc. The metric includes (1) Item data (item metadata, keys, resident and non-resident values); (2) Structural overhead (checkpoint queues, persistence queues, DCP queues, hash table and collections/scope metadata); and (3) Component allocations (Magma storage buffers, bloom filters, durability monitor memory, arena allocator overhead and any other internal allocations).", "stability": "committed", "type": "gauge", "unit": "bytes" @@ -3529,6 +3913,19 @@ "stability": "committed", "type": "counter" }, + "kv_pager_deleted_bytes": { + "added": "7.6.11", + "help": "Number of bytes deleted by pager", + "stability": "committed", + "type": "gauge", + "unit": "bytes" + }, + "kv_pager_throttled": { + "added": "7.6.11", + "help": "Total number of times pager has been throttled", + "stability": "committed", + "type": "counter" + }, "kv_read_bytes": { "added": "7.0.0", "help": "The number bytes received from all connections bound to this bucket", @@ -3685,6 +4082,12 @@ "type": "gauge", "unit": "seconds" }, + "kv_tls_certificate_verification_problems": { + "added": "totoro", + "help": "The number of times TLS certificate verification has found issues with the certificate presented by the client during connection establishment", + "stability": "committed", + "type": "gauge" + }, "kv_total_connections": { "added": "7.0.0", "help": "The total number of connections to this system since the process started (or reset)", diff --git a/modules/rest-api/pages/file-based-data-rebalance.adoc b/modules/rest-api/pages/file-based-data-rebalance.adoc new file mode 100644 index 0000000000..a386a52ee9 --- /dev/null +++ b/modules/rest-api/pages/file-based-data-rebalance.adoc @@ -0,0 +1,190 @@ += Configure File-Based Data Rebalance +:description: pass:q[You can turn Data Service File-Based Rebalance (FBR) on or off using the `/internalSettings` REST API endpoint.] +:page-edition: Enterprise Edition +:page-topic-type: reference +:page-toclevels: 3 + +[abstract] +{description} + +== Description + +The file-based rebalance option for the Data Service directly copies vBucket data files between nodes during rebalance. +It's enabled by default. +It usually improves rebalance speed compared to DCP-based backfill method. + +See xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance] for more information about file-based rebalance and how it compares to DCP-based rebalance. + + + +== HTTP Methods + +This API endpoint supports the following methods: + +* <> +* <> + +[#get-settings] +== Determine if FBR is Enabled + +You can get the current status of FBT by posting a GET request to the `/internalSettings` endpoint. +The FBR settings are part of the internal settings, so this API returns all internal settings. + +.List Internal Settings +---- +GET /internalSettings +---- + +=== curl Syntax + +[source,bash] +---- +curl -u ${USER}:${PASSWORD} -X GET \ + http[s]://${HOST}:${PORT}/internalSettings +---- + +.Path Parameters +:required-privileges: get-privs +include::partial$user_pwd_host_port_params.adoc[] + + +[#get-privs] +=== Required Privileges + +You must have at least 1 of the following roles to get the FBR settings: + +* xref:learn:security/roles.adoc#full-admin[Full Admin] +* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin] +* xref:learn:security/roles.adoc#ro-security-admin[Read-Only Security Admin] +* xref:learn:security/roles.adoc#security-admin[Security Admin] +* xref:learn:security/roles.adoc#local-user-security-admin[Local User Admin] + +=== Responses + +`200 OK`:: +Returns the internal settings. +See examples for an example of the settings. + +`401 Unauthorized`:: +Returned when authentication fails, such as when the password is incorrect. + +`403 Forbidden`:: +Returned if you do not have 1 of the roles listed in <>. + +[#get-settings-example] +=== Examples + +The following example gets the current value of the FBR setting. +It filters the internal settings using the `jq` command to show only the setting that controls whether FBR is enabled. + +[source,bash] +---- +curl -s -u $USERNAME:$PASSWORD \ + -X GET http://node1.example.com:8091///internalSettings \ + | jq '{dataServiceFileBasedRebalanceEnabled}' +---- + +Running the example returns a JSON object containing the current FBR settings, such as the following: + +[source,json] +---- +{ + "dataServiceFileBasedRebalanceEnabled": true, +} +---- + +The key returned by the previous example controls whether FBR is enabled. +When `true`, eligible vBucket moves use FBR; when `false`, all vBucket moves use DCP. + +[#set-settings] +== Turn FBR On or Off + +You can turn FBR on or off by making a `POST` request to `dataServiceFileBasedRebalanceEnabled`. + + +.Change FBR Settings +---- +POST /internalSettings +---- + +=== curl Syntax + +[source,bash] +---- +curl -sS -u ${USER}:${PASSWORD} \ + -X POST http[s]://${host}:${port}/internalSettings \ + [-d dataServiceFileBasedRebalanceEnabled={true|false}] \ +---- + +=== Path Parameters +:priv-link: settings-privs +include::partial$user-pw-host-port-params.adoc[] + +=== Parameters + +`dataServiceFileBasedRebalanceEnabled`:: +Set to `true` (the default) to allow the Data Service to use FBR when rebalancing data in eligible vBuckets. +Set to `false` to force the Data Service to use DCP when rebalancing data. + ++ +NOTE: This setting can be overridden on a per-bucket basis using the bucket's `dataServiceRebalanceType` setting. +See xref:manage:manage-buckets/edit-bucket.adoc#bucket-rebalance-type[Bucket-Level Rebalance Type] for more information. + +[#settings-privs] +=== Required Privileges + +You must have 1 of the following roles to change the FBR settings: + +* xref:learn:security/roles.adoc#full-admin[Full Admin] +* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin] + +=== Responses + +`200 OK`:: +Updating the value or values succeeded. + +`400 Bad Request`:: +Returned if you specify an invalid value for . + ++ +Also returns a JSON object that describes the error, such as the following: + ++ +[source,json] +---- +{ + "errors": [ + "dataServiceFileBasedRebalanceEnabled - Accepted values are 'true' and 'false'." + ] +} + +---- + +`401 Unauthorized`:: +Returned when authentication fails, such as when the password is incorrect. + +`403 Forbidden`:: +Returned if you do not have the proper roles to call this API. +See <<#settings-privs>>. + + +[#settings-examples] +=== Examples + +[#disable-fbr-example] +.Disable FBR + +The following example prevents the Data Service from using FBR cluster-wide: + +[source,bash] +---- +curl -u $USERNAME:$PASSWORD \ + -X POST http://node1.example.com:8091/internalSettings \ + -d dataServiceFileBasedRebalanceEnabled=false +---- + +=== See Also + +* xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)]. +* xref:manage:manage-settings/general-settings.adoc#rebalance-settings[Rebalance Settings]. +* xref:manage:manage-buckets/edit-bucket.adoc#bucket-rebalance-type[Bucket-Level Rebalance Type]. \ No newline at end of file diff --git a/modules/rest-api/pages/rest-bucket-create.adoc b/modules/rest-api/pages/rest-bucket-create.adoc index 8b15cfcf28..8e34fee155 100644 --- a/modules/rest-api/pages/rest-bucket-create.adoc +++ b/modules/rest-api/pages/rest-bucket-create.adoc @@ -94,9 +94,10 @@ curl -X POST -u : -d warmupBehavior=[ background | blocking | none ] -d memoryLowWatermark= -d memoryHighWatermark= + -d dataServiceRebalanceType=[ auto | preferFileBased | preferDcp ] ---- -All parameters are described in the following subsections. +The following subsections describe these parameters. NOTE: The `threadsNumber` parameter, which sets the number of threads for the bucket, has not had any effect since version Couchbase Server 7.0.0. It's deprecated and is no longer listed in the syntax. @@ -139,6 +140,7 @@ All other parameters are optional and have a default value. ** <> ** <> ** <> +** <> + NOTE: When migrating a bucket between storage backends, you can edit only the bucket's xref:rest-api:rest-bucket-create.adoc#ramQuota[ramQuota], xref:rest-api:rest-bucket-create.adoc#evictionpolicy[evictionPolicy], and xref:rest-api:rest-bucket-create.adoc#storagebackend[storageBackend] parameters. For more information, see xref:manage:manage-buckets/migrate-bucket.adoc[]. @@ -600,6 +602,62 @@ curl -v -X POST http://localhost:8091/pools/default/buckets/testBucket / Success returns `200 OK`, and changes the `rank` of `testBucket` to `200`. +[#dataservicerebalancetype] +=== dataServiceRebalanceType +[.edition]#{enterprise}# + +Controls the rebalance method used for this bucket's vBucket moves during Data Service rebalance. +This setting overrides the cluster-level `dataServiceFileBasedRebalanceEnabled` setting for the bucket you set it on. + +The valid values are: + +* `auto` (default): The server uses File-Based Rebalance (FBR) to transfer data between nodes if it can. +If it cannot, it falls back to using data-change protocol (DCP) to transfer vBuckets. +This is the recommended setting for most workloads. + +* `preferFileBased`: This setting is a synonym for `auto`. +If the server can use FBR to move vBucket files, it does. +Otherwise it uses DCP. + +* `preferDcp`: All vBucket moves for this bucket use DCP, regardless of the cluster-level FBR setting. +Couchbase Server never uses FBR for the bucket when you select this setting. + +You can set this parameter when creating or modifying the bucket. + +See xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)] for more information about FBR and how the `dataServiceRebalanceType` bucket-level setting interacts with the cluster-level `dataServiceFileBasedRebalanceEnabled` setting. + +[#example-dataservicerebalancetype-create] +==== Example: Setting a Rebalance Type During Bucket Creation + +The following creates a bucket named `testBucket` with `preferFileBased` rebalance. + +[source,bash] +---- +curl -X POST http://localhost:8091/pools/default/buckets \ + -u $USERNAME:$PASSWORD \ + -d name=testBucket \ + -d ramQuota=125 \ + -d dataServiceRebalanceType=preferFileBased +---- + +A successful call returns `202 Accepted`. + +[#example-dataservicerebalancetype-edit] +==== Example: Updating the Rebalance Type + +The following updates the rebalance type for `testBucket` to `auto`: + +[source,bash] +---- +curl -v -X POST http://localhost:8091/pools/default/buckets/testBucket \ + -u Administrator:password \ + -d dataServiceRebalanceType=auto +---- + +Success returns `200 OK`. + +For conceptual information on FBR and the bucket-level setting, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)] and xref:manage:manage-buckets/edit-bucket.adoc#bucket-rebalance-type[Bucket-Level Rebalance Type]. + [#replicanumber] === replicaNumber diff --git a/modules/rest-api/pages/rest-get-internal-setting.adoc b/modules/rest-api/pages/rest-get-internal-setting.adoc index 2d01ed7722..b4cbcf0f17 100644 --- a/modules/rest-api/pages/rest-get-internal-setting.adoc +++ b/modules/rest-api/pages/rest-get-internal-setting.adoc @@ -1,67 +1,97 @@ = Managing Internal Settings -:description: Couchbase-Server internal settings can be retrieved and modified.. +:description: Couchbase Server lets you retrieve and modify internal settings that control cluster performance. :page-topic-type: reference [abstract] {description} -== HTTP methods and URIs +== HTTP Method and URI +.Get General Internal Settings +[source,url] ---- GET /internalSettings +---- +.Set General Internal Settings +[source] +---- POST /internalSettings +---- +.Get Max Parallel Indexers Setting +[source,url] +---- GET /settings/maxParallelIndexers +---- +.Set Max Parallel Indexers Setting +[source] +---- POST /settings/maxParallelIndexers ---- +[#description] == Description -Couchbase-Server internal settings can be inspected and modified, in order to control cluster-performance. +Couchbase Server lets you inspect and modify internal settings to control cluster performance. -WARNING: The settings detailed on this page are meant for performance tuning. -Rigorous and careful testing of any changes to these settings should be made in a non-production environment that is representative, in load and scale, of production data before implementing in production. -Incorrect settings can cause severe impacts to cluster performance or operation due to resource consumption. -Couchbase makes no recommendations for tuning beyond the default settings. -Any changes to these settings not explicitly recommended by Couchbase Employees are not supported by Couchbase. +[#warning] +[WARNING] +.Use With Care +==== +Couchbase intends the settings on this page for performance tuning. +Always test any changes to these settings in a non-production environment that represents the load and scale of your production data before you implement them in production. +Incorrect settings can severely degrade cluster performance and operation through excessive resource consumption. +Couchbase does not recommend tuning beyond the default settings. +Couchbase does not support any changes to these settings unless a Couchbase employee explicitly recommends them. +==== -[#curl-syntax] -== Curl Syntax +== Getting Internal Settings + +To get the current general internal settings for Couchbase Server, send a GET request to the `/internalSettings` endpoint. + +[#get-curl-syntax] +=== curl Syntax +[source,bash] +---- +curl -X GET http[s]://{host}:{port}/internalSettings + -u $USER:$PASSWORD ---- -curl -u : -X GET \ - http://:8091/internalSettings - curl -u : -X POST \ - http://:8091/internalSettings - [-d =]* +==== Path Parameters +:required-privileges: required-privileges-get-internal +include::partial$user_pwd_host_port_params.adoc[] -curl -u : -X GET \ - http://:8091/internalSettings/maxParallelIndexers +[#required-privileges-get-internal] +=== Required Privileges -curl -u : -X POST \ - http://:8091/internalSettings/maxParallelIndexers - -d globalValue= ----- +Your account must have at least 1 of the following roles to make a GET request to this endpoint: -The `/internalSettings/maxParallelIndexers` URI relates to _View_ index configuration only. -Note that it does _not_ apply to the _GSI_ indexes provided by the Index Service. -When `POST` is used with this URI, the `integer-value` specified by means of the `globalValue` parameter must be between `1` and `1024`. -The default is `4`. +* xref:learn:security/roles.adoc#full-admin[Full Admin] +* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin] +* xref:learn:security/roles.adoc#ro-security-admin[Read-Only Security Admin] +* xref:learn:security/roles.adoc#security-admin[Security Admin] +* xref:learn:security/roles.adoc#local-user-security-admin[Local User Admin] -All commands require either the Full Admin or Cluster Admin role. +=== Responses -[#responses] -== Responses +200 OK:: +The request was successful. +Returns a JSON object containing all current general internal settings. -If successful, `GET` and `POST` commands return `200 OK`; and a JSON object containing all current _general_ internal settings. +401 Unauthorized:: +Authentication failed. -Failure to authenticate returns `401 Unauthorized`. -A URI featuring an incorrect hostname or port returns `404 Object Not Found` and an object containing an error message. -For example: +403 Forbidden:: +The account making the request does not have 1 of the roles listed in <>. +404 Not Found:: +The URI specified an incorrect hostname or port. +Also returns a JSON object containing an error message, such as the following: ++ +[source,json] ---- { "error": "not_found", @@ -69,124 +99,326 @@ For example: } ---- -Incorrect specification of a setting-change returns `400 Bad Request` and an object containing an error message, such as the following: - ----- -{"errors":["Unknown key rebalanceMovesPerNoede"]} ----- - -If a cluster contains any node running a version of Couchbase Server less that 7.1, an attempt to change the `eventLogLimit` produces the following error message: `not supported in mixed version clusters`. - -[#examples] -== Examples +=== Example -The following example returns general internal settings for the cluster. -Note that the output is piped to the https://stedolan.github.io/jq/[jq^] command, to facilitate readability. +The following example gets the general internal settings for the cluster. +It pipes the output to the https://stedolan.github.io/jq/[jq^] command to make it easier to read. +[source,bash] ---- -curl -u Administrator:password -X GET http://10.144.220.101:8091/internalSettings | jq '.' +curl -X GET -u $USERNAME:$PASSWORD \ +http://node1.example.com:8091/internalSettings | jq '.' ---- -If successful, the command returns output such as the following: +If successful, the call returns a JSON object such as the following: +[source,json] ---- { - "enforceLimits": false, "indexAwareRebalanceDisabled": false, "rebalanceIndexWaitingDisabled": false, "rebalanceIndexPausingDisabled": false, "rebalanceIgnoreViewCompactions": false, - "rebalanceMovesPerNode": 4, + "internalTlsMinVersion": "tlsv1.3", "rebalanceMovesBeforeCompaction": 64, "maxParallelIndexers": 4, "maxParallelReplicaIndexers": 2, "maxBucketCount": 30, + "magmaMinMemoryQuota": 100, "eventLogsLimit": 10000, "gotraceback": "single", - "indexAutoFailoverDisabled": true, + "failoverBulkJanitorFactor": 1, "certUseSha1": false, - "httpNodeAddition": false + "clientCertNotAfterDurationSeconds": 71193600, + "httpNodeAddition": false, + "appTelemetryMinScrapeIntervalSeconds": 60, + "appTelemetryMaxScrapeIntervalSeconds": 600, + "argon2idTime": 3, + "argon2idMem": 524288, + "pbkdf2HmacSha512Iterations": 10000, + "scramShaIterations": 15000, + "useRelativeWebRedirects": true, + "maxDocsSkip": 1000, + "maxDocsLimit": 1000, + "includeUsernameInUICookie": false, + "dataServiceFileBasedRebalanceEnabled": true } ---- -To change a setting, use the `POST` method, specifying the setting as an argument, and specifying its new value. -For example: +== Setting Internal Settings ----- -curl -u Administrator:password -X POST http://10.144.210.101:8091/internalSettings -d rebalanceMovesPerNode=4 ----- +To change a general internal setting, send a `POST` to the `/internalSettings` endpoint, specifying the setting as an argument along with its new value. -If successful, the command returns `200 OK`, and an empty array. -To check the result, use the `GET` method again: +[#curl-syntax] +=== curl Syntax +[source,bash] ---- -curl -u Administrator:password -X GET http://10.144.210.101:8091/internalSettings | jq '.' | grep rebalanceMovesPer +curl -X POST http[s]://{host}:{port}/internalSettings + -u $USER:$PASSWORD + [-d eventLogsLimit=] + [-d dataServiceFileBasedRebalanceEnabled=] + [-d =] ---- -This returns the following output: +==== Path Parameters +:required-privileges: required-privileges-post-internal +include::partial$user_pwd_host_port_params.adoc[] ----- - "rebalanceMovesPerNode": 4, ----- +==== Parameters -This confirms that the change was successful. -To inspect the current setting for `maxParallelIndexers`, enter the following: +`eventLogsLimit`:: +(optional) Sets the maximum number of entries Couchbase Server retains in the xref:learn:clusters-and-availability/system-events.adoc[system event log]. +Valid values are integers from `3000` to `20000`. ----- -curl -u Administrator:password -X GET http://10.144.210.101:8091/settings/maxParallelIndexers ----- ++ +You can only change this value when every node in the cluster runs Couchbase Server 7.1 or higher. +If a cluster contains any node running a version of Couchbase Server less than 7.1, an attempt to change `eventLogsLimit` produces the following error message: `not supported in mixed version clusters`. -If successful, this returns output such as the following: ----- -{"globalValue":4,"nodes":{"ns_1@10.144.210.101":4}} ----- +`dataServiceFileBasedRebalanceEnabled`:: +(optional) Controls whether the data service's rebalance can use file-based backfills. +This value defaults to `true`. +See xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)] for more information. -To change the value, specify the new value by means of the `globalValue` argument: +``:: +(optional) The name of any general internal setting, set to its new value. +See the <<#warning,warning at the top of this page>> before changing any settings in this REST API endpoint. ----- -curl -u Administrator:password -X POST http://10.144.210.101:8091/settings/maxParallelIndexers -d globalValue=3 ----- -The output displays that the value has been successfully changed: +[#required-privileges-post-internal] +=== Required Privileges + +Your account must have at least 1 of the following roles to make a POST request to this endpoint: + +* xref:learn:security/roles.adoc#full-admin[Full Admin] +* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin] +=== Responses + +200 OK:: +The request was successful. +Returns an empty array. + +400 Bad Request:: +The request specified an unknown setting or an invalid value. +Also returns a JSON object containing an error message, such as the following: ++ +[source,json] ---- -{"globalValue":3,"nodes":{"ns_1@10.144.210.101":3}} +{"errors":["Unknown key rebalanceMovesPerNoede"]} ---- -Note that the value is thus established cluster-wide. +401 Unauthorized:: +Authentication failed. + +403 Forbidden:: +The account making the request does not have 1 of the roles listed in <>. + +404 Not Found:: +The URI specified an incorrect hostname or port. + +=== Examples -To change the current size for the _system log_ (which is described in xref:learn:clusters-and-availability/system-events.adoc[System Events]), use the `eventLogsLimit` argument. -The value should be an integer, specifying the number of logs. -The smallest acceptable value is `3000`, and the highest `20000`. -Note that this value can only be changed when every node in the cluster is running Couchbase Server Version 7.1 or higher. +The following example changes the size of the system log to 15,000 entries: + +[source,bash] ---- -curl -u Administrator:password -X POST http://10.144.220.101:8091/internalSettings -d eventLogsLimit=15000 +curl -X POST -u $USERNAME:$PASSWORD \ +http://node1.example.com:8091/internalSettings \ +-d eventLogsLimit=15000 ---- If successful, the call returns an empty array. -The modified value can be checked by means of the `GET` method, which returns output such as the following: +Use the `GET` method to verify the modified value, which returns output such as the following: +[source,json] ---- { - "enforceLimits": false, "indexAwareRebalanceDisabled": false, "rebalanceIndexWaitingDisabled": false, "rebalanceIndexPausingDisabled": false, "rebalanceIgnoreViewCompactions": false, - "rebalanceMovesPerNode": 4, + "internalTlsMinVersion": "tlsv1.3", "rebalanceMovesBeforeCompaction": 64, "maxParallelIndexers": 4, "maxParallelReplicaIndexers": 2, "maxBucketCount": 30, + "magmaMinMemoryQuota": 100, "eventLogsLimit": 15000, "gotraceback": "single", - "indexAutoFailoverDisabled": true, + "failoverBulkJanitorFactor": 1, "certUseSha1": false, - "httpNodeAddition": false + "clientCertNotAfterDurationSeconds": 71193600, + "httpNodeAddition": false, + "appTelemetryMinScrapeIntervalSeconds": 60, + "appTelemetryMaxScrapeIntervalSeconds": 600, + "argon2idTime": 3, + "argon2idMem": 524288, + "pbkdf2HmacSha512Iterations": 10000, + "scramShaIterations": 15000, + "useRelativeWebRedirects": true, + "maxDocsSkip": 1000, + "maxDocsLimit": 1000, + "includeUsernameInUICookie": false, + "dataServiceFileBasedRebalanceEnabled": true } ---- -This confirms that the current size of the `eventLogsLimit` is now 15k. +The output confirms that `eventLogsLimit` is now `15000`. + +== Getting the Max Parallel Indexers Setting + +To get the current setting for the maximum number of parallel indexers, send a GET request to the `/settings/maxParallelIndexers` endpoint. + +This setting applies only to View index configuration. +It does not apply to the GSI indexes that the Index Service provides. + +[#get-mpi-curl-syntax] +=== curl Syntax + +[source,bash] +---- +curl -X GET http[s]://{host}:{port}/settings/maxParallelIndexers + -u $USER:$PASSWORD +---- + +==== Path Parameters +:required-privileges: required-privileges-get-mpi +include::partial$user_pwd_host_port_params.adoc[] + +[#required-privileges-get-mpi] +=== Required Privileges + +Your account must have at least 1 of the following roles to make a GET request to this endpoint: + +* xref:learn:security/roles.adoc#full-admin[Full Admin] +* xref:learn:security/roles.adoc#backup-full-admin[Backup Full Admin] +* xref:learn:security/roles.adoc#bucket-admin[Bucket Admin] +* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin] +* xref:learn:security/roles.adoc#eventing-full-admin[Eventing Full Admin] +* xref:learn:security/roles.adoc#query-list-index[Query List Index] +* xref:learn:security/roles.adoc#query-manage-index[Query Manage Index] +* xref:learn:security/roles.adoc#query-system-catalog[Query System Catalog] +* xref:learn:security/roles.adoc#xdcr-admin[XDCR Admin] +* xref:learn:security/roles.adoc#read-only-admin[Read-Only Admin] +* xref:learn:security/roles.adoc#ro-security-admin[Read-Only Security Admin] +* xref:learn:security/roles.adoc#security-admin[Security Admin] +* xref:learn:security/roles.adoc#external-user-security-admin[External User Admin] +* xref:learn:security/roles.adoc#local-user-security-admin[Local User Admin] +* xref:learn:security/roles.adoc#views-admin[Views Admin (deprecated)] + + +=== Responses + +200 OK:: +The request was successful. +Returns a JSON object containing the current global value and the value for each node. + +401 Unauthorized:: +Authentication failed. + +403 Forbidden:: +The account making the request does not have 1 of the roles listed in <>. + +404 Not Found:: +The URI specified an incorrect hostname or port. + +=== Example + +The following example gets the current setting for `maxParallelIndexers`: + +[source,bash] +---- +curl -X GET -u $USERNAME:$PASSWORD \ +http://node1.example.com:8091/settings/maxParallelIndexers +---- + +If successful, the call returns a JSON object such as the following: + +[source,json] +---- +{"globalValue":4,"nodes":{"ns_1@node1.example.com":4}} +---- + +== Setting the Max Parallel Indexers Setting + +To change the maximum number of parallel indexers, send a `POST` to the `/settings/maxParallelIndexers` endpoint, specifying the new value with the `globalValue` parameter. + +[#set-mpi-curl-syntax] +=== curl Syntax + +[source,bash] +---- +curl -X POST http[s]://{host}:{port}/settings/maxParallelIndexers + -u $COUCHBASE_USER:$PASSWORD + -d globalValue= +---- + +==== Path Parameters +:required-privileges: required-privileges-post-mpi +include::partial$user_pwd_host_port_params.adoc[] + +==== Parameters + +`globalValue`:: +(required) Sets the maximum number of parallel indexers cluster-wide. + ++ +Valid values are integers from `1` to `1024`. +The default value is `4`. + +[#required-privileges-post-mpi] +=== Required Privileges + +Your account must have at least 1 of the following roles to make a POST request to this endpoint: + +* xref:learn:security/roles.adoc#full-admin[Full Admin] +* xref:learn:security/roles.adoc#backup-full-admin[Backup Full Admin] +* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin] + + +=== Responses + +200 OK:: +The request was successful. +Returns a JSON object containing the new global value and the value for each node. + +400 Bad Request:: +The request specified a `globalValue` outside the valid range. + +401 Unauthorized:: +Authentication failed. + +403 Forbidden:: +The account making the request does not have 1 of the roles listed in <>. + +404 Not Found:: +The URI specified an incorrect hostname or port. + +=== Example + +The following example sets `maxParallelIndexers` to `3`: + +[source,bash] +---- +curl -X POST -u $USERNAME:$PASSWORD \ +http://node1.example.com:8091/settings/maxParallelIndexers \ +-d globalValue=3 +---- + +If successful, the call returns a JSON object confirming the new value cluster-wide: + +[source,json] +---- +{"globalValue":3,"nodes":{"ns_1@node1.example.com":3}} +---- + +[#see-also] +== See Also + +* xref:rest-api:rest-limit-rebalance-moves.adoc[Limiting Concurrent vBucket Moves] +* xref:learn:clusters-and-availability/system-events.adoc[System Events] +* xref:manage:manage-settings/manage-settings.adoc[Manage Settings] diff --git a/modules/rest-api/pages/rest-limit-rebalance-moves.adoc b/modules/rest-api/pages/rest-limit-rebalance-moves.adoc index 7c14c94218..ad4445428a 100644 --- a/modules/rest-api/pages/rest-limit-rebalance-moves.adoc +++ b/modules/rest-api/pages/rest-limit-rebalance-moves.adoc @@ -1,86 +1,206 @@ = Limiting Concurrent vBucket Moves -:description: Couchbase Server places a limit on the number of concurrent vBucket moves that can occur during rebalance, applying the limit to every node in the cluster. +:description: You can set a limit on the number of vBucket moves a node can take part in concurrently during rebalance. :page-topic-type: reference [abstract] {description} -This limit can be inspected and reset by means of the REST API. +Increasing the limit can reduce the duration of a rebalance. +However, it can also affect other database operations such as KV operations. -== HTTP methods and URI +== HTTP Method and URI +.Get vBucket Move Limit +[source,url] ---- GET /settings/rebalance +---- +.Set vBucket Move Limit +[source] +---- POST /settings/rebalance ---- [#description] == Description -These respectively retrieve and establish the setting whereby the number of concurrent vBucket moves that any node can handle during rebalance is limited. -See xref:learn:clusters-and-availability/rebalance.adoc#limiting-vbucket-moves[Limiting vBucket Moves], for detailed information. +Couchbase Server limits the number of vBucket moves a node can take part in concurrently during rebalance. +This endpoint lets you inspect and change that limit. +See xref:learn:clusters-and-availability/rebalance.adoc#limiting-concurrent-vbucket-moves[Limiting Concurrent vBucket Moves] for more information. -[#curl-syntax] -== Curl Syntax +== Getting the Current vBucket Move Limit ----- -curl -X GET -u : - http://:/settings/rebalance +To get the current vBucket move limit for Couchbase Server, send a GET request to the `/settings/rebalance` endpoint. + +[#get-curl-syntax] +=== curl Syntax -curl -X POST -u : - http://:/settings/rebalance - -d rebalanceMovesPerNode= +[source,bash] ---- +curl -X GET http[s]://{host}:{port}/settings/rebalance + -u $USER:$PASSWORD +---- + +==== Path Parameters +:required-privileges: required-privileges-get +include::partial$user_pwd_host_port_params.adoc[] + +[#required-privileges-get] +=== Required Privileges + +Your account must have at least 1 of the following roles to make a GET request to this endpoint: + -The `positive-integer` used as the value for the `rebalanceMovesPerNode` parameter must be between `1` and `64` inclusive, with `4` being the default. +* xref:learn:security/roles.adoc#full-admin[Full Admin] +* xref:learn:security/roles.adoc#backup-full-admin[Backup Full Admin] +* xref:learn:security/roles.adoc#bucket-admin[Bucket Admin] +* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin] +* xref:learn:security/roles.adoc#eventing-full-admin[Eventing Full Admin] +* xref:learn:security/roles.adoc#xdcr-admin[XDCR Admin] +* xref:learn:security/roles.adoc#read-only-admin[Read-Only Admin] +* xref:learn:security/roles.adoc#ro-security-admin[Read-Only Security Admin] +* xref:learn:security/roles.adoc#security-admin[Security Admin] +* xref:learn:security/roles.adoc#external-user-security-admin[External User Admin] +* xref:learn:security/roles.adoc#local-user-security-admin[Local User Admin] +* xref:learn:security/roles.adoc#views-admin[Views Admin] -[#responses] -== Responses +=== Responses -If successful, `200 OK` is given, and an object containing group-related information is returned. +200 OK:: +The request was successful. +Returns a JSON object containing the current vBucket move limit. -A malformed URI gives `400 Object Not Found`. -Failure to authenticate gives `401 Unauthorized`. +401 Unauthorized:: +Authentication failed. -If an inappropriate value is specified, the following error-notification is returned: `{"errors":{"rebalanceMovesPerNode":"The value must be in range from 1 to 64"}}`. +403 Forbidden:: +The account making the request does not have 1 of the roles listed in <>. -[#examples] -== Examples +=== Example -The following example retrieves the current setting for vBucket move limitation: +The following example gets the current vBucket move limit for Couchbase Server: +[source,bash] ---- -curl -X GET -u Administrator:password \ -http://localhost:8091/settings/rebalance +curl -X GET -u $USERNAME:$PASSWORD \ +http://localhost:8091/settings/rebalance | jq ---- -If successful, the call returns an object such as the following: +If successful, the call returns a JSON object containing the current setting: +[source,json] ---- -{"rebalanceMovesPerNode":4} +{ + "rebalanceMovesPerNode": 4 +} ---- -This indicates that the current setting is `4`, which is the default. +This shows that the current setting is `4`, which is the default. + +== Setting the vBucket Move Limit + +To limit the number of vBucket moves a node can perform concurrently, send a `POST` to the `/settings/rebalance` endpoint. + +[#curl-syntax] +=== curl Syntax + +[source,bash] +---- +curl -X POST http[s]://{host}:{port}/settings/rebalance + -u $USER:$PASSWORD + [-d rebalanceMovesPerNode=] + [-d dataServiceFileBasedRebalanceMovesPerNode=] +---- + +==== Path Parameters +:required-privileges: required-privileges-post +include::partial$user_pwd_host_port_params.adoc[] + +==== Parameters -The following example establishes a new value for the setting: +`rebalanceMovesPerNode`:: +(optional) Sets the maximum number of vBucket moves any node can take part in concurrently during a DCP rebalance. ++ +Valid values are integers from `1` to `64`. +The default value is `4`. + ++ +Increasing this value can reduce the duration of a rebalance. +However, it can also affect other database operations, such as KV operations, so test any changes to this setting before applying them to production systems. + +`dataServiceFileBasedRebalanceMovesPerNode`:: +(optional) Sets the number of simultaneous transfers a node can participate in when performing a file-based rebalance (FBR) backfill. +This limit applies to vBucket file transfers where the node is the source or target of the transfer. + ++ +Valid values are integers from `1` to `64`. +The default value is `4`. + ++ +See xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)] for more information. + +[#required-privileges-post] +=== Required Privileges + +Your account must have at least 1 of the following roles to make a POST request to this endpoint: + +* xref:learn:security/roles.adoc#full-admin[Full Admin] +* xref:learn:security/roles.adoc#backup-full-admin[Backup Full Admin] +* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin] + + +=== Responses + +200 OK:: +The request was successful. +Returns a JSON object containing the new vBucket move limit. + +400 Bad Request:: +The request had errors or an invalid value for `rebalanceMovesPerNode` or `dataServiceFileBasedRebalanceMovesPerNode`. +Also returns a JSON object containing an error message. + +401 Unauthorized:: +Authentication failed. + +403 Forbidden:: +The account making the request does not have 1 of the roles listed in <>. + +=== Example + +The following example sets the vBucket move limit to `12`: + +[source,bash] ---- -curl -X POST -u Administrator:password \ +curl -X POST -u $USERNAME:$PASSWORD \ http://localhost:8091/settings/rebalance \ --d rebalanceMovesPerNode=12 +-d rebalanceMovesPerNode=12 | jq ---- -If successful, the call returns an object such as the following: +If successful, the call returns a JSON object confirming the new setting: +[source,json] ---- -{"rebalanceMovesPerNode":12} +{ + "rebalanceMovesPerNode": 12 +} ---- -This confirms that the value has been successfully reset to `12`. +If you specify a value outside the valid range, Couchbase Server returns an error such as the following: + +[source,json] +---- +{ + "errors": { + "rebalanceMovesPerNode": "The value must be in range from 1 to 64" + } +} +---- [#see-also] == See Also -A complete conceptual overview of rebalance is provided in xref:learn:clusters-and-availability/rebalance.adoc[Rebalance]. -Information on inspecting and modifying Couchbase-Server settings is provided in xref:manage:manage-settings/manage-settings.adoc[Manage Settings]. -Various examples of performing rebalance and other node-management operations can be found in xref:manage:manage-nodes/node-management-overview.adoc[Manage Nodes and Clusters]. +* xref:learn:clusters-and-availability/rebalance.adoc[Rebalance] +* xref:rest-api:file-based-data-rebalance.adoc[Configure File-Based Data Rebalance] +* xref:manage:manage-settings/manage-settings.adoc[Manage Settings] +* xref:manage:manage-nodes/node-management-overview.adoc[Manage Nodes and Clusters] diff --git a/modules/rest-api/pages/rest-reader-writer-thread-config.adoc b/modules/rest-api/pages/rest-reader-writer-thread-config.adoc index 797f2b369c..5ce0d34f8c 100644 --- a/modules/rest-api/pages/rest-reader-writer-thread-config.adoc +++ b/modules/rest-api/pages/rest-reader-writer-thread-config.adoc @@ -30,6 +30,8 @@ For more information, see xref:learn:data/durability.adoc[Durability]. However, allocating too many threads can reduce performance on nodes with limited resources. Test any changes to thread allocation before applying them to production systems. +This endpoint also lets you limit the bandwidth each node uses when downloading vBucket snapshots during a file-based rebalance (FBR) backfill. +See xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)] for more information. == Getting Current Thread Settings @@ -75,11 +77,13 @@ The following example shows how to get the current global thread settings for Co [source,bash] ---- -curl -X GET -u Administrator:password \ -http://localhost:8091/pools/default/settings/memcached/global | jq +curl -X GET -u $USERNAME:$PASSWORD \ +http://node1.example.com:8091/pools/default/settings/memcached/global | jq ---- -If successful, the call returns a JSON object containing any overrides to the default global thread settings: +If successful, the call returns a JSON object containing any overrides to the default global thread settings. +If there are no overrides, the command returns an empty JSON object. +The following example shows the result after two settings have been overridden. [source,json] ---- @@ -106,6 +110,7 @@ curl -X POST http[s]://{host}:{port}/pools/default/settings/memcached/global [-d num_auxio_threads=] [-d num_storage_threads=] [-d magma_flusher_thread_percentage=] + [-d snapshot_download_throttle_bytes=] ---- @@ -165,6 +170,22 @@ NOTE: `num_storage_threads` and `magma_flusher_thread_percentage` are advanced s Contact Couchbase Support before making changes to them. Support can help you determine the best settings for your workload and hardware. +[#fbr-throttle] +`snapshot_download_throttle_bytes`:: +(optional) Limits the number of bytes per second each node can read to download vBucket snapshots during a xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance] (FBR) backfill. +The default value is 0, which does not limit the download rate. +The maximum is the maximum unsigned 64-bit integer value (2^64^-1). + ++ +[NOTE] +==== +Setting this value to too high (or leaving it set at the default value of 0, which lets FBR use unlimited bandwidth) can increase the latency of KV operations. +If KV latency during rebalance become an issue, consider adjusting this value to limit the network bandwidth available to FBR. + +You can monitor the xref:metrics-reference:data-service-metrics.adoc#kv_ep_snapshot_read_bytes[`kv_ep_snapshot_read_bytes`] metric to find the bandwidth FBR is consuming when transferring vBucket snapshots. +See xref:manage:monitor/set-up-prometheus-for-monitoring.adoc[] for more information about monitoring metrics. +==== + [#required-privileges-post] === Required Privileges @@ -192,8 +213,8 @@ To set the numbers of reader, writer, NonIO, and AuxIO threads for Couchbase Ser [source,bash] ---- -curl -X POST -u Administrator:password \ -http://localhost:8091/pools/default/settings/memcached/global \ +curl -X POST -u $USERNAME:$PASSWORD \ +http://node1.example.com:8091/pools/default/settings/memcached/global \ -d num_reader_threads=12 \ -d num_writer_threads=8 \ -d num_nonio_threads=6 \ @@ -217,8 +238,8 @@ The following example increases the Magma storage engine's thread pool to 30 thr [source,bash] ---- -curl -X POST -u Administrator:password \ - http://localhost:8091/pools/default/settings/memcached/global \ +curl -X POST -u $USERNAME:$PASSWORD \ + http://node1.example.com:8091/pools/default/settings/memcached/global \ -d num_storage_threads=30 \ -d magma_flusher_thread_percentage=25 | jq ---- @@ -233,6 +254,30 @@ If successful, the call returns an object confirming your new settings: } ---- +[#fbr-throttle-example] +This example limits FBR file downloads to 600MiB/s. + +[source,bash] +---- +curl -X POST -u $USERNAME:$PASSWORD \ + http://node1.example.com:8091/pools/default/settings/memcached/global \ + -d snapshot_download_throttle_bytes=629145600 | jq +---- + +If successful, the result returns the current settings: + +[source,json] +---- +{ + "num_reader_threads": "balanced", + "num_writer_threads": "balanced", + "num_storage_threads": "default", + "snapshot_download_throttle_bytes": 629145600, + "throttle_enabled": false, + "node_capacity": 18446744073709551615 +} +---- + [#see-also] == See Also diff --git a/modules/rest-api/pages/rest-rebalance-overview.adoc b/modules/rest-api/pages/rest-rebalance-overview.adoc index ebbe8b5c2c..ab77e5f837 100644 --- a/modules/rest-api/pages/rest-rebalance-overview.adoc +++ b/modules/rest-api/pages/rest-rebalance-overview.adoc @@ -1,5 +1,5 @@ = Rebalance -:description: pass:q[When one or more nodes have been brought into or taken out of a cluster, _rebalance_ redistributes data, indexes, event processing, and query processing among available nodes.] +:description: pass:q[When one or more nodes have been brought into or taken out of a cluster, rebalance redistributes data, indexes, event processing, and query processing among available nodes.] :page-topic-type: reference [abstract] @@ -8,7 +8,7 @@ Rebalance can be performed and configured by means of the REST API. == APIs in this Section -_Rebalance_ must be performed whenever the number of nodes in a cluster have changed, and whenever buckets have been added or removed. +Rebalance must be performed whenever the number of nodes in a cluster have changed, and whenever buckets have been added or removed. A complete overview is provided in xref:learn:clusters-and-availability/rebalance.adoc[Rebalance]. The REST API for rebalance is as follows: diff --git a/modules/rest-api/partials/rest-rebalance-table.adoc b/modules/rest-api/partials/rest-rebalance-table.adoc index 3bb6fb516e..b284ee1719 100644 --- a/modules/rest-api/partials/rest-rebalance-table.adoc +++ b/modules/rest-api/partials/rest-rebalance-table.adoc @@ -42,4 +42,12 @@ | `/internalSettings` | xref:rest-api:rest-cluster-disable-query.adoc[Disabling Consistent View Query Results on Rebalance] +| `GET` +| `/internalSettings` +| xref:rest-api:file-based-data-rebalance.adoc#get-settings[Get File-Based Rebalance (FBR) Settings] + +| `POST` +| `/internalSettings` +| xref:rest-api:file-based-data-rebalance.adoc#set-settings[Configure FBR] + |=== diff --git a/modules/rest-api/partials/user_pwd_host_port_params.adoc b/modules/rest-api/partials/user_pwd_host_port_params.adoc index 6f52d5f4eb..302ebabaac 100644 --- a/modules/rest-api/partials/user_pwd_host_port_params.adoc +++ b/modules/rest-api/partials/user_pwd_host_port_params.adoc @@ -1,14 +1,14 @@ // Be sure to set the 'required-privileges' attribute before including this partial. // It must be the anchor to the Required Privileges section for the specific endpoint. -`host`:: -Hostname or IP address of a Couchbase Server. +`HOST`:: +Hostname or IP address of a Couchbase Server node. -`port`:: +`PORT`:: Port number for the REST API. Defaults are 8091 for unencrypted and 18901 for encrypted connections. -`$USER`:: +`USER`:: The name of a user who has at least 1 of the roles listed in xref:{required-privileges}[Required Privileges]. -`$PASSWORD`:: +`PASSWORD`:: The password for the user.