Update GHES 3.13, 3.14, 3.15, 3.16 release notes (Elasticsearch data loss with ghe-repl-teardown | ghe-repl-promote) (#54639)

kyanny · sophietheking · web-flow · commit f46a691d24bd · 2025-03-12T11:02:54.000Z
Co-authored-by: Sophie &lt;29382425+sophietheking@users.noreply.github.com&gt;
diff --git a/data/release-notes/enterprise-server/3-13/0.yml b/data/release-notes/enterprise-server/3-13/0.yml
@@ -191,6 +191,10 @@ sections:
       {% data reusables.release-notes.2024-11-ghe-repl-promote-primary-down %}
 
       [Updated: 2024-11-29]
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
 
   deprecations:
     # https://github.com/github/releases/issues/2732
diff --git a/data/release-notes/enterprise-server/3-13/10.yml b/data/release-notes/enterprise-server/3-13/10.yml
@@ -52,3 +52,7 @@ sections:
       When restoring data originally backed up from a 3.13 appliance onto a 3.13 appliance, the elasticsearch indices need to be reindexed before some of the data will show up.  This happens via a nightly scheduled job.  It can also be forced by running `/usr/local/share/enterprise/ghe-es-search-repair`.
     - |
       When restoring from a backup snapshot, a large number of `mapper_parsing_exception` errors may be displayed.
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
diff --git a/data/release-notes/enterprise-server/3-13/11.yml b/data/release-notes/enterprise-server/3-13/11.yml
@@ -62,7 +62,10 @@ sections:
       When restoring data originally backed up from a 3.13 appliance onto a 3.13 appliance, the elasticsearch indices need to be reindexed before some of the data will show up.  This happens via a nightly scheduled job.  It can also be forced by running `/usr/local/share/enterprise/ghe-es-search-repair`.
     - |
       When restoring from a backup snapshot, a large number of `mapper_parsing_exception` errors may be displayed.
-      
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
   errata:
     - |
       The warning and known issues section have been updated to accurately reflect that instances installed on GCP will face issues while hotpatching to 3.13.11. Previously, the warning and known issue indicated that customers would face issues either while upgrading or hotpatching to version 3.13.11. [Updated: 2025-03-11]
diff --git a/data/release-notes/enterprise-server/3-13/2.yml b/data/release-notes/enterprise-server/3-13/2.yml
@@ -175,3 +175,7 @@ sections:
       {% data reusables.release-notes.2024-11-ghe-repl-promote-primary-down %}
 
       [Updated: 2024-11-29]
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
diff --git a/data/release-notes/enterprise-server/3-13/3.yml b/data/release-notes/enterprise-server/3-13/3.yml
@@ -131,6 +131,10 @@ sections:
       {% data reusables.release-notes.2024-11-ghe-repl-promote-primary-down %}
 
       [Updated: 2024-11-29]
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
 
   errata:
     - |
diff --git a/data/release-notes/enterprise-server/3-13/4.yml b/data/release-notes/enterprise-server/3-13/4.yml
@@ -80,6 +80,10 @@ sections:
       {% data reusables.release-notes.2024-11-ghe-repl-promote-primary-down %}
 
       [Updated: 2024-11-29]
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
 
   errata:
     - 'The [Known issues](/admin/release-notes#3.13.4-known-issues) section previously indicated that `Instance setup in AWS with IMDSv2 enforced fails if no public IP is present` is still an issue. The issue is resolved and is documented in the [Bug fixes](/admin/release-notes#3.13.4-bugs) section. [Updated: 2024-09-30]'
diff --git a/data/release-notes/enterprise-server/3-13/5.yml b/data/release-notes/enterprise-server/3-13/5.yml
@@ -62,3 +62,7 @@ sections:
       {% data reusables.release-notes.2024-11-ghe-repl-promote-primary-down %}
 
       [Updated: 2024-11-29]
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
diff --git a/data/release-notes/enterprise-server/3-13/6.yml b/data/release-notes/enterprise-server/3-13/6.yml
@@ -68,3 +68,7 @@ sections:
       {% data reusables.release-notes.2024-11-ghe-repl-promote-primary-down %}
 
       [Updated: 2024-11-29]
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
diff --git a/data/release-notes/enterprise-server/3-13/7.yml b/data/release-notes/enterprise-server/3-13/7.yml
@@ -32,3 +32,7 @@ sections:
       {% data reusables.release-notes.2024-11-ghe-repl-promote-primary-down %}
 
       [Updated: 2024-11-29]
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
diff --git a/data/release-notes/enterprise-server/3-13/8.yml b/data/release-notes/enterprise-server/3-13/8.yml
@@ -33,3 +33,7 @@ sections:
       Services may respond with a `503` status due to an out of date `haproxy` configuration. This can usually be resolved with a `ghe-config-apply` run.
     - |
       Attempting to stop replications after stopping GitHub Actions on a GitHub Enterprise Server instance would fail, reporting that MSSQL was not responding. The can be avoided by start MSSQL prior to stopping replication  `/usr/local/share/enterprise/ghe-nomad-jobs queue /etc/nomad-jobs/mssql/mssql.hcl`
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
diff --git a/data/release-notes/enterprise-server/3-13/9.yml b/data/release-notes/enterprise-server/3-13/9.yml
@@ -58,3 +58,7 @@ sections:
       When restoring data originally backed up from a 3.13 appliance onto a 3.13 appliance, the elasticsearch indices need to be reindexed before some of the data will show up.  This happens via a nightly scheduled job.  It can also be forced by running `/usr/local/share/enterprise/ghe-es-search-repair`.
     - |
       When restoring from a backup snapshot, a large number of `mapper_parsing_exception` errors may be displayed.
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
diff --git a/data/release-notes/enterprise-server/3-14/0.yml b/data/release-notes/enterprise-server/3-14/0.yml
@@ -221,13 +221,17 @@ sections:
       {% data reusables.release-notes.2024-11-ghe-repl-promote-primary-down %}
 
       [Updated: 2024-11-29]
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
 
   deprecations:
     - |
       The Manage GHES API reached feature parity with the Management Console API in GHES 3.12. As a result, we will remove the Management Console API in GitHub Enterprise Server 3.15. For information about updating tooling that relies on the Management Console API, see [AUTOTITLE](/rest/enterprise-admin/management-console).
     - |
       Team discussions have been removed from {% data variables.product.prodname_ghe_server %}. The sunset of this feature was announced in 2023. See the [GitHub Blog post](https://github.blog/changelog/2023-02-08-sunset-notice-team-discussions/). [Updated: 2025-02-13]
-  
+
   errata:
     - |
       These release notes previously indicated as a known issue that on GitHub Enterprise Server 3.14.0 when log forwarding is enabled, some forwarded log entries may be duplicated.
diff --git a/data/release-notes/enterprise-server/3-14/1.yml b/data/release-notes/enterprise-server/3-14/1.yml
@@ -80,3 +80,7 @@ sections:
       {% data reusables.release-notes.2024-11-ghe-repl-promote-primary-down %}
 
       [Updated: 2024-11-29]
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
diff --git a/data/release-notes/enterprise-server/3-14/2.yml b/data/release-notes/enterprise-server/3-14/2.yml
@@ -84,9 +84,11 @@ sections:
       {% data reusables.release-notes.2024-11-ghe-repl-promote-primary-down %}
 
       [Updated: 2024-11-29]
-      
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
+
   deprecations:
     - |
       The option to "copy Storage settings from Actions" in the Management Console ("GitHub Packages" > "Packages Storage Settings") has been removed. [Updated: 2024-11-20]
-      
-  
diff --git a/data/release-notes/enterprise-server/3-14/3.yml b/data/release-notes/enterprise-server/3-14/3.yml
@@ -82,3 +82,7 @@ sections:
       {% data reusables.release-notes.2024-11-ghe-repl-promote-primary-down %}
 
       [Updated: 2024-11-29]
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
diff --git a/data/release-notes/enterprise-server/3-14/4.yml b/data/release-notes/enterprise-server/3-14/4.yml
@@ -42,3 +42,7 @@ sections:
       {% data reusables.release-notes.2024-11-ghe-repl-promote-primary-down %}
 
       [Updated: 2024-11-29]
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
diff --git a/data/release-notes/enterprise-server/3-14/5.yml b/data/release-notes/enterprise-server/3-14/5.yml
@@ -43,3 +43,7 @@ sections:
       Services may respond with a `503` status due to an out of date `haproxy` configuration. This can usually be resolved with a `ghe-config-apply` run.
     - |
       Attempting to stop replications after stopping GitHub Actions on a GitHub Enterprise Server instance would fail, reporting that MSSQL was not responding. The can be avoided by start MSSQL prior to stopping replication  `/usr/local/share/enterprise/ghe-nomad-jobs queue /etc/nomad-jobs/mssql/mssql.hcl`
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
diff --git a/data/release-notes/enterprise-server/3-14/6.yml b/data/release-notes/enterprise-server/3-14/6.yml
@@ -76,3 +76,7 @@ sections:
       When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.
     - |
       When restoring from a backup snapshot, a large number of `mapper_parsing_exception` errors may be displayed.
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
diff --git a/data/release-notes/enterprise-server/3-14/7.yml b/data/release-notes/enterprise-server/3-14/7.yml
@@ -62,3 +62,7 @@ sections:
       When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.
     - |
       When restoring from a backup snapshot, a large number of `mapper_parsing_exception` errors may be displayed.
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
diff --git a/data/release-notes/enterprise-server/3-14/8.yml b/data/release-notes/enterprise-server/3-14/8.yml
@@ -80,7 +80,10 @@ sections:
       When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.
     - |
       When restoring from a backup snapshot, a large number of `mapper_parsing_exception` errors may be displayed.
-  
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
   errata:
     - |
       The warning and known issues section have been updated to accurately reflect that instances installed on GCP will face issues while hotpatching to 3.14.8. Previously, the warning and known issue indicated that customers would face issues either while upgrading or hotpatching to version 3.14.8. [Updated: 2025-03-11]
diff --git a/data/release-notes/enterprise-server/3-15/0.yml b/data/release-notes/enterprise-server/3-15/0.yml
@@ -203,6 +203,10 @@ sections:
       {% data reusables.release-notes.2024-11-ghe-repl-promote-primary-down %}
     - |
       On Azure instances, a failed pre-upgrade check due to insufficient user disk size can result in the Management Console displaying an `Internal Server Error`. To restore access to the Management Console, run `sudo rm /var/log/preflight-check-report.json` to remove the file. If enabled, the `automatic update checks` need to be disabled from the Management Console until user disk size is increased to minimum 500 GB. To increase the user disk size, see [AUTOTITLE](/admin/monitoring-and-managing-your-instance/updating-the-virtual-machine-and-physical-resources/increasing-storage-capacity#increasing-the-data-partition-size).
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
 
   closing_down:
     # https://github.com/github/releases/issues/3525
diff --git a/data/release-notes/enterprise-server/3-15/1.yml b/data/release-notes/enterprise-server/3-15/1.yml
@@ -90,6 +90,11 @@ sections:
       An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning.
     - |
       When initializing a new GHES cluster, nodes with the `consul-server` role should be added to the cluster before adding additional nodes. Adding all nodes simultaneously creates a race condition between nomad server registration and nomad client registration.
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
+
   deprecations:
     # https://github.com/github/releases/issues/4037
     - heading: Upcoming deprecation of projects (classic)
diff --git a/data/release-notes/enterprise-server/3-15/2.yml b/data/release-notes/enterprise-server/3-15/2.yml
@@ -40,7 +40,7 @@ sections:
   closing_down:
     - |
       All users should be aware that GitHub Projects (classic) are closing down. They should migrate to new Projects powered by GitHub Issues. See [AUTOTITLE](/enterprise-server@3.15/issues/planning-and-tracking-with-projects/creating-projects/migrating-from-projects-classic).
-      
+
       GitHub Projects (classic) will be retired in GitHub Enterprise Server 3.17. For more information, see the [Projects (classic) sunset](https://github.blog/changelog/2024-05-23-sunset-notice-projects-classic/) blog post.
   known_issues:
     - |
@@ -77,3 +77,7 @@ sections:
       When initializing a new GHES cluster, nodes with the `consul-server` role should be added to the cluster before adding additional nodes. Adding all nodes simultaneously creates a race condition between nomad server registration and nomad client registration.
     - |
       Admins setting up cluster high availability (HA) may encounter a spokes error when running ghe-cluster-repl-status if a new organization and repositories are created before using the ghe-cluster-repl-bootstrap command. To avoid this issue, complete the cluster HA setup with ghe-cluster-repl-bootstrap before creating new organizations and repositories.
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
diff --git a/data/release-notes/enterprise-server/3-15/3.yml b/data/release-notes/enterprise-server/3-15/3.yml
@@ -88,7 +88,10 @@ sections:
       When initializing a new GHES cluster, nodes with the `consul-server` role should be added to the cluster before adding additional nodes. Adding all nodes simultaneously creates a race condition between nomad server registration and nomad client registration.
     - |
       Admins setting up cluster high availability (HA) may encounter a spokes error when running `ghe-cluster-repl-status` if a new organization and repositories are created before using the `ghe-cluster-repl-bootstrap` command. To avoid this issue, complete the cluster HA setup with `ghe-cluster-repl-bootstrap` before creating new organizations and repositories.
-  
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
   errata:
     - |
       The warning and known issues section have been updated to accurately reflect that instances installed on GCP will face issues while hotpatching to 3.15.3. Previously, the warning and known issue indicated that customers would face issues either while upgrading or hotpatching to version 3.15.3. [Updated: 2025-03-11]
diff --git a/data/release-notes/enterprise-server/3-16/0-rc1.yml b/data/release-notes/enterprise-server/3-16/0-rc1.yml
@@ -236,6 +236,10 @@ sections:
       After a restore, existing outside collaborators cannot be added to repositories in a new organization. This issue can be resolved by running `/usr/local/share/enterprise/ghe-es-search-repair` on the appliance.
     - |
       After a geo-replica is promoted to primary by running `ghe-repl-promote`, the actions workflow of a repository does not have any suggested workflows.
+    - |
+      {% data reusables.release-notes.2025-03-03-elasticsearch-data-loss %}
+
+      [Updated: 2025-03-12]
 
   closing_down:
     # https://github.com/github/releases/issues/4683
diff --git a/data/reusables/release-notes/2025-03-03-elasticsearch-data-loss.md b/data/reusables/release-notes/2025-03-03-elasticsearch-data-loss.md
@@ -0,0 +1,6 @@
+For appliances in a high availability configuration, Elasticsearch indices are deleted in two situations:
+
+* On failover
+* When running `ghe-repl-teardown <REPLICA_HOSTNAME>` from the primary instance
+
+All indices are recoverable, except for Audit Log indices. Since Elasticsearch itself is the source of truth for these logs, they may only be recoverable from a backup. If you need assistance, visit {% data variables.contact.contact_ent_support %}.
