Documented db corruption analysis and recovery procedures - #1539
Open
daravep wants to merge 2 commits into
Open
Conversation
Before this pr, we didn't explain how to detect and recover broken databases. Now, we have a summary from past experience. Signed-off-by: Ratko Veprek <ratko@digitalasset.com>
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
mziolekda
reviewed
Sep 3, 2026
|
|
||
| # Use a Professional Cloud Operator | ||
|
|
||
| While there are procedures to recover a node's data from the network, they are rather complex (there is an incoming improvement with GetRemoteAcs in 3.7), time-consuming, and only allow you to recover the current state, not history. The simpler approach is therefore to avoid performing these procedures by not corrupting your data. |
There was a problem hiding this comment.
Suggested change
| While there are procedures to recover a node's data from the network, they are rather complex (there is an incoming improvement with GetRemoteAcs in 3.7), time-consuming, and only allow you to recover the current state, not history. The simpler approach is therefore to avoid performing these procedures by not corrupting your data. | |
| While there are procedures to recover a node's data from the network, they are rather complex, time-consuming, and only allow you to recover the current state, not history. The simpler approach is therefore to avoid performing these procedures by not corrupting your data. |
We will not remember to change it back
mziolekda
reviewed
Sep 3, 2026
|
|
||
| While there are procedures to recover a node's data from the network, they are rather complex (there is an incoming improvement with GetRemoteAcs in 3.7), time-consuming, and only allow you to recover the current state, not history. The simpler approach is therefore to avoid performing these procedures by not corrupting your data. | ||
|
|
||
| Operating a database is not trivial, especially when dealing with faults and issues. The simplest way is to rely on cloud-based SQL services offered by all cloud providers, and additionally export the database and store it securely in a second location (ideally encrypted). |
There was a problem hiding this comment.
Suggested change
| Operating a database is not trivial, especially when dealing with faults and issues. The simplest way is to rely on cloud-based SQL services offered by all cloud providers, and additionally export the database and store it securely in a second location (ideally encrypted). | |
| Operating a database is not trivial, especially when handling faults and operational issues. The simplest approach leverages cloud-native SQL services offered by all cloud providers, complemented by exporting the database and storing a secure and encrypted backup in a secondary location. |
mziolekda
reviewed
Sep 3, 2026
| The following action items may help you to prevent data corruption. | ||
|
|
||
| - **Prevent Duplicate Volume Access:** If two Postgres instances access the same data volume, they can corrupt the data directory. Postgres does not auto-detect this situation in containerized deployments (as each process runs in its own namespace with pid=1). Therefore, you must ensure that this doesn’t happen at the orchestration and operations layers! | ||
| In Kubernetes, use StatefulSets and use at least ReadWriteOnce (or even better ReadWriteOncePod) to protect the volume. |
There was a problem hiding this comment.
Why not say use ReadWriteOncePod full stop?
mziolekda
reviewed
Sep 3, 2026
| In Kubernetes, use StatefulSets and use at least ReadWriteOnce (or even better ReadWriteOncePod) to protect the volume. | ||
| If you are using Docker Compose, be very careful when manipulating compose.yaml or changing volume and Postgres service names. Always check what is running before operating the cluster, and manually shut down any existing ones before starting new containers. Never run with `--scale=2` and similar. | ||
|
|
||
| - **Ensure Postgres Shuts Down Cleanly:** Postgres is designed to recover from crashes and sudden system shutdowns. However, recovering from a crash delays startup and therefore the restart time, and increases the risk of data corruption by triggering bugs in either the containerization, the file system, or the disk driver. It may also be a sign of duplicate access to the volume. Therefore, monitor the postgres.log of your database, and check whether Postgres is invoking crash recovery, which is indicated by the following log line: |
There was a problem hiding this comment.
Suggested change
| - **Ensure Postgres Shuts Down Cleanly:** Postgres is designed to recover from crashes and sudden system shutdowns. However, recovering from a crash delays startup and therefore the restart time, and increases the risk of data corruption by triggering bugs in either the containerization, the file system, or the disk driver. It may also be a sign of duplicate access to the volume. Therefore, monitor the postgres.log of your database, and check whether Postgres is invoking crash recovery, which is indicated by the following log line: | |
| - **Ensure Postgres Shuts Down Cleanly:** Postgres is designed to recover from crashes and sudden system shutdowns. However, recovering from a crash delays startup and lengthens the overall restart time. That in turn increases the risk of data corruption by triggering bugs in either the containerization, the file system, or the disk driver. It may also trigger the edge cases leading to duplicate access to the volume. Therefore, monitor the postgres.log of your database, and check whether Postgres is invoking crash recovery, which is indicated by the following log line: |
mziolekda
reviewed
Sep 3, 2026
|
|
||
| The only case where a database might be repaired is when some indexes on auxiliary tables are broken. In this case, a simple REINDEX TABLE <tablename> will rebuild the database index. This might be an option in case the issue affects a section of the database where errors cannot propagate. Note that broken indexes may result from hardware failure; as such, further investigation is necessary. | ||
|
|
||
| # Final Summary |
There was a problem hiding this comment.
We do not summarize the whole article here. I would rephrase it to
Suggested change
| # Final Summary | |
| # Final Takeaway |
mziolekda
reviewed
Sep 3, 2026
|
|
||
| # Final Summary | ||
|
|
||
| There has been no reported case of database corruption in Canton operated on Cloud SQL. The same cannot be said for databases run in containers, whether on K8S or Docker Compose setups. |
There was a problem hiding this comment.
Suggested change
| There has been no reported case of database corruption in Canton operated on Cloud SQL. The same cannot be said for databases run in containers, whether on K8S or Docker Compose setups. | |
| There has been no reported case of database corruption in Canton operated on cloud SQL. The same cannot be said for databases run in containers, whether on K8S or Docker Compose setups. Therefore, if you want to avoid troubles use a cloud-native SQL solution. |
Signed-off-by: Ratko Veprek <ratko@digitalasset.com>
brianweir-da
reviewed
Sep 4, 2026
|
|
||
| The following action items may help you to prevent data corruption. | ||
|
|
||
| - **Prevent Duplicate Volume Access:** If two Postgres instances access the same data volume, they can corrupt the data directory. Postgres does not auto-detect this situation in containerized deployments (as each process runs in its own namespace with pid=1). Therefore, you must ensure that this doesn’t happen at the orchestration and operations layers! |
There was a problem hiding this comment.
I believe you mean pod, not namespace in this regard?
Suggested change
| - **Prevent Duplicate Volume Access:** If two Postgres instances access the same data volume, they can corrupt the data directory. Postgres does not auto-detect this situation in containerized deployments (as each process runs in its own namespace with pid=1). Therefore, you must ensure that this doesn’t happen at the orchestration and operations layers! | |
| - **Prevent Duplicate Volume Access:** If two Postgres instances access the same data volume, they can corrupt the data directory. Postgres does not auto-detect this situation in containerized deployments (as each process runs in its own pod with pid=1). Therefore, you must ensure that this doesn’t happen at the orchestration and operations layers! |
brianweir-da
reviewed
Sep 4, 2026
|
|
||
| # Repairing Data Corruption | ||
|
|
||
| There are several ways to recover from disasters, documented already here. https://docs.canton.network/global-synchronizer/production-operations/validator-disaster-recovery and global-synchronizer/production-operations/disaster-recovery |
There was a problem hiding this comment.
Suggested change
| There are several ways to recover from disasters, documented already here. https://docs.canton.network/global-synchronizer/production-operations/validator-disaster-recovery and global-synchronizer/production-operations/disaster-recovery | |
| There are several ways to recover from disasters. Please review the documentation https://docs.canton.network/global-synchronizer/production-operations/validator-disaster-recovery and https://docs.canton.network/global-synchronizer/production-operations/disaster-recovery. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before this pr, we didn't explain how to detect and recover broken databases. Now, we have a summary from past experience.