Backups

While all components in the system are designed to be crash-fault-tolerant, there is always a chance of failures from which the system cannot immediately recover, e.g. due to misconfiguration or bugs. In such cases, you will need to restore a component or the a full validator node from backups or from dumps of the components that are still operational.

Backup of Node Identities

Once your validator node is up and onboarded, please make sure to backup the node identities of your node. Note that this information is highly sensitive, and contains the private keys of your participant (unless you configured your validator to use an external KMS), so make sure to store it in a secure location, such as a Secret Manager. On the other hand, it is crucial for maintaining your identity (and thus, e.g. access to your Canton Coin holdings), so must be backed up outside of the cluster.

Your identites may be fetched from your node through the following endpoint:

curl "https://wallet.validator.YOUR_HOSTNAME/api/validator/v0/admin/participant/identities" -H "authorization: Bearer <token>"

where <token> is an OAuth2 Bearer Token with enough claims to access the Validator app, as obtained from your OAuth provider. For context, see the Authentication section here.

If you are using a docker-compose deployment, replace https://wallet.validator.YOUR_HOSTNAME with http://wallet.localhost. If you are running the docker-compose deployment with no auth, you can use the utility Python script get-token.py to generate a token for the curl command by running python get-token.py administrator (requires pyjwt).

Backups of postgres instances

To backup a validator node, we recommend to back up all Postgres instances at least every 4 hours. Note that there is a strict order requirement between the backups: the backup of the validator app’s postgres instance must be taken at a point in time strictly earlier than that of the participant. Please make sure the app’s instance backup is completed before starting the participant one.

If you are running your own Postgres instances in the cluster, backups can be taken either using tools like pg_dump, or through snapshots of the underlying Persistent Volume. Similarly, if you are using Cloud-hosted Postgres, you can either use tools like pg_dump or backup tools provided by the Cloud provider.

If you are running a docker-compose deployment, you can run the following commands to backup the Postgres databases. This will create two dump files, one for the participant and one for the validator app.

docker exec -i compose-postgres-splice-1 pg_dump -U cnadmin validator > "${backup_dir}"/validator-"$(date -u +"%Y-%m-%dT%H:%M:%S%:z")".dump
active_participant_db=$(docker exec compose-participant-1 bash -c 'echo $CANTON_PARTICIPANT_POSTGRES_DB')
docker exec compose-postgres-splice-1 pg_dump -U cnadmin "${active_participant_db}" > "${backup_dir}"/"${active_participant_db}"-"$(date -u +"%Y-%m-%dT%H:%M:%S%:z")".dump

Historical backups

If you have enabled participant pruning, but wish to preserve participant data for longer, e.g. for future auditability, then you should preserve historical backups such that they are apart by no more than the size of your pruning window

This means that backups must be preserved with a time difference between two historical backups smaller than the retention set for the participant.

Furthermore, backups should be retained for previous major upgrades, both the historical backups for the participant and all the backups for the other apps.

Comments