Skip to main content

Recurring Snapshots

Recurring snapshots are handled differently based on your version of RKE.

To schedule automatic recurring etcd snapshots, you can enable the etcd-snapshot service with extra configuration options. etcd-snapshot runs in a service container alongside the etcd container. By default, the etcd-snapshot service takes a snapshot for every node that has the etcd role and stores them to local disk in /opt/rke/etcd-snapshots.

If you set up the options for S3, the snapshot will also be uploaded to the S3 backend.

Snapshot Service Logging​

When a cluster is launched with the etcd-snapshot service enabled, you can view the etcd-rolling-snapshots logs to confirm backups are being created automatically.

$ docker logs etcd-rolling-snapshots

time="2018-05-04T18:39:16Z" level=info msg="Initializing Rolling Backups" creation=1m0s retention=24h0m0s
time="2018-05-04T18:40:16Z" level=info msg="Created backup" name="2018-05-04T18:40:16Z_etcd" runtime=108.332814ms
time="2018-05-04T18:41:16Z" level=info msg="Created backup" name="2018-05-04T18:41:16Z_etcd" runtime=92.880112ms
time="2018-05-04T18:42:16Z" level=info msg="Created backup" name="2018-05-04T18:42:16Z_etcd" runtime=83.67642ms
time="2018-05-04T18:43:16Z" level=info msg="Created backup" name="2018-05-04T18:43:16Z_etcd" runtime=86.298499ms

Options for the Etcd-Snapshot Service​

OptionDescriptionS3 Specific
interval_hoursThe duration in hours between recurring backups. This supercedes the creation option (which was used in RKE before v0.2.0) and will override it if both are specified. (Default: 12)
retentionThe number of snapshots to retain before rotation. If the retention is configured in both etcd.retention (time period to keep snapshots in hours), which was required in RKE before v0.2.0, and at etcd.backup_config.retention (number of snapshots), the latter will be used. (Default: 6)
bucket_nameS3 bucket name where backups will be stored*
folderFolder inside S3 bucket where backups will be stored. This is optional. Available as of v0.3.0*
access_keyS3 access key with permission to access the backup bucket.*
secret_keyS3 secret key with permission to access the backup bucket.*
regionS3 region for the backup bucket. This is optional.*
endpointS3 regions endpoint for the backup bucket.*
custom_caCustom certificate authority to use when connecting to the endpoint. Only required for private S3 compatible storage solutions. Available for RKE v0.2.5+.*

The --access-key and --secret-key options are not required if the etcd nodes are AWS EC2 instances that have been configured with a suitable IAM instance profile.

Using a custom CA certificate for S3​

The backup snapshot can be stored on a custom S3 backup like minio. If the S3 backend uses a self-signed or custom certificate, provide a custom certificate using the option custom_ca to connect to the S3 backend.

IAM Support for Storing Snapshots in S3​

In addition to API access keys, RKE supports using IAM roles for S3 authentication. The cluster etcd nodes must be assigned an IAM role that has read/write access to the designated backup bucket on S3. Also, the nodes must have network access to the S3 endpoint specified.

Below is an example IAM policy that would allow nodes to store and retrieve backups from S3:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListObjectsInBucket",
            "Effect": "Allow",
            "Action": ["s3:ListBucket"],
            "Resource": ["arn:aws:s3:::bucket-name"]
        },
        {
            "Sid": "AllObjectActions",
            "Effect": "Allow",
            "Action": "s3:*Object",
            "Resource": ["arn:aws:s3:::bucket-name/*"]
        }
    ]
}

For details on giving an application access to S3, refer to the AWS documentation on Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances.

Configuring the Snapshot Service in YAML​

services:
  etcd:
    backup_config:
      interval_hours: 12
      retention: 6
      s3backupconfig:
        access_key: S3_ACCESS_KEY
        secret_key: S3_SECRET_KEY
        bucket_name: s3-bucket-name
        region: ""
        folder: "" # Optional - Available as of v0.3.0
        endpoint: s3.amazonaws.com
        custom_ca: |-
          -----BEGIN CERTIFICATE-----
          $CERTIFICATE
          -----END CERTIFICATE-----