Skip to main content

Elasticsearch — Snapshots

Brandon Langley
Updated

Overview

This guide covers how to use Elasticsearch snapshots to back up and migrate indices between Diskover instances, using AWS S3 as the snapshot repository.

Compatible Versions

Elasticsearch supports restoring a snapshot taken from an older version into a newer version. However, if the repository version is newer than the target Elasticsearch version, the restore will fail.

See the official Elasticsearch snapshot compatibility reference for the full compatibility matrix.

Permissions — S3

To use an S3 bucket for snapshots, create or use an IAM user with access to the target bucket. The following IAM policy grants the required S3 permissions (replace <YOUR-BUCKET-NAME> with your actual bucket):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*",
                "s3-object-lambda:*"
            ],
            "Resource": "arn:aws:s3:::<YOUR-BUCKET-NAME>"
        }
    ]
}

Once the user is created, generate an access key and secret key. These credentials will be added to the Elasticsearch keystore in the next step.

Taking a Snapshot

Elasticsearch Setup

For Elasticsearch v7 or earlier, install the repository-s3 plugin:

/usr/share/elasticsearch/bin/elasticsearch-plugin install repository-s3
systemctl restart elasticsearch

Add AWS credentials to the Elasticsearch keystore:

/usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.access_key
/usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.secret_key

Creating a Snapshot Repository

Register the S3 bucket as a snapshot repository:

curl -X PUT -u elastic:<ES PASSWORD> "https://<ES-HOST-IP>:9200/_snapshot/<REPO NAME>" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "s3",
    "settings": {
      "bucket": "<YOUR BUCKET NAME>",
      "region": "<YOUR BUCKET REGION>"
    }
  }' \
  --cacert "/etc/elasticsearch/certs/http_ca.crt"

Note: If Elasticsearch is not using SSL, omit -u elastic:<ES PASSWORD> and --cacert ..., and use http:// instead of https://.

Verify the repository was created:

curl -X GET -u elastic:<ES PASSWORD> "https://localhost:9200/_snapshot/<REPO NAME>" \
  --cacert "/etc/elasticsearch/certs/http_ca.crt"

Creating a Snapshot

To snapshot all indices:

curl -X PUT -u elastic:<ES PASSWORD> \
  "https://<ES-HOST-IP>:9200/_snapshot/<REPO NAME>/<SNAPSHOT NAME>?wait_for_completion=true" \
  --cacert "/etc/elasticsearch/certs/http_ca.crt"

To snapshot specific indices:

curl -X PUT -u elastic:<ES PASSWORD> \
  "https://<ES-HOST-IP>:9200/_snapshot/<REPO NAME>/<SNAPSHOT NAME>?wait_for_completion=true" \
  -H "Content-Type: application/json" \
  -d '{"indices": "<index_name_1>,<index_name_2>"}' \
  --cacert "/etc/elasticsearch/certs/http_ca.crt"

Verify the snapshot completed successfully:

curl -X GET -u elastic:<ES PASSWORD> \
  "https://<ES-HOST-IP>:9200/_snapshot/<REPO NAME>/<SNAPSHOT NAME>" \
  --cacert "/etc/elasticsearch/certs/http_ca.crt"

Restoring a Snapshot

To restore a snapshot, connect to the same S3 repository used when taking the snapshot. Once connected, you can restore indices into the target Diskover Elasticsearch instance.

Elasticsearch Setup

For Elasticsearch v7 or earlier, install the repository-s3 plugin:

/usr/share/elasticsearch/bin/elasticsearch-plugin install repository-s3
systemctl restart elasticsearch

Add the same AWS credentials used when creating the snapshot:

/usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.access_key
/usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.secret_key

Connecting to the Repository

Use the same repository name that was used when the snapshot was created:

curl -X PUT -u elastic:<ES PASSWORD> "https://<ES-HOST-IP>:9200/_snapshot/<REPO NAME>" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "s3",
    "settings": {
      "bucket": "<YOUR BUCKET NAME>",
      "region": "<YOUR BUCKET REGION>"
    }
  }' \
  --cacert "/etc/elasticsearch/certs/http_ca.crt"

Verify the available snapshots and indices:

curl -X GET -u elastic:<ES PASSWORD> \
  "https://<ES-HOST-IP>:9200/_snapshot/<REPO NAME>/<SNAPSHOT NAME>" \
  --cacert "/etc/elasticsearch/certs/http_ca.crt"

Restoring Indices

Restore one or more specific indices:

curl -X POST -u elastic:<ES PASSWORD> \
  "https://<ES-HOST-IP>:9200/_snapshot/<REPO NAME>/<SNAPSHOT NAME>/_restore?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "indices": "index1,index2",
    "ignore_unavailable": true,
    "include_global_state": false
  }' \
  --cacert "/etc/elasticsearch/certs/http_ca.crt"

Restore the full snapshot (all indices):

curl -X POST -u elastic:<ES PASSWORD> \
  "https://<ES-HOST-IP>:9200/_snapshot/<REPO NAME>/<SNAPSHOT NAME>/_restore?pretty" \
  --cacert "/etc/elasticsearch/certs/http_ca.crt"

Deleting Repositories and Snapshots

Delete a repository:

curl -X DELETE -u elastic:<ES PASSWORD> \
  "https://<ES-HOST-IP>:9200/_snapshot/<REPO NAME>?pretty" \
  --cacert "/etc/elasticsearch/certs/http_ca.crt"

Delete a snapshot:

curl -X DELETE -u elastic:<ES PASSWORD> \
  "https://<ES-HOST-IP>:9200/_snapshot/<REPO NAME>/<SNAPSHOT NAME>?pretty" \
  --cacert "/etc/elasticsearch/certs/http_ca.crt"

Development Environment — Example Restore Command

The following command is a full example restoring the entire data set of a Development environment.

curl -X POST "http://localhost:9200/_snapshot/s3_repository/snapshot_all_1/_restore?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "indices": "diskover-heatmap-broadcast-202503010903,diskover-heatmap-studio-lot-202502220903,diskover-burbank-media-202412220900,diskover-powerscale-202412060900,diskover-xytech-202412170901,diskover-bam-info-202412190901,diskover-workflow-202501250900,diskover-chicago-lung-cancer-202501031000,diskover-burbank-archive-202501020900,diskover-shotgrid-202411210643,diskover-tokyo-pokemon-sprites-202412291300,diskover-demo-content-202501310900,diskover-chicago-cancer-research-202412261000,diskover-pdfs-resumes-202412290900,diskover-tokyo-game-images-202412191658,diskover-pdfs-legal-202412250900,diskover-s3-sports-202503060900,diskover-s3-workout-202503050900,diskover-images-sample-images-202411210707,diskover-images-test-images-202411210707,diskover-images-imagenet-sample-images-202412100611,diskover-azure_blob-storage-diskover-241111110411,diskover-images-exif-samples-202412100614,diskover-sharepoint",
    "ignore_unavailable": true,
    "include_global_state": false
  }'

Related articles

Comments

0 comments

Please sign in to leave a comment.

Powered by Zendesk