Elasticsearch 6.2 backups on CentOS 7/RHEL 7

I need to run Elasticsearch 6.2 and upgrade to Elasticsearch 7.11, making snapshots at each stage (prior to 6.8 upgrade and 7.11). In order to install ES 6.2.2, this was downloaded and built from a zipfile as the RPM format is incompatible with newer rpm releases. This method might not be a complete as installing a package from the repo, The problem I had was with the path.repo not being defined and this needed a particular format to get working. Here is the setup prior to upgrading.

Problem

[[email protected] elasticsearch]# curl -X PUT "es-node-01:9200/_snapshot/my_backup?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "fs",
  "settings": {
    "location": "/var/lib/elasticsearch_backup/feb2021"
  }
}
'

Error
........
location [/var/lib/elasticsearch_backup] doesn't match any of the locations specified by path.repo because this setting is empty

This is because the path.repo needs adding to the elasticsearch.yml after a shared directory is created on all 3 nodes. I used NFS for this.

Cluster layout

3 Node Elasticsearch cluster, with the following IPs and /etc/hosts entries:

[[email protected] elasticsearch]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.122.106   centos8-3
192.168.122.101   es-node-01
192.168.122.102   es-node-02

Install the nfs-utils, open the firewalld ports if necessary and share the /var/lib/elasticsearch_backup via NFS:

[[email protected] elasticsearch]# firewall -cmd --add-service nfs --permanent; firewall-cmd --reload
[[email protected] elasticsearch]# ls -ld /var/lib/elasticsearch_backup/
drwxrwx---. 3 elasticsearch elasticsearch 21 Feb 21 15:19 /var/lib/elasticsearch_backup/

[[email protected] elasticsearch]# cat /etc/exports
/var/lib/elasticsearch_backup 192.168.122.0/24(rw,no_root_squash)

[[email protected] sysconfig]# mkdir /var/lib/elasticsearch_backup
[[email protected] sysconfig]# mount es-node-01:/var/lib/elasticsearch_backup /var/lib/elasticsearch_backup

[[email protected] sysconfig]# mkdir /var/lib/elasticsearch_backup
[[email protected] sysconfig]# mount es-node-01:/var/lib/elasticsearch_backup /var/lib/elasticsearch_backup

You may wish to make these mounts permanent in the /etc/fstab. Ensure that the path.repo is defined in the /etc/elasticsearch/elasticsearch.yml. I had to doublequote this and leave inside square braces. YMMV.

[[email protected] elasticsearch]# egrep -v "^#" /etc/elasticsearch/elasticsearch.yml
cluster.name: es-cluster1
node.name: es-node-01
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

path.repo: ["/var/lib/elasticsearch_backup"]
network.host: 192.168.122.101
discovery.zen.ping.unicast.hosts: ["es-node-01", "es-node-02", "es-node-03"]
discovery.zen.minimum_master_nodes: 2
node.master: true
node.data: true

Restart elasticsearch on all nodes, then register my_backup to be feb2021:

[[email protected] elasticsearch]# curl -X PUT "es-node-01:9200/_snapshot/my_backup?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "fs",
  "settings": {
    "location": "/var/lib/elasticsearch_backup/feb2021"
  }
}
'
{
  "acknowledged" : true
}

Perform the first snapshot into this diretory

[[email protected] elasticsearch]# curl -X PUT "es-node-01:9200/_snapshot/my_backup/snapshot_1?wait_for_completion=true&pretty"

{
  "snapshot" : {
    "snapshot" : "snapshot_1",
    "uuid" : "Y0kOqvFmT2a5zaIv7gj5pw",
    "version_id" : 6020299,
    "version" : "6.2.2",
    "indices" : [
      "test_data"
    ],
    "include_global_state" : true,
    "state" : "SUCCESS",
    "start_time" : "2021-02-21T15:56:47.203Z",
    "start_time_in_millis" : 1613923007203,
    "end_time" : "2021-02-21T15:56:47.644Z",
    "end_time_in_millis" : 1613923007644,
    "duration_in_millis" : 441,
    "failures" : [ ],
    "shards" : {
      "total" : 2,
      "failed" : 0,
      "successful" : 2
    }
  }
}

Check that there is a reasonable amount of data in /var/lib/elasticsearch_backups:

[[email protected] elasticsearch]# find /var/lib/elasticsearch_backup/feb2021/ | wc -l ; du -hs /var/lib/elasticsearch_backup

Restore

In the event of a disaster during the upgrade, for example /var/lib/elasticsearch is destroyed, this can be moved to one side and /var/lib/elasticsearch recreated. Remember to reset the my_backup location:

[[email protected] elasticsearch]# curl -X PUT "es-node-01:9200/_snapshot/my_backup?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "fs",
  "settings": {
    "location": "/var/lib/elasticsearch_backup/feb2021"
  }
}
'
{
  "acknowledged" : true
}
[[email protected] elasticsearch]# curl -X POST "es-node-01:9200/_snapshot/my_backup/snapshot_1/_restore?pretty"

More information

Elasticsearch documentation