ElasticSearch Index Lifecycle Management – Diskover Data

ElasticSearch indices can accumulate overtime and there is an upper limit to how many shards can be associated to a node. Because of this, it is best practice to setup Index Lifecycle Management (ILM) policies to remove unneeded indices.

The Maximum number of shards per node is 1000

Contents of this page:

How to configure Index Lifecycle Management Policies in Kibana UI
How to configure Index Lifecycle Management Policies via CLI
Having issues creating policies

How to know you should set up lifecycle policies:

If your tasks are failing to run within Diskover and after inspecting the logs, you’re seeing the following error, you will need to follow these steps to create lifecycle policies:

Elasticsearch error creating index RequestError(400, 'validation_exception', 'Validation Failed: 1: this action would add [1] shards, but this cluster currently has [1000]/[1000] maximum normal shards open;') (Exit code: 1)

Temporary workaround without creating lifecycle policies:

This should only be used to clear out a few indices and allow users to execute tasks within Diskover again.

For an immediate fix, you can manually remove indices in Diskover to clear up space. Navigate to the Indices page by clicking the Gear icon in the top right corner of Diskover. From here, select any indices that are OK to remove, and click Delete:

How to configure Index Lifecycle Management Policies in Kibana:

Prerequisites:

Kibana must be installed and running. This is typically installed on the Diskover Web host. You can check if it is online by running sudo systemctl status kibana.
The Kibana UI must be accessible - http://<KIBANA-IP>:5601, or if using https, https://<YOUR DNS>:5601

Creating and configuring the policy:

In Kibana navigate to the Elasticsearch Stack Management page:

First we will create an Index Management Template. This is how we will associate our desired indices to our new policy, which will remove old indices. Navigate to Index Management and then Index Templates:

In Index Management → Index Templates click Create Template:

Creating the Index Template:
1. This template we’ll create will associate with all indices that start with our desired index pattern. In the example below, we’re setting up a policy to get rid of any S3 indices with the prefix of: diskover-s3*
  1. Name: A unique name for this template. Elastic prevents you from having a name with spaces or capital letters.
  2. Index patterns: The index pattern for this template to apply to. Example: diskover-s3*
  3. Data stream: Disable this.
  4. The remaining items are optional and can be left blank, click Create Template
2. Leave steps 2-6 default
3. Click Review Template. After reviewing the summary click Create template.
4. Below is a completed view of step 1 for creating the template:
After Creating the Template we need to navigate back to Elasticsearch Stack Management → Index Lifecycle Policies → Create policy

Creating the policy:
1. Policy name: The name of the policy
2. Under Hot phase expand the Advanced settings as we’re going to disable the following:
  1. Use Recommended Defaults
  2. Set Index Priority
  3. Enable Rollover
3. Click the trash can icon next to Keep data in this phase forever. This will create the Delete phase just below.
4. Under Delete phase, you’ll get the chance to set when indices get moved into this page. This can be set in days, hours, or minutes.
5. Click Save policy when ready.
In Elasticsearch Stack Management → Index Lifecycle Policies you’ll find the new policy you’ve created.
The policy we’ve just created only applies to new indices. It will not affect any historical indices that have already been created in Diskover. To have this policy manage older indices, we need to add the policy to existing indices.
1. Under the “hamburger” icon in the top left, look under Management for Dev Tools. This will allow us to run API calls against the cluster.
2. Get rid of any of the examples that may be here and replace it with the following curl:
```
PUT /diskover-INDEXNAME*/_settings
{
  "index.lifecycle.name": "INDEX-LIFECYCLE-POLICY-NAME"
}
```
  1. Note: INDEXNAME should be replaced with the actual name of the indices you’d like to target.
3. Once you’re ready to run the curl, hit the “play” button in this box. You should see following for a successful API call:
```
{
  "acknowledged": true
}
```

How to configure Index Lifecycle Management Policies via CLI

Create the ILM Policy:

curl -XPUT "http://localhost:9200/_ilm/policy/my-policy" -H 'Content-Type: application/json' -d'
  {
    "policy": {
      "phases": {
        "hot": {
          "actions": {}
        },
        "delete": {
          "min_age": "90d",
          "actions": {
            "delete": {}
          }
        }
      }
    }
  }'

Create the Index Template:

curl -XPUT "http://localhost:9200/_index_template/my-template" -H 'Content-Type: application/json' -d'
  {
    "index_patterns": ["my-index-*"],
    "template": {
      "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 1,
        "index.lifecycle.name": "my-policy"
      },
      "mappings": {
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "message": {
            "type": "text"
          }
        }
      }
    },
    "priority": 500
  }'

Verify the Policy: curl -XGET "http://localhost:9200/_ilm/policy/my-policy"
Verify the Index Template: curl -XGET "http://localhost:9200/_index_template/my-template"
The policy we’ve just created only applies to new indices. It will not affect any historical indices that have already been created in Diskover. To have this policy manage older indices, we need to add the policy to existing indices.
1. Connect to your ES server
2. Run the following curl to see all indices that match your index pattern from your index template: curl <http://<ES> HOST IP>:9200/<Index Pattern>/_ilm/explain?pretty
3. Now add the policy:
```
HTTP:
curl -XPUT http://<ES HOST IP>:9200/diskover-s3*/_settings -H "Content-Type: application/json" -d '{
"index.lifecycle.name": "<POLICY NAME>"
}'

HTTPS:
curl -u elastic:<YOUR ELASTIC USER PASSWORD> -XPUT https://<ES HOST IP>:9200/diskover-<INDEX PATTERN>/_settings --cacert /etc/elasticsearch/certs/http_ca.crt -H "Content-Type: application/json" -d '{
"index.lifecycle.name": "<POLICY NAME>"
}'
```

Check Policy Status on an Index: curl -XGET "http://localhost:9200/my-index/_ilm/explain".

Having issues?

If you’re having issues creating these policies, or have questions, please reach out to the Diskover Support team via a Zendesk ticket.