ES-Curator

Objective

Installing Curator

Simple use pip install elasticsearch-curator to install Curator on your machine. I prefer to install it on the Elasticsearch machine itself.

Configuring Curator

Create a file curator.yml with following contents.

---
client:
  hosts:
    - 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

Now, we need to define an action. i.e. What will curator do. There are many actions to choose from. Check the documentation for more information

• Alias
• Allocation
• Close
• Cluster Routing
• Create Index
• Delete Indices
• Delete Snapshots
• Open
• forceMerge
• Replicas
• Restore
• Snapshot

For this dicussion, we will use Delete Indices as the action, since this is what we want to do.

Below is the sample action file delete_indices.yml , which will delete the logstash indices which are older than 10 days.

---
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 45 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 10
      exclude:

To run this action, simple use the command

curator ./delete_index.yml --config ./curator.yml --dry-run
2017-04-09 17:27:46,075 INFO      Preparing Action ID: 1, "delete_indices"
2017-04-09 17:27:46,080 INFO      Trying Action ID: 1, "delete_indices": Delete indices older than 45 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2017-04-09 17:27:46,538 INFO      DRY-RUN MODE.  No changes will be made.
2017-04-09 17:27:46,538 INFO      (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2017-04-09 17:27:46,538 INFO      Action ID: 1, "delete_indices" completed.
2017-04-09 17:27:46,538 INFO      Job completed.

The --dry-run mode will not actually delete the index. It can be used to test the output of the action.

If you want to schedule it in a cron, you can do so using crontab -e

00 8 * * * root curator /path/delete_index.yml --config /path/curator.yml 

The above configuration will cleanup the indices older than 10 days everyday at 8 AM.

Success metrics

Requirements

User interaction and design

Open Questions

Out of Scope