/
ES-Curator

ES-Curator

Target release2019-M1
Epic EDOP-38 - Getting issue details... STATUS
Document statusDRAFT
Document owner

Gajendran C (Unlicensed)

Objective

Installing Curator

Simple use pip install elasticsearch-curator to install Curator on your machine. I prefer to install it on the Elasticsearch machine itself.

Configuring Curator

Create a file curator.yml with following contents.

---
client:
  hosts:
    - 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

Now, we need to define an action. i.e. What will curator do. There are many actions to choose from. Check the documentation for more information

  • Alias
  • Allocation
  • Close
  • Cluster Routing
  • Create Index
  • Delete Indices
  • Delete Snapshots
  • Open
  • forceMerge
  • Replicas
  • Restore
  • Snapshot

For this dicussion, we will use Delete Indices as the action, since this is what we want to do.

Below is the sample action file delete_indices.yml , which will delete the logstash indices which are older than 10 days.

---
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 45 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 10
      exclude:

To run this action, simple use the command

curator ./delete_index.yml --config ./curator.yml --dry-run
2017-04-09 17:27:46,075 INFO      Preparing Action ID: 1, "delete_indices"
2017-04-09 17:27:46,080 INFO      Trying Action ID: 1, "delete_indices": Delete indices older than 45 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2017-04-09 17:27:46,538 INFO      DRY-RUN MODE.  No changes will be made.
2017-04-09 17:27:46,538 INFO      (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2017-04-09 17:27:46,538 INFO      Action ID: 1, "delete_indices" completed.
2017-04-09 17:27:46,538 INFO      Job completed.

The --dry-run mode will not actually delete the index. It can be used to test the output of the action.

If you want to schedule it in a cron, you can do so using crontab -e

00 8 * * * root curator /path/delete_index.yml --config /path/curator.yml 

The above configuration will cleanup the indices older than 10 days everyday at 8 AM.

Success metrics

GoalMetric


Requirements

#RequirementUser StoryImportanceJira IssueNotes
1HIGH

2




User interaction and design

Open Questions

QuestionAnswerDate Answered

Out of Scope

Related content

Documents
Documents
Read with this
es-curator to clear old logs/indices using Helm Chart
es-curator to clear old logs/indices using Helm Chart
More like this
Using kafka-connect in egov-indexer reindexing jobs to push records to elastic search
Using kafka-connect in egov-indexer reindexing jobs to push records to elastic search
More like this
Indexing Resources
Indexing Resources
More like this
Steps for Collection live indexing
Steps for Collection live indexing
More like this
How to create snapshot and restore elasticsearch cluster data
How to create snapshot and restore elasticsearch cluster data
More like this