Curator Elasticsearch indices 管理工具

Curator Elasticsearch indices 管理工具

一月 11, 2019

ES索引数据太大，kibana查询数据总是内存爆满挂掉，为了方便管理索引文件，使用curator工具

curator下载地址：
https://github.com/elastic/curator

ES-curator/curator-config/curator配置文件编写

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
##curator.yml
---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - 'es-curator-ip'
    - 'es-curator-ip'
    - 'es-curator-ip'
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  aws_key:
  aws_secret_key:
  aws_region:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile: /data/ES-curator/curator-config/log/curator.log
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
##close.yml 按时间匹配，关闭15天以前的index
actions:
  1:
    action: close
    description: >-
      Close indices older than 15 days (based on index name), for logstash-
      prefixed indices.
    options:
      delete_aliases: False
      timeout_override: 180
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: kbilogs-logics
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 15
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
##delete.yml 按时间匹配，删除365天之前的index
actions:
  1:
    action: delete_indices
    description: "delete pay index"
    options:
      ignore_empty_list: True
      timeout_override: 500
      disable_action: False
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(kbilogs-logics-pay-200|kbilogs-logics-pay-201|kbilogs-logics-pay-202|kbilogs-logics-pay-203|kbilogs-logics-pay-204).*$'
    - filtertype: age
      source: creation_date
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 365
  2:
    action: delete_indices
    description: "delete game index"
    options:
      ignore_empty_list: True
      timeout_override: 500
      disable_action: False
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(kbilogs-logics-gamex-200|kbilogs-logics-gamex-201|kbilogs-logics-gamex-202|kbilogs-logics-gamex-203|kbilogs-logics-gamex-204).*$'
    - filtertype: age
      source: creation_date
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 60
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
##merge_index_rule.yml  合并一天前的index
actions:
  1:
    action: forcemerge
    description: "forcemerge pay index"
    options:
      ignore_empty_list: True
      timeout_override: 50000
      disable_action: False
      max_num_segments: 1
      delay: 120
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(kbilogs-logics-pay-200|kbilogs-logics-pay-201|kbilogs-logics-pay-202|kbilogs-logics-pay-203|kbilogs-logics-pay-204).*$'
    - filtertype: age
      source: creation_date
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
## /etc/init.d/curator-manage-index     启动脚本

#!/bin/bash

DOCKER_IMG='curator:v5.5.4'
DOCKER_MOUNT_DIR='/data/ES-curator/curator-config/:/data/ES-curator/curator-config'
CURATOR_MAIN_CONFIG='/data/ES-curator/curator-config/curator.yml'
CURATOR_ACTION_CLOSE_CONFIG='/data/ES-curator/curator-config/close.yml'
CURATOR_ACTION_OPEN_CONFIG='/data/ES-curator/curator-config/open_index_rule.yml'
CURATOR_ACTION_MERGE_CONFIG='/data/ES-curator/curator-config/merge_index_rule.yml'
CURATOR_ACTION_DELETE_CONFIG='/data/ES-curator/curator-config/delete.yml'

function close(){
docker run -d --rm -v ${DOCKER_MOUNT_DIR} ${DOCKER_IMG} --config ${CURATOR_MAIN_CONFIG} ${CURATOR_ACTION_CLOSE_CONFIG}
}

function open(){
docker run -d --rm -v ${DOCKER_MOUNT_DIR} ${DOCKER_IMG} --config ${CURATOR_MAIN_CONFIG} ${CURATOR_ACTION_OPEN_CONFIG}
}

function merge(){
docker run -d --rm -v ${DOCKER_MOUNT_DIR} ${DOCKER_IMG} --config ${CURATOR_MAIN_CONFIG} ${CURATOR_ACTION_MERGE_CONFIG}
}

function delete(){
docker run -d --rm -v ${DOCKER_MOUNT_DIR} ${DOCKER_IMG} --config ${CURATOR_MAIN_CONFIG} ${CURATOR_ACTION_DELETE_CONFIG}
}

case "$1" in
    close)
    close
    ;;
    open)
    echo "Edit ${CURATOR_ACTION_OPEN_CONFIG} then run this comand again"
    open
    ;;
    merge)
    merge
    ;;
    delete)
    delete
    ;;
    cron)
    delete && merge && close
    ;;
    *)
    echo "Usage: $0 close|open|merge|delete|cron"
esac

配置文件和启动脚本编写好后，将启动脚本挂到定时任务里，每天定时执行即可