Curator Elasticsearch indices 管理工具

Curator Elasticsearch indices 管理工具

一月 11, 2019

ES索引数据太大,kibana查询数据总是内存爆满挂掉,为了方便管理索引文件,使用curator工具

curator下载地址:
https://github.com/elastic/curator

ES-curator/curator-config/curator配置文件编写

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
##curator.yml
---
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
client:
hosts:
- 'es-curator-ip'
- 'es-curator-ip'
- 'es-curator-ip'
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
aws_key:
aws_secret_key:
aws_region:
ssl_no_validate: False
http_auth:
timeout: 30
master_only: False

logging:
loglevel: INFO
logfile: /data/ES-curator/curator-config/log/curator.log
logformat: default
blacklist: ['elasticsearch', 'urllib3']
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
##close.yml 按时间匹配,关闭15天以前的index
actions:
1:
action: close
description: >-
Close indices older than 15 days (based on index name), for logstash-
prefixed indices.
options:
delete_aliases: False
timeout_override: 180
continue_if_exception: False
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: kbilogs-logics
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 15
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
##delete.yml 按时间匹配,删除365天之前的index
actions:
1:
action: delete_indices
description: "delete pay index"
options:
ignore_empty_list: True
timeout_override: 500
disable_action: False
filters:
- filtertype: pattern
kind: regex
value: '^(kbilogs-logics-pay-200|kbilogs-logics-pay-201|kbilogs-logics-pay-202|kbilogs-logics-pay-203|kbilogs-logics-pay-204).*$'
- filtertype: age
source: creation_date
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 365
2:
action: delete_indices
description: "delete game index"
options:
ignore_empty_list: True
timeout_override: 500
disable_action: False
filters:
- filtertype: pattern
kind: regex
value: '^(kbilogs-logics-gamex-200|kbilogs-logics-gamex-201|kbilogs-logics-gamex-202|kbilogs-logics-gamex-203|kbilogs-logics-gamex-204).*$'
- filtertype: age
source: creation_date
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 60
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
##merge_index_rule.yml  合并一天前的index
actions:
1:
action: forcemerge
description: "forcemerge pay index"
options:
ignore_empty_list: True
timeout_override: 50000
disable_action: False
max_num_segments: 1
delay: 120
filters:
- filtertype: pattern
kind: regex
value: '^(kbilogs-logics-pay-200|kbilogs-logics-pay-201|kbilogs-logics-pay-202|kbilogs-logics-pay-203|kbilogs-logics-pay-204).*$'
- filtertype: age
source: creation_date
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
## /etc/init.d/curator-manage-index     启动脚本

#!/bin/bash

DOCKER_IMG='curator:v5.5.4'
DOCKER_MOUNT_DIR='/data/ES-curator/curator-config/:/data/ES-curator/curator-config'
CURATOR_MAIN_CONFIG='/data/ES-curator/curator-config/curator.yml'
CURATOR_ACTION_CLOSE_CONFIG='/data/ES-curator/curator-config/close.yml'
CURATOR_ACTION_OPEN_CONFIG='/data/ES-curator/curator-config/open_index_rule.yml'
CURATOR_ACTION_MERGE_CONFIG='/data/ES-curator/curator-config/merge_index_rule.yml'
CURATOR_ACTION_DELETE_CONFIG='/data/ES-curator/curator-config/delete.yml'

function close(){
docker run -d --rm -v ${DOCKER_MOUNT_DIR} ${DOCKER_IMG} --config ${CURATOR_MAIN_CONFIG} ${CURATOR_ACTION_CLOSE_CONFIG}
}

function open(){
docker run -d --rm -v ${DOCKER_MOUNT_DIR} ${DOCKER_IMG} --config ${CURATOR_MAIN_CONFIG} ${CURATOR_ACTION_OPEN_CONFIG}
}

function merge(){
docker run -d --rm -v ${DOCKER_MOUNT_DIR} ${DOCKER_IMG} --config ${CURATOR_MAIN_CONFIG} ${CURATOR_ACTION_MERGE_CONFIG}
}

function delete(){
docker run -d --rm -v ${DOCKER_MOUNT_DIR} ${DOCKER_IMG} --config ${CURATOR_MAIN_CONFIG} ${CURATOR_ACTION_DELETE_CONFIG}
}

case "$1" in
close)
close
;;
open)
echo "Edit ${CURATOR_ACTION_OPEN_CONFIG} then run this comand again"
open
;;
merge)
merge
;;
delete)
delete
;;
cron)
delete && merge && close
;;
*)
echo "Usage: $0 close|open|merge|delete|cron"
esac

配置文件和启动脚本编写好后,将启动脚本挂到定时任务里,每天定时执行即可