ES索引数据太大,kibana查询数据总是内存爆满挂掉,为了方便管理索引文件,使用curator工具
curator下载地址:https://github.com/elastic/curator
ES-curator/curator-config/curator配置文件编写
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 --- client: hosts: - 'es-curator-ip' - 'es-curator-ip' - 'es-curator-ip' port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: aws_key: aws_secret_key: aws_region: ssl_no_validate: False http_auth: timeout: 30 master_only: False logging: loglevel: INFO logfile: /data/ES-curator/curator-config/log/curator.log logformat: default blacklist: ['elasticsearch', 'urllib3' ]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 actions: 1: action: close description: >- Close indices older than 15 days (based on index name), for logstash- prefixed indices. options: delete_aliases: False timeout_override: 180 continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: prefix value: kbilogs-logics - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 actions: 1: action: delete_indices description: "delete pay index" options: ignore_empty_list: True timeout_override: 500 disable_action: False filters: - filtertype: pattern kind: regex value: '^(kbilogs-logics-pay-200|kbilogs-logics-pay-201|kbilogs-logics-pay-202|kbilogs-logics-pay-203|kbilogs-logics-pay-204).*$' - filtertype: age source: creation_date direction: older timestring: '%Y.%m.%d' unit: days unit_count: 365 2: action: delete_indices description: "delete game index" options: ignore_empty_list: True timeout_override: 500 disable_action: False filters: - filtertype: pattern kind: regex value: '^(kbilogs-logics-gamex-200|kbilogs-logics-gamex-201|kbilogs-logics-gamex-202|kbilogs-logics-gamex-203|kbilogs-logics-gamex-204).*$' - filtertype: age source: creation_date direction: older timestring: '%Y.%m.%d' unit: days unit_count: 60
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 actions: 1: action: forcemerge description: "forcemerge pay index" options: ignore_empty_list: True timeout_override: 50000 disable_action: False max_num_segments: 1 delay: 120 filters: - filtertype: pattern kind: regex value: '^(kbilogs-logics-pay-200|kbilogs-logics-pay-201|kbilogs-logics-pay-202|kbilogs-logics-pay-203|kbilogs-logics-pay-204).*$' - filtertype: age source: creation_date direction: older timestring: '%Y.%m.%d' unit: days unit_count: 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 #!/bin/bash DOCKER_IMG='curator:v5.5.4' DOCKER_MOUNT_DIR='/data/ES-curator/curator-config/:/data/ES-curator/curator-config' CURATOR_MAIN_CONFIG='/data/ES-curator/curator-config/curator.yml' CURATOR_ACTION_CLOSE_CONFIG='/data/ES-curator/curator-config/close.yml' CURATOR_ACTION_OPEN_CONFIG='/data/ES-curator/curator-config/open_index_rule.yml' CURATOR_ACTION_MERGE_CONFIG='/data/ES-curator/curator-config/merge_index_rule.yml' CURATOR_ACTION_DELETE_CONFIG='/data/ES-curator/curator-config/delete.yml' function close (){ docker run -d --rm -v ${DOCKER_MOUNT_DIR} ${DOCKER_IMG} --config ${CURATOR_MAIN_CONFIG} ${CURATOR_ACTION_CLOSE_CONFIG} }function open (){ docker run -d --rm -v ${DOCKER_MOUNT_DIR} ${DOCKER_IMG} --config ${CURATOR_MAIN_CONFIG} ${CURATOR_ACTION_OPEN_CONFIG} }function merge (){ docker run -d --rm -v ${DOCKER_MOUNT_DIR} ${DOCKER_IMG} --config ${CURATOR_MAIN_CONFIG} ${CURATOR_ACTION_MERGE_CONFIG} }function delete (){ docker run -d --rm -v ${DOCKER_MOUNT_DIR} ${DOCKER_IMG} --config ${CURATOR_MAIN_CONFIG} ${CURATOR_ACTION_DELETE_CONFIG} }case "$1 " in close) close ;; open) echo "Edit ${CURATOR_ACTION_OPEN_CONFIG} then run this comand again" open ;; merge) merge ;; delete) delete ;; cron) delete && merge && close ;; *) echo "Usage: $0 close|open|merge|delete|cron" esac
配置文件和启动脚本编写好后,将启动脚本挂到定时任务里,每天定时执行即可