Skip to content


Splunk replacement – ElasticSearch with Kibana and Beats on RedHat 7

ElasticSearch/Kibana Server Setup where both services run on the same physical host and “elasticsearch” is not clustered.

ElasticSearch repo:

[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Kibana repo:

[kibana-5.x]
name=Kibana repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Install packages

# yum -y install elasticsearch kibana

Modify ElasticSearch – /etc/eleasticsearch/elasticsearch.yml

cluster.name: elasticsearch
network.host: 0.0.0.0
http.port: 9200

Modify Kibana – /etc/kibana/kibana.yml

server.port: 5601
server.host: "YourElasticSearchServerIpAddress"
server.maxPayloadBytes: 1048576
server.name: "YourElasticSearchHostName"
elasticsearch.url: "http://localhost:9200"
kibana.index: ".kibana"
elasticsearch.username: "kibana"
elasticsearch.password: "changeme"

Enable and Start

# systemctl enable elasticsearch
# systemctl restart elasticsearch.service
# systemctl enable kibana
# systemctl restart kibana.service

ElasticSearch indices maintenance – download package

# yum -y install elasticsearch-curator

Configure it

# cat /etc/cron.daily/del_indices.yml
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True. If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
1:
action: delete_indices
description: >-
Delete indices older than 30 days
options:
ignore_empty_list: True
continue_if_exception: False
disable_action: False
filters:
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 30 

Remove (daily) indices older than 30 days daily

# cat /etc/cron.daily/elasticIndexPrune
#!/bin/sh
#remove older indices from ElastiSearch
curator /etc/cron.daily/del_indices.yml

Install sample dashboards and searches (elasticsearch + kibana on the same host)

# cd /usr/share/metricbeat/scripts
# ./import_dashboards


  

Client Setup
Install ElasticSearch repo

[elasticsearch]
baseurl = https://artifacts.elastic.co/packages/5.x/yum
gpgcheck = 1
gpgkey = https://packages.elastic.co/GPG-KEY-elasticsearch
name = Elasticsearch repository for 5.x packages

Install MetricBeat (performance stats)

# yum -y install metricbeat

Configure MetricBeat

# cat metricbeat.yml
metricbeat.modules:
- module: system
metricsets:
- cpu
- load
- core
- diskio
- filesystem
- fsstat
- memory
- network
- process

enabled: true
period: 30s

processes: ['.*']
cpu_ticks: true
cgroups: true

filters:
- drop_event.when.regexp.mount_point: '^/(sys|run|proc|dev)|(rpc_pipefs|gvfs|pts)$'

output.elasticsearch:
hosts: ["YourElasticSearchServerIpAddress:9200"]
template.enabled: true
template.path: "/etc/metricbeat/metricbeat.template.json"
template.name: "metricbeat"
template.overwrite: false
index: "metricbeat-%{+yyyy.MM.dd}"
logging.level: debug

Install FileBeat (logs redirect)

# yum -y install filebeat

Configure FileBeat

# # cat filebeat.yml
filebeat:
prospectors:
-
paths:
- /var/log/messages
- /var/log/yum.log
- /var/log/dmesg
- /var/log/secure
exclude_files: ['\.gz$', '\*-20[0-9]\*$']
exclude_lines: ['does not contain uidNumber', 'DHCPREQUEST', 'slice', 'svcUNXAgent']
ignore_older: 2h
input_type: log
document_type: syslog
output:
elasticsearch:
hosts: ["YourElasticSearchServerIpAddress:9200"]
template:
path: "filebeat.template.json"

Enable and Start

# systemctl enable metricbeat
# systemctl restart metricbeat.service
# systemctl enable filebeat
# systemctl restart filebeat.service

Posted in LINUX.

Tagged with , , , , , , , .


0 Responses

Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.



Some HTML is OK

or, reply to this post via trackback.

WordPress Anti Spam by WP-SpamShield



Copyright © 2016 - 2017 Waldemar Mark Duszyk. All Rights Reserved. Created by Blog Copyright.