Using Monit to monitor Elasticsearch and Content Gateway

Owned by Tony Lokko (Unlicensed)
Last updated: May 25, 2018 by Aaron Enfield

When installing an Elasticsearch or Content Gateway server on a RedHat or Centos 7 operating system you may want to automatically restart the service if it stops and be able to check the service from a dashboard.

While restarting the service can easily be done via systemd monit provides a more granular method with a lightweight http dashboard to do the same.

Monit is available from the epel repository on standard Redhat and Centos installs. The current version at the time of writing this article is 5.25

Step-by-step guide

First install the epel repository if you don't have it already.

For Centos installs its as simple as 

yum install epel-release

For Red hat installs

yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-$(rpm -E '%{rhel}').noarch.rpm


Then install monit

yum install monit


The monit installation has one main configuration file called monitrc which is located in /etc/ and then various service checks above the base server checks are placed in monit.d/

The basic configuration of monit has a few things uncommented by default.


set daemon 30

# check services at 30 seconds intervals

set log syslog


set httpd port 2812 and

use address localhost # only accept connection from localhost

# allow localhost to connect to the server and

allow admin:monit # require user 'admin' with password 'monit'

#with ssl { # enable SSL/TLS and set path to server certificate

# pemfile: /etc/ssl/certs/monit.pem

#}

This last section willi enable the http interface for monit. Setting the address to an outward facing one on your network will allow you to reach monit on that port.

For example

use address 192.168.1.10

would listen for http requests on that interface only.


The allow section is only for use with authentication.

The user admin is the default.

Below these sections there are various examples for monit which show you how it can be configured to check services.

Lastly there is an include directive.


include /etc/monit.d/*

Checking gateway

create a new file in monit.d/ called something that reflects the service being checked.

In my case i picked cloudgateway as the filename.


Open it in your text editor of choice and add the following **note these instructions cover Centos/Rhel 7

check process cloudgateway with pidfile /var/run/cloudgateway.pid

start program = "/bin/systemctl start cloudgateway"

stop program = "/bin/systemctl stop cloudgateway"

Thats it.
Save the file and then issue

systemctl reload monit
systemctl start monit

and for the paranoid-
systemctl status monit

Once you've done that you can either query the http interface on port 2812 or issue a status command via cli.


monit status cloudgateway
Monit 5.25.1 uptime: 23h 38m

Process 'cloudgateway'
status OK
monitoring status Monitored
monitoring mode active
on reboot start
pid 125456
parent pid 125405
uid 0
effective uid 0
gid 0
uptime 8d 21h 3m
threads 320
children 0
cpu 0.0%
cpu total 0.0%
memory 0.9% [811.0 MB]
memory total 0.9% [811.0 MB]
security attribute (null)
disk write 0 B/s [1.0 GB total]
data collected Fri, 25 May 2018 12:49:30


For elasticsearch the monitoring file will be very similar.

check process elasticsearch with pidfile /var/run/elasticsearch/elasticsearch.pid

    start program = "/bin/systemctl start elasticsearch"

    stop program = "/bin/systemctl stop elasticsearch"


monit status
Monit 5.25.1 uptime: 0m

Process 'elasticsearch'
status OK
monitoring status Monitored
monitoring mode active
on reboot start
pid 3468
parent pid 1
uid 996
effective uid 996
gid 994
uptime 10d 0h 22m
threads 268
children 0
cpu -
cpu total -
memory 36.1% [33.5 GB]
memory total 36.1% [33.5 GB]
security attribute (null)
disk read 0 B/s [93.0 MB total]
disk write 0 B/s [95.5 GB total]
data collected Fri, 25 May 2018 13:01:25


To find the pid information you can use ps ax

ps ax | grep elasticsearch


ps ax | grep elasticsearch
3468 ? SLsl 435:04 /bin/java -Xms32g -Xmx32g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.conf=/etc/elasticsearch
284724 pts/0 S+ 0:00 grep --color=auto elasticsearch

Summing up

Monit can be configured for email alerting and each service check can be extended to run custom scripts so this is very much the basic configuration for it.

More information on monit can be found in their useful and readable documentation here.


https://mmonit.com/monit/documentation/monit.html





© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.