Linux HOWTO, Missing Linux Packages

NRDP for Nagios Core on CentOS 7.x

Leave a comment

Introduction

In continuation to part 1 of this blog series… NRDP (Nagios Remote Data Processor) is a Client/Server plugin for Nagios. With the NRDP model, it is the Application (or the server hosting it) that is responsible for reporting in it’s status. The beauty of this is you can have applications installed all throughout your business, your home, across data centers, overseas, etc all reporting to 1 single Nagios instance. You can monitor your entire infrastructure with this tool.

NRDP Overview
NRDP Overview

The diagram above illustrates how the paradigm works.

  • The A represents the Application Server. There is no limit to the number of these guys.
  • The N represents the Nagios Server. You’ll only ever need one Nagios server.
  • The Application (and/or the server hosting it) will make periodic status reports to the Nagios server. It should report that everything is okay, or report that it isn’t. Out of the box (using my rpms) Nagios will wait for up to X minutes (configurable) for a message to be received before reporting the service/host as being offline. Ideally you should send at least 2 or 3 notifications within this grace period. Currently X is set to 5 minutes with the templates in the RPMs I provide. But you can change this.
  • If Nagios doesn’t hear from the Application for an extended period of time, then it assumes the worst and reports to the user that there is a problem.

NRDP is a newer chapter in the Nagios world which allows you to monitor more applications you wouldn’t have otherwise been able to do. NRDP replaces it’s predecessor NSCA which performs the exact same task/function. One of the key differences is that NRDP is much easier to set up and use than NSCA is. NRDP is also much less lines of code too! The RPMs I provide make the setup even easier. That said, if you’re interested in using NSCA instead, I still package it here (and client here). But lets get back to talking about NRDP….

NRDP Clients pushing to Central NRDP Server
NRDP Clients pushing to Central NRDP Server

Here is what you’re getting with this blog and the packaged rpms that go with it:

  • Nagios Core v4.x: updated RPM packaging which I continued to maintain and carry forward from my previous blogs.
    Note: You will need this installed in order to use NRDP. See part 1 of this blog series for more information if you don’t already have it set up.
  • NRDP (Nagios Remote Data Processor) v1.x: My custom RPM packaging to just work for everyone right out of the box. Here are some modifications I made to the package to make our lives easier:
    • The default insecure port is configured to use 5668.
    • The default secure port is configured to use 5669.
    • I included my own Apache configuration so that the service would run right out of the box.
    • There are a several patches I had to make in order for the NRDP package to even work in our CentOS environment.
    • There is firewall configuration all ready to use with FirewallD.

Best of all, with my RPMs, you can run SELinux in full Enforcing mode for that extra piece of mind from a security standpoint!

The Goods

For those who really don’t care and want to just jump right in with the product. Here you go!

You can download the packages manually if you choose, or reference them using my repository:

Package Download Description
nrdp el7.rpm The NRDP Server: This must be installed on the same server running Nagios. It is in charge of listening to requests sent by nrdp-clients and reporting their status directly to Nagios. Don’t forget to change your tokens which we’ll discuss further down if you don’t know what I’m talking about!
nrdp-selinux el7.rpm An SELinux add-on package that allows the NRDP Server to operate under Enforcing Mode.

Note: This RPM is not required by the NRDP server to run correctly.
nrdp-client el7.rpm The NRDP Client: You’ll only need to optionally install this on to the remote servers that will be sending their status along to your NRDP Server. Since the protocol is so simple to adapt, you can also just choose to build the client aspect into the program you want to monitor with. If you do choose to install this RPM, you get access to a small program called send_nrdp.php which allows you to post status messages to the Nagios Server.
nrdp-doc el7.rpm Just some documentation that is already publicly available on NRDP’s website.
Note: This RPM is not required by Nagios to run correctly.

Note: The source rpm can be obtained here which builds everything you see in the table above. It’s not required for the application to run, but might be useful for developers or those who want to inspect how I put the package together.

NRDP Server Side Configuration

The NRDP Server is only going to work if it’s installed on the same server as Nagios.

Assuming you hooked up to my repository here, the NRDP server can can be easily installed with the following command:

# Install NRDP (Server) on the same server as the one running Nagios
yum install nrdp nrdp-selinux

NRDP was written in PHP and therefore depends on a few small tweaks on your part. For one, you’ll want to make sure you have a timezone configured too to silence any potential errors and keep the reporting consistent. You’ll want to do this from your /etc/php.ini file; look for a directive called date.timezone. Here is a list of supported timezones by PHP. But you can also just do this which works too:

# This clever trick (a 1 liner) lets the timezone in the /etc/php.ini
# to that of whatever your current system is set to!
sed -i -e "s/^[; \t]\?\(date\.timezone\)[ \t]*=.*/\1 = \"$(date +%Z)\"/g" \
   /etc/php.ini

If Apache isn’t running; now’s the time to start it up.

# This simple command should be ran as root and will
# start Apache if it isn't running already, otherwise it
# will send a reload signal if it is.  Regardless,
# after you finish this command, the new NRDP Configuration
# will be running and ready to go.
systemctl status httpd.service && \
   systemctl reload httpd.service || \
   systemctl start httpd.service

# Make sure we start up on reboots (if we haven't done so already)
systemctl enable httpd.service

NRDP/Nagios Configuration

You need to let Nagios know what services it should expect notifications from. The key thing that separates these entries from any other Nagios entries you specified is the ‘use‘ directive. For hosts reporting in, you’ll want to use the generic-nrdp-host directive. For services reporting in, you’ll want to use the generic-nrdp-service entry.

A Configuration file on your Nagios Server (the same server running your NRDP Server) might look like this:

cat << _EOF > /etc/nagios/conf.d/somehost.cfg
# We might have a test server defined like this
define host {
    # Name of host template to use (notice we're using the
    # 'generic-nrdp-host' template).  If you want to actively ping
    # this server and not await to hear from it, you can change this
    # to use the 'linux-server' template instead.
    use                         generic-nrdp-host
    # Hostname
    host_name                   somehost
    # Human Readable
    alias                       My Test Server
}

# Now we might identify a component to associate with our test server called <em>somehost</em>.
define service{
   use                 generic-nrdp-service
   service_description someservice
   host_name           somehost
}
_EOF

With the templates I provide, Nagios will wait for up to 5 minutes for a notification before assuming nothing is coming (and changing the state to CRITICAL). If this is too long (or too short) of a wait time for you, you can change it by modifying the NRDP templates (or creating your own). The provided NRDP templates can be found here: /etc/nagios/conf.d/nrdp.conf. You’ll want to focus on the entry entitled freshness_threshold which is currently set to 300 (300 seconds is equal to 5 minutes) and change it to your liking.

Alternatively, you can over-ride the template value by just adding the freshness_threshold to the service configuration you create like so:

cat << _EOF > /etc/nagios/conf.d/somehost.10min.service.cfg
# Over-ride our template timeout with a new one specified
# in our service declaration
define service{
   use                 generic-nrdp-service
   service_description someservice
   host_name           somehost
   # Over-ride the defaults in our template and only go stale
   # if 10 minutes elapses (600 seconds = 10 min)
   freshness_threshold 600
}
_EOF

If you’ve modified your configuration in anyway; be sure to reload Nagios so that it can take in our new configuration:

# Reload our service (if it's running)
systemctl reload nagios.service

# Be sure to start it if it isn't already:
systemctl start nagios.service

Testing your NRDP Server Setup

Once NRDP is up and running it will provide you a simple website you can use to interact with it. This interface allows you to manually send passive checks through to Nagios which is really useful for testing if everything is working okay!

Here is what the test page looks like:

NRDP Server Side
NRDP Server Side – use nuxref as a token to test with.

Note: The blog does not touch on the Submit Nagios Command portion of this page; but that’s another topic for another day. You want to focus on the Submit Check Data section.

To use NRDP you will be required to provide a token with every submission you ever make to it. This is just a security feature to prevent people from sending messages into your system who shouldn’t be. By default (if you’re using my rpms), the token is nuxref. You’ll notice in the above example, I’ve gone ahead and filed this field in to show you where it goes.

To change this token (which you really should do) just open up the php file /etc/nrdp/config.php. You can specify as many tokens as you like that you wish to accept. Here is what you’re looking for:

// ...
// look for the authorized_tokens directive
// and add your tokens here:
$cfg['authorized_tokens'] = array(
   "the-longer-and-more-encrypted-your-token-is-the-better",
   "specify-as-many-tokens-as.-you-want",
);

If you followed the blog so far and created the test examples I provided (above), then you can now submit a passive check using the default settings; don’t forget to provide your token!

If everything goes according to plan, you should be able to check out your message safely being acknowledged:

NRDP Test Submission Results (using defaults)
NRDP Test Submission Results (using defaults)

NRDP Server Security

Since you can access a very simple NRDP webpage on the same server hosting Nagios via http://localhost:5668/ (insecure) or https://localhost:5669/ (secure), you’ll want to protect it from those who shouldn’t be visiting it.

The below commands open up the protocol to everyone on your network. You should only perform this command if your Nagios Server will be running on a local private network:

# Enable insecure NRDP port (5668)
firewall-cmd --permanent --add-service=nrdp
firewall-cmd --add-service=nrdp

# Enable secure NRDP port (5669)
firewall-cmd --permanent --add-service=nrdps
firewall-cmd --add-service=nrdps

If you’re opening this up to the internet, then you might want to just open the (NRDP) port(s) exclusively used by the remote applications you intend to allow reporting from. It might be worth investing time into fail2ban as well as an additional precaution. The following presumes you know the IP of the application server and will open access to ‘JUST’ that system:

# Assuming 1.2.3.4 belongs to a system you trust who will be sending you
# Nagios status reports via NRDP:
firewall-cmd --permanent --zone=public \
   --add-rich-rule='\
       rule family="ipv4" \
       source address="1.2.3.4/32" \
       port protocol="tcp" \
       port="5668" \
       accept'

# Here is the Secure version of the same command
firewall-cmd --permanent --zone=public \
   --add-rich-rule='\
       rule family="ipv4" \
       source address="1.2.3.4/32" \
       port protocol="tcp" \
       port="5669" \
       accept'

Note: Again, I want to stress you should only open ports you intend to use and to the Application Servers who intend to use it. If you don’t intend on using the insecure port, then don’t even open it on your firewall!

Once you’re satisfied that everything is working. You might want to disable GET requests on your NRDP Server. This will prevent people from using the test website and only accept POST requests. To do this, just have a look in /etc/httpd/conf.d/nrdp.conf and uncomment (hence, remove the #) the line that reads:

RewriteCond %{REQUEST_METHOD} !^(POST) [NC]

You’ll need to reload your Apache server for the changes to take effect. If ever you want to test again, just comment the line back out again:

# This simple command should be ran as root and will
# start Apache if it isn't running already, otherwise it
# will send a reload signal if it is.  Regardless,
# after you finish this command, the new NRDP Configuration
# will be running and ready to go.
systemctl status httpd.service && \
   systemctl reload httpd.service || \
   systemctl start httpd.service

# Make sure we start up on reboots (if we haven't done so already)
systemctl enable httpd.service

NRDP Client Side Configuration

The protocol is so simple, that you can easily adapt this into programs you write or tool-sets. You can also just simple use the tool provided 

# Install Nagios Core using NuxRef
# See: https://nuxref.com/repo for more information
# Install NRDP Client (CLI) on any server you want to
# be able to report it's status to (onto Nagios)
yum install nrdp-client

Now… from our application server we would want to install our nrdp-client package. so that we can access our send_nrdp.php tool.

The tool is very simple to use; here is an example how you can send a simple status message to our Nagios server:

# The below sends a host acknowledgment to our NRDP server
# url:   web address to our Nagios server
# token: the security token (this is to prevent anyone from
#        sending status requests to your Nagios server.
#        The default is 'nuxref' until you change it (which you
#        really should consider doing)
# state: This will work the color coding of Nagios, you can
#        specify one of the following:
#           0 - OKAY (green)
#           1 - WARNING (yellow/orange)
#           2 - CRITICAL (red)
#           3 - UNKNOWN (blue)
# output: Attach a message with our state; this will be visible
#         as well from the Nagios display screen:

# Here is a NRDP notification:
send_nrdp.php --url=http://nagios.server.addr:5668 --token=nuxref \
   --host=somehost \
   --state=0 \
   --output="Test is is working great."

# This will send a warning status to our component we identified
send_nrdp.php --url=http://nagios.server.addr:5668 --token=nuxref \
   --host=somehost \
   --state=0 \
   --output="Test is is working great."

Note: It’s worth noting that there are no WARNING states for hosts. A zero (0) will report that it is online, and anything else will report it as CRITICAL.

NRDP Protocol

Are you a developer? If you are, you may want to avoid using the script and just talk directly to the NRDP Server from your application. It’s incredibly easy to do. If you’re not, don’t worry; the send_nrdp.php script looks after all of this for you. You can still read this section if you’re interested none the less.

NRDP is literally just a small PHP website that relays XML it receives via it’s restful API into a language that Nagios can interpret. In fact NRDP is just a website that writes specially formatted files into the /var/nagios/spool/checkresults/ directory to which Nagios scans for processing.

The NRDP Payload

The payload is XML; the schema looks like this:

<?xml version='1.0'?> 
<!-- For checking a host -->
<checkresults>
    <!-- The type can be either 'host' or 'service' -->
    <!-- The checktype is always 1 for relaying passive status updates -->
  <checkresult type='host' checktype='1'>
    <hostname>somehost</hostname>
    <!-- 0 = Success, 1 = Warning, 2 = Critical -->
    <state>0</state>
    <!-- A message we might want to associate  -->
    <output>Our somehost is behaving perfectly!</output>
  </checkresult>
</checkresults>

Easy peasy right? A Service message looks like this:

<?xml version='1.0'?> 
<!-- For checking a service -->
<checkresults>
    <!-- The type can be either 'host' or 'service' -->
    <!-- The checktype is always 1 for relaying passive status updates -->
  <checkresult type='service' checktype='1'>
    <!-- This has to be the hostname associated with the service -->
    <hostname>somehost</hostname>
    <!-- This has to be the service we configured using our NRDP
         template.  It's identified by the `service_description`
         field as part of our nagios configuration. -->
    <servicename>someservice</servicename>
    <!-- 0 = Success, 1 = Warning, 2 = Critical -->
    <state>2</state>
    <!-- A message we might want to associate  -->
    <output>Danger Will Robinson! Danger!</output>
  </checkresult>
</checkresults>

You can stack as many messages as you want in a single payload which really adds to NRDPs flexibility:

<?xml version='1.0'?> 
<!-- A mix of status updates -->
<checkresults>
  <checkresult type='host' checktype='1'>
    <hostname>somehost</hostname>
    <state>0</state>
    <output>Our testserver is behaving perfectly!</output>
  </checkresult>

  <checkresult type='service' checktype='1'>
    <hostname>somehost</hostname>
    <servicename>someservice</servicename>
    <state>2</state>
    <output>Houston, we have a problem.</output>
  </checkresult>

  <checkresult type='service' checktype='1'>
    <hostname>somehost</hostname>
    <servicename>my-other-service</servicename>
    <state>1</state>
    <output>Let's hope the problem goes away.</output>
  </checkresult>

</checkresults>

The Transaction

Since it’s a PHP website, it’s hosted via Apache, NginX, etc. This blog sets up Apache but nothing is stopping you from using another service.

This means however that all interactions are web requests. A transaction must be an URL encoded HTTP POST and it must include a special token as part of its payload (part of the verification process). If the token specified is invalid or missing, then the NRDP server will just reject the message. The same rules apply if you don’t POST the message.

You could accomplish this with Python like so:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Site: https://nuxref.com
# Desc: A simply python snip-it to show you how easy it is
#       to interface with NRDP server:
#
# Note: 'requests' (which is required) can be installed 
#       like so (if you don't already have it):
#  #> yum install python-requests
import requests

xml = """<?xml version='1.0'?> 
<checkresults>
  <checkresult type='service' checktype='0'>
    <hostname>somehost</hostname>
    <servicename>someservice</servicename>
    <state>0</state>
    <output>Everything is running great!</output>
  </checkresult>
</checkresults>"""

params = {
   'token': 'nuxref',
   'cmd': 'submitcheck',
   'XMLDATA': xml,
}
r = requests.post(
    'https://nagios.server.addr:5669',
    params=params,
    headers={'Content-Type': 'application/xml'},
    # This is only in place for those who have self signed secure
    # certificates for private networks; you may wish to change
    # this to True in a real environment if you have the ability
    # to verify the host
    verify=False,
)

# Ideally you want to see: 200
print(r.status_code)

# Print our data
print(r.text)

Credit

This blog took me a very (,very) long time to put together and test! The repository hosting alone accommodates all my blog entries up to this date. All of the custom packaging described here was done by me personally. I took the open source available to me and rebuilt it to make it an easier solution and decided to share it.

NRDP certainly does not work like this out of the box; here were the patches I created (see pull request #12 on NRDPs Repository for upstream pushes of this):

  1. nrdp-php_headerfix.patch: Fix PHP header so tool can be executed on the shell.
  2. nrdp-basic_authorized_token.patch:Default (nuxref) token for an out-of-the-box working solution.
  3. nrdp-date.timezone.warning.patch: Silence potential Timezone Warning Message.
  4. nrdp-permissions.patch: Better passive check permission handling for NRDP/Nagios
  5. nrdp-client-support_ports.patch: Support more then just port http 80 (as this blog sets up our NRDP server using ports 5669 and/or 5668).

If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

Sources

Leave a Reply

Your email address will not be published. Required fields are marked *