Tag Archives: NRPE

NRPE for Nagios Core on CentOS 7.x

Introduction

In continuation to part 1 and part 2 of this blog series… NRPE (Nagios Remote Plugin Executor) is yet another Client/Server plugin for Nagios (but can work with other applications too). Unlike NRDP, Nagios isn’t required for NRPE to function which means you can harness the power of this tool and apply it to many other applications too. It does however work ‘very’ well with Nagios and was originally designed for it.

If you read my blog on the NRDP protocol, then you’re already familiar with it’s push architecture where the Applications are responsible for reporting in their status. NRPE however works in the reverse fashion; NRPE requires you to pull the information from the Application Server instead. The status checking responsibility falls on the Nagios Server (instead of the Applications it monitors).

NRPE Overview
NRPE Overview

The diagram above illustrates how the paradigm works.

  • The A represents the Application Server. There is no limit to the number of these guys.
  • The N represents the Nagios Server. You’ll only ever need one Nagios server.

NRPE Query Overview

  1. The Nagios Server will make periodic status checks to the to the Application Server via the NRPE Client (check_nrpe).
  2. The Application Server will analyze the request it received and perform the status check (locally).
  3. When the check completes, it will pass the results back to the Nagios Server (via the same connection the NRPE Client started).
  4. Nagios will take the check_nrpe results and display it accordingly. If the check_nrpe tool can’t establish a connection to the NRPE Server (running on the Application Server), then it will report a failure.

    If the call to the check_nrpe tool takes to long to get a response (or a result) back, then it will be reported as a (timeout) failure. By default check_nrpe will wait up to a maximum of 10 seconds before it times out and gives up. You can change this if you find it too short (or to long).

Here is what you’re getting with this blog and the packaged rpms that go with it:

  • Nagios Core v4.x: updated RPM packaging which I continued to maintain and carry forward from my previous blogs.
    Note: You will need this installed in order to monitor applications with NRPE. See part 1 of this blog series for more information if you don’t already have it set up.
  • NRPE (Nagios Remote Plugin Executor) v3.x: My custom RPM packaging bringing NRPE v3.x to CentOS 7.x for the first time especially since I couldn’t find it available anywhere else (at the time of the this blog entry). I had to make a few modifications to it so that it would be easier used our environment such as:
    • I forwarded the useful patches from the old NRPE v2.x branch to the new NRPE 3.x branch that were applicable still.
    • I patched the SystemD startup script to work with CentOS/Red Hat systems.
    • A sudoer’s file is already to go for those who want to use sudo with their NRPE remote calls.
    • There is firewall configuration all ready to use with FirewallD.

    Best of all, with my RPMs, you can run SELinux in full Enforcing mode for that extra piece of mind from a security standpoint!

    The Goods

    For those who really don’t care and want to just jump right in with the product. Here you go!

    You can download the packages manually if you choose, or reference them using my repository:

    Package Download Description
    nrpe el7.rpm The NRPE Server: This should be installed on any server you want to monitor. The server will allow the machine to respond to requests sent to it via the check_nrpe (Nagios) plugin.
    nrpe-selinux el7.rpm An SELinux add-on package that allows the NRDP Server to operate under Enforcing Mode.

    Note: This RPM is not required by the NRDP server to run correctly.
    nagios-plugins-nrpe el7.rpm Our NRPE Client; this is the Nagios Plugin used to request information from the NRPE Server(s). You’ll only need to install this on the Nagios server (for the purpose of this blog). The RPM provides a small tool called check_nrpe that gets installed into the Nagios Plugin Directory (/usr/lib64/nagios/plugins).

    Through some simple configuration; Nagios can use the check_nrpe tool to monitoring anything you want just as long as the NRPE server is (installed and) running at the other end.

    Note: The source rpm can be obtained here which builds everything you see in the table above. It’s not required for the application to run, but might be useful for developers or those who want to inspect how I put the package together.

    NRPE Server Side Configuration

    The NRPE Server is usually installed onto all of the Application servers you want Nagios to monitor remotely. It provides a means of accepting requests to process (such as, what is your system load like?) and handling the request and returning the response back.

    Assuming you hooked up to my repository here, the NRDP server can can be easily installed with the following command:

    # Install NRPE (Server) on your Application Server
    yum install nrpe nrpe-selinux
    

    NRPE communicates through TCP port 5666; which means you may need to enable the ports in your firewall first to allow remote connections from Nagios. The below commands open up the protocol to everyone attached to your network. You should only perform this command if your Application Server will be running on a local private network:

    # Enable NRPE port (5666)
    firewall-cmd --permanent --add-service=nrpe
    firewall-cmd --add-service=nrpe
    

    If you’re opening this up to the internet, then you might want to just open the (NRPE) port exclusively for the Nagios Server. The following presumes you know the IP of the Nagios Server and will open access to ‘JUST’ that system:

    # Assuming 5.6.7.8 belongs to the Nagios Server you intend to
    # allow to monitor you; you might do the following:
    firewall-cmd --permanent --zone=public \
       --add-rich-rule='\
           rule family="ipv4" \
           source address="5.6.7.8/32" \
           port protocol="tcp" \
           port="5666" \
           accept'
    

    You’ll want to have a look at your NRPE Configuration as you will probably have to update a few lines. See /etc/nagios/nrpe.cfg and have a look for the following directives:

    Directive Details
    allowed_hosts This is added security for NRPE but can come back and haunt you if you ignore it (as nothing will work). You should specify the IP address of your Nagios server here and not allow anyone else! For example, if your Nagios server was 6.7.8.9, then you would put 6.7.8.9/32 here.
    dont_blame_nrpe This allows you to pass options into your remote checks. I personally think this is awesome, but there is no question that depending on the checks that accept arguments, it could could exploit content from your system you wouldn’t have otherwise wanted to share. This is specially the case if you grant NRPE Sudoers permission (discussed a bit lower). Set this value to one (1) to enable argument passing and zero (0) to disable it.
    allow_bash_command_substitution Bash substitution is something like $(hostname) (which might return something like node01.myserver). Set this value to one (1) to enable argument passing and zero (0) to disable it. I don’t use personally use this and therefore have it disabled on my system.

    Setting up NRPE Tokens

    Once your server is set up, there is one last thing you need to do. You need to associate tokens with some of the status checks you want to do. You see, NRPE won’t just execute any program you tell it to, it will only execute programs a specific way that you’ve allowed for. This is purely for security and it makes sense to do so! It’s really not that complicated, consider a token/command mapping like this (as an example):

    Token Command
    check_load # Check the system load:
    # – warning if greater than 10.0
    # – critical if grater then 20.0
    # – we map the check_load token to this command:
    /usr/lib64/nagios/plugins/check_load -w 10 -c 20

    You pass this information into NRPE by creating a .conf file in /etc/nrpe.d (assuming you’re using my RPMs) with the syntax:

    command[token]=/path/to/mapped/command

    So with respect to the example we started; it would look like this:

    command[check_load]=/usr/lib64/nagios/plugins/check_load -w 10 -c 20

    You can specify as many tokens as you like (each one has to be unique from the other). If you have a look in /etc/nrpd.d/, you’ll see one called common_checks.cfg which has a handful of useful commands to start you off with. You can add to this file or start another if you like (in the same directory with a .conf extension). Each time you make changes to the configuration (or another) you’ll need to signal NRPE to have it load any new changes you provided:

    # Restart our NRPE Service
    systemctl restart nrpe.service
    

    Here is a conceptual diagram that will help illustrate what was just explained here using the check_load example above:

    NRPE check_load Example
    NRPE check_load Example

    Sudoer’s Permission

    Substitute User Do (or sudo) allows you to run commands as other users. Most commonly people use sudo to elevate there permission to the root (superuser) privilege to execute a command. When the command has completed, they return back to their normal privileges.

    Some actions require you to have higher system authority to get anything good from them. This includes retrieve certain kinds of system/status information. For example, you can’t check how much mail is spooled and ready for remote delivery (on a Mail Server) unless you reduce the privileges of all of your stored mail (making it accessible by anyone); not a very good idea! But alternatively you can use sudo to grant one user permission to just run a command that can only fetch the number of mail items queued; this is a much better approach! Thus elevating a users permissions for just "special status checking only " commands isn’t so much of a risk. NRPE can be configured to have superuser permissions for it’s status checks which can allow you to monitor a lot more things! Here is how to do it:

    # First make sure that the sudoer's configuration doesn't
    # require tty (teletype terminals) endpoints to use. You
    # do so by commenting out the line that reads 'Defaults requiretty'
    # in the /etc/sudoers file (access it with the command visudo)
    # You can also do this with the below one-liner too
    sed -i -e 's/^\(Defaults[ \t]\+requiretty.*\)$/#\1/g' \
             /etc/sudoers
    
    # By doing this, you allow pseudo-teletype (pty) endpoints
    # to use the sudo command too. NRDP would be a pty service
    # for example since it's not an actual person running the
    # command.
    
    # Update our configuration to use the sudo command
    sed -i -e "s|^[ \t#]*\(command_prefix\)=.*|\1=/usr/bin/sudo|g" \
          /etc/nagios/nrpe.cfg
    
    # Restart our server if it's running
    systemctl restart nrpe.service
    

    SELinux users will want to also do this:

    # Allow NRPE/Nagios calls to run /usr/bin/sudo
    setsebool -P nagios_run_sudo on
    

    NRPE Client Side Configuration

    The client side simply consists of a small application (called check_nrpe) which connects remotely to any NRPE Server you tell it to and hands it a token for processing (provided your NRPE Server is configured correctly). The NRPE Server will take this token and execute a command that was associated with it and return the results back to you.

    # You'll want to be connected to my repositories for this to work:
    # See: https://nuxref.com/repo for more information
    # Install NRPE on the same server running Nagios
    yum install nagios-plugins-nrpe
    

    There is configuration already in place for you if you’re using my RPMS located in /etc/nagios/conf.d called nrpe.cfg. But you can feel free to create your own (or over-write it like so:

    cat << _EOF > /etc/nagios/conf.d/nrpe.cfg
    ; simple wrapper to check_nrpe for remote calls to NRPE servers
    define command{
       command_name check_nrpe
       command_line /usr/lib64/nagios/plugins/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
    }
    
    ; check_nrpe with arguments enabled (enable the dont_blame_nrpe for this
    ; to work properly otherwise simply don't use it)
    define command{
       command_name check_nrpe_args
       command_line /usr/lib64/nagios/plugins/check_nrpe -H $HOSTADDRESS$ -c $ARG1$ -a $ARG2$
    }
    

    You can now send command tokens to servers running NRPE via Nagios. A Nagios configuration might look like this:

    cat << _EOF > /etc/nagios/conf.d/my.application.server01.cfg
    ; first we define a host that we want to monitor.
    ; If you already have a host configured; you can skip this part
    define host{
    ; Name of host template to use. This host definition will inherit
    ; all variables that are defined in (or inherited by) the
    ; linux-server host template definition. You can find this
    ; in /etc/nagios/objects/templates.cfg if you're interested
            use                     linux-server
    
    ; Now we define the server we're going to monitor
            host_name               my-application-server01.nuxref.com
            alias                   my-application-server01.nuxref.com
            address                 192.168.1.2
    
            statusmap_image         redhat.png
            icon_image              redhat.png
            icon_image_alt          CentOS 7.x
    }
    
    ; Now we want to define our monitoring service that talks to our
    ; Application server with NRPE installed on it:
    define service{
    ; Name of service template to use. This service definition will inherit
    ; all variables that are defined in (or inherited by) the
    ; local-service service template definition. You can find this
    ; in /etc/nagios/objects/templates.cfg if you're interested
    	use				local-service
            host_name                       my-application-server01.nuxref.com
    	service_description		Our System Load
    
    ; The Exclamation mark (!) lets nagios know that check_load is the argument
    ; we want to pass into our check_nrpe command we defined earlier (as $ARG1$)
    	check_command			check_nrpe!check_load
    }
    _EOF
    

    If you want to make sure it’s going to work, before you go any further you can run the same command you just told Nagios to do (above):

    # The syntax 'check_nrpe!check_load' gets translated to:
    #  /usr/lib64/nagios/plugins/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
    #
    # Which we can further translate (for testing purposes) to:
    /usr/lib64/nagios/plugins/check_nrpe -H 192.168.1.2 -c check_load
    

    If everything worked okay you should see something similar to the output:

    OK – load average: 0.00, 0.00, 0.00|load1=0.000;10.000;20.000;0; load5=0.000;10.000;20.000;0; load15=0.000;10.000;20.000;0

    You’ll want to reload Nagios to pick up on your new configuration so it can start calling this command too:

    # But before you reload it; it doesn't hurt to just check and make
    # sure your configuration is okay; You can test it out with:
    nagios -v /etc/nagios/nagios.cfg
    
    # correct any errors that display and rerun the above command until
    # everything checks out okay!
    
    # Now reload Nagios so it reads in it's new configuration:
    systemctl reload nagios.service
    

    NRPE vs. NRDP

    Which one should you use? Both NRDP and NRPE have their Pro’s and Cons.

    Benefits of using…
    NRPE NRDP
    You have central control over the checking periods of an application. You only ever need to open 1 TCP port; from a security standpoint; this is awesome!
    You don’t need Nagios to use this. If used properly, it can provide a great way to access remote servers for information and even execute administrative and maintenance commands. You only need to manage the Nagios configuration when a new server is added.
    Can send multiple status messages in one single (TCP) transaction.
    Can reflect a remote application status change immediately oppose to NRPE which one reflect the change until the next status check is performed.

    It should be worth noting that nothing is stopping you from using both NRDP and NRPE at the same time. You might choose an NRDP strategy for remote systems while choosing an NRPE strategy for all your systems residing in your private network. NRPE works over the internet as well, but just exercise caution and be sure to have SELinux running in Enforcing mode on all of the server end points.

    Credit

    This blog took me a very (,very) long time to put together and test! The repository hosting alone accommodates all my blog entries up to this date. All of the custom packaging described here was done by me personally. I took the open source available to me and rebuilt it to make it an easier solution and decided to share it. If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

    Sources

Configuring and Installing NRPE and NSCA into Nagios Core 4 on CentOS 6

Introduction

About a month ago I wrote (and updated) an article on how to install Nagios Core 4 onto your system. I’m a bit of a perfectionist, so I’ve rebuilt the packages a little to accommodate my needs. Now I thought it might be a good idea to introduce some of the powerful extensions you can get for Nagios.

For an updated solution, you may wish to check out the following:

  • NRDP for Nagios Core on CentOS 7.x: This blog explains how awesome NRDP really is and why it might become a vital asset to your own environment. This tool can be used to replace NSCA’s functionality. The blog also provides the first set of working RPMs (with SELinux support of course) of it’s kind to support it.
  • NRPE for Nagios Core on CentOS 7.x: This blog explains how to set up NRPE (v3.x) for your Nagios environment. At the time this blog was written, there was no packaging of it’s kind for this version.

RPM Solution

RPMs provide a version control and an automated set of scripts to configure the system how I want it. The beauty of them is that if you disagree with something the tool you’re packaging does, you can feed RPMs patch files to accommodate it without obstructing the original authors intention.

Now I won’t lie and claim I wrote these SPEC files from scratch because I certainly didn’t. I took the stock ones that ship with these products (NRPE and NSCA) and modified them to accommodate and satisfy my compulsive needs. πŸ™‚

My needs required a bit more automation in the setup as well as including:

  • A previous Nagios requirement I had was a /etc/nagios/conf.d directory to behave similar to how Apache works. I wanted to be able to drop configuration files into here and just have it work without re-adjusting configuration files. In retrospect of this, these plugins are a perfect example of what can use this folder and work right out of the box.
  • These new Nagios plugins should adapt to the new nagiocmd permissions. The nagioscmd group permission was a Nagios requirement I had made in my previous blog specifically for the plugin access.
  • NSCA should prepare some default configuration to make it easier on an administrator.
  • NSCA servers that don’t respond within a certain time should advance to a critical state. This should be part of the default (optional) configuration one can use.
  • Both NRPE and NSCA should plug themselves into Nagios silently without human intervention being required.
  • Both NRPE and NSCA should log independently to their own controlled log file that is automatically rotated by the system when required.

Nagios Enhancement Focus

The key things I want to share with you guys that you may or may not find useful for your own environment are the following:

  • Nagios Remote Plugin Executor (NRPE): NRPE (officially accessed here) provides a way to execute all of the Nagios monitoring tools on a remote server. These actions are all preformed through a secure (private) connection to the remote server and then reported back to Nagios. NRPE can allow you to monitor servers that are spread over a WAN (even the internet) from one central monitoring server. This is truly the most fantastic extension of Nagios in my opinion.
    NRPE High Level Overview
    NRPE High Level Overview
  • Nagios Service Check Acceptor (NSCA): NSCA (officially accessed here) provides a way for external applications to report their status directly to the Nagios Server on their own. This solution still allows the remote monitoring of a system by taking the responsibility off of the status checks off of Nagios. However the fantastic features of Nagios are still applicable: You are still centrally monitoring your application and Nagios will immediately take action in notifying you if your application stops responding or reports a bad status. This solution is really useful when working with closed systems (where opening ports to other systems is not an option).
    NSCA High Level Overview
    NSCA High Level Overview

Just give me your packaged RPMS

Here they are:

How do I make these packages work for me?

In all cases, the RPMs take care of just about everything for you, so there isn’t really much to do at this point. Some considerations however are as follows:

  • NRPE
    NRPE - Nagios Remote Plugin Executor
    NRPE – Nagios Remote Plugin Executor

    In an NRPE setup, Nagios is always the client and all of the magic happens when it uses the check_nrpe plugin. Most of NRPE’s configuration resides at the remote server that Nagios will monitor. In a nutshell, NRPE will provide the gateway to check a remote system’s status but in a much more secure and restrictive manor than the check_ssh which already comes with the nagios-plugins package. The check_ssh requires you to create a remote user account it can connect with for remote checks. This can leave your system vulnerable to an attack since you can do a lot more damage with a compromised SSH account. However check_nrpe uses the NRPE protocol and can only return what you let it; therefore making it a MUCH safer choice then check_ssh!

    You’ll want to install nagios-plugins-nrpe on the same server your hosting Nagios on:

    # Download NRPE
    wget --output-document=nagios-plugins-nrpe-2.15-1.el6.x86_64.rpm http://repo.nuxref.com/centos/6/en/x86_64/custom/nagios-plugins-nrpe-2.15-4.el6.nuxref.x86_64.rpm
    
    # Now install it
    yum -y localinstall nagios-plugins-nrpe-2.15-1.el6.x86_64.rpm
    

    Again I must stress, the above setup will work right away presuming you chose to use my custom build of Nagios introduced in my blog that went with it.

    Just to show you how everything works, we’ll make the Nagios Server the NRPE Server as well. In real world scenario, this would not be the case at all! But feel free to treat the setup example below on a remote system as well because it’s configuration will be identical! πŸ™‚

    # Install our NRPE Server
    wget --output-document=nrpe-2.15-1.el6.x86_64.rpm http://repo.nuxref.com/centos/6/en/x86_64/custom/nrpe-2.15-4.el6.nuxref.x86_64.rpm
    
    # Install some Nagios Plugins we can configure NRPE to use
    wget --output-document=nagios-plugins-1.5-1.x86_64.rpm http://repo.nuxref.com/centos/6/en/x86_64/custom/nagios-plugins-1.5-5.el6.nuxref.x86_64.rpm
    
    # Now Install it
    yum -y localinstall nrpe-2.15-1.el6.x86_64.rpm 
       nagios-plugins-1.5-1.x86_64.rpm
    # This tool requires xinetd to be running; start it if it isn't
    # already running
    service xinetd status || service xinetd start
    
    # Make sure our system will always start xinetd
    # even if it's rebooted
    chkconfig --level 345 xinetd on
    

    Now we can test our server by creating a test configuration:

    # Create a NRPE Configuration our server can accept
    cat << _EOF > /etc/nrpe.d/check_mail.cfg
    command[check_mailq]=/usr/lib64/nagios/plugins/check_mailq -c 100 -w 50
    _EOF
    
    # Create a temporary test configuration to work with:
    cat << _EOF > /etc/nagios/conf.d/nrpe_test.cfg
    define service{
       use                 local-service
       service_description Check Users
       host_name           localhost
       # check_users is already defined for us in /etc/nagios/nrpe.cfg
    	check_command		  check_nrpe!check_users
    }
    
    # Test our new custom one we just created above
    define service{
       use                 local-service
       service_description Check Mail Queue
       host_name           localhost
       # Use the new check_mailq we defined above in /etc/nrpe.d/check_mail.cfg
    	check_command		  check_nrpe!check_mailq
    }
    _EOF
    
    # Reload Nagios so it sees our new configuration defined in
    # /etc/nagios/conf.d/*
    service nagios reload
    
    # Reload xinetd so nrpe sees our new configuration defined in
    # /etc/nrpe.d/*
    service xinetd reload
    

    We can even test our connection manually by calling the command:

    # This is what the output will look like if everything is okay:
    /usr/lib64/nagios/plugins/check_nrpe -H localhost -c check_mailq
    OK: mailq is empty|unsent=0;50;100;0
    

    Another scenario you might see (when setting on up on your remote server) is:

    /usr/lib64/nagios/plugins/check_nrpe -H localhost -c check_mailq
    CHECK_NRPE: Error - Could not complete SSL handshake.
    

    Uh oh, Could not complete SSL handshake.! What does that mean?
    This is the most common error people see with the NRPE plugin. If you Google it, you’ll get an over-whelming amount of hits suggesting how you can resolve the problem. I found this link useful.
    That all said, I can probably tell you right off the bat why it isn’t working for you. Assuming you’re using the packaging I provided then it’s most likely because your NRPE Server is denying the requests your Nagios Server is making to it.

    To fix this, access your NRPE Server and open up /etc/xinetd/nrpe in an editor of your choice. You need to allow your Nagios Server access by adding it’s IP address to the only_from entry. Or you can just type the following:

    # Set your Nagios Server IP here:
    NAGIOS_SERVER=192.168.192.168
    
    # If you want to keep your previous entries and append the server
    # you can do the following (spaces delimit the servers):
    sed -i -e "s|^(.*only_from[^=]+=)[ t]*(.*)|1 2 $NAGIOS_SERVER|g" 
       /etc/xinetd.d/nrpe
    
    # The below command is fine too to just replace what is there
    # with the server of your choice (you can use either example
    sed -i -e "s|^(.*only_from[^=]+=).*|1 $NAGIOS_SERVER|g" 
       /etc/xinetd.d/nrpe
    
    # When your done, restart xinetd to update it's configuration
    service xinetd reload
    

    Those who didn’t receive the error I showed above, it’s only because your using your Nagios Server as your NRPE Server too (which the xinetd tool is pre-configured to accept by default). So please pay attention to this when you start installing the NRPE server remotely.

    You will want to install nagios-plugins-nrpe on to your NRPE Server as well granting you access to all the same great monitoring tools that have already been proven to work and integrate perfectly with Nagios. This will save you a great deal of effort when setting up the NRPE status checks.

    As a final note, you may want to make sure port 5666 is open on your NRPE Server’s firewall otherwise the Nagios Server will not be able to preform remote checks.

    ## Open NRPE Port (as root)
    iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 5666 -j ACCEPT
    
    # consider adding this change to your iptables configuration
    # as well so when you reboot your system the port is
    # automatically open for you. See: /etc/sysconfig/iptables
    # You'll need to add a similar line as above (without the
    # iptables reference)
    # -A INPUT -m state --state NEW -m tcp -p tcp --dport 5666 -j ACCEPT
    
  • NSCA
    NSCA - Nagios Service Check Acceptor
    NSCA – Nagios Service Check Acceptor

    Remember, NSCA is used for systems that connect to you remotely (instead of you connecting to them (what NRPE does). This is a perfect choice plugin for systems you do not want to open ports up to unnecessarily on your remote system. That said, it means you need to open up ports on your Monitoring (Nagios) server instead.

    You’ll want to install nsca on the same server your hosting Nagios on:

    # Download NSCA
    wget --output-document=nsca-2.7.2-9.el6.x86_64.rpm http://repo.nuxref.com/centos/6/en/x86_64/custom/nsca-2.7.2-10.el6.nuxref.x86_64.rpm
    
    # Now install it
    yum -y localinstall nsca-2.7.2-9.el6.x86_64.rpm
    
    # This tool requires xinetd to be running; start it if it isn't
    # already running
    service xinetd status || service xinetd start
    
    # Make sure our system will always start xinetd
    # even if it's rebooted
    chkconfig --level 345 xinetd on
    
    # SELinux Users may wish to turn this flag on if they intend to allow it
    # to call content as root (using sudo) which it must do for some status checks.
    setsebool -P nagios_run_sudo on
    

    The best way to test if everything is working okay is by also installing the nsca-client on the same machine we just installed NSCA on (above). Then we can simply create a test passive service to test everything with. The below setup will work presuming you chose to use my custom build of Nagios introduced in my blog that went with it.

    # First install our NSCA client on the same machine we just installed NSCA
    # on above.
    wget http://repo.nuxref.com/centos/6/en/x86_64/custom/nsca-client-2.7.2-10.el6.nuxref.x86_64.rpm
    
    # Now install it
    yum -y localinstall nsca-client-2.7.2-9.el6.x86_64.rpm
    
    # Create a temporary test configuration to work with:
    cat << _EOF > /etc/nagios/conf.d/nsca_test.cfg
    # Define a test service. Note that the service 'passive_service'
    # is already predefined in /etc/nagios/conf.d/nsca.cfg which was
    # placed when you installed my nsca rpm
    define service{
       use                 passive_service
       service_description TestMessage
       host_name           localhost
    }
    _EOF
    
    # Now reload Nagios to it reads in our new configuration
    # Note: This will only work if you are using my Nagios build
    service nagios reload
    

    Now that we have a test service set up, we can send it different nagios status through the send_nsca binary that was made available to us after installing nsca-client.

    # Send a Critical notice to Nagios using our test service
    # and send_nsca. By default send_nsca uses the '<tab>' as a
    # delimiter, but that is hard to show in a blog (it can get mixed up
    # with the space.  So in the examples below i add a -d switch
    # to adjust what the delimiter in the message.
    # The syntax is simple:
    #    hostname,nagios_service,status_code,status_msg
    #
    # The test service we defined above identifies both the
    # 'host_name' and 'service_description' define our first 2
    # delimited columns below. The status_code is as simple as:
    #       0 : Okay
    #       1 : Warning
    #       2 : Critical
    # The final delimited entry is just the human readable text
    # we want to pass a long with the status.
    #
    # Here we'll send our critical message:
    cat << _EOF | /usr/sbin/send_nsca -H 127.0.0.1 -d ','
    localhost,TestMessage,2,This is a Test Error
    _EOF
    
    # Open your Nagios screen (http://localhost/nagios) at this point and watch the
    # status change (it can take up to 4 or 5 seconds or so to register
    # the command above).
    
    # Cool?  Here is a warning message:
    cat << _EOF | /usr/sbin/send_nsca -H 127.0.0.1 -d ',' -c /etc/nagios/send_nsca.cfg
    localhost,TestMessage,1,This is a Test Warning
    _EOF
    
    # Check your warning on Nagios, when your happy, here is your
    # OKAY message:
    cat << _EOF | /usr/sbin/send_nsca -H 127.0.0.1 -d ',' -c /etc/nagios/send_nsca.cfg
    localhost,TestMessage,0,Life is good!
    _EOF
    

    Since NSCA requires you to listen to a public port, you’ll need to know this last bit of information to complete your NSCA configuration. Up until now the package i provide only open full access to localhost for security reasons. But you’ll need to take the next step and allow your remote systems to talk to you.

    NSCA uses port 5667, so you’ll want to make sure your firewall has this port open using the following command:

    ## Open NSCA Port (as root)
    iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 5667 -j ACCEPT
    
    # consider adding this change to your iptables configuration
    # as well so when you reboot your system the port is
    # automatically open for you. See: /etc/sysconfig/iptables
    # You'll need to add a similar line as above (without the
    # iptables reference)
    # -A INPUT -m state --state NEW -m tcp -p tcp --dport 5667 -j ACCEPT
    

    Another security in place with the NSCA configuration you installed out of
    the box is that it is being managed by xinetd. The configuration can
    be found here: /etc/xinetd.d/nsca. The security restriction in place that you’ll want to pay close attention to is line 16 which reads:

    only_from = 127.0.0.1 ::1

    If you remove this line, you’ll allow any system to connect to yours; this is a bit unsafe but an option. Personally, I recommend that you individually add each remote system you want to monitor to this line. Use a space to separate more the one system.

    You can consider adding more security by setting up a NSCA paraphrase which will reside in /etc/nagios/nsca.cfg to which you can place the same paraphrase in all of the nsca-clients you set up by updating /etc/nagios/send_nsca.cfg.

    Consider our example above; I can do the following to add a paraphrase:

    # Configure Client
    sed -i -e 's/^#*password=/password=ABCDEFGHIJKLMNOPQRSTUVWXYZ/g' 
       /etc/nagios/send_nsca.cfg
    # Configure Server
    sed -i -e 's/^#*password=/password=ABCDEFGHIJKLMNOPQRSTUVWXYZ/g' 
       /etc/nagios/nsca.cfg
    # Reload xinetd so it rereads /etc/nagios/nsca.cfg
    service xinetd reload
    

I don’t trust you, I want to repackage this myself!

As always, I will always provide you a way to build the source code from scratch if you don’t want to use what I’ve already prepared. I use mock for everything I build so I don’t need to haul in development packages into my native environment. You’ll need to make sure mock is setup and configured properly first for yourself:

# Install 'mock' into your environment if you don't have it already.
# This step will require you to be the superuser (root) in your native
# environment.
yum install -y mock

# Grant your normal every day user account access to the mock group
# This step will also require you to be the root user.
usermod -a -G mock YourNonRootUsername

At this point it’s safe to change from the ‘root‘ user back to the user account you granted the mock group privileges to in the step above. We won’t need the root user again until the end of this tutorial when we install our built RPM.

Just to give you a quick summary of what I did, here are the new spec files and patch files I created:

  • NSCA RPM SPEC File: Here is the enhanced spec file I used (enhancing the one already provided in the EPEL release found on pkgs.org). At the time I wrote this blog, the newest version of NSCA was v2.7.2-8. This is why I repackaged it as v2.7.2-9 to include my enhancements. I created 2 patches along with the spec file enhancements.
    nrpe.conf.d.patch was created to provide a working NRPE configuration right out of the box (as soon as it was installed) and nrpe.xinetd.logrotate.patch was created to pre-configure a working xinetd server configuration.
  • NRPE RPM SPEC File: Here is the enhanced spec file I used (enhancing the one already provided in the EPEL release found on pkgs.org). At the time I wrote this blog, the newest version of NRPE was v2.14-5. However v2.15 was available off of the Nagios website so this is why I repackaged it as v2.15-1 to include my enhancements.
    nsca.xinetd.logrotate.patch was the only patch I needed to create to prepare a NSCA xinetd server working out of the box.

Everything else packaged (patches and all) are the same ones carried forward from previous versions by their package managers.

Rebuild your external monitoring solutions:

Below shows the long way of rebuilding the RPMs from source.

# Perhaps make a directory and work within it so it's easy to find
# everything later
mkdir nagiosbuild
cd nagiosbuild
###
# Now we want to download all the requirements we need to build
###
# Prepare our mock environment
###
# Initialize Mock Environment
mock -v -r epel-6-x86_64 --init

# NRPE (v2.15)
wget http://repo.nuxref.com/centos/6/en/source/custom/nrpe-2.15-4.el6.nuxref.src.rpm 
mock -v -r epel-6-x86_64 --copyin nrpe-2.15-1.el6.src.rpm /builddir/build

# NSCA (v2.7.2)
wget http://repo.nuxref.com/centos/6/en/source/custom/nsca-2.7.2-10.el6.nuxref.src.rpm 
mock -v -r epel-6-x86_64 --copyin nsca-2.7.2-9.el6.src.rpm /builddir/build

#######################
### THE SHORT WAY #####
#######################
# Now, the short way to rebuild everything is through these commands:
mock -v -r epel-6-x86_64 --resultdir=$(pwd)/results 
   --rebuild  nrpe-2.15-1.el6.src.rpm  nsca-2.7.2-9.el6.src.rpm

# You're done; You can find all of your rpms in a results directory
# in the same location you typed the above command in.  You can 
# alternatively rebuild everything the long way allowing you to
# inspect the content in more detail and even change it for your
# own liking

#######################
### THE LONG WAY  #####
#######################
# Install NRPE Dependencies
mock -v -r epel-6-x86_64 --install 
   autoconf automake libtool openssl-devel tcp_wrappers-devel

# Install NSCA Dependencies
mock -v -r epel-6-x86_64 --install 
   tcp_wrappers-devel libmcrypt-devel

###
# Build Stage
###
# Shell into our enviroment
mock -v -r epel-6-x86_64 --shell

# Change to our build directory
cd builddir/build

# Install our SRPMS (within our mock jail)
rpm -Uhi nsca-*.src.rpm nrpe-*.src.rpm

# Now we'll have placed all our content in the SPECS and SOURCES
# directory (within /builddir/build).  Have a look to verify
# content if you like

# Build our RPMS
rpmbuild -ba SPECS/*.spec

# we're now done with our mock environment for now; Press Ctrl-D to
# exit or simply type exit on the command line of our virtual
# environment
exit

###
# Save our content that we built in the mock environment
###

#NRPE
mock -v -r epel-6-x86_64 --copyout /builddir/build/SRPMS/nrpe-2.15-1.el6.src.rpm .
mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nrpe-2.15-1.el6.x86_64.rpm .
mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nagios-plugins-nrpe-2.15-1.el6.x86_64.rpm .
mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nrpe-debuginfo-2.15-1.el6.x86_64.rpm .

#NSCA
mock -v -r epel-6-x86_64 --copyout /builddir/build/SRPMS/nsca-2.7.2-9.el6.src.rpm .
mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nsca-2.7.2-9.el6.x86_64.rpm .
mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nsca-client-2.7.2-9.el6.x86_64.rpm .
mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nsca-debuginfo-2.7.2-9.el6.x86_64.rpm .

# *Note that all the commands that interact with mock I pass in 
# the -v which outputs a lot of verbose information. You don't
# have to supply it; but I like to see what is going on at times.

# **Note: You may receive this warning when calling the '--copyout'
# above:
# WARNING: unable to delete selinux filesystems 
#    (/tmp/mock-selinux-plugin.??????): #
#    [Errno 1] Operation not permitted: '/tmp/mock-selinux-plugin.??????'
#
# This is totally okay; and is safe to ignore, the action you called
# still worked perfectly; so don't panic!

So where do I go from here?
NRPE and NSCA are both fantastic solutions that can allow you to tackle any monitoring problem you ever had. In this blog here I focus specifically on Linux, but these tools are also available on Microsoft Windows as well. You can easily have 1 Nagios Server manage thousands of remote systems (of all operating system flavours). There are hundreds of fantastic tools to monitor all mainstream applications used today (Databases, Web Servers, etc). Even if your trying to support a custom application you wrote. If you can interface with your application using the command line interface, well then Nagios can monitor it for you. You only need to write a small script with this in mind:

  • Your script should always have an exit code of 0 (zero) if everything is okay, 1 (one) if you want to raise a warning, and 2 (two) if you want to raise a critical alarm.
  • No matter what the exit code is, you should also echo some kind of message that someone could easily interpret what is going on.

There is enough information in this blog to do the rest for you (as far as creating a Nagios configuration entry for it goes). If you followed the 2 rules above, then everything should ‘just work’. It’s truely that easy and powerful.

How do I decide if I need NSCA or NRPE?

NRPE & NSCA High Level Overview
NRPE & NSCA High Level Overview

NRPE makes it Nagios’s responsibility to check your application where as NSCA makes it your applications responsible to report its status. Both have their pros and cons. NSCA could be considered the most secure approach because at the end of the day the only port that requires opening is the one on the Nagios server. NSCA does not use a completely secure connection (but there is encryption none the less). NRPE is very secure and doesn’t require you to really do much since it just simply works with the nagios-plugins already available. It litterally just extends these existing Nagios local checks to remote ones. NSCA requires you to configure a cron, or adjust your applications in such a way that it frequently calls the send_nsca command. NSCA can be a bit more difficult to set up but creates some what of a heartbeat between you and the system monitoring it (which can be a good thing too). I pre-configured the NSCA server with a small tweak that will automatically set your application to a critical state if a send_nsca call is missed for an extended period of time.

Always consider that the point of this blog was to state that you can use both at the same time giving you total flexibility over all of your systems that require monitoring.

Credit

All of the custom packaging in this blog was done by me personally. I took the open source available to me and rebuilt it to make it an easier solution and decided to share it. If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

Sources

I referenced the following resources to make this blog possible:

  • The blog I wrote earlier that is recommended you read before this one:Configuring and Installing Nagios Core 4 on CentOS 6
  • Official NRPE download link; I used all of the official documentation to make the NRPE references on this blog possible.
  • A document identifying the common errors you might see and their resolution here.
  • Official NSCA download link; I used all of the official documentation to make the NSCA references on this blog possible.
  • The NRPE and NSCA images I’m reposting on this blog were taking straight from their official sites mentioned above.
  • Linux Packages Search (pkgs.org) was where I obtained the source RPMs as well as their old SPEC files. These would be a starting point before I’d expand them.
  • A bit outdated, but a great (and simple) representation of how NSCA works with Nagios can be seen here.