Introduction

In continuation to part 1 and part 2 of this blog series… NRPE (Nagios Remote Plugin Executor) is yet another Client/Server plugin for Nagios (but can work with other applications too). Unlike NRDP, Nagios isn’t required for NRPE to function which means you can harness the power of this tool and apply it to many other applications too. It does however work ‘very’ well with Nagios and was originally designed for it.

If you read my blog on the NRDP protocol, then you’re already familiar with it’s push architecture where the Applications are responsible for reporting in their status. NRPE however works in the reverse fashion; NRPE requires you to pull the information from the Application Server instead. The status checking responsibility falls on the Nagios Server (instead of the Applications it monitors).

• The A represents the Application Server. There is no limit to the number of these guys.
• The N represents the Nagios Server. You’ll only ever need one Nagios server.

1. The Nagios Server will make periodic status checks to the to the Application Server via the NRPE Client (check_nrpe).
2. The Application Server will analyze the request it received and perform the status check (locally).
3. When the check completes, it will pass the results back to the Nagios Server (via the same connection the NRPE Client started).
4. Nagios will take the check_nrpe results and display it accordingly. If the check_nrpe tool can’t establish a connection to the NRPE Server (running on the Application Server), then it will report a failure.

   If the call to the check_nrpe tool takes to long to get a response (or a result) back, then it will be reported as a (timeout) failure. By default check_nrpe will wait up to a maximum of 10 seconds before it times out and gives up. You can change this if you find it too short (or to long).

Here is what you’re getting with this blog and the packaged rpms that go with it:

• Nagios Core v4.x: updated RPM packaging which I continued to maintain and carry forward from my previous blogs.
  Note: You will need this installed in order to monitor applications with NRPE. See part 1 of this blog series for more information if you don’t already have it set up.
• NRPE (Nagios Remote Plugin Executor) v3.x: My custom RPM packaging bringing NRPE v3.x to CentOS 7.x for the first time especially since I couldn’t find it available anywhere else (at the time of the this blog entry). I had to make a few modifications to it so that it would be easier used our environment such as:
  • I forwarded the useful patches from the old NRPE v2.x branch to the new NRPE 3.x branch that were applicable still.
  • I patched the SystemD startup script to work with CentOS/Red Hat systems.
  • A sudoer’s file is already to go for those who want to use sudo with their NRPE remote calls.
  • There is firewall configuration all ready to use with FirewallD.

  Best of all, with my RPMs, you can run SELinux in full Enforcing mode for that extra piece of mind from a security standpoint!

  The Goods

  For those who really don’t care and want to just jump right in with the product. Here you go!

  You can download the packages manually if you choose, or reference them using my repository:

  Package	Download	Description
  nrpe	el7.rpm	The NRPE Server: This should be installed on any server you want to monitor. The server will allow the machine to respond to requests sent to it via the check_nrpe (Nagios) plugin.
  nrpe-selinux	el7.rpm	An SELinux add-on package that allows the NRDP Server to operate under Enforcing Mode. Note: This RPM is not required by the NRDP server to run correctly.
  nagios-plugins-nrpe	el7.rpm	Our NRPE Client; this is the Nagios Plugin used to request information from the NRPE Server(s). You’ll only need to install this on the Nagios server (for the purpose of this blog). The RPM provides a small tool called check_nrpe that gets installed into the Nagios Plugin Directory (/usr/lib64/nagios/plugins). Through some simple configuration; Nagios can use the check_nrpe tool to monitoring anything you want just as long as the NRPE server is (installed and) running at the other end.

  Note: The source rpm can be obtained here which builds everything you see in the table above. It’s not required for the application to run, but might be useful for developers or those who want to inspect how I put the package together.

  NRPE Server Side Configuration

  The NRPE Server is usually installed onto all of the Application servers you want Nagios to monitor remotely. It provides a means of accepting requests to process (such as, what is your system load like?) and handling the request and returning the response back.

  Assuming you hooked up to my repository here, the NRDP server can can be easily installed with the following command:

  # Install NRPE (Server) on your Application Server
  yum install nrpe nrpe-selinux
  

  NRPE communicates through TCP port 5666; which means you may need to enable the ports in your firewall first to allow remote connections from Nagios. The below commands open up the protocol to everyone attached to your network. You should only perform this command if your Application Server will be running on a local private network:

  # Enable NRPE port (5666)
  firewall-cmd --permanent --add-service=nrpe
  firewall-cmd --add-service=nrpe
  

  If you’re opening this up to the internet, then you might want to just open the (NRPE) port exclusively for the Nagios Server. The following presumes you know the IP of the Nagios Server and will open access to ‘JUST’ that system:

  # Assuming 5.6.7.8 belongs to the Nagios Server you intend to
  # allow to monitor you; you might do the following:
  firewall-cmd --permanent --zone=public \
     --add-rich-rule='\
         rule family="ipv4" \
         source address="5.6.7.8/32" \
         port protocol="tcp" \
         port="5666" \
         accept'
  

  You’ll want to have a look at your NRPE Configuration as you will probably have to update a few lines. See /etc/nagios/nrpe.cfg and have a look for the following directives:

  Directive	Details
  allowed_hosts	This is added security for NRPE but can come back and haunt you if you ignore it (as nothing will work). You should specify the IP address of your Nagios server here and not allow anyone else! For example, if your Nagios server was 6.7.8.9, then you would put 6.7.8.9/32 here.
  dont_blame_nrpe	This allows you to pass options into your remote checks. I personally think this is awesome, but there is no question that depending on the checks that accept arguments, it could could exploit content from your system you wouldn’t have otherwise wanted to share. This is specially the case if you grant NRPE Sudoers permission (discussed a bit lower). Set this value to one (1) to enable argument passing and zero (0) to disable it.
  allow_bash_command_substitution	Bash substitution is something like $(hostname) (which might return something like node01.myserver). Set this value to one (1) to enable argument passing and zero (0) to disable it. I don’t use personally use this and therefore have it disabled on my system.

  Setting up NRPE Tokens

  Once your server is set up, there is one last thing you need to do. You need to associate tokens with some of the status checks you want to do. You see, NRPE won’t just execute any program you tell it to, it will only execute programs a specific way that you’ve allowed for. This is purely for security and it makes sense to do so! It’s really not that complicated, consider a token/command mapping like this (as an example):

  Token	Command
  check_load	# Check the system load: # – warning if greater than 10.0 # – critical if grater then 20.0 # – we map the check_load token to this command: /usr/lib64/nagios/plugins/check_load -w 10 -c 20

  You pass this information into NRPE by creating a .conf file in /etc/nrpe.d (assuming you’re using my RPMs) with the syntax:

  command[token]=/path/to/mapped/command

  So with respect to the example we started; it would look like this:

  command[check_load]=/usr/lib64/nagios/plugins/check_load -w 10 -c 20

  You can specify as many tokens as you like (each one has to be unique from the other). If you have a look in /etc/nrpd.d/, you’ll see one called common_checks.cfg which has a handful of useful commands to start you off with. You can add to this file or start another if you like (in the same directory with a .conf extension). Each time you make changes to the configuration (or another) you’ll need to signal NRPE to have it load any new changes you provided:

  # Restart our NRPE Service
  systemctl restart nrpe.service
  

  Here is a conceptual diagram that will help illustrate what was just explained here using the check_load example above:

  

  Sudoer’s Permission

  Substitute User Do (or sudo) allows you to run commands as other users. Most commonly people use sudo to elevate there permission to the root (superuser) privilege to execute a command. When the command has completed, they return back to their normal privileges.

  Some actions require you to have higher system authority to get anything good from them. This includes retrieve certain kinds of system/status information. For example, you can’t check how much mail is spooled and ready for remote delivery (on a Mail Server) unless you reduce the privileges of all of your stored mail (making it accessible by anyone); not a very good idea! But alternatively you can use sudo to grant one user permission to just run a command that can only fetch the number of mail items queued; this is a much better approach! Thus elevating a users permissions for just "special status checking only " commands isn’t so much of a risk. NRPE can be configured to have superuser permissions for it’s status checks which can allow you to monitor a lot more things! Here is how to do it:

  # First make sure that the sudoer's configuration doesn't
  # require tty (teletype terminals) endpoints to use. You
  # do so by commenting out the line that reads 'Defaults requiretty'
  # in the /etc/sudoers file (access it with the command visudo)
  # You can also do this with the below one-liner too
  sed -i -e 's/^\(Defaults[ \t]\+requiretty.*\)$/#\1/g' \
           /etc/sudoers
  
  # By doing this, you allow pseudo-teletype (pty) endpoints
  # to use the sudo command too. NRDP would be a pty service
  # for example since it's not an actual person running the
  # command.
  
  # Update our configuration to use the sudo command
  sed -i -e "s|^[ \t#]*\(command_prefix\)=.*|\1=/usr/bin/sudo|g" \
        /etc/nagios/nrpe.cfg
  
  # Restart our server if it's running
  systemctl restart nrpe.service
  

  SELinux users will want to also do this:

  # Allow NRPE/Nagios calls to run /usr/bin/sudo
  setsebool -P nagios_run_sudo on
  

  NRPE Client Side Configuration

  The client side simply consists of a small application (called check_nrpe) which connects remotely to any NRPE Server you tell it to and hands it a token for processing (provided your NRPE Server is configured correctly). The NRPE Server will take this token and execute a command that was associated with it and return the results back to you.

  # You'll want to be connected to my repositories for this to work:
  # See: https://nuxref.com/repo for more information
  # Install NRPE on the same server running Nagios
  yum install nagios-plugins-nrpe
  

  There is configuration already in place for you if you’re using my RPMS located in /etc/nagios/conf.d called nrpe.cfg. But you can feel free to create your own (or over-write it like so:

  cat << _EOF > /etc/nagios/conf.d/nrpe.cfg
  ; simple wrapper to check_nrpe for remote calls to NRPE servers
  define command{
     command_name check_nrpe
     command_line /usr/lib64/nagios/plugins/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
  }
  
  ; check_nrpe with arguments enabled (enable the dont_blame_nrpe for this
  ; to work properly otherwise simply don't use it)
  define command{
     command_name check_nrpe_args
     command_line /usr/lib64/nagios/plugins/check_nrpe -H $HOSTADDRESS$ -c $ARG1$ -a $ARG2$
  }
  

  You can now send command tokens to servers running NRPE via Nagios. A Nagios configuration might look like this:

  cat << _EOF > /etc/nagios/conf.d/my.application.server01.cfg
  ; first we define a host that we want to monitor.
  ; If you already have a host configured; you can skip this part
  define host{
  ; Name of host template to use. This host definition will inherit
  ; all variables that are defined in (or inherited by) the
  ; linux-server host template definition. You can find this
  ; in /etc/nagios/objects/templates.cfg if you're interested
          use                     linux-server
  
  ; Now we define the server we're going to monitor
          host_name               my-application-server01.nuxref.com
          alias                   my-application-server01.nuxref.com
          address                 192.168.1.2
  
          statusmap_image         redhat.png
          icon_image              redhat.png
          icon_image_alt          CentOS 7.x
  }
  
  ; Now we want to define our monitoring service that talks to our
  ; Application server with NRPE installed on it:
  define service{
  ; Name of service template to use. This service definition will inherit
  ; all variables that are defined in (or inherited by) the
  ; local-service service template definition. You can find this
  ; in /etc/nagios/objects/templates.cfg if you're interested
  	use				local-service
          host_name                       my-application-server01.nuxref.com
  	service_description		Our System Load
  
  ; The Exclamation mark (!) lets nagios know that check_load is the argument
  ; we want to pass into our check_nrpe command we defined earlier (as $ARG1$)
  	check_command			check_nrpe!check_load
  }
  _EOF
  

  If you want to make sure it’s going to work, before you go any further you can run the same command you just told Nagios to do (above):

  # The syntax 'check_nrpe!check_load' gets translated to:
  #  /usr/lib64/nagios/plugins/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
  #
  # Which we can further translate (for testing purposes) to:
  /usr/lib64/nagios/plugins/check_nrpe -H 192.168.1.2 -c check_load
  

  If everything worked okay you should see something similar to the output:

  OK – load average: 0.00, 0.00, 0.00|load1=0.000;10.000;20.000;0; load5=0.000;10.000;20.000;0; load15=0.000;10.000;20.000;0

  You’ll want to reload Nagios to pick up on your new configuration so it can start calling this command too:

  # But before you reload it; it doesn't hurt to just check and make
  # sure your configuration is okay; You can test it out with:
  nagios -v /etc/nagios/nagios.cfg
  
  # correct any errors that display and rerun the above command until
  # everything checks out okay!
  
  # Now reload Nagios so it reads in it's new configuration:
  systemctl reload nagios.service
  

  NRPE vs. NRDP

  Which one should you use? Both NRDP and NRPE have their Pro’s and Cons.

  Benefits of using…
  NRPE	NRDP
  You have central control over the checking periods of an application.	You only ever need to open 1 TCP port; from a security standpoint; this is awesome!
  You don’t need Nagios to use this. If used properly, it can provide a great way to access remote servers for information and even execute administrative and maintenance commands.	You only need to manage the Nagios configuration when a new server is added.
  	Can send multiple status messages in one single (TCP) transaction.
  	Can reflect a remote application status change immediately oppose to NRPE which one reflect the change until the next status check is performed.

  It should be worth noting that nothing is stopping you from using both NRDP and NRPE at the same time. You might choose an NRDP strategy for remote systems while choosing an NRPE strategy for all your systems residing in your private network. NRPE works over the internet as well, but just exercise caution and be sure to have SELinux running in Enforcing mode on all of the server end points.

  Credit

  This blog took me a very (,very) long time to put together and test! The repository hosting alone accommodates all my blog entries up to this date. All of the custom packaging described here was done by me personally. I took the open source available to me and rebuilt it to make it an easier solution and decided to share it. If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

  Sources

  • Nagios Core 4.x Setup for CentOS 7.x: Part 1 of this blog series.
  • NRDP for Nagios Core on CentOS 7.x: Part 2 of this blog series.
  • Use Apprise to greatly extend the notifications Nagios can deliver to (to such locations like Telegram, Discord, Amazon SNS, Twilio, etc.
  • Installing Nagios Core 4 on CentOS 6 is a blog I wrote 3 years ago which is still applicable today. All of the packages identified here have been forward packaged for CentOS 7.x.
  • Configuring and installing NRPE and NSCA into Nagios Core 4 on CentOS 6 is another blog that is still got great information that is applicable to this blog.
  • The Nagios Official Website
  • The Nagios Plugins
  • Nagios Exchange: Specifically the link about NRPE.
  Tweet