Tag Archives: Slack

Integrate Apprise into Nagios for More Notification Support

Introduction

Apprise is an open source tool that allows you to send a notification through a wide range of messaging services out there (such as Discord, Slack, Telegram, Microsoft Teams, etc). Well when you combine this with Nagios, you open it up to a much larger scope then simply emailing on an alert.

With Apprise you can configure Nagios to text your mobile phone using Amazon’s Web Service, or notify your devops team on Slack and/or Microsoft Teams. You can even trigger an IFTTT event. But it doesn’t just stop there, Apprise already supports over 35+ notification services today (and is always expanding) which means Nagios could leverage all of this too. This blog will explain how you can set up your instance of Nagios to notify more end points then just email.

The Installation

I’m going to presume you have a copy of Nagios already installed. If you don’t you can check out my blog here on how to set up your own copy with CentOS 7. Those who are not using CentOS are certainly not out of luck though, there are lots of blogs out there to get you started.

This blog will assume you have root privileges or have sudoers privileges.

Apprise can be easily added to your system through pip:

# Install Apprise onto the system currently also hosting Nagios
sudo pip install apprise

Configure Nagios

The Nagios configuration files can vary in their location depending on what Linux distribution you’re using. I’m going to just refer to some standard paths used by the stuff I host here (for CentOS/RedHat).

Step 1: Nagios Import Directory

If you’re using the Nuxref RPMs, then you can skip this step and move to the next as you’ll already be configured for this. Those using another distribution will want to update their nagios.cfg to point to a directory we can use to drop in and remove configuration from. The file is presumably going to be located as: /etc/nagios/nagios.cfg):

# Place this anywhere in /etc/nagios/nagios.cfg
# preferably put it near the bottom of the file.

# Definitions for global configuration directory
cfg_dir=/etc/nagios/conf.d

Now make sure this directory exists because this is where we’ll place our new apprise configuration:

# Ensure our global include directory exists that we
# just defined in our nagios.cfg file:
mkdir -p /etc/nagios/conf.d

# Place a dummy file in here so that Nagios doesn't
# throw any errors (as it isn't a fan of include directories
# without configuration files in it).
touch /etc/nagios/conf.d/dummy.cfg

Step 2: Apprise/Nagios Integration

Now we need to let Nagios know about Apprise. We’ll do this by creating the following files called /etc/nagios/conf.d/apprise.cfg

#
# Apprise to Nagios Configuration File
# Place this file as /etc/nagios/conf.d/apprise.cfg
#
# 'notify-host-by-apprise' command definition
define command{
   command_name   notify-host-by-apprise
   command_line   /usr/bin/printf "%b" "- *Notification Type*: $NOTIFICATIONTYPE$\n- *Host*: $HOSTNAME$\n- *State*: \n- *Address*: $HOSTADDRESS$\n- *Info*: $HOSTOUTPUT$\n\n- *Date/Time*: $LONGDATETIME$\n" | /usr/bin/apprise -c /etc/nagios/apprise.yml -n "$HOSTSTATE$" -g "$NOTIFICATIONTYPE$" -t "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **"
}
 
# 'notify-service-by-apprise' command definition
define command{
   command_name   notify-service-by-apprise
   command_line   /usr/bin/printf "%b" "*Notification Type*: $NOTIFICATIONTYPE$\n- *Service*: $SERVICEDESC$\n- *Host*: $HOSTALIAS$\n- *Address*: $HOSTADDRESS$\n- *State*: $SERVICESTATE$\n- *Date/Time*: $LONGDATETIME$\n\n*Additional Info*:\n$SERVICEOUTPUT$\n" | /usr/bin/apprise -c /etc/nagios/apprise.yml -n "$HOSTSTATE$" -g "$NOTIFICATIONTYPE$" -t "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **"
}

# Register our contact template that we can reference
define contact{
  ; The name of this contact template
  name                             apprise-contact

  ; service notifications can be sent anytime
  service_notification_period      24x7

  ; host notifications can be sent anytime
  host_notification_period         24x7

  ; send notifications for all service states, flapping events,
  ; and scheduled downtime events
  service_notification_options     w,u,c,r,f,s

  ; send notifications for all host states, flapping events,
  ; and scheduled downtime events
  host_notification_options        d,u,r,f,s

  ; send service notifications via email
  service_notification_commands    notify-service-by-apprise

  ; send host notifications via email
  host_notification_commands       notify-host-by-apprise

  ; Don't register this as it is just a template for future
  ; references by contacts who wish to use the apprise plugin
  register                         0
}

Now for every contact we set up going forward, we can point it to use Apprise. By default Nagios usually provides us a contact.cfg file that contains the generic user nagiosadmin. For those using my packaging, you can find this file at /etc/nagios/objects/contacts.cfg; you’ll want to change it to looks like this:

define contact{
  ; Short name of (Nagios) user
  contact_name    nagiosadmin

  ; This next line used to read generic-contact; but we want to switch it
  ; over to our new Apprise based one:
  use             apprise-contact

  ; Full name of user
  alias           Nagios Admin

  ; not important if using apprise-contact (defined above)
  email           nagios@localhost
}

Before you advance to the next step, you’ll want to run a test flight check on your configuration and make sure it validates okay.

# Perform a flight check on our new configuration (as root)
sudo nagios -v /etc/nagios/nagios.cfg

If you get any errors, you should revisit the first part of this blog and try to iron them out before continuing. If everything is error free, then the next step is to reload our instance of Nagios (if it’s running) so it can re-read this configuration. This can be done with the command:

# You will need to be root to do this; send a SIGHUP
# to all instances of nagios running in memory:
sudo killall -HUP nagios

Step 3: Apprise Configuration

Now we need to prepare our Apprise configuration (/etc/nagios/apprise.yml) and fill it with the notification services we want listen for and who we want to pass it to.

We can associate with Nagios notifications types passed to us through tags. Nagios will pass these along one of the following $NOTIFICATIONTYPE$ when an event occurs; these are:

  • PROBLEM: There was an issue with one of the checks.
  • RECOVERY: The issue previously set has been cleared.
  • ACKNOWLEDGEMENT: An outstanding issue has been acknowledged.
  • FLAPPINGSTART: Flapping is a state where a service has a PROBLEM associated with it and then moments later has a RECOVERY. This is the state called FLAPPING. When this process occurs too many times in a row, this alert gets set.
  • FLAPPINGSTOP: The service that was previously FLAPPING is no longer doing so.
  • FLAPPINGDISABLED: Someone just disabled FLAPPING for this service/host.
  • DOWNTIMESTART: The scheduled downtime for this service/host has begun.
  • DOWNTIMEEND: The scheduled downtime is over.
  • DOWNTIMECANCELLED: Someone just cancelled the scheduled downtime for this service/host.

Knowing the above notification types that we’ll receive, here is what an Apprise configuration file located at /etc/nagios/apprise.yml might look like:

# This file should be placed in /etc/nagios/apprise.yml

# NOTE: THIS IS JUST AN EXAMPLE CONFIGURATION FILE. YOU WILL WANT
#       TO CUSTOMIZE YOUR OWN WITH THE SERVICE(S) OF YOUR CHOICE
#       VISIT https://github.com/caronc/apprise TO SEE WHAT IS
#       AVAILABLE AND HOW THEY WORK.

# Identify all of the global notification types we want to flag on.
tag:
  - PROBLEM
  - RECOVERY
  - FLAPPINGSTART
  - FLAPPINGSTOP

# Now we want to define our Apprise URLS; you'll want to visit
# https://github.com/caronc/apprise to see all of the supported
# services and how to build their URLs.
urls:

  # Maybe we want to notify a custom service we're hosting to
  # monitor and track Nagios status; Check out the following 
  # for more details https://github.com/caronc/apprise/wiki/Notify_json

  - json://localhost

  # Maybe we want to notify a Slack channel; more details on this
  # are here: https://github.com/caronc/apprise/wiki/Notify_slack
  - slack://T1JJ3T3L2/A1BRTD4JD/TIiajkdnlazkcOXrIdevi7F/#nuxref

  # the Apprise YAML configuration is quite powerful, the
  # following prepares the email URL and sends an email to each
  # user identified below:
  - email://user:password@gmail.com
      - to: george@example.com
      - to: admin@example.com

  # More details on the emails can be found here:
  # https://github.com/caronc/apprise/wiki/Notify_email

  # We can also individually disperse the tags in the same config
  # file.  The below tags will override the globals defined above.
  # A use case for this would be that maybe we just want to
  # send certain notification types to say... the DevOps team:
  - email://user:password@gmail.com
      - to: devops@example.com
        tag: DOWNTIMESTART, DOWNTIMEEND, DOWNTIMECANCELLED

Once you’re file is all ready, be sure this file is readable by Nagios (to keep it away from prying eyes), but otherwise you’re all set and ready to go!

# Here is what one might do to protect this apprise configuration
chmod 640 /etc/nagios/apprise.yml
chown nagios.root /etc/nagios/apprise.yml

Verification

To be sure everything works, you may want to just test that you got all of your configuration right You can test this using manually as follows:

# Test our configuration with apprise using the PROBLEM tag
# -vvv for some verbose debugging in-case we need it.
apprise -c /etc/nagios/apprise.yml \
    -n CRITICAL -g PROBLEM \
    -vvv \
    -t "A Test Title" \
    -b "a Test Body"

Here is a screenshot of a test error displayed on gitter.im that was sent by Nagios using Apprise:
Gitter Example

Sources

Apprise Makes Sending Notifications via the CLI Easy

Introduction

Apprise is a free Python based notification tool that just makes our lives easier. It allows you to interact with just about all of the most popular Notification Services available to us today! I mean right now while writing this blog, there are already 28 supported services you can use! We’re talk about services like Discord, Telegram, Slack, Pushbullet, IFTTT, Amazon Web Service (SNS), Email, etc, etc.

How Does It Work?

You could be sending our system stats via an email from the command line:

# top will print all of the processes running on our Linux system
# Then we pipe that result into apprise and send it in an email:
top -b -n 1 -c | \
   apprise -t "my system stats" mailto://user:password@hotmail.com

Or maybe you’re a Microsoft Windows user! No problem, Apprise is not OS dependent at all! You could still send your services running to your email like so:

# TASKLIST will print all of the services running on our Widows system
# Then we pipe that result into apprise and send it in an email:
TASKLIST /svc ^ \
   apprise -t "my system stats" mailto://user:password@hotmail.com

The tool itself is pretty simple to use:

apprise [Options] [[[URI1] [URI2] ...]

Here are what the options look like:

Option Description
-t, –title TEXT Specify the message title. This field is optional.
-b, –body TEXT Specify the message body. If you don’t specify this value, then STDIN is used instead.
-n, –notification-type TYPE Specify the message type (default=info). The other options are success, warning, and error.
-T, –theme THEME Specify the default theme. This isn’t discussed in this blog entry, but I’ll talk about it soon!
-v, –verbose By default apprise is pretty quiet. By specifying -v (or a few more -vv and -vvv) you can increase the verbosity of the script. This is useful if you’re trying to figure out why a service might not be working for you.

To make a notification you must provide:

  1. One or more Uniform Resource Identifier(s) (URI). These are used to identify and configure Apprise with the service you wish to access (Email, Discord, etc). I’ll explain a more about this in the next section.
  2. A message body to pass along with the notification you’re going to make. This could be piped in from STDIN like in the examples above, or you can pass the message in via the –body (-b) switch.

Note: In the not so distant future, the message body requirement won’t be required for all services as it is today. An example where the body isn’t always necessary is for IFTTT requests. The message body also isn’t required for most home automation services that Apprise may support in the future!

The Apprise Notification URI

The Uniform Resource Identifier (URI) is the magic one-liner configuration that tells Apprise everything it needs to in order to notify your service(s).

Generally URI looks like this:

  • service://user@host:port
  • service://password@host:port
  • service://user:password@host:port
  • service://host:port
  • service://host

We see these every day when we access a website such as https://nuxref.com.

Take an Apprise Email URI for example:

mailtos://nuxref:mypass@gmail.com
  ^         ^      ^       ^
  |         |      |       |
service   login    |   email domain
                   |
               password

In the above example, Apprise knows you want to send an email because of the mailto:// service identified in front. From there it can acquire all of the server information it needs about this service based upon the rest of the URI provided. Email (URI) configuration can get complicated depending on what you want to do… But for most email providers, Apprise makes it really easy! You can read more about email URIs here.

There are other Apprise URIs that are really easy to use too. For example, if you’re on a Microsoft Windows PC, you can use the windows:// URI and send your system a broadcast message without any effort at all:

apprise -t "Apprise Is Awesome" -b "A Windows Notification" windows://

Note: This only works on a Microsoft Windows PC. It also requires you to have the pypiwin32 bundle installed (if you don’t have it already).

If you’re on a Linux PC you can use the gnome:// or kde:// to achieve the same effects as The MS Windows users get:

# Gnome Messages:
apprise -t "Apprise Is Awesome" -b "A Gnome Notification" gnome://

# KDE Messages
apprise -t "Apprise Is Awesome" -b "A KDE Notification" kde://

There are so many supported notification services, and the list continues to grow.

The other advantage of having all of your configuration in a single URI is that can chain them all together. Apprise will notify each and every one you specify with the message you put in place. There is no limit to the number of servers that you specify.

# The below fires off a pushbullet (pbul://) notification
# and an email (mailto://)
apprise -b "visit nuxref.com" \
   mailto://myusername:mypassword@yahoo.com \
   pbul://o.gn5kj6nfhv736I7jC3cj3QLRiyhgl98b

By uniquely mapping services to their own URIs, we can additionally allow for custom options this way too. For example, let’s say we want our email to be text formatted instead of the default html. No problem:

# Send all of our log files in a text based email format:
cat /var/log/messages | apprise -t "Logging" \
   mailtos://nuxref:mypass@gmail.com?format=text 

Notification Types

Notification Type Image
info
success
warning
failure

By default all of your notifications are sent as an info message. This comes into play depending on what kind of notification service you’re using. Some services really don’t care, while others will give you a richer experience as a result of knowing the message type.

Notification services like Slack and Telegram can decorate the message you send it with an icon.

Meanwhile other services like SMS Messages (sent from AWS) or Emails won’t use this feature at all.

# The below sends a success message to the general channel on
# slack (slack://):
apprise -b "visit nuxref.com" -t "success" \
   slack:///T1JJ3T3L2/A1BRTD4JD/TIiajkdnlazkcOXrIdevi7F/#general

Note: You will need to build your own URI for your own services that you use. the URIs I use in my example are made up just to demonstrate how it works.

Wait… Exposing My Password On The Command Line Is a Terrible Idea!

I couldn’t agree more! Thankfully Apprise supports configuration files

# Place your URLs into a configuration file in our home directory:
cat << _EOF > ~/.apprise
mailto://myusername:mypassword@yahoo.com
pbul://o.gn5kj6nfhv736I7jC3cj3QLRiyhgl98b
slack:///T1JJ3T3L2/A1BRTD4JD/TIiajkdnlazkcOXrIdevi7F/#nuxref
_EOF

# Now secure the file so no one can peak at what is personal
# to you:
chmod 400 ~/.apprise

# Now we can call apprise and source the file for our service URIs:
# By default apprise looks in ~/.apprise so now the command line
# gets even easier to use (and more secure too!)
apprise -b 'Apprise Rocks!'

# The above command would fire the notification to all 3 of the
# services you identified in the ~/.apprise file.

# If you want to manage multiple files, you can easily do this too
# just use the --config switch on the command line and point to
# your file:
apprise -b 'My Message Body' --config=/path/to/config

Microsoft Windows users could do a similar solution:


# Prepare ourselves a file (use a text editor is more ideal
# then what I'm putting here):
mkdir  %APPDATA%/Apprise
echo "mailto://myusername:mypassword@yahoo.com" > %APPDATA%/Apprise/apprise
echo "pbul://o.gn5kj6nfhv736I7jC3cj3QLRiyhgl98b" >> %APPDATA%/Apprise/apprise
echo "slack:///T1JJ3T3L2/A1BRTD4JD/TIiajkdnlazkcOXrIdevi7F/#nuxref" >> %APPDATA%/Apprise/apprise

# Set permissions on this file at this point if you want

# Now we can call apprise and source the file for our service URIs:
apprise -b 'Apprise Rocks!'

# If you want to manage multiple files, you can easily do this too
# just use the --config switch on the command line and point to
# your file:
apprise -b 'My Message Body' --config=C:\path\to\config

How Do I Get Apprise?

Assuming you’ve got Python (2.7 or 3.x) already installed, then PyPI is probably the best place to go. But if you have pip installed, just could also just do the following:

pip install apprise

That’s it! You’re ready to go!

Sources