Mail Server Solution

Configuring a Secure Mail Server on CentOS 6


Introduction
As you know, Mail Server tutorials are all over the internet. So if you stumbled upon mine, then you were like me trying to assemble one based on how others do it to save reading the manual. A mail server is NOT a walk through the park like some other servers are (such as Web, Database, etc). They can be very complicated to master; and frankly, when you’re short on time, it’s easier to just get one going right away. Don’t dismiss the need to learn how it works once you’ve set it up otherwise you may not know how to recover from a unexpected situation. But instead learn about how it works (and how to further tweak it) after the deadlines have been met. Besides, your boss will be impressed when you show him your successful results sooner then later.

You’re reading this blog because your needs were/are similar to what mine were:

  • You’re looking for an automated solution to installing a mail server with little effort because you’re on a tight deadline.
  • You want a central configuration point; you want everything to be easy to maintain after it’s all set up.
  • You want everything to just work the first time and you want to leave the figuring it out part to the end.
  • Package management and version control is incredibly important to you.

Here is what my tutorial will be focused on:

  • A Database Backend (PostgreSQL) giving you central configuration. This tutorial focuses on version 8.4 because that is what ships with CentOS and Red Hat. But most (if not all) of this tutoral should still work fine if you choose to use version 9.x of the database instead.
  • Spam Control (SpamAssasin v3.3.x, Amavisd-new v2.8, and Clam AntiVirus v0.98) gives you some spam and antivirus control. I’ve been looking into DSPAM but haven’t implemented it yet. I figure I’ll make a part 2 to this blog once i get around to working with it and mastering it’s setup.
  • Dovecot Pigeonhole v2.0.x provides you a way of routing mail in the users inbox based on it’s content. You can key off of certain content within a received message and mark it as spam, or flag it as important because it was sent by a certain individual, etc. It basically gives you the ability to custom some of the post processing of a received message that passed through all the other checks (spam, antivirus, etc).
  • Security Considered
  • Mail Delivery Agent (MDA) (DoveCot v2.0.x):
    • Secure POP3 (POP3S) Access
    • Secure IMAP (IMAPS) Access
    • WebMail (RoundCube) Access
Message Delivery Agent Configuration

MDA Configuration

  • Mail Transport Agent (MTA) (PostFix v2.6.x): Secure SMTP (SMTPS)
Mail Transport Agent

MTA Configuration

  • Web Based Administration (PostFixAdmin v2.3.6). Life is made easier when you don’t have to do very much once everything has been set up. Run your server like an ISP would:
    • Virtual Domain Administration: Add/Remove as many as you want
    • Unlimited Virtual Aliases (or restrict if you want) per domain
    • Unlimited Mailboxes (or restrict if you want) per domain
    • Administrative Delegation: Grant enough privileges to another administrator who only has rights to his/her domain you granted them access to.
    • Away on Vacation Support (automatic replies)
Central Configuration

Central Configuration

Please note the application versions identified above as this tutorial focuses specifically on only them. One big issue I found while researching how to set up a mail server was that the tutorials I found didn’t really mention the version they were using. Hence, when I stumble across these old article(s) with new(er) software, it can make for quite a painful experience when things didn’t work.

Please also note that other tutorials will imply that you setup one feature at a time. Then test it to see if it worked correctly before moving on to the next step. This is no doubt the proper way to do things. However, I’m just going to give it all to you at once. If you stick with the versions an packages I provide and follow my instructions, it will just work for you. Debugging on your end will be a matter of tracing back to see what step you missed.

I tried to make this tutorial as cookie cutter(ish) as I could. Therefore you can literally just copy and paste what I share right to your shell prompt and the entire setup will be automated for you.

Hurdles
Just to let you know, I had many hurdles in order to pull this feat off. They were as follows:

  • postfix as shipped with CentOS and in the EPEL repository is not compiled with PostgreSQL support. I had to recompile this package as well just to enable this extra database support.
  • postfixadmin in the EPEL repository has qwirks I wasn’t happy with. I needed to fix a php error that kept coming up. I needed to adjust the database schema and permissions as well as fixing the Vacation Notification feature. I also did not want the mandatory MySQL dependency; so I removed that too.
  • perl Yes… that’s right, I had to patch perl :(. I had to recompile it due to a bug that I don’t belive was correctly addressed. In a nutshell, prior to my custom rebuild, perl-devel would haul in absolutely every development package including kernel headers and the GCC compiler. In the past the perl-devel package could be installed by itself providing us some tools spamassassin and amavisd-new depended on. You’re welcome to use the official version of perl over the one I recompiled; but be prepared to have a ton of compilation tools and source code in your production environment. This is not something I wanted at all. Interestingly enough; after the developers at RedHat said they wouldn’t tackle the issue of rolling their changes back, they seem to be entertaining this new guy’s request who’s asking for the same thing. So who knows, maybe newer versions of perl will accommodate mail servers again! :)
  • This blog itself was a massive hurdle. There are just so many configuration files and important points to account for that I found it easier to package my own rpm (nuxref-templates-mxserver) containing a series of configuration templates. The templates themselves took a while to construct in such a way that they could all be used together.

Step 1 of 7: Setup Your Environment
This is the key to my entire blog; it’s going to make all of the remaining steps just work the first time for you. All; I repeat All of the steps below (after this one) assume that you’ve set this environment up. You will need to reset up your environment at least once before running through any of the remaining steps below or they will not work.

It’s also important to mention that you will need to be root to configure this entire mail server. This applies to all of the steps identified below throughout this blog.

I started hosting some of these packages on one of my own servers to make it easier to do all the tasks. I can’t guarantee other packages will work as well; this is definitely the case for the packages I had to rebuild (identified above). If you don’t trust my packages, you can access their source files and rebuild them for yourself if you wish.

Content/Repositories are grouped as follows:

  • nuxref - Repo/Source: All custom packages rebuilt reside here (including those from past blog entries).
  • nuxref-shared - Repo/Source: This is really just a snapshot in time of packages i used from the different external repositories I don’t maintain. It may not always be up to date, but it can be thought of as a baseline (where things worked for me and with respect to this blog). If I get around to testing a new package and everything is still working fine, I’ll update the repositories here. I also resign everything with my own key to mark that I’ve tested it.

Install all of the necessary packages:

# I signed everything I host to add a level of security to
# all of the people following my blog and using the packages
# I'm hosting. You'll want to install that here:
rpm --import http://repo.lead2gold.org/pub/NUXREF-GPG-KEY

# Connect to my repository to which I've had to rebuild a few
# packages to support PostgreSQL as well as fix some bugs in
# other bugs. This step will really make your life easy and let
# us compare apples to apples with package versions. It also
# allows you to haul in a working setup right out of the box.

# Don't worry, we're defaulting it to a disabled state, for
# the paranoid.  (note that enable=0 below for both
# repositories)
cat << _EOF > /etc/yum.repos.d/nuxref.repo
[nuxref]
name=NuxRef Core - Enterprise Linux 6 - \$basearch
baseurl=http://repo.lead2gold.org/centos/6/en/\$basearch/custom
enabled=0
priority=1
gpgcheck=1

[nuxref-shared]
name=NuxRef Shared - Enterprise Linux 6 - \$basearch
baseurl=http://repo.lead2gold.org/centos/6/en/\$basearch/shared
enabled=0
priority=1
gpgcheck=1
_EOF

# You should use yum priorities (if you're not already) to
# eliminate version problems later, but if you intend on 
# just pointing to my repository and not the others, then you'll
# be fine.
yum install -y yum-plugin-priorities

################################################################
# Install our required products
################################################################
yum install -y \
       --enablerepo=nuxref \
       --enablerepo=nuxref-shared \
           postgresql-server postgresql \
           php-pgsql php-imap php-mcrypt php-mbstring  \
           dovecot dovecot-pgsql dovecot-pigeonhole \
           clamav amavisd-new spamassassin \
           postfix postfix-perl-scripts \
           roundcubemail postfixadmin \
           nuxref-templates-mxserver

# Choose between NginX or Apache
## NginX Option (a) - This one is my preferred choice:
yum install -y \
       --enablerepo=nuxref \
       --enablerepo=nuxref-shared \
           nginx php-fpm

## Apache Option (b):
yum install -y \
       --enablerepo=nuxref \
       --enablerepo=nuxref-shared \
            httpd php

# Make sure your build of Postfix supports PostgreSQL
# execute the following to see: 
postconf -c /etc/postfix -m | \
   egrep -q '^pgsql$' && echo "Supported!" || \
      echo "Not Supported!"

# If it's not supported then you ran into the first of many
# problems I had. It also means you didn't haul in my version
# from the previous command.  This can happen if you already had
# postfix installed on your machine and it's of a newer version
# then what I'm hosting. Your options at this point is to either
# uninstall your copy of postfix and install mine or recompile 
# your version of postfix (but this time with PostgreSQL support).

# Setup Default Timezone for PHP. For a list of supported
# timezones you can visit here: http://ca1.php.net/timezones
TIMEZONE="America/Montreal"
sed -i -e "s|^[ \t]*;*\(date\.timezone[ \t]*=\).*|\1 $TIMEZONE|g" /etc/php.ini

################################################################
# Setup PostgreSQL (v8.4)
################################################################
# The commands below should all work fine on a PostgreSQL v9.x
# database too; but your mileage may vary as I've not personally
# tested it yet.

# Only init the database if you haven't already. This command
# could otherwise reset things and you'll loose everything.
# If your database is already setup and running, then you can
# skip this line
service postgresql initdb

# Now that the database is initialized, configure it to trust
# connections from 'this' server (localhost)
sed -i -e 's/^[ \t]*\(local\|host\)\([ \t]\+.*\)/#\1\2/g' \
    /var/lib/pgsql/data/pg_hba.conf
cat << _EOF >> /var/lib/pgsql/data/pg_hba.conf
# Configure all local database access with trust permissions
local   all         all                               trust
host    all         all         127.0.0.1/32          trust
host    all         all         ::1/128               trust
_EOF

# Make sure PostgreSQL is configured to start up each time
# you start up your system
chkconfig --levels 345 postgresql on

# Start the database now too because we're going to need it
# very shortly in this tutorial
service postgresql start

To simplify your life, I’ve made the configuration of all the steps below reference a few global variables. The ones identified below are the only ones you’ll probably want to change. May I suggest you paste the below information in your favourite text editor (vi, emacs, etc) and adjust the variables to how you want them making it easier to paste them back to your terminal screen.

# Identify the domain name of your server here
DOMAIN=nuxref.com
# Setup what you want to be your Administrative email account
# Note: This does 'NOT' have to be of the same domain even though
#       thats how I set it up to be. Don't worry if the email
#       address doesn't exist, because when you're all done
#       following this blog, you'll be able to create it!
ADMIN=hostmaster@$DOMAIN

# The following is only used for our SSL Key Generation
COUNTRY_CODE="7K"
PROV_STATE="Westerlands"
CITY="Lannisport"
SITE_NAME="NuxRef"

Now for the rest of the global configuration; There really should be no reason to change any of these values (but feel free to). It’s important that you paste the above information (tailored to your liking’s) as well as the below information below to your command line interface (CLI) of the server you wish to set up.

# PostgreSQL Database
PGHOST=localhost
PGPORT=5432
PGNAME=system_mail
# This is the Read Only access user (or very limited access)
PGROUSER=mailreader
PGROPASS=mailreader
# This is for administration
PGRWUSER=mailwriter
PGRWPASS=mailwriter

# VHost Mail Directory
VHOST_HOME=/var/mail/vhosts
VHOST_UID=5000
# No real reason to make them differ
# but define tis variable anyway for
# below configuration to work
VHOST_GID=$VHOST_UID

# RoundCube Configuration
MXHOST=$PGHOST
PGRQNAME=system_roundcube
PGRQUSER=roundcube_user
PGRQPASS=roundcube_pass

# This is where our templates get installed to make your life
# incredibly easy and the setup to be painless. These files are
# installed from the nuxref-templates-mxserver RPM package you
# installed above. If you do not have this RPM package then you
# must install it or this blog simply won't work for you.
# > yum install --enablerepo=nuxref nuxref-templates-mxserver
NUXREF_TEMPLATES=/usr/share/nuxref

Once all of this has been set you can proceed to do any of the below steps! Keep in mind that if you decide to change any of the variables above, you may need to redo every single step identified below.
Step 2 of 7: System Preparation
First off, Make sure you’ve set up your environment correctly (defined in Step 1 above) or you will have problems with the outcome of this step!
General:

# Create vmail user; this will be a secure user no one else has
# permissions to that we can lock and keep our mail private
# from any prying eyes of people who have or gain access to our
# server.
useradd -u $VHOST_UID -d /var/mail/vhosts -M -s /sbin/nologin vmail
mkdir -p  /var/mail/vhosts/
chown vmail.vmail /var/mail/vhosts/
chmod 700 /var/mail/vhosts/

# Create a clam user we can preform our anti-virus scans as
usermod -G amavis clam

# Fix ClamD Sock Reference
sed -i -e 's|/var/spool/amavisd/clamd.sock|/var/run/clamav/clamd.sock|g' /etc/amavisd/amavisd.conf

# Fix Amavis Directory Permission
chmod 1770 /var/spool/amavisd/tmp/
# Amavis Domain Configuration
sed -i -e '/NuxRef BulletProofing/d' \
       -e "s/\(# \$myhostname.*\)/\1\n\$myhostname = 'mail.$DOMAIN'; # NuxRef BulletProofing/g" /etc/amavisd/amavisd.conf
sed -i -e "s/^\(\$mydomain[ \t]*=[ \t]*\).*$/\1'$DOMAIN';/g" /etc/amavisd/amavisd.conf

# Generate SSL Keys (if you don't have any already) that we
# will secure all our inbound and outbound mail as.
openssl req -nodes -new -x509 -days 730 -sha1 -newkey rsa:2048 \
   -keyout /etc/pki/tls/private/$DOMAIN.key \
   -out /etc/pki/tls/certs/$DOMAIN.crt \
   -subj "/C=$COUNTRY_CODE/ST=$PROV_STATE/L=$CITY/O=$SITE_NAME/OU=IT/CN=$DOMAIN"

# Permissions; protect our Private Key
chmod 400 /etc/pki/tls/private/$DOMAIN.key

# Permissions; protect our Public Key
chmod 444 /etc/pki/tls/certs/$DOMAIN.crt

Create PostgreSQL Mail Database:

# Optionally Eliminate Reset Database.
/bin/su -c "/usr/bin/dropdb -h $PGHOST -p $PGPORT $PGNAME 2>&1" postgres &>/dev/null
/bin/su -c "/usr/bin/dropuser -h $PGHOST -p $PGPORT $PGRWUSER 2>&1" postgres &>/dev/null
/bin/su -c "/usr/bin/dropuser -h $PGHOST -p $PGPORT $PGROUSER 2>&1" postgres &>/dev/null

# Create Read/Write User (our Administrator)
echo "Enter the role password of '$PGRWPASS' when prompted"
/bin/su -c "/usr/bin/createuser -h $PGHOST -p $PGPORT -S -D -R $PGRWUSER -P 2>&1" postgres

# Create Read-Only User
echo "Enter the role password of '$PGROPASS' when prompted"
/bin/su -c "/usr/bin/createuser -h $PGHOST -p $PGPORT -S -D -R $PGROUSER -P 2>&1" postgres

# Create our Database and assign it our Administrator as it's owner
/bin/su -c "/usr/bin/createdb -h $PGHOST -p $PGPORT -O $PGRWUSER $PGNAME 2>&1" postgres 2>&1

# Secure and protect a temporary file to work with
touch /tmp/pgsql.postfix.schema.sql
chmod 640 /tmp/pgsql.postfix.schema.sql
chown root.postgres /tmp/pgsql.postfix.schema.sql

# the below seems big; but will work fine if you just copy and
# it as is right to your terminal: This will prepare the SQL
# statement needed to build your mail server's database backend
sed -e '/^--?/d' \
    -e "s/%PGROUSER%/$PGROUSER/g" \
    -e "s/%PGRWUSER%/$PGRWUSER/g" \
    -e "s/%DOMAIN%/$DOMAIN/g" \
        $NUXREF_TEMPLATES/pgsql.postfix.template.schema.sql > \
          /tmp/pgsql.postfix.schema.sql

# load DB
/bin/su -c "/usr/bin/psql -h $PGHOST -p $PGPORT -f /tmp/pgsql.postfix.schema.sql $PGNAME 2>&1" postgres 2>&1
# cleanup
/bin/rm -f /tmp/pgsql.postfix.schema.sql

Step 3 of 7: Setup our Mail Transfer Agent (MTA): Postfix
First off, Make sure you’ve set up your environment correctly (defined in Step 1 above) or you will have problems with the outcome of this step!

################################################################
# Configure Postfix (MTA)
################################################################
# Create backup of configuration files
[ ! -f /etc/postfix/main.cf.orig ] && \
   cp /etc/postfix/main.cf /etc/postfix/main.cf.orig
[ ! -f /etc/postfix/master.cf.orig ] && \
   cp /etc/postfix/master.cf /etc/postfix/master.cf.orig

# Directory to store our configuration in
[ ! -d /etc/postfix/pgsql ] && \
    mkdir -p /etc/postfix/pgsql

# Secure this new directory since it will contain passwords
# information
chmod 750 /etc/postfix/pgsql
chown root.postfix /etc/postfix/pgsql

# Now using our templates, build our SQL files:
for FILE in relay_domains.cf \
            transport_maps.cf \
            virtual_alias_maps.cf \
            virtual_domains_maps.cf \
            virtual_mailbox_limit_maps.cf \
            virtual_mailbox_maps.cf
do
    sed -e "/^#?/d" \
        -e "s/%PGROUSER%/$PGROUSER/g" \
        -e "s/%PGROPASS%/$PGROPASS/g" \
        -e "s/%PGHOST%/$PGHOST/g" \
        -e "s/%PGNAME%/$PGNAME/g" \
            $NUXREF_TEMPLATES/pgsql.postfix.template.$FILE > \
                /etc/postfix/pgsql/$FILE
done

# main.cf
sed -e "/^#?/d" \
    -e "s/%DOMAIN%/$DOMAIN/g" \
    -e "s|%VHOST_HOME%|$VHOST_HOME|g" \
    -e "s/%VHOST_UID%/$VHOST_UID/g" \
    -e "s/%VHOST_GID%/$VHOST_GID/g" \
        $NUXREF_TEMPLATES/pgsql.postfix.template.main.cf > \
            /etc/postfix/main.cf

# master.cf
cat pgsql.postfix.template.master.cf > /etc/postfix/master.cf

# Run the newaliases command to generate /etc/aliases.db
newaliases

# Vacation Support
echo "autoreply.$DOMAIN vacation:" > /etc/postfix/transport
postmap /etc/postfix/transport

# Update to latest Spam Assassin Rules/Filters
sa-update
# Update to latest Antivirus
freshclam

# Setup Auto-Cron Entries (so future antivirus updates
# can just be automatic).
sed -i -e '|/etc/cron.hourly|d' /etc/crontab
sed -i -e '|/etc/cron.daily|d' /etc/crontab
sed -i -e '|/etc/cron.weekly|d' /etc/crontab
sed -i -e '|/etc/cron.monthly|d' /etc/crontab
cat << _EOF >> /etc/crontab
  01 * * * * root run-parts /etc/cron.hourly 
  02 4 * * * root run-parts /etc/cron.daily 
  22 4 * * 0 root run-parts /etc/cron.weekly 
  42 4 1 * * root run-parts /etc/cron.monthly
_EOF

# Enable our Services On Reboot
chkconfig --levels 345 spamassassin on
chkconfig --levels 345 clamd on
chkconfig --levels 345 clamd.amavisd on
chkconfig --levels 345 amavisd on
chkconfig --levels 345 postfix on

# Start all of our other services if they aren't already
service spamassassin start
service clamd start
service amavisd start
service clamd.amavisd start

# Restart our postfix service to on our new configuration
service postfix restart

Step 4 of 7: Setup our Mail Delivery Agent (MDA): Dovecot
First off, Make sure you’ve set up your environment correctly (defined in Step 1 above) or you will have problems with the outcome of this step!

################################################################
# Configure Dovecot (MDA)
################################################################
# Create backup of configuration files
[ ! -f /etc/dovecot/dovecot.conf.orig ] && \
   cp /etc/dovecot/dovecot.conf /etc/dovecot/dovecot.conf.orig

# dovcot.conf
sed -e "/^#?/d" \
    -e "s/%DOMAIN%/$DOMAIN/g" \
    -e "s|%VHOST_HOME%|$VHOST_HOME|g" \
    -e "s/%VHOST_UID%/$VHOST_UID/g" \
    -e "s/%VHOST_GID%/$VHOST_GID/g" \
        $NUXREF_TEMPLATES/pgsql.dovecot.template.dovecot.conf > \
            /etc/dovecot/dovecot.conf

# Create our Sieve Directories
[ ! -d /var/lib/sieve/users ] && \
   mkdir -p /var/lib/sieve/users
[ ! -d /var/lib/sieve/before.d ] && \
   mkdir -p /var/lib/sieve/before.d
[ ! -d /var/lib/sieve/after.d ] && \
   mkdir -p /var/lib/sieve/after.d
chown -R vmail.vmail /var/lib/sieve
chmod 750 /var/lib/sieve

# Dovecot PostgreSQL Configuration
for FILE in dovecot-sql.conf \
            dovecot-dict-user-quota.conf \
            dovecot-dict-domain-quota.conf
do
   sed -e "/^#?/d" \
       -e "s|%VHOST_HOME%|$VHOST_HOME|g" \
       -e "s/%VHOST_UID%/$VHOST_UID/g" \
       -e "s/%VHOST_GID%/$VHOST_GID/g" \
       -e "s/%PGROUSER%/$PGROUSER/g" \
       -e "s/%PGROPASS%/$PGROPASS/g" \
       -e "s/%PGHOST%/$PGHOST/g" \
       -e "s/%PGNAME%/$PGNAME/g" \
           $NUXREF_TEMPLATES/pgsql.dovecot.template.$FILE > \
               /etc/dovecot/$FILE
   # permissions
   chmod 640 /etc/dovecot/$FILE
   chown root.dovecot /etc/dovecot/$FILE
done

# Warning Message when mailbox is almost full
sed -e "/^#?/d" \
    -e "s/%DOMAIN%/$DOMAIN/g" \
        $NUXREF_TEMPLATES/pgsql.dovecot.template.mail-warning.sh > \
            /usr/libexec/dovecot/mail-warning.sh

# Make Script Executable
chmod 755 /usr/libexec/dovecot/mail-warning.sh

# Ensure Dovecot starts after each system reboot:
chkconfig --levels 345 dovecot on

# Start Dovecot (otherwise restart it if it's already running)
service dovecot status && service dovecot restart || service dovecot start

Step 5 of 7: Setup Postfix Admin
First off, Make sure you’ve set up your environment correctly (defined in Step 1 above) or you will have problems with the outcome of this step!

################################################################
# Configure PostfixAdmin
################################################################
sed -e "/^\/\/?/d" \
    -e "s/%DOMAIN%/$DOMAIN/g" \
    -e "s/%ADMIN%/$ADMIN/g" \
    -e "s/%PGHOST%/$PGHOST/g" \
    -e "s/%PGNAME%/$PGNAME/g" \
    -e "s/%PGRWUSER%/$PGRWUSER/g" \
    -e "s/%PGRWPASS%/$PGRWPASS/g" \
        $NUXREF_TEMPLATES/pgsql.postfixadmin.template.config.local.php > \
            /etc/postfixadmin/config.local.php

# Protect file since it contains passwords
chmod 640 /etc/postfixadmin/config.local.php
chown root.apache /etc/postfixadmin/config.local.php

# Vacation Auto-respond Support
sed -e "/^#?/d" \
    -e "s/%DOMAIN%/$DOMAIN/g" \
    -e "s/%PGROUSER%/$PGROUSER/g" \
    -e "s/%PGROPASS%/$PGROPASS/g" \
    -e "s/%PGHOST%/$PGHOST/g" \
    -e "s/%PGNAME%/$PGNAME/g" \
        $NUXREF_TEMPLATES/pgsql.postfixadmin.template.vacation.conf > \
            /etc/postfixadmin/vacation.conf

# Protect file since it contains passwords
chmod 640 /etc/postfixadmin/vacation.conf
chown root.vacation /etc/postfixadmin/vacation.conf

# Log Rotation
cat << _EOF > /etc/logrotate.d/postfix-vacation
/var/log/vacation.log {
        missingok
        notifempty
        create 644 vacation root
}
_EOF

Now you can setup NginX to host your administration; in the below example, I set up https://postfixadmin.<your.domain>/ to go to the postfixadmin page.

# Create dummy favicon.ico for now (silences some log entries)
touch /usr/share/postfixadmin/favicon.ico

# PostfixAdmin NginX Configuration
sed -e "/^#?/d" \
    -e "s/%DOMAIN%/$DOMAIN/g" \
        $NUXREF_TEMPLATES/nginx.postfixadmin.template.conf > \
            /etc/nginx/conf.d/postfixadmin.conf

# You may have to bump php-fpm to be safe (if it's not already running)
service php-fpm status 2>/dev/null && service php-fpm restart || service php-fpm start
# make sure it starts on every reboot too:
chkconfig php-fpm --level 345 on
# Restart NginX if it's not already
service nginx status 2>/dev/null && service nginx restart || service nginx start
chkconfig nginx --level 345 on

Once you’re complete that, you’re now ready to access the administrator interface and set up a new account. Simply visit https://postfixadmin.<your.domain>/setup.php. The templates I provided will set the system password to admin. You’ll need to always enter this value prior to creating an account below.
PostfixAdmin Setup
Once you’re done creating an account, just change the setup.php script to read-only as a security precaution. You can preform every other action you’ll ever need using the account you already created.

################################################################
# Disable Future System Administrator Creation
################################################################
chmod 600 /usr/share/postfixadmin/setup.php

# later on, you can re-enable the <i>setup.php</i> file to create a new
# account in the distant future by just typing:
#
# chmod 644 /usr/share/postfixadmin/setup.php
#

You can simply access https://postfixadmin.<your.domain>/ now (without the setup.php) and login using the new administrator you created.

Step 6 of 7: Setup Roundcube
First off, Make sure you’ve set up your environment correctly (defined in Step 1 above) or you will have problems with the outcome of this step!

################################################################
# Configure RoundCube Mailer
################################################################
# RoundCube NginX Configuration
sed -e "/^#?/d" \
    -e "s/%DOMAIN%/$DOMAIN/g" \
        $NUXREF_TEMPLATES/nginx.roundcubemail.template.conf > \
            /etc/nginx/conf.d/roundcubemail.conf

# Optionally Eliminate Reset RoundCube Database
/bin/su -c "/usr/bin/dropdb -h  $PGHOST -p $PGPORT $PGRQNAME 2>&1" postgres &>/dev/null
/bin/su -c "/usr/bin/dropuser -h $PGHOST -p $PGPORT $PGRQUSER 2>&1" postgres &>/dev/null

# Create RoundCube Admistrator User
echo "Enter the role password of '$PGRQPASS' when prompted"
/bin/su -c "/usr/bin/createuser -h $PGHOST -p $PGPORT -S -D -R $PGRQUSER -P 2>&1" postgres 2>&1

# Create our Database and assign it our Administrator as it's owner
/bin/su -c "/usr/bin/createdb -h $PGHOST -p $PGPORT -O $PGRQUSER $PGRQNAME 2>&1" postgres 2>&1
/usr/bin/psql -h $PGHOST -p $PGPORT -U $PGRQUSER $PGRQNAME -f /usr/share/doc/roundcubemail-0.9.5/SQL/postgres.initial.sql

# Configure Roundmail
sed -i -e "s|\(^[ \t]*\$rcmail_config\[[']db_dsnw['\"]\][ \t]*=\).*$|\1 'pgsql://$PGRQUSER:$PGRQPASS@$PGHOST/$PGRQNAME';|g" /etc/roundcubemail/db.inc.php
sed -i -e "s|\(^[ \t]*\$rcmail_config\[[']default_host['\"]\][ \t]*=\).*$|\1 'ssl://$MXHOST:993';|g" /etc/roundcubemail/main.inc.php
sed -i -e "s|\(^[ \t]*\$rcmail_config\[[']smtp_server['\"]\][ \t]*=\).*$|\1 'tls://$MXHOST';|g" /etc/roundcubemail/main.inc.php
sed -i -e "s|\(^[ \t]*\$rcmail_config\[[']smtp_user['\"]\][ \t]*=\).*$|\1 '%u';|g" /etc/roundcubemail/main.inc.php
sed -i -e "s|\(^[ \t]*\$rcmail_config\[[']smtp_pass['\"]\][ \t]*=\).*$|\1 '%p';|g" /etc/roundcubemail/main.inc.php

# Some extra Roundmail options i preferred:
sed -i -e "s|\(^[ \t]*\$rcmail_config\[[']identities_level['\"]\][ \t]*=\).*$|\1 3;|g" /etc/roundcubemail/main.inc.php
sed -i -e "s|\(^[ \t]*\$rcmail_config\[[']quota_zero_as_unlimited['\"]\][ \t]*=\).*$|\1 true;|g" /etc/roundcubemail/main.inc.php
sed -i -e "s|\(^[ \t]*\$rcmail_config\[[']htmleditor['\"]\][ \t]*=\).*$|\1 1;|g" /etc/roundcubemail/main.inc.php
sed -i -e "s|\(^[ \t]*\$rcmail_config\[[']preview_pane['\"]\][ \t]*=\).*$|\1 true;|g" /etc/roundcubemail/main.inc.php

# You may have to bump php-fpm to be safe (if it's not already running)
service php-fpm status 2>/dev/null && service php-fpm restart || service php-fpm start
# make sure it starts on every reboot too:
chkconfig php-fpm --level 345 on
# Restart NginX if it's not already
service nginx status 2>/dev/null && service nginx restart || service nginx start
chkconfig nginx --level 345 on

You can simply access https://roundcube.<your.domain>/ now and access any mailboxes you configured. Just remember to tell your users that they must specify their full email address as their username.

Each mailbox you create using PostfixAdmin you’ll be able to access with your Roundcube webpage.

Step 7 of 7: Security
If you aren’t familiar with Fail2Ban; now would be an excellent time to learn about it. I wrote a blog about securing your CentOS system a while back and encourage you to read it. At the very least, read the section on Fail2Ban. The below explains how you can protect yourself from brute force.

# Monitor for multiple failed SASL Logins into postfix
cat << _EOF > /etc/fail2ban/filter.d/postfix-sasl.conf
[Definition]
failregex = (?i): warning: [-._\w]+\[<HOST>\]: SASL (?:LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed: [A-Za-z0-9+/ ]*)?\$
ignoreregex =
_EOF
# Monitor for multiple failed postfixadmin logins
cat << _EOF > /etc/fail2ban/filter.d/postfixadmin-auth.conf
[Definition]
failregex = ^<HOST> -.*POST.*login.php HTTP[^ \t]+ 500\$
ignoreregex =
_EOF

# Now in /etc/fail2ban/jail.conf you will want the following:
cat << _EOF >> /etc/fail2ban/jail.conf
[dovecot-iptables]
enabled  = true
filter   = dovecot
backend  = polling
action   = iptables[name=dovecot, port=110,143,993,995, protocol=tcp]
           sendmail-whois[name=dovecot, dest=root, sender=fail2ban@$DOMAIN]
logpath  = /var/log/mail.log

[sasl-iptables]
enabled  = true
filter   = postfix-sasl
backend  = polling
action   = iptables[name=sasl, port=smtp, protocol=tcp]
           sendmail-whois[name=sasl, dest=root, sender=fail2ban@$DOMAIN]
logpath  = /var/log/mail.log

[roundcube-iptables]
enabled  = true
filter   = roundcube-auth
backend  = polling
action   = iptables[name=RoundCube, port="http,https"]
           sendmail-whois[name=RoundCube, dest=root, sender=fail2ban@$DOMAIN]
logpath  = /var/log/roundcubemail/errors

[postfixadmin-iptables]
enabled  = true
filter   = postfixadmin-auth
backend  = polling
action   = iptables[name=PostfixAdmin, port="http,https"]
           sendmail-whois[name=PostfixAdmin, dest=root, sender=fail2ban@$DOMAIN]
logpath  = /var/log/nginx/postfixadmin.access.log
_EOF

You will want to additionally add the following to your iptables /etc/sysconfig/iptables:

#---------------------------------------------------------------
# Web Traffic (for PostfixAdmin and RoundCube)
#---------------------------------------------------------------
# Allow non-encrypted port so we can redirect these users to the
# encrypted version.  It's just a nicer effect to support
# redirection
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT

#---------------------------------------------------------------
# SMTP /Message Transfer Agent Communication
#---------------------------------------------------------------
-A INPUT -m state --state NEW -m tcp -p tcp --dport 465 -j ACCEPT

#---------------------------------------------------------------
# IMAP 
#---------------------------------------------------------------
-A INPUT -m state --state NEW -m tcp -p tcp --dport 993 -j ACCEPT

#---------------------------------------------------------------
# POP3
#---------------------------------------------------------------
-A INPUT -m state --state NEW -m tcp -p tcp --dport 995 -j ACCEPT

Useful Commands When Running Your Mail Server

  • doveconf -a: Display Dovecot configuration
  • postconf -n: Display Postfix configuration
  • postqueue -p: Display mail queue information

Sometimes when first playing with quotas, you may or may not want to recalculate them against a user. This can be done as follows:

#  Recalculate a specific users quota
doveadm quota recalc -u foobar@your.domain.com

# Or you can do this and recalculate ALL user quotas
doveadm quota recalc -A

# You will also want to run the following command if you decide
# to recalculate someone (or all) in your database:
# UPDATE domain_quota SET bytes=sq.sb, messages=sq.sm 
#   FROM (SELECT 'your.domain.com',
#           sum(bytes) as sb, sum(messages) as sm from quota2 WHERE
#            username like '%@your.domain.com') AS sq 
#   WHERE domain = 'your.domain.com';

Note: If you delete a mailbox for a specified domain, remember to manually remove: /var/mail/vhosts/domain/user

So… That’s it? Now I’m done?
Yes and No… My blog pretty much hands over a working mail server with little to no extra configuration needed on your part. But to properly accept mail from other people around the world, you will need:

  1. This mail server (ideally) must be accessible to the internet via a static IP address. This means that if you’re hosting this at home, the IP address your ISP provides you may not work (Dynamic IP vs Static IP). That said; a lot of ISPs can offer you a static IP if you don’t already have one for little (to no) extra cost.
  2. Your own domain name (if you don’t have an official one already) because you can’t have an email@your.domain.com if your.domain.com isn’t publicly recognized.
  3. A Mail Exchange (MX) Record that points to your new mail server by via it’s accessible IP on the internet. This is the only way the outside world will be able to send mail back to you; this step is not required if the only thing you’re doing is sending email out.

    Most Domain Registars allow you to set your own MX record (GoDaddy.com, NameCheap.com) which can simply be entered right from their webpage with little to no effort. If you’re paying someone else to host your websites for the domain you own, then most likely they have the MX record already pointing to them. You may need to open a support ticket (or call them) and tell them you want the MX record changed to point back to your own server instead or forward it.

Please keep in mind that there are risks involved with running your own mail server. You can get yourself temporarily (or permantenly) blacklisted if you’re not careful. Once you’re blacklisted, you’ll have a very hard time getting other mail servers on the internet to accept your emails for delivery. Blacklisting occurs when mail servers (which you will now be interacting with) detect abuse. In most cases, the mail server administrator (this is YOU) won’t even know you’re abusing other peoples servers. The abuse will start under your nose from emails that originated from your system by those you given mailboxes too. In fact, once your domain is blacklisted; it can be a pain in the @$$ to get de-listed later. A Blacklisted domain’s emails usually never reaches their intended recipient. Instead they are immediately flagged as spam and sent to the deleted items (by the remote server). The configuration I provided considers most of the cases, but you still need to consider:

  • Don’t create mailboxes for people that you know intend to use it for derogatory purposes or for the intentions of spamming others. Hence; don’t allow users to send out hundreds of thousands of emails a day to a massive distribution on a regular bases even if it’s meaningful mail. Consider that the same stuff that you don’t like in your inbox is the same stuff that nobody else likes in theirs either. :)
  • Don’t allow your mail server to relay mail from untrusted sources. Hence; make sure you only allow users you create accounts for to send mail from your server.
  • Throttle all outbound mail delivery to each of their relay locations. With respect to the first point, even if you have to send massive amounts of mail from your system on a regular basis, do it in small batches. This way you won’t overwhelm the remote servers accepting your mail you want delivered.

If you followed my blog and are using the settings I put in place, then you’re already configured for the last 2 options above. The first option is governed by your own decisions.

No system is bulletproof; disaster can always strike when you’re least expecting it. To cover yourself, always consider backups of the following:

  • Your PostgreSQL Database: This is where all of your mail configuration is for both your MTA and MDA. You definitely do not want to lose this. May I suggest you reference my other blog entry here where I wrote a really simple backup/restore tool for a PostgreSQL database.
  • /etc/postfixadmin/*: Your Postfix Admin flat file configuration allowing you to centrally manage everything via a webpage.
  • /etc/postfix/*: Your Postfix flat file configuration which defines the core of your MTA. It’s configuration allows you to centrally manage everything else through the Postfix Administration website.
  • /etc/roundcube/*: Your Roundcube flat file configuration which allowing users to check their mail via a webpage you host.
  • /etc/dovecot/*: Your Dovecot flat file configuration which defines the core of your MDA. It’s configuration allows you to centrally manage everything through the Postfix Administration website.
  • /var/mail/vhosts/*: All of your user’s mailboxes are defined here. This is a vast storage of unread and read mail that resides on your server.

What about Apache?
Apache is a perfectly fine alternative solution as well! I simply chose NginX because it is much more lightweight approach. In fact, PostfixAdmin and RoundCube mail already come with Apache configuration out of the box located in /etc/httpd/conf.d/. Thus, if you simply start up your Apache instance (service httpd start), you will be hosting its services right away. Please keep in mind that the default (Apache) configuration does not come with all the SSL and added security I provided with the NginX templates. Perhaps later on, I will update the template rpm to include an Apache secure setup as well.

Credit
This blog took me a very (,very) long time to put together and test! The repository hosting alone now accomodates all my blog entries up to this date. If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

Repository
This blog required me to set up my own repository of which I was thinking that some people might want me to continue to maintain. Such as fetching and applying the latest security updates after testing them first for the continued promise of stability. Personally, I believe I may be setting up a new can of worms for myself by offering this service because bandwidth costs money and updates cost time. But I’ll share it publicly and see how things go anyway.

If you’d like me to continue to monitor and apply updates as well as hosting the repository for long terms, please consider donating or offering a mirror server to help me out! This would would be greatly appreciated!

Sources
This blog could not have been made possible without the tons of resources that I used. Some of these resources including other peoples blogs too which I picked and chose approaches they took.

Software Resources

PostgreSQL Backup & Restore

Backing Up and Restoring Your PostgreSQL Database


Introduction
As a Developer or even a System Administrator, experiencing database corruption is a scary thing. Usually when something you’re responsible fails… it fails hard. It’s important to always make frequent backups of your database(s) your in charge of. The internet is filled with solutions and one liners you can consider. But there is a lot of contradiction everywhere too as each site tells you how it should be done. Personally, I’m not going to tell you the way I do it is the correct way or not. But I will tell you it works, and it’s saved me in numerous occasions.

I’m lazy, and when I look for solutions online, I always hope someone is just going to spell it right out for me. It’s so much easier when someone else does all the work; but usually that’s wasn’t the case for me. Stack Overflow is fantastic for getting the quick 1 liners you need to build your solution, but I prefer automation. I don’t want to remember all of the different options one tool has from another. I want simplicity because when something fails, I don’t want to learn the complicated steps to do a database restoration.

As a strong supporter of PostgreSQL, I want to provide the solution I use to hopefully speed along someone else’s research in the future. I can’t stress enough also that if you aren’t taking regular backups of databases you are responsible for, then you really should reconsider and at use the scripts defined below.

Database Backup
Assuming you you are the global system administrator responsible for a PostgreSQL database and have root privileges, here is a simple backup script you can use:

#!/bin/bash
# Author: Chris Caron <lead2gold at gmail.com>
# Name: pgnux_backup
# Description: Preform a backup of one ore more PostgreSQL Database(s)
# Returns: 0 on success, a non-zero value if a failure occurred
# Syntax: pgnux_backup [db1 [db2 [db3 [...]]]
#
# Note: If no database is specified to be backed up, then all of
#       databases hosted by the postgresql instance are used.

##################################
# Define some simple variables
##################################
PGENGINE=/usr/bin
PGPORT=5432
PGHOST=localhost
DUMPDIR=/var/pgbackup

##################################
# System Tweaks
##################################
# if COMPRESSION is enabled, then all database backups are
# gzipped to save space.  The trade off is more cpu power will
# be used to generate your backups and they will take longer
# to create.  But you can save yourself significant amounts of
# disk space.  Set this value to 0 if you do not wish to use
# compression. COMPRESSION can be set to a value between 0 and
# 9 where a setting of 0 is used to disable it.
# Please keep in mind also, that if you choose to use
# compression, the PostgreSQL restores will be a bit slower since
# the data needs to be first uncompressed before it can be
# reloaded.
COMPRESSION=6

# For SELinux we need to use 'runuser' not 'su'
if [ -x /sbin/runuser ]; then
    SU=/sbin/runuser
else
    SU=/bin/su
fi

# List of databases you wish to back up by default.  If you
# leave this blank, then the system will automatically detect
# all of the databases hosted by the PostgreSQL instance.
#
# You can use Space, Tab or some form of white space delimiter
# between each database you specify if you need to specify more
# than one.
# If nothing is specified on the command line (when this script
# is called, then this is the default list that will be used in
# it's place.
DBLIST=""

# If anything was specified on the command line; we over-ride it here
[ ! -z "$@" ] && DBLIST="$@"

# Capture the current time to make it easy to locate our backups later
SNAPTIME=$(date +'%Y%m%d%H%M%S')

# Build DBLIST if it isn't already defined:
[ -z "$DBLIST" ] && DBLIST=$($SU -l postgres -c "$PGENGINE/psql \
                          -p $PGPORT -h $PGHOST -lt | \
                   awk 'NF >=5 { print \$1}' | \
                   egrep -v '^(template[0|1])\$'")

# Create our backup directory if it doesn't already exist
[ ! -d $DUMPDIR ] && mkdir -p $DUMPDIR

# Ensure our directory is protected from prying eyes
chmod 770 $DUMPDIR
chown root.postgres $DUMPDIR

# Backup our Database Globals (exit on failure)
$SU -l postgres -c "$PGENGINE/pg_dumpall --globals-only \
   -p $PGPORT -h $PGHOST | \
   gzip -c > $DUMPDIR/${SNAPTIME}-globals.sql.gz"
if [ $? -ne 0 ]; then
   echo -n "$(date +'%Y-%m-%d %H:%M:%S') - $$ - ERROR - "
   echo "Failed to dump system globals; backup aborted."
   exit 2
fi

# Protect our backup and ensure the user postgres user can manage
# this file if need be.
chmod ug+rw "$DUMPDIR/${SNAPTIME}-globals.sql.gz" &>/dev/null
chown postgres.root $DUMPDIR/${SNAPTIME}-globals.sql.gz &>/dev/null

FAIL_COUNT=0
# Iterate over our database list and perform our backup:
for DBNAME in $DBLIST; do
   echo -n "$(date +'%Y-%m-%d %H:%M:%S') - $$ - INFO - "
   echo "Dumping database '$DBNAME' ..."
   PGBKUP=
   PGFLAGS="-p $PGPORT -h $PGHOST --create --oids --format=t --verbose"
   if [ $COMPRESSION -gt 0 ]; then
      # Use Compression
      PGBKUP=$SNAPTIME-$DBNAME.db.gz
      $SU -l postgres -c "$PGENGINE/pg_dump $PGFLAGS $DBNAME | \
                 gzip -cq$COMPRESSSION > $DUMPDIR/$PGBKUP"
      RETVAL=$?
   else
      # Do not compres (flag is not set to 1)
      PGBKUP=$SNAPTIME-$DBNAME.db
      $SU -l postgres -c "$PGENGINE/pg_dump $PGFLAGS \
                 --file=$DUMPDIR/$PGBKUP $DBNAME"
      RETVAL=$?
   fi
   if [ $RETVAL -eq 0 ]; then
      echo -n "$(date +'%Y-%m-%d %H:%M:%S') - $$ - INFO - "
      echo "Backup Complete: $PGBKUP"
      chmod ug+rw $DUMPDIR/$PGBKUP &>/dev/null
      chown postgres.root $DUMPDIR/$PGBKUP &>/dev/null
   else
      [ -f $DUMPDIR/$PGBKUP ] && \
          rm -f $DUMPDIR/$PGBKUP
      echo -n "$(date +'%Y-%m-%d %H:%M:%S') - $$ - ERROR - "
      echo "Backup Failed: $PGBKUP"
      let FAIL_COUNT+=1
   fi
done

if [ $FAIL_COUNT -ne 0 ]; then
   exit 1
fi
exit 0

Database Restore
Backups are one thing, but the ability to restore is another. This script pairs with the above and will allow you to restore different points in time.

#!/bin/bash
# Author: Chris Caron <lead2gold at gmail.com>
# Name: pgnux_restore
# Description: Preform a restore of one ore more PostgreSQL Database(s)
# Returns: 0 on success, a non-zero value if a failure occurred
# Syntax: pgnux_restore <database_name> [snapshot_time]
#
# Note: This script will not work if people are connected to the
#       databases in question you are trying to restore.
 
##################################
# Define some simple variables
##################################
PGENGINE=/usr/bin
PGPORT=5432
PGHOST=localhost
DUMPDIR=/var/pgbackup
 
# How many backups in the past to list when no restore
# has taken place
HISTORY=10
 
# Specify a keyword that when in place of the <database_name>
# it triggers all databases for specified snapshot time to be
# restored including database system globals. Set this to
# 'nothing' if you want the restore to do a global restore every
# time (might be dangerous)
# The below allows you to type 'pgnux_restore --' to restore
# everything.
RESTORE_ALL_KEYWORD="--"

# For SELinux we need to use 'runuser' not 'su'
if [ -x /sbin/runuser ]; then
    SU=/sbin/runuser
else
    SU=/bin/su
fi
  
# Not worth continuing if there is no dump directory
if [ ! -d $DUMPDIR ]; then
   echo -n "$(date +'%Y-%m-%d %H:%M:%S') - $$ - ERROR - "
   echo "Directory '$DUMPDIR' does not exist."
   exit 2
fi
  
# Capture Database(s) and allow for delimiters to separate more
# then one. Keep in mind that the first database specified will
# be the one used to detect the SNAPTIME if one isn't specified
DBLIST=$(echo "$1" | sed -e 's/[,:+%!&]/ /g')
 
# Take the value specified on the command line (argument 2)
shift
_SNAPTIME=$(echo "$@" | sed -e 's/[^0-9]//g')
SNAPTIME=
if [ -z "$_SNAPTIME" ]; then
   # Fetch most recent backup time
   SNAPTIME=$(find -L $DUMPDIR -maxdepth 1 -mindepth 1 \
        -type f -regex ".*/[0-9]+-.*\.\(db\|db.gz\)$" \
        -exec basename {} \; | \
        sed -e 's/^\([0-9]\+\)-\(.*\)\.\(db\|db\.gz\)$/\1/g' | \
        sort -r -n | uniq | head -n1)
   if [ -z "$SNAPTIME" ]; then
      echo -n "$(date +'%Y-%m-%d %H:%M:%S') - $$ - ERROR - "
      echo "No backups detected."
      exit 2
   fi
fi
 
# Initialize Globals Loaded Flag 
GLOBALS_LOADED=1
if [ "$DBLIST" == "$RESTORE_ALL_KEYWORD" ]; then
   # Now we build a list of the databases based on the
   # snapshot time specified
   DBLIST=$(find -L $DUMPDIR -maxdepth 1 -mindepth 1 \
      -type f -regex ".*/$SNAPTIME-.*\.\(db\|db.gz\)\$" \
      -exec basename {} \; | \
      sed -e 's/^[0-9]\+-\(.*\)\.\(db\|db\.gz\)$/\1/g')
   GLOBALS_LOADED=0
fi
 
FAILLIST=""
RESTORE_COUNT=0
for DBNAME in $DBLIST; do
   # Toggle 'Globals Loaded Flag' if 'Restore All Keyword' found
   # This neat little trick allows you to force the restores of
   # globals associated with a specified date of the backup
   # hence: pgnux_restore --,foobar_database 2014010100000
   # would cause the globals taken at this time to be loaded
   # with the database,
   [ "$DBLIST" == "$RESTORE_ALL_KEYWORD" ] && GLOBALS_LOADED=0

   # If no Snapshot time was specified, we need to
   # keep detecting it based on the database specified
   [ -z "$_SNAPTIME" ] &&
      SNAPTIME=$(find -L $DUMPDIR -maxdepth 1 -mindepth 1 \
        -type f -regex ".*/[0-9]+-$DBNAME\.\(db\|db.gz\)$" \
        -exec basename {} \; | \
        sed -e 's/^\([0-9]\+\)-\(.*\)\.\(db\|db\.gz\)$/\1/g' | \
        sort -r -n | uniq | head -n1)
   PGBKUP=$DUMPDIR/$SNAPTIME-$DBNAME.db

   if [ $GLOBALS_LOADED -eq 0 ] && \
      [ -f $DUMPDIR/globals-$SNAPTIME.sql.gz ]; then
      echo -n "$(date +'%Y-%m-%d %H:%M:%S') - $$ - INFO - "
      echo -n $"Restoring Database Globals ..."
         $SU -l postgres -c "gunzip \
             -c $DUMPDIR/globals-$SNAPTIME.sql.gz | \
             $PGENGINE/psql -a -p $PGPORT -h $PGHOST"
      # Toggle Flag
      GLOBALS_LOADED=1
   fi
 
   # Detect Compression
   if [ -f $PGBKUP.gz ]; then
      echo -n "$(date +'%Y-%m-%d %H:%M:%S') - $$ - INFO - "
      echo $"Uncompressing Database $DBNAME ($SNAPTIME)..."
      gunzip -c $PGBKUP.gz > $PGBKUP
      if [ $? -ne 0 ]; then
         FAILLIST="$FAILLIST $DBNAME"
         [ -f $PGBKUP ] && rm -f $PGBKUP
      fi
   fi
 
   [ ! -f $PGBKUP ] && continue
 
   echo -n "$(date +'%Y-%m-%d %H:%M:%S') - $$ - INFO - "
   echo $"Restoring Database $DBNAME ($SNAPTIME)..."
   # This action can fail for 2 reasons:
   #  1. the database doesn't exist (this is okay because we
   #     are restoring it anyway.
   #  2. Someone is accessing the database we are trying to
   #     drop.  You need to make sure you've stopped all
   #     processes that are accessing the particular database
   #     being restored or you will have strange results.
   $SU -l postgres -c "$PGENGINE/dropdb \
          -p $PGPORT -h $PGHOST $DBNAME" &>/dev/null
   # use -C & -d 'template1' so we can bootstrap off of the
   # 'template1' database. Note, that this means we are bound
   # by the character encoding of 'template1' (default is UTF-8)
   $SU -l postgres -c "$PGENGINE/pg_restore -e -v -p $PGPORT \
                   -C -d template1 -h $PGHOST $PGBKUP"
   [ $? -ne 0 ] && FAILLIST="$FAILLIST $DBNAME" || \
        let RESTORE_COUNT+=1
   [ -f $PGBKUP.gz ] && [ -f $PGBKUP ] && rm -f $PGBKUP
done
  
# Spit a readable list of databases that were not recovered if
# required.
if [ ! -z "$FAILLIST" ]; then
   for DBNAME in $FAILLIST; do
      echo "Warning: $DBNAME ($SNAPTIME) was not correctly restored."
   done
   exit 1
elif [ $RESTORE_COUNT -eq 0 ]; then
   # Nothing was restored; now is a good time to display to the user
   # their options
   echo -n "$(date +'%Y-%m-%d %H:%M:%S') - $$ - WARNING - "
   echo $"There were no databases restored."
   echo
   # Display last X backups and the databases associated with each
   BACKUPS=$(find -L $DUMPDIR -maxdepth 1 -mindepth 1 \
        -type f -regex ".*/[0-9]+-.*\.\(db\|db.gz\)$" \
        -exec basename {} \; | \
        sed -e "s/^\([0-9]\+\)-\(.*\)\.\(db\|db\.gz\)$/\1|\2/g" | \
        sort -r -n | head -n$HISTORY)
   [ -z "$BACKUPS" ] && exit 1
 
   LAST_SNAPTIME=""
   printf "    %-16s %s\n" "TIMESTAMP" "DATABASE(S)"
   for BACKUP in $BACKUPS; do
      SNAPTIME=$(echo $BACKUP | cut -d'|' -f1)
      DBNAME=$(echo $BACKUP | cut -d'|' -f2)
      if [ "$LAST_SNAPTIME" != "$SNAPTIME" ]; then
          printf "\n%s: %s" $(echo "$SNAPTIME" | \
              sed -e 's/\([0-9]\{4\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)/\1-\2-\3_\4:\5:\6/g') "$DBNAME"
          LAST_SNAPTIME=$SNAPTIME
      else
         printf ", %s" "$DBNAME"
      fi
   done
   printf "\n"
   exit 1
fi
exit 0

Automation
Consider creating a few crontab entries to manage backups during off hours as well as some tool that can keep your backups from growing out of control on your hard drive. Feel free to adjust the numbers as you see fit. The below assumes you copied the 2 scripts above in /usr/bin

cat << EOF > /etc/cron.d/nuxref_postgresql_backup
# Create a backup of all databases every hour
0  * * * * root /usr/bin/pgnux_backup &>/dev/null
# Only keep the last 30 days of backups (check run every hour on the 40 min mark)
40 * * * * root find /var/pgbackup -maxdepth 1 -type f -mtime +30 -delete &>/dev/null
EOF

Usage
Backups are as simple as the following:

# Backup All databases
pgnux_backup

# Backup just the foobar_database:
pgnux_backup foobar_database

# Backup just the foobar_database and the barfoo_database:
pgnux_backup foobar_database barfoo_database

Restoring requires that no one is accessing the database (if it already exists). Otherwise it’s just as easy:

# Call the script with no parameters, to see a nice list of the last 10 backups.
# This list can be really helpful because it also identifies all of the
# snapshot times that they were taken.  You need these times to preform
# restores with.
pgnux_restore

# Restore ALL databases that resided in the last database snapshot
# Caution: If you just backed up a single database on it's own last
#          time, this might be a bit unsafe to run. If you backed
#          up other databases at a different time, then they won't
#          get included in this restore. The -- also forces the
#          system globals to get reloaded as well.
pgnux_restore --

# Restore the last snapshot taken of the foobar_database
pgnux_restore foobar_database

# Restore the last snapshot taken of the foobar_database and
# barfoo_database notice we can't use space as a delimiter (like
# pgnux_backup does) only because the second argument identifies
# the snapshot time (in case you want your restore to be specific).
pgnux_restore foobar_database,barfoo_database

# With respect to the last point, this will restore a database
# snapshot further back in the past (not the latest one). This
# also assumes you have a backup at this point.  You can find
# this out by running pgnux_restore all by itself as the first
# example given above. The script will take care of the formatting
# of the time so long as you provide YYYYMMDDHHmmss in that format.
# You can stick in as many delimiters to make the time readable as
# you want. The below restore a specific database snapshot:
pgnux_restore foobar_database 2014-03-13_15:47:06

# You can restore more then one database at a specific time as
# well using the a comma (,) to delimit your databases and
# specifying the snapshot time at the end. Here is how to restore
# the foobar_database and the barfoo_database at a specific
# instance of time:
pgnux_restore foobar_database,barfoo_database 2014-03-13_15:47:06

# the -- also plays another role, it can be used to force the loading
# of the postgresql globals again when it's paired with databases
# consider the above line but with this small tweak:
pgnux_restore --,foobar_database,barfoo_database 2014-03-13_15:47:06

Catastrophic Recoveries
If worst case scenario happens, and the restore process seems to be failing you, your not out of options. As long as you’ve been taking frequent backups, there is always a way back!

Remember that restores will fail if the database is being accessed by someone else. The below is a dirty little scripted trick that boots everyone accessing your system off. It’s somewhat dangerous to do, but if restoring is your final goal, then the danger is mitigated.

#!/bin/bash
# Author: Chris Caron <lead2gold at gmail.com>
# Name: pgnux_kickusers
# Description: Kicks all users off of a PostgreSQL instance being
#              accessed.
# Syntax: pgnux_kickusers

##################################
# Define some simple variables
##################################
PGENGINE=/usr/bin
PGPORT=5432
PGHOST=localhost

# For SELinux we need to use 'runuser' not 'su'
if [ -x /sbin/runuser ]; then
    SU=/sbin/runuser
else
    SU=/bin/su
fi
# Get a list of all PIDS accessing our system
PIDS=$($SU -l postgres \
      -c "$PGENGINE/psql -h $PGHOST -p $PGPORT -e \
         -c \"SELECT procpid from pg_stat_activity;\"" \
            2>/dev/null | \
      egrep '^[ \t]+[0-9]+' | tr -s '[:space:]' ' ')

# Iterate over our list and attempt to kick our users
# by sending each of them a SIGTERM.  Since we're impatient
# if they don't die we try 3 more times before sending a
# SIGKILL
for PID in $PIDS; do
   RETRY=3
   while [ $RETRY -gt 0 ]; do
      kill $PID &>/dev/null
      ps -p $PID &>/dev/null
      if [ $? -ne 0 ]; then break; fi
      let RETRY-=1
   done
   ps -p $PID &>/dev/null
   [ $? -eq 0 ] && kill -9 $PID
done

They can also fail if PostgreSQL data directory contents has been corrupted. Corruption can occur:

  • From a system that is experiencing harddrive failures.
  • From another processes accessing /var/lib/pgsql/data/* that shouldn’t be. Only the database itself should be modifying/accessing the contents of this directory unless you know what you’re doing.
  • From just a simple power outage. A busy system that isn’t backed up by a UPS device or redundant power sources can have it’s file system corrupted easily during a loss of power. What usually happens is PostgreSQL is writing content to the database when the power is pulled from the machine. At the end of the day, you’re left with a database that wasn’t correctly written to and in some cases can be seriously damaged from this as a result.

In some cases, the corruption can be so bad the database instance (PostgreSQL) won’t even start. Evidently, you can’t restore a working copy of a database if the service itself isn’t working for you.

Sometimes attempting to repair the write-ahead logging can help you recover. Obviously this completely depends on the type of failure you’re experiencing. /var/lib/pgsql/data/pg_xlog/* can help shed some light on this subject as well. The below script assumes your instance of PostgreSQL is stopped.

# Make sure your database is NOT running if you intend to run the below
# command:
service postgresql stop

Here is the the simple wrapper to the read-ahead write log repair:

#!/bin/bash
# Author: Chris Caron <lead2gold at gmail.com>
# Name: pgnux_repair
# Description: Simple wrapper to pg_resetxlog
# Syntax: pgnux_repair
 
##################################
# Define some simple variables
##################################
PGENGINE=/usr/bin
PGDATA=/var/lib/pgsql/data

# For SELinux we need to use 'runuser' not 'su'
if [ -x /sbin/runuser ]; then
    SU=/sbin/runuser
else
    SU=/bin/su
fi

# Attempt to repair the database
$SU -l postgres -c "$PGENGINE/pg_resetxlog -f $PGDATA"

If it still doesn’t work, as a last resort consider these simple commands to bring your system back up again. Note that you should be the root user prior to preforming these steps:

# Stop the database (if it isn't already):
service postgresql stop

# Backup your configuration files
mkdir -p /tmp/postgresql_conf
cp -a /var/lib/pgsql/data/*.conf /tmp/postgresql_conf

# Destroy the database data directory. Unless you changed the
# default directory, the below command will do it:
# This is IRREVERSIBLE, be sure you have a recovery point to
# restore to before doing this otherwise you will make your
# situation worse!
find /var/lib/pgsql/ -mindepth 1 -maxdepth 1 rm -rf {} \;

# Initialize a new empty database; this will rebuild
# the /var/lib/pgsql/data directory for you
service postgresql initdb

# Restore all of your configuration you backed up above:
cp -af /tmp/postgresql_conf/*.conf /var/lib/pgsql/data/

# Just in case, for SELinux Users, make sure we're set okay
restorecon -R /var/lib/pgsql

# Start your database up
service postgresql start

# Now restore your databases using pgnux_restore and make sure you
# include the double dash '--' so that you load the system globals
# back as well:
#   pgnux_restore --,db1,db2,db3,db4,...
#
# If you've always been backing them all together, then you can
# restore them with a single command:
pgnux_restore --

# Your done! You should be back and running now and hopefully with
# minimal downtime.

Disclaimer
I can not be held responsible or liable for anything that goes wrong with your system. This blog is intended to be used as a resource only and will hopefully help you out in the time of need. These tools work for me and have been tested thoroughly using PostgreSQL v8.4 (which ships with CentOS 6.x). I have not tried any of these tools for PostgreSQL v9.x or higher and can not promise you anything, but I honestly can’t see a reason why they wouldn’t work.

Credit
Please note that this information took me several days to put together and test thoroughly. I may not blog often; but I want to re-assure the stability and testing I put into everything I intend share.

If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

FTPS Solution

Configuring and Installing VSFTPD on CentOS 6


Introduction
FTP Servers have been around for a very long time. On could easily argue that they aren’t always the best option to choose anymore. But for those dealing with legacy systems, the FTP protocol provides one the easiest ways to share and distribute files across external systems. Despite the hate some might give the protocol; it still receives a lot a love from others just because it’s still very compatible with just about everything; heck, even web browsers such as Internet Explorer, Chrome, and Firefox (and many more) have the protocol built right into it.

In my case, I needed to set up an FTP server to help a client with some legacy software they use (and are familiar with). This blog is more or less just the steps I took to make it work in case anyone else is interested.

Security Concerns
FTP preforms all of it’s transactions in ‘plain text’ including it’s authentication. This wasn’t a problem back in 1980 when online security wasn’t an issue. This also isn’t a problem for sites offering anonymous file hosting services. But for everyone else, it pretty much leaves you susceptible to privacy issues and possible intrusions.

FTPS (not to be confused with SFTP) is a way of securing the FTP protocol for the systems that require it allowing you to eliminate the ‘plain text’ problem. But this requires the client uses software that can take advantage of it.

Some additional security concerns I wanted to consider:

  • Separate user accounts from the system ones. We don’t want people trying to guess our root password or access anyone’s home directory unless with specifically configure the server to allow it.

Setup VSFTPD
VSFTPD stands for Very Secure FTP Daemon and provides all the flexibility we need. It’s official website can be found here.

The following will set up VSFTPD into your CentOS/RedHat environment in a isolated manor. You see we want to disconnect the users that currently access your system from the users we create for VSFTPD for more flexibility and control.

# First fetch the required packages
# db4:       The Berkeley Database (Berkeley DB) which is a
#             quick and dirty way of storing our user accounts.
# db4-utils: This provides a tool we'll use to build a small user
#             database with.  Technically you can uninstall (just)
#             this package afterwards for added security after.
# openssl:   This is used to grant our FTP server FTPS support
# vsftpd:    The FTP/FTPS Daemon itself
yum -y install db4 db4-utils openssl vsftpd

# Create a password file: /etc/vsftpd/users.passwd
# This file contains all of the users you want to allow on the
# site in the structure:
# Line: Entry
#  1  | USERNAME1
#  2  | PASSWORD1
#  3  | USERNAME2
#  4  | PASSWORD2
# This below creates a simple user 'foobar' and a password
# of 'barfoo'.  This isn't the safest way to build your password
# file because it leaves a persons password set available in
# the shell history... But for the purpose of this tutorial:
echo foobar > /etc/vsftpd/users.passwd
echo barfoo >> /etc/vsftpd/users.passwd

# Protect our password file now from prying eyes:
chmod 600 /etc/vsftpd/users.passwd
chown root.root /etc/vsftpd/users.passwd

# Prepare a directory we want the foobar user to send it's files
# to. You can also just use a directory you already have in place
# or someone elses home directory.
mkdir -p /var/ftp/foobar

# Set a comfortable permission to this directory granting the ftp
# user (or any user account your later going to assign to this user
# read/write access)
chown root.ftp /var/ftp/foobar
chmod 775 /var/ftp/foobar

# Convert our plain-text password file into the Berkley Database
# format. This is the only command that requires the db4-utils
# rpm package which you can uninstall if you want for security
# reasons after you run the below command.  You'll need to
# re-installl it though if you ever want to add or update accounts
db_load -T -t hash \
   -f /etc/vsftpd/users.passwd /etc/vsftpd/virtual.users.db

# Protect our new (Berkley) database:
chmod 600 /etc/vsftpd/virtual.users.db
chown root.root /etc/vsftpd/virtual.users.db

# Prepare our virtual user directory; this is where we can
# optionally place over-riding configuration for each user
# we create in the Berkley database above.
mkdir -p /etc/vsftpd/virtual.users
chmod 700 /etc/vsftpd/virtual.users
chown root.root /etc/vsftpd/virtual.users

# Create PAM Module that points to our new database.
# Note: you do not provide the '.db' extension when creating
#       this file. The file is valid as you see it below.
cat << _EOF > /etc/pam.d/vsftpd-virtual
auth     required pam_userdb.so db=/etc/vsftpd/virtual.users
account  required pam_userdb.so db=/etc/vsftpd/virtual.users
session  required pam_loginuid.so
_EOF

# Protect our Module
chmod 644 /etc/pam.d/vsftpd-virtual
chown root.root /etc/pam.d/vsftpd-virtual

# Create an empty jail directory.  This is used for default
# configurations only. A well configured system won't even use
# this; but it's still good to have since we'll be referencing
# it in our configuration. This will become the default
# directory a user connects to if they aren't otherwise
# configured to go to another location.
mkdir -p /var/empty/vsftpd/
chown nobody.ftp /var/empty/vsftpd/
chmod 555 /var/empty/vsftpd/

# Now we want to allow FTPS support, we'll need an SSL key to
# do it with.  If you already have one, you can skip this step.
# Otherwise, the following will just generate you a self-signed
# key as a temporary solution.
openssl req -nodes -new -x509 -days 730 -sha1 -newkey rsa:2048 \
   -keyout /etc/pki/tls/private/nuxref.com.key \
   -out /etc/pki/tls/certs/nuxref.com.crt \
   -subj "/C=CA/ST=Ontario/L=Ottawa/O=NuxRef/OU=IT/CN=nuxref.com"

# Protect our Keys
chmod 400 /etc/pki/tls/private/nuxref.com.key; # Private Key
chmod 444 /etc/pki/tls/certs/nuxref.com.crt; # Public Certificate

# Create ourselves a little banner we can use to at least alert
# human intruders that they are in fact being monitored.  This
# scare tactic may or may not work, but if you ever have a breach
# of security, you may need to reference that you gave the user
# ample warning that they were violating someones rights by
# continuing.  Feel free to adjust the banner to your likings.
cat << _EOF > /etc/banner
* - - - - - - W A R N I N G - - - - - - - W A R N I N G - - - - - *
*                                                                 *
* The use of this system is restricted to authorized users. All   *
* information and communications on this system are subject to    *
* review, monitoring and recording at any time, without notice or *
* permission.                                                     *
*                                                                 *
* Unauthorized access or use shall be subject to prosecution.     *
*                                                                 *
* - - - - - - W A R N I N G - - - - - - - W A R N I N G - - - - - *
_EOF

# Protect our banner
chmod 640 /etc/banner

At this point we have our environment set up the way we want it. The next step is to create our VSFTPD configuration.

# Lets first backup original configuration file
mv /etc/vsftpd/vsftpd.conf /etc/vsftpd/vsftpd.conf.orig

# Create new configuration
cat << _EOF > /etc/vsftpd/vsftpd.conf
# --------------------------------------------------------------
# Base Configuration
# --------------------------------------------------------------
anon_world_readable_only=NO
anonymous_enable=NO
chroot_local_user=YES
hide_ids=YES
listen=YES
local_enable=YES
max_clients=10
max_per_ip=3
nopriv_user=ftp
pasv_min_port=64000
pasv_max_port=64100
session_support=NO
user_config_dir=/etc/vsftpd/virtual.users
userlist_enable=YES
use_localtime=YES
xferlog_enable=YES
xferlog_std_format=NO
log_ftp_protocol=YES
pam_service_name=vsftpd-virtual
banner_file=/etc/banner
reverse_lookup_enable=NO
# --------------------------------------------------------------
# Secure Configuration (FTPS)
# --------------------------------------------------------------
ssl_enable=YES
virtual_use_local_privs=NO
allow_anon_ssl=NO
# forcing SSL makes the FTP portion of your site disabled and it
# will only operate as FTPS.  This may or may not be what you
# want.
force_local_data_ssl=NO
force_local_logins_ssl=NO
ssl_tlsv1=YES
ssl_sslv2=NO
ssl_sslv3=NO
# Point to our certificates
rsa_cert_file=/etc/pki/tls/certs/nuxref.com.crt
rsa_private_key_file=/etc/pki/tls/private/nuxref.com.key
require_ssl_reuse=NO
ssl_ciphers=HIGH:!MD5:!ADH
# --------------------------------------------------------------
# FTP Configuration
# --------------------------------------------------------------
async_abor_enable=YES
ftp_data_port=20
connect_from_port_20=YES
# --------------------------------------------------------------
# Default Anonymous Restrictions (over-ride per virtual user)
# --------------------------------------------------------------
guest_enable=NO
guest_username=nobody
# Default home directory once logged in
local_root=/var/empty/vsftpd
# write_enabled is required if the user is to make use of any of
# the anon_* commands below
write_enable=NO
# give the user the ability to make directories
anon_mkdir_write_enable=NO
# give the user the ability delete and overwrite files
anon_other_write_enable=NO
# give the user the ability upload new files
anon_upload_enable=NO
# Give the user permission to do a simple directory listings
dirlist_enable=NO
# Give the user permission to download files
download_enable=NO
# if the user has can upload or make new directories, then this
# will be the umask applied to them
anon_umask=0002
# delete failed uploads (speaks for itself)
delete_failed_uploads=YES
_EOF

# Protect our configuration
chmod 600 /etc/vsftpd/vsftpd.conf
chown root.root /etc/vsftpd/vsftpd.conf

Technically we’re done now, but because we intentionally specified very restrictive user access rights, our foobar user we created will only connect to the /var/empty/vsftpd directory with no access rights. Therefore, our final step is to create an additional configuration file for the foobar account granting him read/write access to /var/ftp/foobar.

# The file you write to in the /etc/vsftpd/virtual.users/ 'must' be the same
# name as the user(s) you created to over-ride their permissions!
cat << _EOF > /etc/vsftpd/virtual.users/foobar
local_root=/var/ftp/foobar
# --------------------------------------------------------------
# User
# --------------------------------------------------------------
guest_enable=YES
# Set this to any system user you want
guest_username=ftp
local_root=/var/ftp/foobar
# --------------------------------------------------------------
# Permissions
# --------------------------------------------------------------
# write_enabled is required if the user is to make use of any of
# the anon_* commands below
write_enable=YES
# give the user the ability to make directories
anon_mkdir_write_enable=YES
# give the user the ability delete and overwrite files
anon_other_write_enable=YES
# give the user the ability upload new files
anon_upload_enable=YES
# Give the user permission to do a simple directory listings
dirlist_enable=YES
# Give the user permission to download files
download_enable=YES
# if the user has can upload or make new directories, then this
# will be the umask applied to them
anon_umask=0002
# delete failed uploads (speaks for itself)
delete_failed_uploads=NO
_EOF

# Protect our foobar permission file
chmod 600 /etc/vsftpd/virtual.users/foobar
chown root.root /etc/vsftpd/virtual.users/foobar

You are now complete, you can start VSFTPD at any time:

# Have VSFTPD start after each system reboot
chkconfig --level 345 vsftpd on

# Start VSFTPD if it isn't already running
service vsftpd status || service vsftpd start

It’s worth noting that if you ever change any of the configuration or add more, you will need to restart the VSFTPD server in order for the changes you made to take effect:

# Restarting the vsftpd server is simple:
service vsftpd restart

Firewall Configuration
The FTP firewall configuration can get complicated especially when the ephemeral ports it chooses to open are random (when operating it in Passive mode). If you scan through the configuration file above, you’ll see that we’ve specified this range to be between 64000 and 64100.

In a nutshell, if you choose to use/enable FTPS (which I strongly recommend you do), the firewall configuration will look like this (/etc/sysconfig/iptables):

#....
#---------------------------------------------------------------
# FTP Traffic
#---------------------------------------------------------------
-A INPUT -p tcp -m state --state NEW --dport 21 -j ACCEPT
# non-encrypted ftp connections (ip_contrack_ftp) module looks
# after these ports, however for the encrypted sessions it can't
# spy so we need to disable ip_contrack_ftp and just open the
# port range himself
-A INPUT -p tcp -m state --state NEW --dport 64000:64100 -j ACCEPT
#...

However, if (and only if) you choose not to use FTPS and strictly operate using FTP only then your configuration will look as follows:

  1. /etc/sysconfig/iptables
    #....
    #---------------------------------------------------------------
    # FTP Traffic
    #---------------------------------------------------------------
    -A INPUT -p tcp -m state --state NEW --dport 21 -j ACCEPT
    #...
  2. /etc/sysconfig/iptables-config
    This file requires you to add ip_conntrack_ftp to the variable IPTABLES_MODULES which is near the top of this file. You may also need to update /etc/sysconfig/ip6tables-config if you are using ip6; the change is the same. This keeps the entire range of 64000 to 64100 ports closed be default and through packet sniffing, they are opened on demand.
#...
IPTABLES_MODULES="ip_conntrack_ftp"
#...

If you’re not running any other modules, you can use the following one liner to update the file:

sed -i -e 's/^IPTABLES_MODULES=.*/IPTABLES_MODULES="ip_conntrack_ftp"/g' \
    /etc/sysconfig/iptables-config
sed -i -e 's/^IPTABLES_MODULES=.*/IPTABLES_MODULES="ip_conntrack_ftp"/g' \
    /etc/sysconfig/ip6tables-config

Be sure to reload your firewall configuration once you have these in place:

# Restart the firewall
service iptables restart

Fail2Ban Bruit Force Configuration
The final thing you should consider if your server will be available via the internet is some bruit force prevention. I really recommend you read my blog on Securing Your CentOS 6 System, specifically my blurb on Fail2Ban which I think all systems should always have running. Fail2ban allows you to track all users hitting your FTP server and take immediate action on preventing further access to this potential intruder.

The configuration is as follows (remember to set the email to what you want it as where I’ve specified your@email.goes.here so you can be notified of any intrusions that take place.

cat << _EOF >> /etc/fail2ban/jail.conf
[my-vsftpd-iptables]

enabled  = true
filter   = vsftpd
action   = iptables[name=VSFTPD, port=ftp, protocol=tcp]
           sendmail-whois[name=VSFTPD, dest=your@email.goes.here]
logpath  = /var/log/vsftpd.log
_EOF

Be sure to reload your Fail2Ban configuration once this is done

# Restart Fail2Ban
service fail2ban restart

Test Our Configuration
Make sure you have an ftp tool installed into your environment like lftp or even a GUI based tool like FileZilla that supports both FTP and FTPS. The old Linux tool ‘ftp’ will only allow you to test the un-encrypted connection.

# Install lftp to keep things simple
yum -y install lftp

# Test our server (FTP)
[root@nuxref ~]# lftp ftp://foobar:barfoo@localhost
lftp foobar@localhost:~> pwd
ftp://foobar:barfoo@localhost
lftp foobar@localhost:~> exit

# Test our server (FTPS)
[root@nuxref ~]# lftp ftps://foobar:barfoo@localhost
lftp foobar@localhost:~> pwd
ftps://foobar:barfoo@localhost
lftp foobar@localhost:~> exit

First lets just test the basic FTP portion (plain-text):

# Connect to our server
[root@nuxref ~]# ftp localhost
Connected to localhost (127.0.0.1).
220-* - - - - - - W A R N I N G - - - - - - - W A R N I N G - - - - - *
220-*                                                                 *
220-* The use of this system is restricted to authorized users. All   *
220-* information and communications on this system are subject to    *
220-* review, monitoring and recording at any time, without notice or *
220-* permission.                                                     *
220-*                                                                 *
220-* Unauthorized access or use shall be subject to prosecution.     *
220-*                                                                 *
220-* - - - - - - W A R N I N G - - - - - - - W A R N I N G - - - - - *
220 
Name (localhost:nuxref): foobar
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls
227 Entering Passive Mode (127,0,0,1,250,83).
150 Here comes the directory listing.
226 Directory send OK.
FileZilla FTPS Configuration

FileZilla FTPS Configuration

If you plan on using FileZilla as your solution, You need to configure it to connect as the FTP protocol with the Encryption set to Requires explicit FTP over TLS similar to the screen shot I provided.

You may or may not have to accept your certificate afterwards that we created earlier in this blog.

FileZilla On Going FTPS Bug
The FileZilla Client is a pretty sweet application for those who like to work with a GUI instead of a command line. Those who choose to test their configuration with this should just know that there is an outstanding bug with FileZilla and the FTPS protocol. Hence, if you’re using Filezilla to to test your new VSFTPD server and it’s not working, it might not be your configuration at the end of the day. The versions seem to be hit and miss of which cause the bug to surface; reports of v3.5.2 working and v3.5.3 not. That all said, I’m using v3.7.3 and am not having a problem.

Here is the ticket #7873 that identifies the problem. One thing that is mentioned is that an earlier version of FileZilla works perfectly fine (specifically v3.5.2). But I’ve also had no problem with the current version (at the time was v3.7.3). I guess my main point is… don’t panic if you see this error; it’s not necessarily anything you’ve configured incorrectly. If you followed this blog then you shouldn’t have any issue at all.

Credit
I may not blog often; but I want to re-assure the stability and testing I put into everything I intend share.

If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least; it’s really all that I ask of you.

Sources:

NSCA & NRPE Solution

Configuring and Installing NRPE and NSCA into Nagios Core 4 on CentOS 6


Introduction
About a month ago I wrote (and updated) an article on how to install Nagios Core 4 onto your system. I’m a bit of a perfectionist, so I’ve rebuilt the packages a little to accommodate my needs. Now I thought it might be a good idea to introduce some of the powerful extensions you can get for Nagios.

RPM Solution
RPMs provide a version control and an automated set of scripts to configure the system how I want it. The beauty of them is that if you disagree with something the tool you’re packaging does, you can feed RPMs patch files to accommodate it without obstructing the original authors intention.

Now I won’t lie and claim I wrote these SPEC files from scratch because I certainly didn’t. I took the stock ones that ship with these products (NRPE and NSCA) and modified them to accommodate and satisfy my compulsive needs. :)

My needs required a bit more automation in the setup as well as including:

  • A previous Nagios requirement I had was a /etc/nagios/conf.d directory to behave similar to how Apache works. I wanted to be able to drop configuration files into here and just have it work without re-adjusting configuration files. In retrospect of this, these plugins are a perfect example of what can use this folder and work right out of the box.
  • These new Nagios plugins should adapt to the new nagiocmd permissions. The nagioscmd group permission was a Nagios requirement I had made in my previous blog specifically for the plugin access.
  • NSCA should prepare some default configuration to make it easier on an administrator.
  • NSCA servers that don’t respond within a certain time should advance to a critical state. This should be part of the default (optional) configuration one can use.
  • Both NRPE and NSCA should plug themselves into Nagios silently without human intervention being required.
  • Both NRPE and NSCA should log independently to their own controlled log file that is automatically rotated by the system when required.

Nagios Enhancement Focus
The key things I want to share with you guys that you may or may not find useful for your own environment are the following:

  • Nagios Remote Plugin Executor (NRPE): NRPE (officially accessed here) provides a way to execute all of the Nagios monitoring tools on a remote server. These actions are all preformed through a secure (private) connection to the remote server and then reported back to Nagios. NRPE can allow you to monitor servers that are spread over a WAN (even the internet) from one central monitoring server. This is truly the most fantastic extension of Nagios in my opinion.
    NRPE High Level Overview

    NRPE High Level Overview

  • Nagios Service Check Acceptor (NSCA): NSCA (officially accessed here) provides a way for external applications to report their status directly to the Nagios Server on their own. This solution still allows the remote monitoring of a system by taking the responsibility off of the status checks off of Nagios. However the fantastic features of Nagios are still applicable: You are still centrally monitoring your application and Nagios will immediately take action in notifying you if your application stops responding or reports a bad status. This solution is really useful when working with closed systems (where opening ports to other systems is not an option).
    NSCA High Level Overview

    NSCA High Level Overview

Just give me your packaged RPMS
Here they are:

How do I make these packages work for me?
In all cases, the RPMs take care of just about everything for you, so there isn’t really much to do at this point. Some considerations however are as follows:

  • NRPE
    NRPE - Nagios Remote Plugin Executor

    NRPE – Nagios Remote Plugin Executor


    In an NRPE setup, Nagios is always the client and all of the magic happens when it uses the check_nrpe plugin. Most of NRPE’s configuration resides at the remote server that Nagios will monitor. In a nutshell, NRPE will provide the gateway to check a remote system’s status but in a much more secure and restrictive manor than the check_ssh which already comes with the nagios-plugins package. The check_ssh requires you to create a remote user account it can connect with for remote checks. This can leave your system vulnerable to an attack since you can do a lot more damage with a compromised SSH account. However check_nrpe uses the NRPE protocol and can only return what you let it; therefore making it a MUCH safer choice then check_ssh!

    You’ll want to install nagios-plugins-nrpe on the same server your hosting Nagios on:

    # Download NRPE
    wget --output-document=nagios-plugins-nrpe-2.15-1.el6.x86_64.rpm https://www.dropbox.com/sh/9dt7klam6ex1kpp/ra9QOooqC4/20131208/nagios-plugins-nrpe-2.15-1.el6.x86_64.rpm?dl=1
    
    # Now install it
    yum -y localinstall nagios-plugins-nrpe-2.15-1.el6.x86_64.rpm
    

    Again I must stress, the above setup will work right away presuming you chose to use my custom build of Nagios introduced in my blog that went with it.

    Just to show you how everything works, we’ll make the Nagios Server the NRPE Server as well. In real world scenario, this would not be the case at all! But feel free to treat the setup example below on a remote system as well because it’s configuration will be identical! :)

    # Install our NRPE Server
    wget --output-document=nrpe-2.15-1.el6.x86_64.rpm https://www.dropbox.com/sh/9dt7klam6ex1kpp/znwk7QJjr2/20131208/nrpe-2.15-1.el6.x86_64.rpm?dl=1
    
    # Install some Nagios Plugins we can configure NRPE to use
    wget --output-document=nagios-plugins-1.5-1.x86_64.rpm https://www.dropbox.com/sh/9dt7klam6ex1kpp/fM6fHoEELL/20131110/nagios-plugins-1.5-1.x86_64.rpm?dl=1
    
    # Now Install it
    yum -y localinstall nrpe-2.15-1.el6.x86_64.rpm \
       nagios-plugins-1.5-1.x86_64.rpm
    # This tool requires xinetd to be running; start it if it isn't
    # already running
    service xinetd status || service xinetd start
    
    # Make sure our system will always start xinetd
    # even if it's rebooted
    chkconfig --level 345 xinetd on
    

    Now we can test our server by creating a test configuration:

    # Create a NRPE Configuration our server can accept
    cat << _EOF > /etc/nrpe.d/check_mail.cfg
    command[check_mailq]=/usr/lib64/nagios/plugins/check_mailq -c 100 -w 50
    _EOF
    
    # Create a temporary test configuration to work with:
    cat << _EOF > /etc/nagios/conf.d/nrpe_test.cfg
    define service{
       use                 local-service
       service_description Check Users
       host_name           localhost
       # check_users is already defined for us in /etc/nagios/nrpe.cfg
    	check_command		  check_nrpe!check_users
    }
    
    # Test our new custom one we just created above
    define service{
       use                 local-service
       service_description Check Mail Queue
       host_name           localhost
       # Use the new check_mailq we defined above in /etc/nrpe.d/check_mail.cfg
    	check_command		  check_nrpe!check_mailq
    }
    _EOF
    
    # Reload Nagios so it sees our new configuration defined in
    # /etc/nagios/conf.d/*
    service nagios reload
    
    # Reload xinetd so nrpe sees our new configuration defined in
    # /etc/nrpe.d/*
    service xinetd reload
    

    We can even test our connection manually by calling the command:

    # This is what the output will look like if everything is okay:
    /usr/lib64/nagios/plugins/check_nrpe -H localhost -c check_mailq
    OK: mailq is empty|unsent=0;50;100;0
    

    Another scenario you might see (when setting on up on your remote server) is:

    /usr/lib64/nagios/plugins/check_nrpe -H localhost -c check_mailq
    CHECK_NRPE: Error - Could not complete SSL handshake.
    

    Uh oh, Could not complete SSL handshake.! What does that mean?
    This is the most common error people see with the NRPE plugin. If you Google it, you’ll get an over-whelming amount of hits suggesting how you can resolve the problem. I found this link useful.
    That all said, I can probably tell you right off the bat why it isn’t working for you. Assuming you’re using the packaging I provided then it’s most likely because your NRPE Server is denying the requests your Nagios Server is making to it.

    To fix this, access your NRPE Server and open up /etc/xinetd/nrpe in an editor of your choice. You need to allow your Nagios Server access by adding it’s IP address to the only_from entry. Or you can just type the following:

    # Set your Nagios Server IP here:
    NAGIOS_SERVER=192.168.192.168
    
    # If you want to keep your previous entries and append the server
    # you can do the following (spaces delimit the servers):
    sed -i -e "s|^\(.*only_from[^=]\+=\)[ \t]*\(.*\)|\1 \2 $NAGIOS_SERVER|g" \
       /etc/xinetd.d/nrpe
    
    # The below command is fine too to just replace what is there
    # with the server of your choice (you can use either example
    sed -i -e "s|^\(.*only_from[^=]\+=\).*|\1 $NAGIOS_SERVER|g" \
       /etc/xinetd.d/nrpe
    
    # When your done, restart xinetd to update it's configuration
    service xinetd reload
    

    Those who didn’t receive the error I showed above, it’s only because your using your Nagios Server as your NRPE Server too (which the xinetd tool is pre-configured to accept by default). So please pay attention to this when you start installing the NRPE server remotely.

    You will want to install nagios-plugins-nrpe on to your NRPE Server as well granting you access to all the same great monitoring tools that have already been proven to work and integrate perfectly with Nagios. This will save you a great deal of effort when setting up the NRPE status checks.

    As a final note, you may want to make sure port 5666 is open on your NRPE Server’s firewall otherwise the Nagios Server will not be able to preform remote checks.

    ## Open NRPE Port (as root)
    iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 5666 -j ACCEPT
    
    # consider adding this change to your iptables configuration
    # as well so when you reboot your system the port is
    # automatically open for you. See: /etc/sysconfig/iptables
    # You'll need to add a similar line as above (without the
    # iptables reference)
    # -A INPUT -m state --state NEW -m tcp -p tcp --dport 5666 -j ACCEPT
    
  • NSCA
    NSCA - Nagios Service Check Acceptor

    NSCA – Nagios Service Check Acceptor


    Remember, NSCA is used for systems that connect to you remotely (instead of you connecting to them (what NRPE does). This is a perfect choice plugin for systems you do not want to open ports up to unnessisarily on your remote system. That said, it means you need to open up ports on your Monitoring (Nagios) server instead.

    You’ll want to install nsca on the same server your hosting Nagios on:

    # Download NSCA
    wget --output-document=nsca-2.7.2-9.el6.x86_64.rpm https://www.dropbox.com/sh/9dt7klam6ex1kpp/p2QklCG6Q3/20131208/nsca-2.7.2-9.el6.x86_64.rpm?dl=1
    
    # Now install it
    yum -y localinstall nsca-2.7.2-9.el6.x86_64.rpm
    
    # This tool requires xinetd to be running; start it if it isn't
    # already running
    service xinetd status || service xinetd start
    
    # Make sure our system will always start xinetd
    # even if it's rebooted
    chkconfig --level 345 xinetd on
    

    The best way to test if everything is working okay is by also installing the nsca-client on the same machine we just installed NSCA on (above). Then we can simply create a test passive service to test everything with. The below setup will work presuming you chose to use my custom build of Nagios introduced in my blog that went with it.

    # First install our NSCA client on the same machine we just installed NSCA
    # on above.
    wget --output-document=nsca-client-2.7.2-9.el6.x86_64.rpm https://www.dropbox.com/sh/9dt7klam6ex1kpp/SOsYLCgGP2/20131208/nsca-client-2.7.2-9.el6.x86_64.rpm?dl=1
    
    # Now install it
    yum -y localinstall nsca-client-2.7.2-9.el6.x86_64.rpm
    
    # Create a temporary test configuration to work with:
    cat << _EOF > /etc/nagios/conf.d/nsca_test.cfg
    # Define a test service. Note that the service 'passive_service'
    # is already predefined in /etc/nagios/conf.d/nsca.cfg which was
    # placed when you installed my nsca rpm
    define service{
       use                 passive_service
       service_description TestMessage
       host_name           localhost
    }
    _EOF
    
    # Now reload Nagios to it reads in our new configuration
    # Note: This will only work if you are using my Nagios build
    service nagios reload
    

    Now that we have a test service set up, we can send it different nagios status through the send_nsca binary that was made available to us after installing nsca-client.

    # Send a Critical notice to Nagios using our test service
    # and send_nsca. By default send_nsca uses the '<tab>' as a
    # delimiter, but that is hard to show in a blog (it can get mixed up
    # with the space.  So in the examples below i add a -d switch
    # to adjust what the delimiter in the message.
    # The syntax is simple:
    #    hostname,nagios_service,status_code,status_msg
    #
    # The test service we defined above identifies both the
    # 'host_name' and 'service_description' define our first 2
    # delimited columns below. The status_code is as simple as:
    #       0 : Okay
    #       1 : Warning
    #       2 : Critical
    # The final delimited entry is just the human readable text
    # we want to pass a long with the status.
    #
    # Here we'll send our critical message:
    cat << _EOF | /usr/sbin/send_nsca -H 127.0.0.1 -d ','
    localhost,TestMessage,2,This is a Test Error
    _EOF
    
    # Open your Nagios screen (http://localhost/nagios) at this point and watch the
    # status change (it can take up to 4 or 5 seconds or so to register
    # the command above).
    
    # Cool?  Here is a warning message:
    cat << _EOF | /usr/sbin/send_nsca -H 127.0.0.1 -d ',' -c /etc/nagios/send_nsca.cfg
    localhost,TestMessage,1,This is a Test Warning
    _EOF
    
    # Check your warning on Nagios, when your happy, here is your
    # OKAY message:
    cat << _EOF | /usr/sbin/send_nsca -H 127.0.0.1 -d ',' -c /etc/nagios/send_nsca.cfg
    localhost,TestMessage,0,Life is good!
    _EOF
    

    Since NSCA requires you to listen to a public port, you’ll need to know this last bit of information to complete your NSCA configuration. Up until now the package i provide only open full access to localhost for security reasons. But you’ll need to take the next step and allow your remote systems to talk to you.

    NSCA uses port 5667, so you’ll want to make sure your firewall has this port open using the following command:

    ## Open NSCA Port (as root)
    iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 5667 -j ACCEPT
    
    # consider adding this change to your iptables configuration
    # as well so when you reboot your system the port is
    # automatically open for you. See: /etc/sysconfig/iptables
    # You'll need to add a similar line as above (without the
    # iptables reference)
    # -A INPUT -m state --state NEW -m tcp -p tcp --dport 5667 -j ACCEPT
    

    Another security in place with the NSCA configuration you installed out of
    the box is that it is being managed by xinetd. The configuration can
    be found here: /etc/xinetd.d/nsca. The security restriction in place that you’ll want to pay close attention to is line 16 which reads:

    only_from = 127.0.0.1 ::1

    If you remove this line, you’ll allow any system to connect to yours; this is a bit unsafe but an option. Personally, I recommend that you individually add each remote system you want to monitor to this line. Use a space to separate more the one system.

    You can consider adding more security by setting up a NSCA paraphrase which will reside in /etc/nagios/nsca.cfg to which you can place the same paraphrase in all of the nsca-clients you set up by updating /etc/nagios/send_nsca.cfg.

    Consider our example above; I can do the following to add a paraphrase:

    # Configure Client
    sed -i -e 's/^#*password=/password=ABCDEFGHIJKLMNOPQRSTUVWXYZ/g' \
       /etc/nagios/send_nsca.cfg
    # Configure Server
    sed -i -e 's/^#*password=/password=ABCDEFGHIJKLMNOPQRSTUVWXYZ/g' \
       /etc/nagios/nsca.cfg
    # Reload xinetd so it rereads /etc/nagios/nsca.cfg
    service xinetd reload
    

I don’t trust you, I want to repackage this myself!
As always, I will always provide you a way to build the source code from scratch if you don’t want to use what I’ve already prepared. I use mock for everything I build so I don’t need to haul in development packages into my native environment. You’ll need to make sure mock is setup and configured properly first for yourself:

# Install 'mock' into your environment if you don't have it already.
# This step will require you to be the superuser (root) in your native
# environment.
yum install -y mock

# Grant your normal every day user account access to the mock group
# This step will also require you to be the root user.
usermod -a -G mock YourNonRootUsername

At this point it’s safe to change from the ‘root‘ user back to the user account you granted the mock group privileges to in the step above. We won’t need the root user again until the end of this tutorial when we install our built RPM.

Just to give you a quick summary of what I did, here are the new spec files and patch files I created:

  • NSCA RPM SPEC File: Here is the enhanced spec file I used (enhancing the one already provided in the EPEL release found on pkgs.org). At the time I wrote this blog, the newest version of NSCA was v2.7.2-8. This is why I repackaged it as v2.7.2-9 to include my enhancements. I created 2 patches along with the spec file enhancements.
    nrpe.conf.d.patch was created to provide a working NRPE configuration right out of the box (as soon as it was installed) and nrpe.xinetd.logrotate.patch was created to pre-configure a working xinetd server configuration.
  • NRPE RPM SPEC File: Here is the enhanced spec file I used (enhancing the one already provided in the EPEL release found on pkgs.org). At the time I wrote this blog, the newest version of NRPE was v2.14-5. However v2.15 was available off of the Nagios website so this is why I repackaged it as v2.15-1 to include my enhancements.
    nsca.xinetd.logrotate.patch was the only patch I needed to create to prepare a NSCA xinetd server working out of the box.

Everything else packaged (patches and all) are the same ones carried forward from previous versions by their package managers.

Rebuild your external monitoring solutions:
Below shows the long way of rebuilding the RPMs from source.

# Perhaps make a directory and work within it so it's easy to find
# everything later
mkdir nagiosbuild
cd nagiosbuild
###
# Now we want to download all the requirements we need to build
###
# Prepare our mock environment
###
# Initialize Mock Environment
mock -v -r epel-6-x86_64 --init

# NRPE (v2.15)
wget --output-document=nrpe-2.15-1.el6.src.rpm https://www.dropbox.com/sh/9dt7klam6ex1kpp/4de9MardB2/20131208/nrpe-2.15-1.el6.src.rpm?dl=1 
mock -v -r epel-6-x86_64 --copyin nrpe-2.15-1.el6.src.rpm /builddir/build

# NSCA (v2.7.2)
wget --output-document=nsca-2.7.2-9.el6.src.rpm https://www.dropbox.com/sh/9dt7klam6ex1kpp/bEzldQuPoM/20131208/nsca-2.7.2-9.el6.src.rpm?dl=1 
mock -v -r epel-6-x86_64 --copyin nsca-2.7.2-9.el6.src.rpm /builddir/build

#######################
### THE SHORT WAY #####
#######################
# Now, the short way to rebuild everything is through these commands:
mock -v -r epel-6-x86_64 --resultdir=$(pwd)/results \
   --rebuild  nrpe-2.15-1.el6.src.rpm  nsca-2.7.2-9.el6.src.rpm

# You're done; You can find all of your rpms in a results directory
# in the same location you typed the above command in.  You can 
# alternatively rebuild everything the long way allowing you to
# inspect the content in more detail and even change it for your
# own liking

#######################
### THE LONG WAY  #####
#######################
# Install NRPE Dependencies
mock -v -r epel-6-x86_64 --install \
   autoconf automake libtool openssl-devel tcp_wrappers-devel

# Install NSCA Dependencies
mock -v -r epel-6-x86_64 --install \
   tcp_wrappers-devel libmcrypt-devel

###
# Build Stage
###
# Shell into our enviroment
mock -v -r epel-6-x86_64 --shell

# Change to our build directory
cd builddir/build

# Install our SRPMS (within our mock jail)
rpm -Uhi nsca-*.src.rpm nrpe-*.src.rpm

# Now we'll have placed all our content in the SPECS and SOURCES
# directory (within /builddir/build).  Have a look to verify
# content if you like

# Build our RPMS
rpmbuild -ba SPECS/*.spec

# we're now done with our mock environment for now; Press Ctrl-D to
# exit or simply type exit on the command line of our virtual
# environment
exit

###
# Save our content that we built in the mock environment
###

#NRPE
mock -v -r epel-6-x86_64 --copyout /builddir/build/SRPMS/nrpe-2.15-1.el6.src.rpm .
mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nrpe-2.15-1.el6.x86_64.rpm .
mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nagios-plugins-nrpe-2.15-1.el6.x86_64.rpm .
mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nrpe-debuginfo-2.15-1.el6.x86_64.rpm .

#NSCA
mock -v -r epel-6-x86_64 --copyout /builddir/build/SRPMS/nsca-2.7.2-9.el6.src.rpm .
mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nsca-2.7.2-9.el6.x86_64.rpm .
mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nsca-client-2.7.2-9.el6.x86_64.rpm .
mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nsca-debuginfo-2.7.2-9.el6.x86_64.rpm .

# *Note that all the commands that interact with mock I pass in 
# the -v which outputs a lot of verbose information. You don't
# have to supply it; but I like to see what is going on at times.

# **Note: You may receive this warning when calling the '--copyout'
# above:
# WARNING: unable to delete selinux filesystems 
#    (/tmp/mock-selinux-plugin.??????): #
#    [Errno 1] Operation not permitted: '/tmp/mock-selinux-plugin.??????'
#
# This is totally okay; and is safe to ignore, the action you called
# still worked perfectly; so don't panic!

So where do I go from here?
NRPE and NSCA are both fantastic solutions that can allow you to tackle any monitoring problem you ever had. In this blog here I focus specifically on Linux, but these tools are also available on Microsoft Windows as well. You can easily have 1 Nagios Server manage thousands of remote systems (of all operating system flavours). There are hundreds of fantastic tools to monitor all mainstream applications used today (Databases, Web Servers, etc). Even if your trying to support a custom application you wrote. If you can interface with your application using the command line interface, well then Nagios can monitor it for you. You only need to write a small script with this in mind:

  • Your script should always have an exit code of 0 (zero) if everything is okay, 1 (one) if you want to raise a warning, and 2 (two) if you want to raise a critical alarm.
  • No matter what the exit code is, you should also echo some kind of message that someone could easily interpret what is going on.

There is enough information in this blog to do the rest for you (as far as creating a Nagios configuration entry for it goes). If you followed the 2 rules above, then everything should ‘just work’. It’s truely that easy and powerful.

How do I decide if I need NSCA or NRPE?

NRPE & NSCA High Level Overview

NRPE & NSCA High Level Overview


NRPE makes it Nagios’s responsibility to check your application where as NSCA makes it your applications responsible to report its status. Both have their pros and cons. NSCA could be considered the most secure approach because at the end of the day the only port that requires opening is the one on the Nagios server. NSCA does not use a completely secure connection (but there is encryption none the less). NRPE is very secure and doesn’t require you to really do much since it just simply works with the nagios-plugins already available. It litterally just extends these existing Nagios local checks to remote ones. NSCA requires you to configure a cron, or adjust your applications in such a way that it frequently calls the send_nsca command. NSCA can be a bit more difficult to set up but creates some what of a heartbeat between you and the system monitoring it (which can be a good thing too). I pre-configured the NSCA server with a small tweak that will automatically set your application to a critical state if a send_nsca call is missed for an extended period of time.

Always consider that the point of this blog was to state that you can use both at the same time giving you total flexibility over all of your systems that require monitoring.

Credit
All of the custom packaging in this blog was done by me personally. I took the open source available to me and rebuilt it to make it an easier solution and decided to share it. If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

Sources
I referenced the following resources to make this blog possible:

  • The blog I wrote earlier that is recommended you read before this one:Configuring and Installing Nagios Core 4 on CentOS 6
  • Official NRPE download link; I used all of the official documentation to make the NRPE references on this blog possible.
  • A document identifying the common errors you might see and their resolution here.
  • Official NSCA download link; I used all of the official documentation to make the NSCA references on this blog possible.
  • The NRPE and NSCA images I’m reposting on this blog were taking straight from their official sites mentioned above.
  • Linux Packages Search (pkgs.org) was where I obtained the source RPMs as well as their old SPEC files. These would be a starting point before I’d expand them.
  • A bit outdated, but a great (and simple) representation of how NSCA works with Nagios can be seen here.

Creating a Swap File System for Your VPS


Introduction
RAM is expensive when compared to physical hard disk (storage) in today’s market. Combine this knowledge with Virtual Private Server (VPS) solutions offered by vendors such as Amazon EC2 who will literately give us our own Linux box in The Cloud. They’ll also grant us full root level privileges to do whatever we want in these new sandboxes of ours. These vendors usually provide their product to us with a ton of disk space and very little RAM. That said, the more cash we’re willing to shell out of our wallet, the more ram and disk space we’ll receive in return (just like everything else in our consumer driven world).

If you chose a VPS as your preliminary (or final) hosting solution, then you (like everybody else) are on a budget. There is a good chance you’re working with a very minimal amount of RAM (probably 512MB to 4GB) regardless of how much cash you forked out of your pocket. Now you could argue that this is all you need, but this is where I say: you still need swap space no matter what.

This is truly worth blogging about because if you install just about any Linux distribution, it will (usually) set up a swap partition for you as part of it’s default out of the box settings. But most VPS solutions out there provide you with a pre-configured distribution of Linux that is not equipped with any swap space whatsoever. I wrote this blog because popular VPS vendors such as Linode and Digital Ocean do not provide swap space with their virtual machine (or at least at the time of writing this).

Did you ever think that maybe they don’t provide it because you don’t need it?
No, that’s simply not the case. The fact of the matter is; they offer packages on their website such as 1GB of RAM and 20GB of storage space and that is exactly what they give you. Could you just imagine how frustrated you’d be as a customer if you paid for that deal and found out you actually only had 512MB (0.5GB) of physical RAM while the remaining 512MB was just your swap partition? That would be just false advertising in my opinion. The issue could get even more complicated then that… would that mean they’d be providing you with 20GB of storage space minus (-) 512MB because it’s reserved for this swap partition? Thankfully at the end of the day we get what we pay for. In this example we’d have purchased a Linux distribution with 1 root partition (/) that is exactly 20GB in size along with 1GB of physical RAM. We could type the command free -m and see this 1GB of physical RAM for ourselves.

Why do I need swap space?
There are a few reasons:

  • You grant your system more (virtual) memory (RAM) to work with. This is especially useful in dire situations when your memory may be almost exhausted. Remember, if you run out of RAM, the services running on your server WILL start to fail on you.
  • You will free up physical RAM while swap space is present.

    Consider this: you’re running a program that has 200 features in it, but you’re only using 10 of them. Without swap space, all 200 features may (and most probably will) be loaded into physical (expensive) memory; even the features you’re not using! But with a swap file/partition, the system will detect the sections of the code not being referenced/used (in RAM) and move it into swap space. This process free’s up physical memory for other services you want to run. The program won’t care you’re doing this; as far as it’s concerned, the resources are still available to it (which they are).

    In other words: The operating system is smart enough to note the frequency of code being executed. These pieces of accessed code will always remain in memory while less frequently accessed code will be swapped to the (swap) disk until it is needed again. The process is called paging when the operating system goes back into the swap space to retrieve this data (returning it to RAM for the program that requested it). We have some control over the swapping process (to limit paging) which I talk about later (swappiness).

Swap files are slow and can cause extensive i/o:
Swapping content from RAM back to a file storage (and back) is slow; there is no question about this. I also will not argue with you that this process does generate i/o. You obviously want to minimize the swapping of the content that you’re quite sure will be used again. But you still never want to eliminate it completely as it plays a huge role in optimizing your system.

In fact, the cons of swap space only outweigh it’s benefits if you abuse it by treating it as if it were additional physical RAM.

Hence you DO NOT create virtual swap space because you are out of memory! You create a swap space to optimize system resources. You create swap space to provide a safety net for your applications during peak operation times. You create swap space to increase system performance by freeing up sections of memory that are rarely being used.

If you only paid for 0.5GB (512MB) of RAM from your VPS provider, then do not consider yourself as having 2.5GB of RAM after you create a 2GB swap file. Sure, let your applications think this, but you yourself don’t want to exceed this 0.5GB memory boundaries any more then you have to. Obviously you’re creating more (virtual) RAM so that you can venture into the higher ranges in dire situations, but thats it. Otherwise you’re not using swap space effectively.

A Final Word of Caution…
With some vendors such as Amazon EC, you’ll pay for the disk i/o you generate. So improper use of swap space would work against you (costing you). Meanwhile companies like Crissic Solutions go out of their way to micro-manage the virtual machines they host. They specifically look for people violating disk i/o unnecessarily and are quick to launch their own internal investigations to see if you’re abusing their privileges. It’s this tight environment they enforce on everyone that allows them to promise their clients a very responsive experience.

Word of Caution

Just a Final Word of Caution

If you decide you need swap space (as I did) along with your services (and you should), then do not treat your results as free RAM.

The decision you make in choosing how much extra RAM you will give yourself with a swap file will not decide if you intend to abuse your hosting privileges. However the amount of this new free memory you intend to exhaust at once could… so just be considerate! Most vendors will generally just trust that you are not doing anything malicious; in fact you acknowledged this as part of the Terms of Service when you first signed up. So with that said, please don’t abuse the information in this blog or your VPS provider WILL catch you if their system disk i/o spikes for long durations of time. Trust me when I say: “it won’t take rocket science to track the source to be your VPS if you abuse their services.”

Its also worth noting that if you already have a swap partition on your system, then it’s really not worth adding another. You probably want to consider adding or paying for more RAM if your system is still struggling.

How much swap space do I need then?
This really is the magic question, there are lots of people who have tons of different considerations for this. Personally, it honestly depends on what your going to run. Some applications utilize an insane amount of ram up front, but once they’re past their start up phase, they only use a fraction of RAM. Meanwhile, other systems are just generally memory hogs.

At the end of the day, the size really isn’t important. In fact it’s probably safer to have to much then too little. I mean if you had 16GB of physical RAM in your VPS, I’d (arbitrarily) say create a 10GB swap partition. This allows you to run a TON of applications and all of them have plenty of space to dump their unused code to. It also leaves you with a bit of a buffer (of extra RAM) in critical situations. If you have 512MB to 4GB of physical RAM, then I’d say create a partition that is about 2GB in size. Disk space is cheap and 2GB really isn’t that much.

Again, the decision is really up to you… more space is better then less (but too much just becomes wasted disk space). There is also no reason for your swap space to ever be larger then your actual alloted physical RAM. You’ll want to watch and make sure that too much paging is taking place at the end of the day regardless of what size you pick. Excessive paging means you’ve exhausted your physical RAM and you’re relying on the swap space you created to fill the void. You need to consider paying for more RAM at this point or not running so many applications on the Virtual Machine your VPS provided you.

Commands
Here is how you can create your own swap space (as a swap file) on an already running system.

# Creating a directory to host our swap file
mkdir -p /var/swap

# Agree on a size... in my case I want a 2GB buffer.
# Now ideally this would be the easy to read command to run
# as the following:
#   dd if=/dev/zero of=/var/swap/swap.fs bs=2G count=512
#
# But.. no... it's not that simple, the above command would
# need 2GB of ram which you may or may not have (yet).... so we
# need to work with a smaller block size and a higher count to
# achieve the same results.
#
# The below command is more i/o intensive but less memory
# intensive which is the situation for must of us reading this today.
#
# 1024MB = 1GB
# (1024 * 1024MB * 2) = 2097152 block size
#
dd if=/dev/zero of=/var/swap/swap.fs bs=1024 count=2097152

# Format swap file (allowing it to be a recognizable swap space
# partition)
mkswap -L swapfs /var/swap/swap.fs

# You may see a warning about including -f; don't bother... you're
# done after the above command, Now you want to protect
# the new file you created
chmod 600 /var/swap/swap.fs
chmod 100 /var/swap

# Update fstab file:
echo '/var/swap/swap.fs swap swap defaults 0 0'  >> /etc/fstab

# Enable the swap partition
swapon /var/swap/swap.fs

You’re done… You can use the following command to test to see if your partition got activated or not:

# Test to see swap correctly activated itself:
free -m

Swappiness
Yes, swappiness is a word… at least it is in respect to disk swapping/paging it is. Your system is most likely set to the same level of swappiness as the rest of us have (60). Here is how you can check what yours is set to:

[root@node01 ~]# cat /proc/sys/vm/swappiness
60

Swappiness is a number from 0 to 100 where the higher values lead to more pages being swapped, and lower values lead to more applications being kept in memory, even if they are idle. Generally 60 is pretty safe to use, but if this isn’t your machine that is hosting it. You may want to tweak this value a bit. Bring it down a bit so you only swap in situations where you absolutely have to. This will minimizing the i/o impact on the server provided to you at such a great cost.

This can be done using the following command:

# Note this is temporary and will be lost on a reboot
echo 40 > /proc/sys/vm/swappiness

Or if your happy with your change, you can permanently save this change by updating your /etc/sysctl.conf file:

# first make sure an existing entry doesn't already exist
sed -i -e '/[ \t]*vm\.swappiness/d' /etc/sysctl.conf

# Now create it with our entry
echo "vm.swappiness=40" >> /etc/sysctl.conf

# Now future reboots will hold this setting for you

It’s not really necessary to play with the default swappiness value unless this isn’t your server you just created a swap file on. Then it becomes a nice way to minimize unnecessary i/o when using a VPS as your solution.

Credit
If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

Sources
Swap file generation is really well documented everywhere and certainly isn’t a new concept. My goal was just to bring it to your attention that most VPS solutions do not provide you with any swap space at all initially and you should consider adding it right away. Here are some helpful links on swapping:

Virtual Private Server (VPS)
I made reference to a few of them in this blog; I don’t want to promote one or the other. In fact, I’m really not up to advertising for any of them. But since I referenced them in the blog, it’s only right of me to include them and more at the end for reference.

Linux Security Solution

Securing Your CentOS 6 System


Introduction
Security is important these days and there are a scary amount of blogs and comments from people out there who solve their problems by turning off the firewall and/or disabling SELinux. Not only am I against this, I’m going to explain how you can use them in this blog. I’ll also suggest some other ways of adding some security to your system.

Here are the topics I’ll cover in this blog

  • Fail2Ban
    Intrusive detection and prevention
  • Firewall
    The front line to the internet (where all the bad stuff happens)
  • Restricting SSH Access
    A couple tweaks to help bulletproof yourself from possible intrusion.
  • System Auditing
    Find out what got accessed and by who and what major system calls were made.
  • System Monitoring
    Monitor your system for key things that could be symptoms of a hacked or compromised system.
  • Enable SELinux
    This is literally your last line of defense, and its a lot better then people make it out to be.
  • Disk Quotas
    Prevent a user from violating disk space on a production system.

Fail2Ban
For those of you who run a system on the front end of the internet have to deal with firewalls and security. Fail2Ban is a python based tool that wraps itself around iptables. It’s can cleverly watch system log files and detect abuse; it then proceeds to temporary (or permanently) block/ban the culprit by using their IP address in conjunction with your firewall.

At the time I blogged this, Fail2Ban was in the pre-release stages of v0.8.11. For this reason I’ll focus on v0.8.10.

Get the Software
Now the EPEL Repositories already provide us with software to do the installation here (and source rpm here).

Alternatively, I rebuilt the source myself and am hosting it here (in case the version changes significantly enough that this tutorial no longer works). My hosted version of version of Fail2Ban rpm can be retrieved here (and source rpm here).

Setting Up Fail2Ban

# Configure epel (if not already)
rpm -Uhi http://fedora.mirror.nexicom.net/epel/6/i386/epel-release-6-8.noarch.rpm

# Install the goods
yum -y install fail2ban

# Optionally install 'jwhois'. This tool gives you great detail on people
# accessing your system by performing a whois lookup on them and including
# the results of this in the automated email sent. Honestly it's worth it.
yum -y install jwhois

Out of the box the settings will work with SSHD only and for now that is all I need. However I’ve made it a bit more restrictive to satisfy my needs:

# var/log/messages is busy enough; I prefer to use
# /var/log/fail2ban.log personally (it's already pre-configured
# with a logrotate.d entry to accomodate this)
sed -i -e 's|^logtarget[ \t]=.*|logtarget = /var/log/fail2ban.log|g' \
	/etc/fail2ban/fail2ban.conf

# The below ;'sed' commands only change the first occurence in the file
# # which is where the [default] is identified
# Default Ban for 1 day (86400 seconds = 1 day)
sed -i -e '0,/^bantime[ \t]*=.*/s//bantime = 86400/' \
	/etc/fail2ban/jail.conf

# Default Ban if more then 3 unsuccessful attempts are made within 30 min
# (1800 seconds = 30 min).
sed -i -e '0,/^maxretry[ \t]*=.*/s//maxretry = 3/' \
	/etc/fail2ban/jail.conf
sed -i -e '0,/^findtime[ \t]*=.*/s//findtime = 1800/' \
	/etc/fail2ban/jail.conf

# These IPs don't conform to our very restrictive checks and will
# bypass the Fail2Ban security. For my own personal privacy, I've
# adjusted them from my own personal IPs. You might want to change
# this to reflect what you want (use spaces to delimit them). You
# can use masks too such as 10.128.3.0/16 (adding the slash (/)).
# but remember you need to escape (\) the slash in the below
# sed statement.  For example, the below will add the following:
#     127.0.0.1, 1.2.3.4, and 7.8.9.0/24
#
# Note: At a minimum, make sure to include 127.0.0.1
sed -i -e '0,/^ignoreip[ \t]*=.*/s//ignoreip = 127.0.0.1 1.2.3.4 7.8.9.0\/24/' \
	/etc/fail2ban/jail.conf

# Configure the system to start fail2ban after every reboot
chkconfig --levels 345 fail2ban on

# Start it up now for the first time (use 'restart' instead
# of 'start)' if it was already running:
service fail2ban start

There are lots of configurations already available that ship with this tool (but disabled by default). Have a look at /etc/fail2ban/jails.conf, perhaps there are others you might be interested in. If you’re uncertain what some of them are, or what they do; just have a look at the fail2ban manual.

Firewall / iptables
This literally your front line runner to all the security between you and the violent internet. Your firewall is your only shield and one of the last line of defense you have in some scenarios. SELinux would be the last line of defense you have which I talk about later. If troubleshooting a product has boiled down to stopping your firewall, then you’ve done something horribly wrong. There are other ways to debug firewall issues and stopping it shouldn’t be one of them.

Try running the following command just to see what ports your system is already listening on:

# The below lists all listening connections that could be
# being access remotely if your firewall is disabled
#
netstat -pnat | egrep LISTEN | \
   tr -s ' ' | cut -f4,7 -d' ' | sed '/^127\.0.0/d'

It’s the above list you’re trying to protect! It’s not uncommon for an application to communicate to another through ip (such as a database does); but these ports do not need to be open to the entire internet!

Setting Up a Simple Firewall
Here is a very simple firewall you can use to get you started:

# Set this to your internet interface (it might be ppp0 too)
# run ifconfig to see what interfaces you have
PUBLIC=eth0
cat << _EOF > /etc/sysconfig/iptables
#
# A Simple firewall that allows access to SSH and Web Based inbound
# connections but will allow you to access everything outside
#
*filter
#---------------------------------------------------------------
# Drop Everything by default
#---------------------------------------------------------------
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]

#---------------------------------------------------------------
# Internal Traffic is Okay
#---------------------------------------------------------------
-A INPUT -i lo -j ACCEPT
-A OUTPUT -o lo -j ACCEPT

#---------------------------------------------------------------
# Always accept already established connections
#---------------------------------------------------------------
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

#---------------------------------------------------------------
# Deny traffic from internet that spoofs addresses used internally
#---------------------------------------------------------------
-A INPUT -i $PUBLIC -s 192.168.0.0/24 -j DROP
-A INPUT -i $PUBLIC -s 127.0.0.0/8 -j DROP
-A INPUT         -d 10.0.0.0/8 -j DROP
-A INPUT         -d 169.254.0.0/16 -j DROP

#---------------------------------------------------------------
#  All Outbound Traffic Accepted (for now)
#---------------------------------------------------------------
-A OUTPUT -o $PUBLIC -j ACCEPT

#---------------------------------------------------------------
# SSH Allowed
#---------------------------------------------------------------
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

#---------------------------------------------------------------
# Web Traffic Allowed
#---------------------------------------------------------------
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT

#---------------------------------------------------------------
# The default is to drop everything else
# but for read-ability and peace of mind
# we force it again anyway
#---------------------------------------------------------------
-A INPUT -i $PUBLIC -j DROP
-A FORWARD -i $PUBLIC -j DROP

# End
COMMIT
_EOF

# Now restart iptables for the new rules to take affect
service iptables restart

# If you're running fail2ban then you'll need to restart it too
# since the extra chains it creates into iptables would have just
# got wiped with the last command.  Restarting it will rebuild
# everything the way it should be
service fail2ban restart

# Ensure this file is not accessible by anyone
chmod 600 /etc/sysconfig/iptables

When or if you add a new program into your environment, it should just work… if you need to share or host it’s services to others, find out what ports it uses and ‘ONLY’ open them. Consider the security risks that become available to you once these ports are open to the world as well. Hence you are putting all your trust in the application listening on every port you open. Also note that every port your open is a potential point of entry a hacker can use into your system.

Restrict SSH Access
SSH is a great way for you to connect remotely to your server and see how things are going, make changes etc. But consider other people (whom you don’t know) might be trying to access it also (usually for malicious reasons). If you’ve set up fail2ban already, then you’re already in really good shape. But consider restricting the the SSH Daemon even more for precautionary reasons. Here is what I’m suggesting:

  • Disable remote SSH access for the root user login:
    # Before you do this, be sure you have an non-root account you
    # can still connect to the system as that you will use instead of
    # root
    useradd nuxref
    # I always add my users to the users group, you don't have to do
    # this:
    usermod -G users nuxref
    # Set a password for the user you created
    passwd nuxref
    
    # In the above example I can use 'nuxref' as an entrance into the
    # system to which I can switch to the root after I establish a
    # my connection.
    # 90% of the constant connections your server will face when
    # directly connected to the internet will be as the 'root' user.
    # So why even bother enabling that account? Fail2Ban will end up
    # blocking them shortly anyway, but why even give them 3 lucky
    # guesses? It's really not worth it. It's much safer to use another
    # account and switch to root if needed later.
    
    # This also means that if for some miraculous reason someone
    # guesses your non-root account to gain access to the system, the
    # damage they can do will be as minimal as the access you've given
    # that account.
    
    # Now Deny Root Login Attempts
    sed -i -e 's/^[# \t]*PermitRootLogin .*/PermitRootLogin no/g' /etc/ssh/sshd_config
    
    # To prevent the system from allowing additional users you add to
    # your system from accessing it remotely you'll want to consider
    # doing the following:
    # First get rid of an existing entry
    sed -i -e '/^[# \t]*AllowUsers .*/d' /etc/ssh/sshd_config
    # Now add our user restrictions (if you have more then one
    # user you want to add, separate them using spaces
    echo "AllowUsers nuxref" >> /etc/ssh/sshd_config
    
  • Consider using a banner message as a warning to let people know they’ve in-invterantly accessed a system they shouldn’t. Sure a hacker will ignore this message, but for the poor fellow who really did just mistype an ip or host; this will give your server some character and notify them that you are monitoring them. Nothing but a scare tactic; but it’s still worth doing.
    # A simple scare banner
    cat << _EOF > /etc/banner
    * - - - - - - - W A R N I N G - - - - - - - - - - W A R N I N G - - - - - - - *
    *                                                                             *
    * The use of this system is restricted to authorized users. All information   *
    * and communications on this system are subject to review, monitoring and     *
    * recording at any time, without notice or permission.                        *
    *                                                                             *
    * Unauthorized access or use shall be subject to prosecution.                 *
    *                                                                             *
    * - - - - - - - W A R N I N G - - - - - - - - - - W A R N I N G - - - - - - - *
    _EOF
    
    # Make sure it's not accessible by others
    chmod 640 /etc/banner
    
    # SELinux Handling
    restorecon /etc/banner
    
    # Now tell the SSH Daemon to reference it:
    sed -i -e 's|^[# \t]*Banner .*|Banner /etc/banner|g' /etc/ssh/sshd_config
    
  • Disable X11 forwarding and Tcp Forwarding; These are what hackers will want to utilize if they ever successfully gain access to your system:
    # X11 grants someone the ability to launch X applications locally
    # at their machine that are associated with your server.  In some
    # cases this is okay. But if you're just hosting web servers and
    # databases; you shouldn't offer free candy to a potential unknown
    # users who may have just connected to your production system.
    sed -i -e 's|^X11Forwarding .*|X11Forwarding no|g' /etc/ssh/sshd_config
    # Yet another service that just shouldn't be open no a production
    # system. Someone is trying to access something they couldn't
    # otherwise have done it if they're invoking this. Not saying this
    # feature isn't powerful, I'm just saying disable it until you
    # find a reason not to.
    sed -i -e 's|^[#]*AllowTcpForwarding .*|AllowTcpForwarding no|g' /etc/ssh/sshd_config
    
  • Disable UseDNS reference. This is optional; I just do this because it’s faster. Also IPs are easier to scan later. The DNS lookup can slow things down sometimes which can be annoying. This isn’t a security thing at all; it just makes things faster.
    # Disable UseDNS
    sed -i -e 's|^[# \t]*UseDNS .*|UseDNS no|g' /etc/ssh/sshd_config
    

When you’re comfortable with all your changes, you can restart the SSH Daemon to take them on. If you were logged in remotely already, don’t worry, you won’t lose your connection when you do this.

# Restart SSHD so it will reread it's configuration 
service sshd restart

Enable Auditing
If you have to resort to auditing, then your system may have already been compromised, but this will play a key role in figuring out what happened. Hopefully you’ll never have to rely on this step, but being cocky and going without it might become problematic in the future. This tool will help make your system better in the long run.

But auditing gives you something more as well; once it’s enabled you can set up the monitoring of it’s log file (/var/log/audit/audit.log) for suspicious activity. You can even go as far to write your own plugin for Fail2Ban to study the audit.log and react if certain suspicious system transaction takes place by a non-root user.

Auditing should be a considered manditory service you impliment on all of your servers.

Setting Up Some Simple Working Audit Rules
Here is a quick audit file you can use (and tailor to your liking) just to get you started; even if you added nothing else to this file, it’s configuration as is may save you one day:

# Install Auditing if it isn't already (most systems install this out of
# the box)
yum -y install audit
# Ensure your system will always run it
chkconfig --levels 345 auditd on
# Start it if it isn't already started:
service auditd status || service auditd start
# Install Audit Rules
cat << _EOF > /etc/audit/audit.rules
# First rule - delete all
-D

# increase the buffers to survive stress events. make this bigger for
# busy systems.
-b 1024

# monitor unlink() and rmdir() system calls.
-a exit,always -F arch=x86_64 -S unlink -S rmdir
# settimeofday so we know no one is adjusting the system times
-a exit,always -F arch=x86_64 -S settimeofday
# setrlimit.* so we know when kernel resources are being adjusted
-a exit,always -F arch=x86_64 -S setrlimit
# Filesystem Mounting (umount = 32bit, umount2 = 64bit)
-a exit,always -F arch=x86_64 -S mount -S umount2

#Ensure that failed use of the following system calls is audited
-a exit,always -F arch=x86_64 -S quotactl -S kill -S chroot -F success=0 -F auid=-1 -F auid=0

# some file and directory watches
-w /var/log/audit/ 
-w /etc/audit/auditd.conf -p rxwa
-w /etc/audit/audit.rules -p rxwa

# monitor write-access and change in file properties (read/write/execute)
# of the following files.
-w /etc/group -p wa
-w /etc/passwd -p wa
-w /etc/shadow -p wa
-w /etc/sudoers -p wa

# monitor write-access to the following directories
-w /etc/fail2ban -p wa
-w /etc/httpd -p wa
-w /etc/cron.d -p wa
-w /var/www -p wa

# lock the audit configuration to prevent any modification of this file.
#-e 2
_EOF

# Restart Audit To take on new configuration
service auditd restart

Now you can check for changes as root using commands like:

# Search for anyone touching the /etc/passwd file
ausearch -f /etc/passwd
# Search for anyone accessing the /etc/fail2ban/fail2ban.conf
ausearch -f /etc/fail2ban/fail2ban.conf

Don’t forget to uncomment the very last line of the /etc/audit/audit.rules if you’re using sample configuration file I have above as your template. The -e 2 will prevent someone from turning the auditing off before they wreck havoc on your system. the -e 2 will make it so a reboot is required for the rules to change. This IS what you want; trust me! Hackers aren’t stupid; disabling auditing is the first thing they’ll attempt before they begin creating their chaos.

Monitoring Strategies
I already wrote a tutorial on using Nagios here. Consider using this; it can even be configured to monitor the audit logs and set alarms off when something unusual is matched.

But consider monitoring things such as the following as well:

  • The System Load
  • Remaining Disk Space
  • Bandwidth Utilization
  • Number of Users Remotely Logged in

All of the suggestions above can help you quickly identify unusual behaviour and allow you to take action immediately (instead of just the next time you happen to be on the system). Obviously you’ll know your system better than anyone else, so if you expect the system load to go high at night for certain tasks, you can consider this in your monitoring as well.

It’s the times that nothing should be out of the ordinary that an alarm could help you resolve a problem just minutes (if not seconds) after it occurs. Monitoring also works in your favour for checking other system commands such as web page availability (if you’re hosting one) etc. Find out something is wrong before your very own customers do is the key here.

In fact, there really isn’t a good reason you should ever consider snuffing monitoring from your security TODO list.

Enable SELinux
There are a lot of people who seem to be really against using SELinux when in fact it is totally amazing and really easy to work with once you understand it. The biggest problem most people have is they don’t ever take the time to learn it. Hell, even I’ll admit it’s a frustrating learning curve in it’s unknown state.

But honestly: it effectively allows you to revoke sections of your file system as well as major system calls based on the executable code (not just the user/group). Why is this so important? Well take Apache for example. If someone were to successfully compromise it from the outside (using a buffer overflow), they will have gained full user access granted to the apache user and can literally browse your entire system. Their goal would be to potentially look for other exploits that they can use to gain higher privileges (such as root access). Heck even as the Apache user, you can run quite a lot of programs including ones that access the internet.

With respect to the Apache example just explained, if the administrator had SELinux running (in Enforcing Mode) the hacker is restricted to only reading and accessing the permissions assigned to the /usr/sbin/httpd binary. At most they’d be able to view your Apache configuration file and read html files… That’s about it.

SELinux is an amazing tool for locking down your system. In fact, SELinux is your very last line of defence. If this isn’t running and an application gets compromised, you’re going to have to rely completely on your audit logs to find what damage was done to your system and attempt to repair it (hopefully from backups).

I won’t lie though, the big problem with SELinux is that the documentation is poor. Not to mention that no one really promotes it’s fantastic functionality. Instead everyone just disables it and moves on.

One of the simplest things you can do upfront when trying to work with SELinux is just keep it in enforcing mode. RHEL/CentOS packages have already looked after most of the settings for you. In the rare case something doesn’t work for you; instead of panicking and disabling SELinux (which all websites tell you to do). Set it to ‘Permissive’ mode instead. This is effectively the same as disabling it except it still records violations in /var/log/audit/audit.log what it would have otherwise denied.

# Set SELinux into Permissive mode
setenforce 0

At this point your application (that may have not worked prior to this call) may suddenly work again. It is at this point you you can begin seeing what SELinux was denying by referencing these very audit logs. These logs play a key in making everything work for you again.

Making a New Product SELinux Compatible
Take Nagios for example and the blog I did for it. This is really easy to protect and still have SELinux running in Enforcing mode. Assuming you set SELinux in Permissive mode already (explained above), you can now generate a fast set of rules to allow the product to work with it enabled. Here is how you can do it:

# First make sure you have the right package installed to work
# with:
yum install -y policycoreutils

# For alarms to be generated, you'll want to run Nagios with
# SELinux in Permissive mode for a while (preferably a day would
# be great to get everything) 

# Filter out only nagios alarms that were generated by SELinux
cat /var/log/audit/audit.log | grep denied | \
	grep nagios > nagios.audit.log

# Now tailor the file if necessary (strip out lines you don't
# want to grant access to). Use a simple editor (like vi) to do
# this. It's really not that hard to read, and the output will
# show you precisely what nagios tried to access.  In some
# cases, you may never want nagios to access these things. so
# remove these entries from this list. Only keep the denied
# messages you want to reverse (and allow)

# Create an installable Module (prefix with 'my' to avoid
# conflicts with other package management that it might be using)
audit2allow -v --input nagios.audit.log -M mynagios

## Install the new module you just created
semodule -i mynagios.pp

That’s it; now you can set SELinux back to enforcing mode for the added security it offers you:

# Set SELinux back to Enforcing mode
setenforce 1

Was that really so hard? Every now and then grep through your audit logs (like you already did above) and scan for the keyword ‘nagios’. It’s possible it may still try to do things to which it’s being denied access to. You may even need to update your mynagios.pp SELinux module with extra entries found using audit2allow again. Consider also that it may be doing things you’re quite happy to leave the way it is. You don’t need to grant it access to absolutely everything, just want is necessary for it to operate.

Remember; if your project is compromised it will try to do all sorts of things it shouldn’t do and SELinux will be the barrier that will save your system.

Enable User Disk Quotas
Even if you’re running SELinux and have locked your system with auditing. A hacker can still try to bring your system down completely by filling it’s disk space until it’s full. This can cause other applications to catastrophically fail or stop behaving as they should which will is not cool at all.

By enabling user disk quota’s you can restrict the amount disk space imposed by a user (and even group level). Above, in the "Restricting SSH Access" points, I created a user called nuxref who i’d use as the entry point into my system. If you followed that piece of advice and significantly restricted remote access to your system you’re already halfway there. But… in the extremely unlikely circumstances that remote entry account gets compromised, you will want to be sure you keep the hackers options to a minimum.

Since I’ll only use that account to access the system and either check a few things or switch to the root user to make a change, I want to eliminate it’s ability to do much anything else. Most hackers start transferring all their hacking tools onto the system the second they gain access. It’s through these tools they can exploit more content. Other hackers might just want to completely fill your disk space causing your production data to crash.

An Example When You Might Want To Consider Disk Quotas:
Some cloud hosting services that provide you a bare bone virtual machine do not partition/carve out it’s disks in ways you’d have otherwise done differently. They do this primarily because they usually offer such small amounts of disk space (like 20-40GB) and it would be unfair of them to carve it in a way you might complain about. It’s better for them to just give your the entire space as one partition. But hey! When you pay for a very cheap hosting service; you take what you get. :)

Here is an example of what one of the virtual hosting providers I use did to my system when they gave it to me:

[root@node01 ~]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/vda              20642428   1437052  18156800   8% /
none                    251228         0    251228   0% /dev/shm

In this example, /home (where users log into) and /tmp are not carved into their own partition. These directories are critical because all users usually have full read/write permissions here. If this were your production environment (and it is for many), a user could completely fill these unprotected directories until the file system was full. So in this example, Disk Quotas would be a very good idea!

Setting Disk Quotas Up
I will set up a assuming the file system I have to work with is identical to that displayed above (where I only have a ‘/’ partition) and a single user account that is accessible by SSHD. I’ll keep using the nuxref account to make things consistent.

Up until now I also gave you ‘one-liners’ to automate everything by simply copying and pasting from your browser to your command line. Well this one here will differ across systems, so it’s not really fair for me to give you this.

  1. First make sure the correct software is installed onto your system:
    yum -y install quota
    
  2. Identify what partition needs to be updated. You need to open up your /etc/fstab file for this. I’m personally a vi fan, but use whatever editor your comforatble with.
    # Here is a before snapshot of the /etc/fstab file before I edited it:
    [root@node01 ~]# cat /etc/fstab             
    LABEL=DOROOT       /               ext4    errors=remount-ro 0       1
    none             /dev/shm      tmpfs   defaults                    0 0
    
  3. I want to adjust the ‘/’ partition (column 4); I need to update the line that already reads errors=remount-ro with errors=remount-ro,usrquota,grpquota. The fstab file will look like this for me when i’m finished:
    # Here is an after snapshot of the /etc/fstab file after I edited it:
    [root@node01 ~]# cat /etc/fstab 
    LABEL=DOROOT       /               ext4    errors=remount-ro,usrquota,grpquota 0       1
    none             /dev/shm      tmpfs   defaults                    0 0
    
  4. Now remount the ‘/’ filesystem (or whatever mount you chose to update in your /etc/fstab file) by typing the following:
    mount -o remount /
    
  5. Now allow your system to scan your filesystem for files so it can begin tracking them:
    quotacheck -avugm
    

    The step above could take hours depending on how busy your system is or the number of files that reside on the partition your scanning.

    You may see some output like this (don’t worry; nothing bad has happened) :)

    quotacheck: Scanning /dev/vda [/] done
    quotacheck: Cannot stat old user quota file: No such file or directory
    quotacheck: Cannot stat old group quota file: No such file or directory
    quotacheck: Cannot stat old user quota file: No such file or directory
    quotacheck: Cannot stat old group quota file: No such file or directory
    quotacheck: Checked 4404 directories and 30017 files
    quotacheck: Old file not found.
    quotacheck: Old file not found.
    

    You may also see a warning like this:

    quotacheck: Your kernel probably supports journaled quota but you are not using it. Consider switching to journaled quota to avoid running quotacheck after an unclean shutdown.
    

    Journaling is a way of managing the transactions that are written and removed from your hard disk. It makes for a faster and more promising data recovery should your server ever suffers a hard crash (power outage or kernel crash). Journals sure do have their merrits, but there are times when you don’t want them either. One reason is because your server is using a Solid State disk. These drives are SO much faster then traditional (mechanical) hardrives, but the trade off is they don’t like un-nessisary writes. Disabling journalling is a way of prologging the life of these kind of drives.

    In a nutshell… if you’re getting this warning about journaled quota being disabled, it’s not a bad thing. In the example I’m using, the ‘/’ (ext4) partition is infact residing on a Solid State hardrive, so i’m already fully aware of my scenario and know that the filesystem was formatted with out journaling enabled. So if it’s disabled already on your system too, think twice before enabling it; it is probably disabled for a reason and is completely safe to leave off (dispite this warning).

  6. Now we can activate the quota system:

    quotaon -av
    

    You should see the following output (/dev/vda will be replaced with whatever device your currently watching quotas with):

    /dev/vda [/]: group quotas turned on
    /dev/vda [/]: user quotas turned on
    
  7. Now you can look at the restrictions put in place:

    [root@node01 ~]# repquota -a
    *** Report for user quotas on device /dev/vda
    Block grace time: 7days; Inode grace time: 7days
                            Block limits                File limits
    User            used    soft    hard  grace    used  soft  hard  grace
    ----------------------------------------------------------------------
    root      -- 1212596       0       0          32284     0     0       
    postfix   --      60       0       0             38     0     0       
    nuxref    --      60       0       0             10     0     0 
    apache    --      12       0       0              3     0     0       
    postgres  --      16       0       0              4     0     0       
    

    The quota are identified under the soft and hard columns which are all zero’s (0). This means there are no restrictions in place at all.

  8. We want to impost a restriction on the nuxref user in my case. I want to restrict this user to 10MB as there is no excuse why I should need more then that if I’m only using the account to check on things or switch to root. This is done using the command edquota username which will pop up an editor allow you to change these fields.

    # Edit the nuxref user's quota
    edquota nuxref
    # Adjusted both the soft and hard limits to 8 and 10 respectively.
    
    [root@node01 ~]# repquota -a    
    *** Report for user quotas on device /dev/vda
    Block grace time: 7days; Inode grace time: 7days
                            Block limits                File limits
    User            used    soft    hard  grace    used  soft  hard  grace
    ----------------------------------------------------------------------
    root      -- 1212596       0       0          32284     0     0       
    postfix   --      60       0       0             38     0     0       
    nuxref    --      60    8000   10000             10     0     0 
    apache    --      12       0       0              3     0     0       
    postgres  --      16       0       0              4     0     0       
    

    As you can see, the nuxref user is now restricted to 10 MB. If soft limit (set to 8MB in this example) is reached, an email will be sent to the user and for a ‘grace’ period (default is 6 days), the user will still be able to write more content to this directory. But after the grace period is reached the ‘soft’ limit is identical to the ‘hard’ limit and restricts further writing (until you clean up some space).

    It’s worth noting too that I could have acomplished the exact same effect (without using an editor) as above with the following one liner:

    # If you script the following; make sure you specify the correct device!
    # /dev/vda is used below only because that follows inline with all my
    # examples above.
    setquota -u nuxref  8000 10000 0 0 /dev/vda
    

Disk Quotas Key Point If You Use Them:
I want to be clear about this: If you choose to use Disk Quotas, you only really need to lock down the accounts that ‘could’ be compromised. In my examples in this blog, I only have one account (nuxref). It’s not necessary to lock down system accounts like apache, postgres, mysql, etc. NEVER lock down the root user!. Most of you may not even need to do this step; It truly is an extreme (and in most cases unnecessary) precautionary step that only ‘some’ should take.

Credit
This blog took me a very long time to put together and test! If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

Sources
A lot of what I wrote here was accumulated knowledge over time. I can’t recall where all my sources came from and for that I’m sorry. This blog is more of a brain dump along content I keep in a personal Wiki containing information I’ve been meaning to share with everyone.

Here are some great links that may help you understand some topics discussed here a bit better though:

Nagios Solution

Configuring and Installing Nagios Core 4 on CentOS 6


Introduction
Nagios is a powerful tool that allows you to monitor just about anything. It’s so easy to even take existing legacy system and adapt it to work with Nagios. When I came across kingkiwix and his blog here I wanted to repackage the content in a more user-friendly way for sharing. I wanted a solution that didn’t require me to pull in a bunch of development libraries and compilers. I also wanted a solution that I could re-use on other systems through a version controlled and packaged interface.

RPM Solution
RPMs provide a version control and an automated set of scripts to configure the system how I want it. The beauty of them is that if you disagree with
something the tool you’re packaging does, you can feed RPMs patch files to accommodate it without obstructing the original authors intention.

Now I won’t lie and claim I wrote these SPEC files from scratch because I certainly didn’t. I took the stock ones that ship with both products (both Nagios and Nagios-Plugins) and modified them significantly to accommodate and satisfy my compulsive needs. :)

My needs required a bit more automation in the setup as well as including:

2013 Dec 1st, UPDATE: I have recently rebuilt Nagios to v4.0.2 and included one more small patch to the application available

  • The use of nagiocmd user group.
  • A /etc/nagios/conf.d directory to behave similar to how Apache works. I want to be able to drop configuration files into here and just have it work without re-adjusting configuration files.
  • Alerting Apache on new updates/installs so just installing the package alone is all you need to do.
  • Nagios Plugins should work right away once they are installed.
  • Nagios Plugins and permissions should adapt to the new nagiocmd user and place in a common directory Nagios was already configured to look in.
  • Nagios Plugins has to many dependencies; I wanted to break this up into separate packages for those who needed them. For example, I don’t use MySQL at all; so why should I need the MySQL Libraries installed on my system just to use Nagios Plugins.

Stop Babbling, Just Give Me All Your Hard Work
Of course here it is:

At a minimum you need to install both Nagios Core (v4.0.1) Nagios Core (v4.0.2) and it’s accompanied Nagios Plugins (v1.5) package.

For those who are interested, here is a quick direct link to my additions to the building environment and deployment:

  • Nagios Core 4 RPM SPEC
    (old SPEC) and a patch to allow it to access a conf.d directory. I additionally created a new patch for Nagios Core 4.0.2 to fix a bug that seems to cause a ton of files to accumulate in /tmp each time you start Nagios.
  • Nagios Plugin RPM SPEC file and a patch to additionally prepare and include some working configuration straight out of the box (taking advantage of the conf.d directory.

I Installed the Packages, Now What?
The RPMs take care of just about everything for you, so there isn’t really much to do at this point.

  • Make sure Apache is running and if it isn’t start it:
    # the following way is a harmless way of checking if Apache is
    # running and starting it if it wasn't (requires root)
    service httpd status || service httpd start
    
    # If it wasn't started in the above command, you may want to
    # consider having it start up each time you reboot your machine
    # through the following command:
    chkconfig httpd --levels 345 on
    
  • Start Nagios
    # Startup Nagios (requires root)
    service nagios start
    
    # Consider having Nagios (Backend) start up each time you reboot
    # your machine through the following command:
    chkconfig nagios --levels 345 on
    

Visiting http://localhost/nagios/ should look similar to this if you followed the steps correctly

Visiting http://localhost/nagios/ should look similar to this if you followed the steps correctly

Yes, that’s right… you’re done! You can access your Nagios Web Interface by visiting http://localhost/nagios. How you configure your system further is beyond the scope of this tutorial/blog. My goal was just to get you started with an out of the box working Nagios solution :).

If your not already familiar with Nagios, you may want to read the documentation so you can monitor more then just the 8 example checks it sets up for you out of the box. There are also some great web configuration tools you can check out as well here.

Package Information

  • Software installs to: /usr/share/nagios
  • Plugins install to: /usr/lib64/nagios/plugins. Keep in mind I packaged everything in such a way that compiling it for 32-bit will work fine. In which case you’ll find plugins installed to: /usr/lib/nagios/plugins
  • Website is accessible at: http://localhost/nagios/ out of the box, but you can adjust it’s web configuration by adjusting /etc/httpd/conf.d/nagios.conf to your likings.
  • Nagios Core 4 configuration file can be found at: /etc/nagios/nagios.cfg and a newly added /etc/nagios/conf.d/ directory has been added to make some things easier to add/remove later.
  • I want to repackage this myself
    All of my blogs provide a way to build everything from scratch yourself for those who don’t want to just download the binary packages and roll with them. I use mock for everything I build so I don’t need to haul in development packages into my native environment. You’ll need to make sure mock is setup and configured properly first for yourself:

    # Install 'mock' into your environment if you don't have it already.
    # This step will require you to be the superuser (root) in your native
    # environment.
    yum install -y mock
    
    # Grant your normal every day user account access to the mock group
    # This step will also require you to be the root user.
    usermod -a -G mock YourNonRootUsername
    

    At this point it’s safe to change from the ‘root‘ user back to the user account you granted the mock group privileges to in the step above. We won’t need the root user again until the end of this tutorial when we install our built RPM.

    # Perhaps make a directory and work within it so it's easy to find
    # everything later
    mkdir nagiosbuild
    cd nagiosbuild
    ###
    # Now we want to download all the requirements we need to build
    ###
    # Nagios Core 4
    wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-4.0.2.tar.gz
    # Nagios Plugins v1.5
    wget https://www.nagios-plugins.org/download/nagios-plugins-1.5.tar.gz
    # Customize Nagios Spec Files
    wget --output-document=nagios.spec https://www.dropbox.com/sh/9dt7klam6ex1kpp/NtFRlX1FuC/20131110/nagios4.0.2.spec?dl=1
    # Customize Nagios Plugin Spec Files
    wget --output-document=nagios-plugins.spec https://www.dropbox.com/sh/9dt7klam6ex1kpp/EB7Auruw87/20131110/nagios-plugins.spec?dl=1
    # Nagios Patch to enable a conf.d directory
    wget --output-document=nagios.conf.d.patch https://www.dropbox.com/sh/9dt7klam6ex1kpp/Jz9fNXE2X6/20131110/nagios.conf.d.patch?dl=1
    # Nagios Patch to fix accumulating files in /tmp/*
    wget --output-document=configtest.tmp.patch https://www.dropbox.com/sh/9dt7klam6ex1kpp/x1aQ7KfmwX/20131110/configtest.tmp.patch?dl=1
    # Nagios Plugin Patch to include some sample configurations and
    # setup Nagios to work out of the box
    wget --output-document=add.sample.command.cfg.patch https://www.dropbox.com/sh/9dt7klam6ex1kpp/7Wots9KUZ4/20131110/add.sample.command.cfg.patch?dl=1
    
    ###
    # Prepare our mock environment
    ###
    # Initialize Mock Environment
    mock -v -r epel-6-x86_64 --init
    
    # Copy in our downloaded content:
    mock -v -r epel-6-x86_64 --copyin nagios-4.0.2.tar.gz \
       nagios-plugins-1.5.tar.gz \
       nagios.conf.d.patch \
       add.sample.command.cfg.patch \
       onfigtest.tmp.patch \
       /builddir/build/SOURCES
    
    mock -v -r epel-6-x86_64 --copyin nagios.spec nagios-plugins.spec \
       /builddir/build/SPECS
    
    # Install Dependencies
    mock -v -r epel-6-x86_64 install gd-devel zlib-devel \
       libpng-devel libjpeg-devel doxygen gperf perl-devel net-snmp-devel
    
    # Install Plugin Dependencies
    mock -v -r epel-6-x86_64 install bind-utils mysql-devel net-snmp-utils \
                   postgresql-devel ntp openldap-devel openssh-clients \
                   samba-client /usr/bin/mailq
    
    ###
    # Build Stage
    ###
    # Shell into our enviroment
    mock -v -r epel-6-x86_64 --shell
    
    # Change to our build directory
    cd builddir/build
    
    # Build our RPMS
    rpmbuild -ba SPECS/nagios.spec
    rpmbuild -ba SPECS/nagios-plugins.spec
    
    # we're now done with our mock environment for now; Press Ctrl-D to
    # exit or simply type exit on the command line of our virtual
    # environment
    exit
    
    ###
    # Save our content that we built in the mock environment
    ###
    mock -v -r epel-6-x86_64 --copyout /builddir/build/SRPMS/nagios-4.0.2-2.el6.src.rpm .
    mock -v -r epel-6-x86_64 --copyout /builddir/build/SRPMS/nagios-plugins-1.5-1.el6.src.rpm .
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nagios-plugins-1.5-1.el6.x86_64.rpm .
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nagios-plugins-mysql-1.5-1.el6.x86_64.rpm .
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nagios-plugins-pgsql-1.5-1.el6.x86_64.rpm .
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nagios-plugins-ldap-1.5-1.el6.x86_64.rpm .
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nagios-plugins-debuginfo-1.5-1.el6.x86_64.rpm .
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nagios-4.0.2-2.el6.x86_64.rpm .
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nagios-contrib-4.0.2-2.el6.x86_64.rpm .
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nagios-devel-4.0.2-2.el6.x86_64.rpm .
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/nagios-debuginfo-4.0.2-2.el6.x86_64.rpm .
    
    # *Note that all the commands that interact with mock I pass in 
    # the -v which outputs a lot of verbose information. You don't
    # have to supply it; but I like to see what is going on at times.
    
    # **Note: You may receive this warning when calling the '--copyout'
    # above:
    # WARNING: unable to delete selinux filesystems 
    #    (/tmp/mock-selinux-plugin.??????): #
    #    [Errno 1] Operation not permitted: '/tmp/mock-selinux-plugin.??????'
    #
    # This is totally okay; and is safe to ignore, the action you called
    # still worked perfectly; so don't panic :)
    

    Considerations
    If you take Nagios seriously, you will really want to consider expanding it to support one (or all) of the following addons:

    • NRPE - Nagios Remote Plugin Executor

      NRPE – Nagios Remote Plugin Executor

      NRPE – Nagios Remote Plugin Executor
      NRPE is an addon that allows you to execute plugins on remote Linux/Unix hosts. This is useful if you need to monitor local resources/attributes like disk usage, CPU load, memory usage, etc. on a remote host. Similiar functionality can be accomplished by using the check_by_ssh plugin, although it can impose a higher CPU load on the monitoring machine – especially if you are monitoring hundreds or thousands of hosts.
    • NSCA - Nagios Service Check Acceptor

      NSCA – Nagios Service Check Acceptor

      NSCA – Nagios Service Check Acceptor
      NSCA is an addon that allows you to send passive check results from remote Linux/Unix hosts to the Nagios daemon running on the monitoring server. This is very useful in distributed and redundant/failover monitoring setups.

    I’m considering packaging these as well to conform with the packages I’ve already provided here, but I’ll do that in another blog some other time if there ends up being a demand for it :).

    Dec 8th, 2013 Update: I ended up creating this blog to extend this one: Configuring and Installing NRPE and NSCA into Nagios Core 4 on CentOS 6.

    Credit
    If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

    Sources
    I referenced the following resources to make this blog possible: