Tag Archives: SELinux

Nagios Core 4.x Setup for CentOS 7.x

Introduction

A few years ago I blogged about setting up Nagios Core 4.x for CentOS 6.x. I later made a blog on setting up NSCA and NRPE too. But times have changed so I figured it was time to do an updated blog for CentOS 7.x and Nagios.

Some new great tools I’ve packaged to make a load and go setup for everyone is:

  • Nagios Core v4.x: updated RPM packaging which I continued to maintain and carry forward from my previous blogs.
  • Nagios Plugins v2.x: A ton of out of the box working plugins.

Best of all, with my RPMs, you can run SELinux in full Enforcing mode for that extra piece of mind from a security standpoint!

Nagios Core

Nagios (for those who don’t know) is an application that allows us to monitor other system/applications we manage. It’s primary function is to immediately bring to our attention any outage or anomaly is detected with our systems. This tool is completely free and should be an essential component of anyone’s business infrastructure.

The current version of Nagios (at the time of writing this blog) is v4.2.2. You can download the latest version from my repository (if you’re set up) as follows:

# Install Nagios Core using NuxRef repositories
# at: https://nuxref.com/repo
yum install -y nagios nagios-selinux

You can also download the packages manually if you wish using this table:

Package Download Description
nagios el7.rpm Nagios Core IV is the the actual monitoring server we can use to monitor our applications.
nagios-selinux el7.rpm An add-on package that allows you to run Nagios in Enforcing Mode.

Note: This RPM is not required by Nagios to run correctly.
nagios-contrib el7.rpm Extra tools that add to the great features Nagios already offers (such as distributed monitoring). These tools are not discussed in this blog entry; but maybe useful to you.
Note: This RPM is not required by Nagios to run correctly.
nagios-devel el7.rpm Header files for developers who want to build using the libnagios shared library.
Note: This RPM is not required by Nagios to run correctly.

Note: The source rpm can be obtained here which builds everything you see in the table above. It’s not required for the application to run, but might be useful for developers or those who want to inspect how I put the package together.

Configure Nagios Core

Once Nagios is installed it can be started like so:

# Start Nagios
systemctl start nagios.service

# If you want Nagios to start after the system is rebooted, you
# can type the following:
systemctl enable nagios.service

# Now we want to turn on Apache if it's not running, otherwise
# reload the configuration if it is:
systemctl status httpd.service && \
   systemctl start httpd.service ||\
   systemctl reload httpd.service

If you followed the instructions above you should be able to access the (Nagios) monitoring website right away by visiting http://localhost/nagios. If you’re installing this on another server you may need to open your web ports to access the Nagios Monitoring site:

# The following commands should be ran on your Nagios Server.
# It will enable our http (and secure https) port on our firewall so our
# monitoring website can be accessed remotely:
firewall-cmd --permanent --add-service=http
firewall-cmd --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --add-service=https
Nagios Credentials
Login nagiosadmin
Password nagiosadmin

You’ll be prompted for a user/pass combination; at this time the default values are defined in the table:

Once you’ve logged in, you can click on links like Services (under the Current Status heading) which will list to you all of the system services you’re currently monitoring and their status:

Nagios > Current Status > Services
Nagios > Current Status > Services

In the example, you can see Nagios has picked up on a high system load (denoted by the yellow warning entry).

Nagios at it’s most basic level is set up at this point and can be maintained by having a look at the following directories:

/etc/nagios/nagios.cfg:
The main configuration file that is read by Nagios when starting up. The only things you may want to change in here are:

Directive Description
date_format The default is us (MM-DD-YYYY HH:MM:SS), but personally I like iso8601 (YYYY-MM-DD HH:MM:SS).
# simple 1-liner to toggle date_format to iso8601:
sed -i -e 's/^\(date_format\)=.*$/\1=iso8601/g' \
   /etc/nagios/nagios.cfg
check_for_updates The default is 1 (which is to check for updates). Personally, I don’t want my web page pinging Nagios every time i access the website for updates. Here is how you can do the same:
# disable update check:
sed -i -e 's/^\(check_for_updates\)=.*$/\1=0/g' \
   /etc/nagios/nagios.cfg

/etc/nagios/objects/contacts.cfg:
This is where a default contact has been created. Feel free to open this up and add your name and (especially the) email address.
/etc/nagios/objects/commands.cfg:
All of the possible checks Nagios can perform on it’s own (without any plugins or extensions) are identified here. It’s as easy as defining a command_name (give it some name) and then tell it what you want to execute with the command_line directive.

A Nagios command is really simple to write; you can write one in any language you want. Here is one written in bash shell:

#!/bin/bash
# Path: /usr/lib64/nagios/plugins/check_temp_file_count
# Please keep in mind this script is pretty useless but
# it's goal is to show you how easy it is to write a
# script for Nagios.
#
# The rules are simple:
# Whatever we echo to the screen get's displayed to Nagios
# and our return value from our script/program will
# determine the color code (and whether or not we alarm)

# A return code of zero (0) tells Nagios everything is okay
RET_OKAY=0

# A return code of one (1) tells Nagios we're reporting a warning
# about whatever it is we're monitoring
RET_WARN=1

# A return code of two (2) tells Nagios we're reporting a critical
# error
RET_CRIT=2

# I define 3 here, but quite honestly, anything you return that
# does not fit in the 0, 1 or 2 response type (as identified
# above) is considered an unknown state.  Try to avoid this
# type if you can.
RET_UNKN=3

# As a test script we'll count all of the files and directories
# in the /tmp folder
COUNT=$(find /tmp 2>/dev/null | wc -l)

# If we have less then 10 files we'll tell Nagios everything
# is okay:
if [ $COUNT -lt 10 ]; then
   echo "$COUNT files; everything is good!"
   exit $RET_OKAY
fi

# If we have less then 30 files we'll tell Nagios that
# it should report a warning
if [ $COUNT -lt 30 ]; then
   echo "$COUNT files; caution!"
   exit $RET_WARN
fi

# Anything more we should report a critical alarm
echo "$COUNT files; Critical!!"
exit $RET_CRIT

If you’re familiar with Perl, the Nagios team has made a framework with it which you use to make your packages too! I’ve already gone ahead and packaged the perl-Nagios-Monitoring-Plugin rpm for you if you want it.

Nagios will associate /usr/lib64/nagios/plugins/ as it’s plugin directory; so you should save any plugins you create there (to keep them all in a common location). Plus if you intend to use SELinux, this is the directory that Nagios is allowed to execute from.

Our entry in the commands.cfg for this new script might look like this:

; 'check_temp_file_count' command definition
; $USER1$ gets translated automatically to our Nagios Plugin directory
; so in our case: /usr/lib64/nagios/plugins/
; Ideally you should call the command the same name as the checking
; tool you wrote.
define command{
	command_name	check_temp_file_count
	command_line	$USER1$/check_temp_file_count
	}

/etc/nagios/objects/localhost.cfg:
This is just a general configuration file for the very machine Nagios is running on. If you open it up, you’ll see that it defines a lot of entries that reference commands (defined already in the commands.cfg file).

A new entry in the commands.cfg for this new script might look like this:

; the 'use' directive identifies a template of information to save
; us from typing it all out here.  For now; just leave this as
; local-service.
;
; The 'hostname' defines our server (defined at the very top of this same
; file. Since the host at the to was defined as 'localhost', we need
; to use this same name here too
;
; The next field is just a description field; it will be how this service
; is presented on Nagios through the website
;
; The check_command is the name we gave it in the commands.cfg
; file.
define service{
        use                             local-service
        host_name                       localhost
        service_description             Count our Temporary Files
        check_command                   check_temp_file_count
}

Check to see if Nagios has any errors with any new configuration you provided:

# This command just tells Nagios to read in it's configuration
# and check if it appears valid:
nagios -v /etc/nagios/nagios.cfg

If everything checks out okay, go ahead and reload Nagios with our
new configuration:

# Reload Nagios
systemctl reload nagios.service

Nagios Plugins

If you’ve installed Nagios, there isn’t really any good reason why you shouldn’t just install the Nagios Plugins too. This is just more tools and checking scripts to make Nagios all that more powerful. The best part is, these tools have been tested over the years, so they’re already proven to be reliable and will allow you to accomplish most monitoring without much effort.

Nagios Plugins
Nagios Plugins

It’s important to note that the Nagios Plugin RPMs are NOT required by Nagios to run correctly. They merely just improve it’s existing functionality. You may however want to install the plugins you’re interested in that monitor systems you’re using.

The current version of the Nagios Plugins (at the time of writing this blog) is v2.1.3. You can download the latest version from my repository (if you’re set up) as follows:

# Install Nagios Core using NuxRef repositories
# See: https://nuxref.com/repo for more information
yum install -y nagios-plugins nagios-plugins-selinux

You can also download the packages manually if you wish using this table:

Package Download Description
nagios-plugins el7.rpm 50+ plugins that are fully adaptable to Nagios in every way. If you’re planning on installing Nagios, don’t forget about adding this package for it’s convenience!
nagios-plugins-selinux el7.rpm An optional add-on package that allows you to use the Nagios Plugins in Enforcing Mode.
nagios-plugins-ldap el7.rpm A Nagios plugin that can be used to check integrity and data entries within an LDAP database.
nagios-plugins-mysql el7.rpm A Nagios plugin that can be used to check integrity and data entries within an MySQL (or Maria) database.
nagios-plugins-ntp el7.rpm A Nagios plugin that can be used to check the NTP status of the machine it’s called on.
nagios-plugins-pgsql el7.rpm A Nagios plugin that can be used to check integrity and data entries within an PostgreSQL database.
nagios-plugins-samba el7.rpm A Nagios plugin that can be used to check status of your Samba mounts and their availability.
nagios-plugins-snmp el7.rpm A Nagios plugin that can query SNMP enabled appliances (routers, firewalls, switches, servers) and convert their output back to something Nagios can monitor or report.

Note: The source rpm can be obtained here which builds everything you see in the table above. It’s not required for the application to run, but might be useful for developers or those who want to inspect how I put the package together.

The main thing to know about this package after it is installed is the slew of new plugins available to you in /usr/lib64/nagios/plugins/ and a config file to get you started which references most of them in /etc/nagios/conf.d/nagios-plugin-commands.cfg.

Extra Plugins

There are a lot of great plugins on Nagios Exchange! I packaged just a few of them because they required patches and tweaks to work out of the box. All of these are available on my repository, but feel free to haul them down directly here:

Package Download Description
nagios-plugins-lvm el7.rpm / src.rpm / NE Source This plugin finds all LVM logical volumes, checks their used space, and compares against the supplied thresholds.
nagios-plugins-crm el7.rpm / src.rpm / NE Source A plugin for monitoring a Pacemaker/Corosync cluster.
Note: that this plugin requires perl-Nagios-Monitoring-Plugin to work.
nagios-plugins-drbd84 el7.rpm / src.rpm / NE Source A plugin for monitoring a DRBD v8.4 setup.

Credit

This blog took me a very (,very) long time to put together and test! The repository hosting alone accommodates all my blog entries up to this date. All of the custom packaging in this blog was done by me personally. If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

Sources

The remaining portions of this series can be found here:

  • Part 2 – NRDP for Nagios Core on CentOS 7.x: This blog explains how awesome NRDP really is and why it might become a vital asset to your own environment. It’s also provides the first set of working RPMs (with SELinux support of course) of it’s kind.
  • Part 3 – NRPE for Nagios Core on CentOS 7.x: This blog explains how to set up NRPE (v3.x) for your Nagios environment. At the time this blog was written, there was no packaging of it’s kind for this version. So allow me be the first to share it with you!

Secure and Protect Your CentOS 6 System

Introduction

Security is important these days but sadly there are a scary amount of blogs and comments from people out there who solve their problems by turning off their firewall and/or disabling SELinux. These steps may work around a problem, but it will make your system much more vulnerable to cyber attacks. Learning alternative ways of solving your applications woes without disabling key safety components on your system is a better approach to keeping it secure. The goal of this blog is to offer a quick primer on some key security components that will protect you an your data!

The topics covered in this blog to keep your systems secure are:

  • Fail2Ban
    Intrusive detection and prevention
  • Firewall
    The front line to the internet (where all the bad stuff happens)
  • Restricting SSH Access
    A couple tweaks to help bulletproof yourself from possible intrusion.
  • System Auditing
    Find out what got accessed and by who and what major system calls were made.
  • System Monitoring
    Monitor your system for key things that could be symptoms of a hacked or compromised system.
  • Enable SELinux
    This is literally your last line of defense, and its a lot better then people make it out to be.
  • Disk Quotas
    Prevent a user from violating disk space on a production system.

Fail2Ban

For those of you who run a system on the front end of the internet have to deal with firewalls and security. Fail2Ban is a python based tool that wraps itself around iptables. It’s can cleverly watch system log files and detect abuse; it then proceeds to temporary (or permanently) block/ban the culprit by using their IP address in conjunction with your firewall.

At the time I blogged this, Fail2Ban was in the pre-release stages of v0.8.11. For this reason I’ll focus on v0.8.10.

Get the Software
Now the EPEL Repositories already provide us with software to do the installation here (and source rpm here).

Alternatively, I rebuilt the source myself and am hosting it here (in case the version changes significantly enough that this tutorial no longer works). My hosted version of version of Fail2Ban rpm can be retrieved here (and source rpm here).

Setting Up Fail2Ban

# Configure epel (if not already)
rpm -Uhi http://fedora.mirror.nexicom.net/epel/6/i386/epel-release-6-8.noarch.rpm

# Install the goods
yum -y install fail2ban

# Optionally install 'jwhois'. This tool gives you great detail on people
# accessing your system by performing a whois lookup on them and including
# the results of this in the automated email sent. Honestly it's worth it.
yum -y install jwhois

Out of the box the settings will work with SSHD only and for now that is all I need. However I’ve made it a bit more restrictive to satisfy my needs:

# Create a backup of our original configuration files before we apply
# any changes to them:
[ ! -f /etc/fail2ban/fail2ban.conf.orig ] && \
   cp -af /etc/fail2ban/fail2ban.conf /etc/fail2ban/fail2ban.conf.orig
[ ! -f /etc/fail2ban/jail.conf.orig ] && \
   cp -af /etc/fail2ban/jail.conf /etc/fail2ban/jail.conf.orig

# var/log/messages is busy enough; I prefer to use
# /var/log/fail2ban.log personally (it's already pre-configured
# with a logrotate.d entry to accommodate this)
sed -i -e 's|^logtarget[ t]=.*|logtarget = /var/log/fail2ban.log|g' \
	/etc/fail2ban/fail2ban.conf

# The below ;'sed' commands only change the first occurrence in the file
# # which is where the [default] is identified
# Default Ban for 1 day (86400 seconds = 1 day)
sed -i -e '0,/^bantime[ t]*=.*/s//bantime = 86400/' \
	/etc/fail2ban/jail.conf

# Default Ban if more then 3 unsuccessful attempts are made within 30 min
# (1800 seconds = 30 min).
sed -i -e '0,/^maxretry[ t]*=.*/s//maxretry = 3/' \
	/etc/fail2ban/jail.conf
sed -i -e '0,/^findtime[ t]*=.*/s//findtime = 1800/' \
	/etc/fail2ban/jail.conf

# These IPs don't conform to our very restrictive checks and will
# bypass the Fail2Ban security. For my own personal privacy, I've
# adjusted them from my own personal IPs. You might want to change
# this to reflect what you want (use spaces to delimit them). You
# can use masks too such as 10.128.3.0/16 (adding the slash (/)).
# but remember you need to escape () the slash in the below
# sed statement.  For example, the below will add the following:
#     127.0.0.1, 1.2.3.4, and 7.8.9.0/24
#
# Note: At a minimum, make sure to include 127.0.0.1
sed -i -e '0,/^ignoreip[ t]*=.*/s//ignoreip = 127.0.0.1 1.2.3.4 7.8.9.0\/24/' \
	/etc/fail2ban/jail.conf

# Configure the system to start fail2ban after every reboot
chkconfig --levels 345 fail2ban on

# Start it up now for the first time (use 'restart' instead
# of 'start)' if it was already running:
service fail2ban start

There are lots of configurations already available that ship with this tool (but disabled by default). Have a look at /etc/fail2ban/jails.conf, perhaps there are others you might be interested in. If you’re uncertain what some of them are, or what they do; just have a look at the fail2ban manual.

Firewall / iptables

This literally your front line runner to all the security between you and the violent internet. Your firewall is your only shield and one of the last line of defense you have in some scenarios. SELinux would be the last line of defense you have which I talk about later. If troubleshooting a product has boiled down to stopping your firewall, then you’ve done something horribly wrong. There are other ways to debug firewall issues and stopping it shouldn’t be one of them.

Try running the following command just to see what ports your system is already listening on:

# The below lists all listening connections that could be
# being access remotely if your firewall is disabled
#
netstat -pnat | egrep LISTEN | 
   tr -s ' ' | cut -f4,7 -d' ' | sed '/^0.0.0/d'

It’s the above list you’re trying to protect! It’s not uncommon for an application to communicate to another through ip (such as a database does); but these ports do not need to be open to the entire internet!

Setting Up a Simple Firewall
Here is a very simple firewall you can use to get you started:

# Set this to your internet interface (it might be ppp0 too)
# run ifconfig to see what interfaces you have
PUBLIC=eth0
cat << _EOF > /etc/sysconfig/iptables
#
# A Simple firewall that allows access to SSH and Web Based inbound
# connections but will allow you to access everything outside
#
*filter
#---------------------------------------------------------------
# Drop Everything by default
#---------------------------------------------------------------
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]

#---------------------------------------------------------------
# Internal Traffic is Okay
#---------------------------------------------------------------
-A INPUT -i lo -j ACCEPT
-A OUTPUT -o lo -j ACCEPT

#---------------------------------------------------------------
# Always accept already established connections
#---------------------------------------------------------------
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

#---------------------------------------------------------------
# Deny traffic from internet that spoofs addresses used internally
#---------------------------------------------------------------
-A INPUT -i $PUBLIC -s 192.168.0.0/24 -j DROP
-A INPUT -i $PUBLIC -s 127.0.0.0/8 -j DROP
-A INPUT         -d 10.0.0.0/8 -j DROP
-A INPUT         -d 169.254.0.0/16 -j DROP

#---------------------------------------------------------------
#  All Outbound Traffic Accepted (for now)
#---------------------------------------------------------------
-A OUTPUT -o $PUBLIC -j ACCEPT

#---------------------------------------------------------------
# SSH Allowed
#---------------------------------------------------------------
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

#---------------------------------------------------------------
# Web Traffic Allowed
#---------------------------------------------------------------
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT

#---------------------------------------------------------------
# The default is to drop everything else
# but for read-ability and peace of mind
# we force it again anyway
#---------------------------------------------------------------
-A INPUT -i $PUBLIC -j DROP
-A FORWARD -i $PUBLIC -j DROP

# End
COMMIT
_EOF

# Now restart iptables for the new rules to take affect
service iptables restart

# If you're running fail2ban then you'll need to restart it too
# since the extra chains it creates into iptables would have just
# got wiped with the last command.  Restarting it will rebuild
# everything the way it should be
service fail2ban restart

# Ensure this file is not accessible by anyone
chmod 600 /etc/sysconfig/iptables

When or if you add a new program into your environment, it should just work… if you need to share or host it’s services to others, find out what ports it uses and ‘ONLY’ open them. Consider the security risks that become available to you once these ports are open to the world as well. Hence you are putting all your trust in the application listening on every port you open. Also note that every port your open is a potential point of entry a hacker can use into your system.

Restrict SSH Access

SSH is a great way for you to connect remotely to your server and see how things are going, make changes etc. But consider other people (whom you don’t know) might be trying to access it also (usually for malicious reasons). If you’ve set up fail2ban already, then you’re already in really good shape. But consider restricting the the SSH Daemon even more for precautionary reasons. Here is what I’m suggesting:

  • Disable remote SSH access for the root user login:
    # Before you do this, be sure you have an non-root account you
    # can still connect to the system as that you will use instead of
    # root
    useradd nuxref
    # I always add my users to the users group, you don't have to do
    # this:
    usermod -G users nuxref
    # Set a password for the user you created
    passwd nuxref
    
    # In the above example I can use 'nuxref' as an entrance into the
    # system to which I can switch to the root after I establish a
    # my connection.
    # 90% of the constant connections your server will face when
    # directly connected to the internet will be as the 'root' user.
    # So why even bother enabling that account? Fail2Ban will end up
    # blocking them shortly anyway, but why even give them 3 lucky
    # guesses? It's really not worth it. It's much safer to use another
    # account and switch to root if needed later.
    
    # This also means that if for some miraculous reason someone
    # guesses your non-root account to gain access to the system, the
    # damage they can do will be as minimal as the access you've given
    # that account.
    
    # Now Deny Root Login Attempts
    sed -i -e 's/^[# t]*PermitRootLogin .*/PermitRootLogin no/g' /etc/ssh/sshd_config
    
    # To prevent the system from allowing additional users you add to
    # your system from accessing it remotely you'll want to consider
    # doing the following:
    # First get rid of an existing entry
    sed -i -e '/^[# t]*AllowUsers .*/d' /etc/ssh/sshd_config
    # Now add our user restrictions (if you have more then one
    # user you want to add, separate them using spaces
    echo "AllowUsers nuxref" >> /etc/ssh/sshd_config
    
  • Consider using a banner message as a warning to let people know they’ve inadvertently accessed a system they shouldn’t. Sure a hacker will ignore this message, but for the poor fellow who really did just mistype an ip or host; this will give your server some character and notify them that you are monitoring them. Nothing but a scare tactic; but it’s still worth doing.
    # A simple scare banner
    cat << _EOF > /etc/banner
    * - - - - - - - W A R N I N G - - - - - - - - - - W A R N I N G - - - - - - - *
    *                                                                             *
    * The use of this system is restricted to authorized users. All information   *
    * and communications on this system are subject to review, monitoring and     *
    * recording at any time, without notice or permission.                        *
    *                                                                             *
    * Unauthorized access or use shall be subject to prosecution.                 *
    *                                                                             *
    * - - - - - - - W A R N I N G - - - - - - - - - - W A R N I N G - - - - - - - *
    _EOF
    
    # Make sure it's not accessible by others
    chmod 640 /etc/banner
    
    # SELinux Handling
    restorecon /etc/banner
    
    # Now tell the SSH Daemon to reference it:
    sed -i -e 's|^[# t]*Banner .*|Banner /etc/banner|g' /etc/ssh/sshd_config
    
  • Disable X11 forwarding and Tcp Forwarding; These are what hackers will want to utilize if they ever successfully gain access to your system:
    # X11 grants someone the ability to launch X applications locally
    # at their machine that are associated with your server.  In some
    # cases this is okay. But if you're just hosting web servers and
    # databases; you shouldn't offer free candy to a potential unknown
    # users who may have just connected to your production system.
    sed -i -e 's|^X11Forwarding .*|X11Forwarding no|g' /etc/ssh/sshd_config
    # Yet another service that just shouldn't be open no a production
    # system. Someone is trying to access something they couldn't
    # otherwise have done it if they're invoking this. Not saying this
    # feature isn't powerful, I'm just saying disable it until you
    # find a reason not to.
    sed -i -e 's|^[#]*AllowTcpForwarding .*|AllowTcpForwarding no|g' /etc/ssh/sshd_config
    
  • Disable UseDNS reference. This is optional; I just do this because it’s faster. Also IPs are easier to scan later. The DNS lookup can slow things down sometimes which can be annoying. This isn’t a security thing at all; it just makes things faster.
    # Disable UseDNS
    sed -i -e 's|^[# t]*UseDNS .*|UseDNS no|g' /etc/ssh/sshd_config
    

When you’re comfortable with all your changes, you can restart the SSH Daemon to take them on. If you were logged in remotely already, don’t worry, you won’t lose your connection when you do this.

# Restart SSHD so it will reread it's configuration 
service sshd restart

Enable Auditing

If you have to resort to auditing, then your system may have already been compromised, but this will play a key role in figuring out what happened. Hopefully you’ll never have to rely on this step, but being cocky and going without it might become problematic in the future. This tool will help make your system better in the long run.

But auditing gives you something more as well; once it’s enabled you can set up the monitoring of it’s log file (/var/log/audit/audit.log) for suspicious activity. You can even go as far to write your own plugin for Fail2Ban to study the audit.log and react if certain suspicious system transaction takes place by a non-root user.

Auditing should be a considered mandatory service you implement on all of your servers.

Setting Up Some Simple Working Audit Rules
Here is a quick audit file you can use (and tailor to your liking) just to get you started; even if you added nothing else to this file, it’s configuration as is may save you one day:

# Install Auditing if it isn't already (most systems install this out of
# the box)
yum -y install audit
# Ensure your system will always run it
chkconfig --levels 345 auditd on
# Start it if it isn't already started:
service auditd status || service auditd start
# Install Audit Rules
cat << _EOF > /etc/audit/audit.rules
# First rule - delete all
-D

# increase the buffers to survive stress events. make this bigger for
# busy systems.
-b 1024

# monitor unlink() and rmdir() system calls.
-a exit,always -F arch=x86_64 -S unlink -S rmdir
# settimeofday so we know no one is adjusting the system times
-a exit,always -F arch=x86_64 -S settimeofday
# setrlimit.* so we know when kernel resources are being adjusted
-a exit,always -F arch=x86_64 -S setrlimit
# Filesystem Mounting (umount = 32bit, umount2 = 64bit)
-a exit,always -F arch=x86_64 -S mount -S umount2

#Ensure that failed use of the following system calls is audited
-a exit,always -F arch=x86_64 -S quotactl -S kill -S chroot -F success=0 -F auid=-1 -F auid=0

# some file and directory watches
-w /var/log/audit/ 
-w /etc/audit/auditd.conf -p rxwa
-w /etc/audit/audit.rules -p rxwa

# monitor write-access and change in file properties (read/write/execute)
# of the following files.
-w /etc/group -p wa
-w /etc/passwd -p wa
-w /etc/shadow -p wa
-w /etc/sudoers -p wa

# monitor write-access to the following directories
-w /etc/fail2ban -p wa
-w /etc/httpd -p wa
-w /etc/cron.d -p wa
-w /var/www -p wa

# lock the audit configuration to prevent any modification of this file.
#-e 2
_EOF

# Restart Audit To take on new configuration
service auditd restart

Now you can check for changes as root using commands like:

# Search for anyone touching the /etc/passwd file
ausearch -f /etc/passwd
# Search for anyone accessing the /etc/fail2ban/fail2ban.conf
ausearch -f /etc/fail2ban/fail2ban.conf

Don’t forget to uncomment the very last line of the /etc/audit/audit.rules if you’re using sample configuration file I have above as your template. The -e 2 will prevent someone from turning the auditing off before they wreck havoc on your system. the -e 2 will make it so a reboot is required for the rules to change. This IS what you want; trust me! Hackers aren’t stupid; disabling auditing is the first thing they’ll attempt before they begin creating their chaos.

Monitoring Strategies

I already wrote a tutorial on using Nagios here. Consider using this; it can even be configured to monitor the audit logs and set alarms off when something unusual is matched.

But consider monitoring things such as the following as well:

  • The System Load
  • Remaining Disk Space
  • Bandwidth Utilization
  • Number of Users Remotely Logged in

All of the suggestions above can help you quickly identify unusual behavior and allow you to take action immediately (instead of just the next time you happen to be on the system). Obviously you’ll know your system better than anyone else, so if you expect the system load to go high at night for certain tasks, you can consider this in your monitoring as well.

It’s the times that nothing should be out of the ordinary that an alarm could help you resolve a problem just minutes (if not seconds) after it occurs. Monitoring also works in your favour for checking other system commands such as web page availability (if you’re hosting one) etc. Find out something is wrong before your very own customers do is the key here.

In fact, there really isn’t a good reason you should ever consider snuffing monitoring from your security TODO list.

Enable SELinux

There are a lot of people who seem to be really against using SELinux when in fact it is totally amazing and really easy to work with once you understand it. The biggest problem most people have is they don’t ever take the time to learn it. Hell, even I’ll admit it’s a frustrating learning curve in it’s unknown state.

But honestly: it effectively allows you to revoke sections of your file system as well as major system calls based on the executable code (not just the user/group). Why is this so important? Well take Apache for example. If someone were to successfully compromise it from the outside (using a buffer overflow), they will have gained full user access granted to the apache user and can literally browse your entire system. Their goal would be to potentially look for other exploits that they can use to gain higher privileges (such as root access). Heck even as the Apache user, you can run quite a lot of programs including ones that access the internet.

With respect to the Apache example just explained, if the administrator had SELinux running (in Enforcing Mode) the hacker is restricted to only reading and accessing the permissions assigned to the /usr/sbin/httpd binary. At most they’d be able to view your Apache configuration file and read html files… That’s about it.

SELinux is an amazing tool for locking down your system. In fact, SELinux is your very last line of defense. If this isn’t running and an application gets compromised, you’re going to have to rely completely on your audit logs to find what damage was done to your system and attempt to repair it (hopefully from backups).

I won’t lie though, the big problem with SELinux is that the documentation is poor. Not to mention that no one really promotes it’s fantastic functionality. Instead everyone just disables it and moves on.

One of the simplest things you can do upfront when trying to work with SELinux is just keep it in enforcing mode. RHEL/CentOS packages have already looked after most of the settings for you. In the rare case something doesn’t work for you; instead of panicking and disabling SELinux (which all websites tell you to do). Set it to ‘Permissive’ mode instead. This is effectively the same as disabling it except it still records violations in /var/log/audit/audit.log what it would have otherwise denied.

# Set SELinux into Permissive mode
setenforce 0

At this point your application (that may have not worked prior to this call) may suddenly work again. It is at this point you you can begin seeing what SELinux was denying by referencing these very audit logs. These logs play a key in making everything work for you again.

Making a New Product SELinux Compatible
Take Nagios for example and the blog I did for it. This is really easy to protect and still have SELinux running in Enforcing mode. Assuming you set SELinux in Permissive mode already (explained above), you can now generate a fast set of rules to allow the product to work with it enabled. Here is how you can do it:

# First make sure you have the right package installed to work
# with:
yum install -y policycoreutils

# For alarms to be generated, you'll want to run Nagios with
# SELinux in Permissive mode for a while (preferably a day would
# be great to get everything) 

# Filter out only nagios alarms that were generated by SELinux
cat /var/log/audit/audit.log | grep denied | 
	grep nagios > nagios.audit.log

# Now tailor the file if necessary (strip out lines you don't
# want to grant access to). Use a simple editor (like vi) to do
# this. It's really not that hard to read, and the output will
# show you precisely what nagios tried to access.  In some
# cases, you may never want nagios to access these things. so
# remove these entries from this list. Only keep the denied
# messages you want to reverse (and allow)

# Create an installable Module (prefix with 'my' to avoid
# conflicts with other package management that it might be using)
audit2allow -v --input nagios.audit.log -M mynagios

## Install the new module you just created
semodule -i mynagios.pp

That’s it; now you can set SELinux back to enforcing mode for the added security it offers you:

# Set SELinux back to Enforcing mode
setenforce 1

Was that really so hard? Every now and then grep through your audit logs (like you already did above) and scan for the keyword ‘nagios’. It’s possible it may still try to do things to which it’s being denied access to. You may even need to update your mynagios.pp SELinux module with extra entries found using audit2allow again. Consider also that it may be doing things you’re quite happy to leave the way it is. You don’t need to grant it access to absolutely everything, just want is necessary for it to operate.

Remember; if your project is compromised it will try to do all sorts of things it shouldn’t do and SELinux will be the barrier that will save your system.

Enable User Disk Quotas

Even if you’re running SELinux and have locked your system with auditing. A hacker can still try to bring your system down completely by filling it’s disk space until it’s full. This can cause other applications to catastrophically fail or stop behaving as they should which will is not cool at all.

By enabling user disk quota’s you can restrict the amount disk space imposed by a user (and even group level). Above, in the "Restricting SSH Access" points, I created a user called nuxref who i’d use as the entry point into my system. If you followed that piece of advice and significantly restricted remote access to your system you’re already halfway there. But… in the extremely unlikely circumstances that remote entry account gets compromised, you will want to be sure you keep the hackers options to a minimum.

Since I’ll only use that account to access the system and either check a few things or switch to the root user to make a change, I want to eliminate it’s ability to do much anything else. Most hackers start transferring all their hacking tools onto the system the second they gain access. It’s through these tools they can exploit more content. Other hackers might just want to completely fill your disk space causing your production data to crash.

An Example When You Might Want To Consider Disk Quotas:
Some cloud hosting services that provide you a bare bone virtual machine do not partition/carve out it’s disks in ways you’d have otherwise done differently. They do this primarily because they usually offer such small amounts of disk space (like 20-40GB) and it would be unfair of them to carve it in a way you might complain about. It’s better for them to just give your the entire space as one partition. But hey! When you pay for a very cheap hosting service; you take what you get. πŸ™‚

Here is an example of what one of the virtual hosting providers I use did to my system when they gave it to me:

[root@node01 ~]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/vda              20642428   1437052  18156800   8% /
none                    251228         0    251228   0% /dev/shm

In this example, /home (where users log into) and /tmp are not carved into their own partition. These directories are critical because all users usually have full read/write permissions here. If this were your production environment (and it is for many), a user could completely fill these unprotected directories until the file system was full. So in this example, Disk Quotas would be a very good idea!

Setting Disk Quotas Up
I will set up a assuming the file system I have to work with is identical to that displayed above (where I only have a ‘/’ partition) and a single user account that is accessible by SSHD. I’ll keep using the nuxref account to make things consistent.

Up until now I also gave you ‘one-liners’ to automate everything by simply copying and pasting from your browser to your command line. Well this one here will differ across systems, so it’s not really fair for me to give you this.

  1. First make sure the correct software is installed onto your system:
    yum -y install quota
    
  2. Identify what partition needs to be updated. You need to open up your /etc/fstab file for this. I’m personally a vi fan, but use whatever editor your comforatble with.
    # Here is a before snapshot of the /etc/fstab file before I edited it:
    [root@node01 ~]# cat /etc/fstab             
    LABEL=DOROOT       /               ext4    errors=remount-ro 0       1
    none             /dev/shm      tmpfs   defaults                    0 0
    
  3. I want to adjust the ‘/’ partition (column 4); I need to update the line that already reads errors=remount-ro with errors=remount-ro,usrquota,grpquota. The fstab file will look like this for me when i’m finished:
    # Here is an after snapshot of the /etc/fstab file after I edited it:
    [root@node01 ~]# cat /etc/fstab 
    LABEL=DOROOT       /               ext4    errors=remount-ro,usrquota,grpquota 0       1
    none             /dev/shm      tmpfs   defaults                    0 0
    
  4. Now remount the ‘/’ filesystem (or whatever mount you chose to update in your /etc/fstab file) by typing the following:
    mount -o remount /
    
  5. Now allow your system to scan your filesystem for files so it can begin tracking them:
    quotacheck -avugm
    

    The step above could take hours depending on how busy your system is or the number of files that reside on the partition your scanning.

    You may see some output like this (don’t worry; nothing bad has happened) πŸ™‚

    quotacheck: Scanning /dev/vda [/] done
    quotacheck: Cannot stat old user quota file: No such file or directory
    quotacheck: Cannot stat old group quota file: No such file or directory
    quotacheck: Cannot stat old user quota file: No such file or directory
    quotacheck: Cannot stat old group quota file: No such file or directory
    quotacheck: Checked 4404 directories and 30017 files
    quotacheck: Old file not found.
    quotacheck: Old file not found.
    

    You may also see a warning like this:

    quotacheck: Your kernel probably supports journaled quota but you are not using it. Consider switching to journaled quota to avoid running quotacheck after an unclean shutdown.
    

    Journaling is a way of managing the transactions that are written and removed from your hard disk. It makes for a faster and more promising data recovery should your server ever suffers a hard crash (power outage or kernel crash). Journals sure do have their merits, but there are times when you don’t want them either. One reason is because your server is using a Solid State disk. These drives are SO much faster then traditional (mechanical) hardrives, but the trade off is they don’t like unnecessary writes. Disabling journaling is a way of prologging the life of these kind of drives.

    In a nutshell… if you’re getting this warning about journaled quota being disabled, it’s not a bad thing. In the example I’m using, the ‘/’ (ext4) partition is in fact residing on a Solid State hardrive, so I’m already fully aware of my scenario and know that the filesystem was formatted with out journaling enabled. So if it’s disabled already on your system too, think twice before enabling it; it is probably disabled for a reason and is completely safe to leave off (despite this warning).

  6. Now we can activate the quota system:

    quotaon -av
    

    You should see the following output (/dev/vda will be replaced with whatever device your currently watching quotas with):

    /dev/vda [/]: group quotas turned on
    /dev/vda [/]: user quotas turned on
    
  7. Now you can look at the restrictions put in place:

    [root@node01 ~]# repquota -a
    *** Report for user quotas on device /dev/vda
    Block grace time: 7days; Inode grace time: 7days
                            Block limits                File limits
    User            used    soft    hard  grace    used  soft  hard  grace
    ----------------------------------------------------------------------
    root      -- 1212596       0       0          32284     0     0       
    postfix   --      60       0       0             38     0     0       
    nuxref    --      60       0       0             10     0     0 
    apache    --      12       0       0              3     0     0       
    postgres  --      16       0       0              4     0     0       
    

    The quota are identified under the soft and hard columns which are all zero’s (0). This means there are no restrictions in place at all.

  8. We want to impost a restriction on the nuxref user in my case. I want to restrict this user to 10MB as there is no excuse why I should need more then that if I’m only using the account to check on things or switch to root. This is done using the command edquota username which will pop up an editor allow you to change these fields.

    # Edit the nuxref user's quota
    edquota nuxref
    # Adjusted both the soft and hard limits to 8 and 10 respectively.
    
    [root@node01 ~]# repquota -a    
    *** Report for user quotas on device /dev/vda
    Block grace time: 7days; Inode grace time: 7days
                            Block limits                File limits
    User            used    soft    hard  grace    used  soft  hard  grace
    ----------------------------------------------------------------------
    root      -- 1212596       0       0          32284     0     0       
    postfix   --      60       0       0             38     0     0       
    nuxref    --      60    8000   10000             10     0     0 
    apache    --      12       0       0              3     0     0       
    postgres  --      16       0       0              4     0     0       
    

    As you can see, the nuxref user is now restricted to 10 MB. If soft limit (set to 8MB in this example) is reached, an email will be sent to the user and for a ‘grace’ period (default is 6 days), the user will still be able to write more content to this directory. But after the grace period is reached the ‘soft’ limit is identical to the ‘hard’ limit and restricts further writing (until you clean up some space).

    It’s worth noting too that we could have acomplished the exact same effect (without using an editor) as above with the following one liner:

    # If you script the following; make sure you specify the correct device!
    # /dev/vda is used below only because that follows inline with all my
    # examples above.
    setquota -u nuxref  8000 10000 0 0 /dev/vda
    

Disk Quotas Key Point If You Use Them:
I want to be clear about this: If you choose to use Disk Quotas, you only really need to lock down the accounts that ‘could’ be compromised. In my examples in this blog, I only have one account (nuxref). It’s not necessary to lock down system accounts like apache, postgres, mysql, etc. NEVER lock down the root user!. Most of you may not even need to do this step; It truly is an extreme (and in most cases unnecessary) precautionary step that only ‘some’ should take.

Heartbleed Security Exploit

This is probably one of the biggest security exploits that has surfaced in the past few years that can compromise many systems (and potentially has been for years). This includes systems such as CentOS 6.5 (or less) allowing attackers to access random (64K) chunks of your RAM (memory) at a time (which is almost never encrypted). This can allow them to extract a lot of sensitive information that they shouldn’t have otherwise been granted access to. If you’ve installed a newer version of CentOS (or Red Hat) than v6.5 you are most likely okay. But with that all said: Make sure your version of OpenSSL is at least 1.0.1e-16!

You can check your version by typing:

# Check the details of your openssl:
rpm -qi openssl
# The output should read this version (at least):
#
Name        : openssl                      Relocations: (not relocatable)
Version     : 1.0.1e                            Vendor: CentOS
Release     : 16.el6_5.7                    Build Date: Mon 07 Apr 2014 10:43:19 PM EDT
# ....

If your version of openssl is not satisfactory, then you should update it immediately to keep your system secure! Download it from the CentOS repositories directly, or use yum:

# Get the latest version of openssl
yum update -y openssl

The website heartbleed.com does a fantastic job explaining the details on this exploit if you are interested. The Common Vulnerabilities and Exposures is defined here: CVE-2014-0160.

Credit

This blog took me a very long time to put together and test! If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

Sources

A lot of what I wrote here was accumulated knowledge over time. I can’t recall where all my sources came from and for that I’m sorry. This blog is more of a brain dump along content I keep in a personal Wiki containing information I’ve been meaning to share with everyone.

Here are some great links that may help you understand some topics discussed here a bit better though: