Tag Archives: Red Hat

Uncategorized

Hosting Your Own RPM Repository

Leave a comment

Introduction

There are lots of reasons why you might want to host your own RPM repository. A few reasons up front:

  • Speed: Performing a yum (or dnf) update from one of your locally hosted repositories is much faster than checking the ones on the internet!
  • Cost: If you run a business, then you know bandwidth can be expensive. If Linux is the operating system your developers and/or employees use, then hosting the repository internally means they won’t waste bandwidth each time one of them performs a system update. You’ll have downloaded everything once, and provided a central controlled share point they can apply all of their updates from.
  • Retention: Over time, your successfully deployed systems will age and the packages that were once readily available for it may be a lot harder to acquire! Sometimes it’s a good thing to create and manage your own internal repository so that you always have access to the exact RPMs you used to deploy/create your development environment with. This is especially the case for that one rare day that arises requiring you to go back and debug a legacy system of yours. If the packages are already available to you locally; you should have no problem reproducing any detected problems since you’ll be able to reconstruct their environment exactly.
  • Mock: I love this tool; mock allows you to dynamically generate different distributions of CentOS/Fedora (new and old) allowing you to build your applications to test them in it. You simply create a mock environment of the distribution of interest. In this new environment, you can do whatever it was you wanted to, then you can easily blow it away when you’re done. The best part about mock is that it allows you to build/test things without hauling in all of the development libraries into your native working environment. Quite frankly, I couldn’t have hosted or tested half of the things I do in all my blogs and my repository without it. Mock generates it’s ‘throw/away’ environments by connection to a yum repository and setting itself up. If you host your own repositories, it can greatly speed up this process.
  • Consistency: If everyone in your department were to reference the same rpm repository instead of one of the hundreds of mirrors available on the internet, you would consistently be hosting and sharing the same packages internally with your team. This is fantastic in a software development environment where everyone should be using the same packages anyway. This isn’t to say that all public CentOS mirrors are different, but they do go up and down from time to time. They also all synchronize themselves with whatever the latest and greatest at different times too.
  • Mirrors: Even if you just host your own mirror publicly, you’d be doing the Linux community good! You’d become another server of the hundreds already out there providing a source for free software! Your efforts would offload network congestion others face and speed up everyone’s Linux experience.
  • Custom Media: Hosting your own RPM packages can grant you the power to build your own custom media. You can accomplish the same task without hosting internally, but it’s much (,much) slower! This will make for a great topic in another blog though since this process is a topic of it’s own. I do however explain how to host your own custom repositories in this blog though!

Local Hosting Environment

The whole hosting process will take up some disk-space… 150GB or so if you decide to host everything I identify here. Nothing major, but worth noting for those with smaller disk drives.

The first thing you need to do is decide where you’ll host everything from. I will use the following directories for this blog entry:

Directory Details
/var/share/repo The location we’ll host the repository from.
/var/share/isos The location we’ll store our ISO image files in.

First let’s make sure our directories exist:

# Repository data will go here: /var/share/repo
[ ! -d /var/share/repo ] && \
   mkdir -p var/share/repo

# ISOs downloaded/kept will go here: /var/share/isos
[ ! -d /var/share/isos ] && \
   mkdir -p var/share/isos

You’ll want to have the following tools on hand as well if you plan on hosting your own repositories:

# Mirroring tools
yum install -y rsync lftp

# Repository management tools
yum install -y createrepo yum-utils find

Repository Hosting

The next few sections focus on the following repositories; you may or may not need them all. So feel free to just use what you need.

Distribution 64-Bit 32-Bit
CentOS 5.11 x86_64 i386
CentOS 6.8 x86_64 i386
CentOS 7.2 x86_64 n/a

Note: The blog utilizes the lftp tool for all of the synchronization, but the rsync command is also documented in the comments if you prefer that route as well. It’s also worth noting that you can re-run these mirror commands again and again to keep your local repository updated with the latest.

CentOS 5.11

64-bit

# Change to our ISO directory
pushd /var/share/isos

wget http://mirror.csclub.uwaterloo.ca/centos/5.11/isos/x86_64/CentOS-5.11-x86_64-bin-DVD-1of2.iso \
   -O /var/share/isos/CentOS-5.11-x86_64-DVD1.iso
wget http://mirror.csclub.uwaterloo.ca/centos/5.11/isos/x86_64/CentOS-5.11-x86_64-bin-DVD-2of2.iso \
   -O /var/share/isos/CentOS-5.11-x86_64-DVD2.iso

# Create temporary directories to work in
mkdir dvd1
mkdir dvd2

# Mount it
mount -o loop,ro CentOS-5.11-x86_64-DVD1.iso dvd1
mount -o loop,ro CentOS-5.11-x86_64-DVD2.iso dvd2

# CentOS Base Mirror
[ ! -d /var/share/repo/centos/5.11/x86_64/os ] && \
    mkdir -p /var/share/repo/centos/5.11/x86_64/os

# CentOS Updates Mirror
[ ! -d /var/share/repo/centos/5.11/x86_64/updates ] && \
    mkdir -p /var/share/repo/centos/5.11/x86_64/updates

# CentOS Extras Mirror
[ ! -d /var/share/repo/centos/5.11/x86_64/extras ] && \
    mkdir -p /var/share/repo/centos/5.11/x86_64/extras

# Setup EPEL location
[ ! -d /var/share/repo/centos/5/x86_64/epel ] && \
    mkdir -p /var/share/repo/centos/5/x86_64/epel

# Create a fixed link to newest repo
pushd /var/share/repo/centos/5/x86_64
ln -snf ../../5.11/x86_64/os os
ln -snf ../../5.11/x86_64/updates updates
ln -snf ../../5.11/x86_64/extras extras
popd

# Set up other repositories
pushd /var/share/repo/centos/5.11/x86_64
ln -snf ../../5/x86_64/epel epel
popd

# Mirror repository
rsync -av --ignore-existing dvd1/ /var/share/repo/centos/5.11/x86_64/os/
rsync -av --ignore-existing dvd2/CentOS/ /var/share/repo/centos/5.11/x86_64/os/Packages/
 
# Cleanup
umount dvd1
umount dvd2
rmdir dvd1
rmdir dvd2
popd

# Mirror (5.11) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/5.11/updates/x86_64/ /var/share/repo/centos/5.11/x86_64/updates/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/5.11/updates/x86_64/ /var/share/repo/centos/5.11/x86_64/updates"

# Mirror (5.11) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/5.11/extras/x86_64/ /var/share/repo/centos/5.11/x86_64/extras/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/5.11/extras/x86_64/ /var/share/repo/centos/5.11/x86_64/extras"

# EPEL: (see https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/5 for list of servers)
# In the future you can run these commands again and again to update your repository
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/fedora-enchilada/epel/6/x86_64/ /var/share/repo/centos/5/x86_64/epel/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /fedora/epel/5/x86_64/ /var/share/repo/centos/5/x86_64/epel"

32-bit

# Change to our ISO directory
pushd /var/share/isos

wget http://mirror.csclub.uwaterloo.ca/centos/5.11/isos/i386/CentOS-5.11-i386-bin-DVD-1of2.iso \
   -O /var/share/isos/CentOS-5.11-i386-DVD1.iso
wget http://mirror.csclub.uwaterloo.ca/centos/5.11/isos/i386/CentOS-5.11-i386-bin-DVD-2of2.iso \
   -O /var/share/isos/CentOS-5.11-i386-DVD2.iso

# Create temporary directories to work in
mkdir dvd1
mkdir dvd2

# Mount it
mount -o loop,ro CentOS-5.11-i386-DVD1.iso dvd1
mount -o loop,ro CentOS-5.11-i386-DVD2.iso dvd2

# CentOS Base Mirror
[ ! -d /var/share/repo/centos/5.11/i386/os ] && \
    mkdir -p /var/share/repo/centos/5.11/i386/os

# CentOS Updates Mirror
[ ! -d /var/share/repo/centos/5.11/i386/updates ] && \
    mkdir -p /var/share/repo/centos/5.11/i386/updates

# CentOS Extras Mirror
[ ! -d /var/share/repo/centos/5.11/i386/extras ] && \
    mkdir -p /var/share/repo/centos/5.11/i386/extras

# Setup EPEL Mirror
[ ! -d /var/share/repo/centos/5/i386/epel ] && \
    mkdir -p /var/share/repo/centos/5/i386/epel

# Create a fixed link to newest repo
pushd /var/share/repo/centos/5/i386
ln -snf ../../5.11/i386/os os
ln -snf ../../5.11/i386/updates updates
ln -snf ../../5.11/i386/extras extras
popd

# Set up other repositories
pushd /var/share/repo/centos/5.11/i386
ln -snf ../../5/i386/epel epel
popd

# Mirror repository
rsync -av --ignore-existing dvd1/ /var/share/repo/centos/5.11/i386/os/
rsync -av --ignore-existing dvd2/CentOS/ /var/share/repo/centos/5.11/i386/os/Packages/
 
# Cleanup
umount dvd1
umount dvd2
rmdir dvd1
rmdir dvd2
popd

# Mirror (5.11) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/5.11/updates/i386/ /var/share/repo/centos/5.11/i386/updates/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/5.11/updates/i386/ /var/share/repo/centos/5.11/i386/updates"

# Mirror (5.11) Extras
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/5.11/extras/i386/ /var/share/repo/centos/5.11/i386/extras/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/5.11/extras/i386/ /var/share/repo/centos/5.11/i386/extras"

# EPEL: (see https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/5 for list of servers)
# In the future you can run these commands again and again to update your repository
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/fedora-enchilada/epel/6/i386/ /var/share/repo/centos/5/i386/epel/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /fedora/epel/5/i386/ /var/share/repo/centos/5/i386/epel"

CentOS 6.8

64-bit

# Change to our ISO directory
pushd /var/share/isos

wget http://mirror.its.dal.ca/centos/6.8/isos/x86_64/CentOS-6.8-x86_64-bin-DVD1.iso \
   -O /var/share/isos/CentOS-6.8-x86_64-DVD1.iso
wget http://mirror.its.dal.ca/centos/6.8/isos/x86_64/CentOS-6.8-x86_64-bin-DVD2.iso \
   -O /var/share/isos/CentOS-6.8-x86_64-DVD2.iso

# Create temporary directories to work in
mkdir dvd1
mkdir dvd2

# Mount it
mount -o loop,ro CentOS-6.8-x86_64-DVD1.iso dvd1
mount -o loop,ro CentOS-6.8-x86_64-DVD2.iso dvd2

# CentOS Base Mirror
[ ! -d /var/share/repo/centos/6.8/x86_64/os ] && \
    mkdir -p /var/share/repo/centos/6.8/x86_64/os

# CentOS Updates Mirror
[ ! -d /var/share/repo/centos/6.8/x86_64/updates ] && \
    mkdir -p /var/share/repo/centos/6.8/x86_64/updates

# CentOS Extras Mirror
[ ! -d /var/share/repo/centos/6.8/x86_64/extras ] && \
    mkdir -p /var/share/repo/centos/6.8/x86_64/extras

# Setup EPEL location
[ ! -d /var/share/repo/centos/6/x86_64/epel ] && \
    mkdir -p /var/share/repo/centos/6/x86_64/epel

# Create a fixed link to newest repo
pushd /var/share/repo/centos/6/x86_64
ln -snf ../../6.8/x86_64/os os
ln -snf ../../6.8/x86_64/updates updates
ln -snf ../../6.8/x86_64/extras extras
popd

# EPEL Repository Mirror
pushd /var/share/repo/centos/6.8/x86_64
ln -snf ../../6/x86_64/epel epel
popd

# Mirror repository
rsync -av --ignore-existing dvd1/ /var/share/repo/centos/6.8/x86_64/os/
rsync -av --ignore-existing dvd2/Packages/ /var/share/repo/centos/6.8/x86_64/os/Packages/

# Cleanup
umount dvd1
umount dvd2
rmdir dvd1
rmdir dvd2
popd

# Mirror (6.8) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/6.8/updates/x86_64/ /var/share/repo/centos/6.8/x86_64/updates/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/6.8/updates/x86_64/ /var/share/repo/centos/6.8/x86_64/updates"

# Mirror (6.8) Extras
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/6.8/extras/x86_64/ /var/share/repo/centos/6.8/x86_64/extras/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/6.8/extras/x86_64/ /var/share/repo/centos/6.8/x86_64/extras"

# EPEL: (see https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/6 for list of servers)
# In the future you can run these commands again and again to update your repository
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/fedora-enchilada/epel/6/x86_64/ /var/share/repo/centos/6/x86_64/epel/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /fedora/epel/6/x86_64/ /var/share/repo/centos/6/x86_64/epel"

32-bit

# Change to our ISO directory
pushd /var/share/isos

wget http://mirror.its.dal.ca/centos/6.8/isos/i386/CentOS-6.8-i386-bin-DVD1.iso \
   -O /var/share/isos/CentOS-6.8-i386-DVD1.iso
wget http://mirror.its.dal.ca/centos/6.8/isos/i386/CentOS-6.8-i386-bin-DVD2.iso \
   -O /var/share/isos/CentOS-6.8-i386-DVD2.iso

# Create temporary directories to work in
mkdir dvd1
mkdir dvd2

# Mount it
mount -o loop,ro CentOS-6.8-i386-DVD1.iso dvd1
mount -o loop,ro CentOS-6.8-i386-DVD2.iso dvd2

# CentOS Base Mirror
[ ! -d /var/share/repo/centos/6.8/i386/os ] && \
    mkdir -p /var/share/repo/centos/6.8/i386/os

# CentOS Updates Mirror
[ ! -d /var/share/repo/centos/6.8/i386/updates ] && \
    mkdir -p /var/share/repo/centos/6.8/i386/updates

# CentOS Extras Mirror
[ ! -d /var/share/repo/centos/6.8/i386/extras ] && \
    mkdir -p /var/share/repo/centos/6.8/i386/extras

# EPEL Repository Mirror
[ ! -d /var/share/repo/centos/6/i386/epel ] && \
    mkdir -p /var/share/repo/centos/6/i386/epel

# Create a fixed link to newest repo
pushd /var/share/repo/centos/6/i386
ln -snf ../../6.8/i386/os os
ln -snf ../../6.8/i386/updates updates
ln -snf ../../6.8/i386/extras extras
popd

# Set up other repositories
pushd /var/share/repo/centos/6.8/i386
ln -snf ../../6/i386/epel epel
popd

# Mirror repository
rsync -av --ignore-existing dvd1/ /var/share/repo/centos/6.8/i386/os/
rsync -av --ignore-existing dvd2/Packages/ /var/share/repo/centos/6.8/i386/os/Packages/

# Cleanup
umount dvd1
umount dvd2
rmdir dvd1
rmdir dvd2
popd

# Mirror (6.8) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/6.8/updates/i386/ /var/share/repo/centos/6.8/i386/updates/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/6.8/updates/i386/ /var/share/repo/centos/6.8/i386/updates"

# Mirror (6.8) Extras
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/6.8/extras/i386/ /var/share/repo/centos/6.8/i386/extras/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/6.8/extras/i386/ /var/share/repo/centos/6.8/i386/extras"

# EPEL: (see https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/6 for list of servers)
# In the future you can run these commands again and again to update your repository
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/fedora-enchilada/epel/6/i386/ /var/share/repo/centos/6/i386/epel/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /fedora/epel/6/i386/ /var/share/repo/centos/6/i386/epel"

CentOS 7.2

64-bit

# Change to our ISO directory
pushd /var/share/isos

# CentOS 7.2
[ ! -d /var/share/isos/ ] && mkdir -p /var/share/isos/
wget http://mirror.its.dal.ca/centos/7.2.1511/isos/x86_64/CentOS-7-x86_64-Everything-1511.iso \
   -O /var/share/isos/CentOS-7.2.1511-x86_64-Everything.iso

# CentOS Base Mirror
[ ! -d /var/share/repo/centos/7.2/x86_64/os ] && \
    mkdir -p /var/share/repo/centos/7.2/x86_64/os

# CentOS Updates Mirror
[ ! -d /var/share/repo/centos/7.2/x86_64/updates ] && \
    mkdir -p /var/share/repo/centos/7.2/x86_64/updates

# CentOS Extras Mirror
[ ! -d /var/share/repo/centos/7.2/x86_64/extras ] && \
    mkdir -p /var/share/repo/centos/7.2/x86_64/extras

# EPEL Repository Mirror
[ ! -d /var/share/repo/centos/7/x86_64/epel ] && \
    mkdir -p /var/share/repo/centos/7/x86_64/epel

# Create a fixed link to newest repo
pushd /var/share/repo/centos/7/x86_64
ln -snf ../../7.2/x86_64/os os
ln -snf ../../7.2/x86_64/updates updates
ln -snf ../../7.2/x86_64/extras extras
popd

# Set up other repositories
pushd /var/share/repo/centos/7.2/x86_64
ln -snf ../../7/x86_64/epel epel
popd

# Mount it
mount -o loop,ro /var/share/isos/CentOS-7.2.1511-x86_64-Everything.iso \
    /var/share/repo/centos/7.2/x86_64/os

# Update our fstab file so we persistently do this on reboots; This saves disk space
sed -i -e '/CentOS-7\.2\.1511-x86_64-Everything\.iso/d' /etc/fstab
echo '/var/share/isos/CentOS-7.2.1511-x86_64-Everything.iso /var/share/repo/centos/7.2/x86_64/os iso9660 loop,ro,auto 0 0' >> /etc/fstab
# If you prefer not to run the above command, you can follow through with the same rsync
# commands identified above and mirror the contents of the iso to disk

# Mirror (7.2) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/7.2.1511/updates/x86_64/ /var/share/repo/centos/7.2/x86_64/updates/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/7.2.1511/updates/x86_64/ /var/share/repo/centos/7.2/x86_64/updates"

# Mirror (7.2) Extras
# In the future you can run these commands again and again to update your repository
# EPEL: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/7.2.1511/extras/x86_64/ /var/share/repo/centos/7.2/x86_64/extras/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/7.2.1511/extras/x86_64/ /var/share/repo/centos/7.2/x86_64/extras"

# Mirror EPEL
# In the future you can run these commands again and again to update your repository
# EPEL: (see https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/7 for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/fedora-enchilada/epel/7/x86_64/ /var/share/repo/centos/7/x86_64/epel/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /fedora/epel/7/x86_64/ /var/share/repo/centos/7/x86_64/epel"

Web Hosting

Once you’ve got your repositories mirrored, you need to host them. Here is the easiest way to do so:
If you’re (planning on) using NginX, then the following will get you going:

# Install nginx (if it's not there already)
yum install nginx -y

# Enable for future reboots
systemctl enable nginx.service

# Start it up
systemctl start nginx.service

cat << _EOF > /etc/nginx/default.d/repo.conf
location /repo/ {
   alias         /var/share/repo/;
   autoindex on;
}
_EOF
# Reload
systemctl reload nginx.service

Now you should be able to access your website by visiting the server you set this up on with /repo as the path. ie: http://localhost/repo

SELinux

Users running SELinux in enforcing mode will want to do the following so that they’re repository can be hosted properly:

# make sure /var/share/repo can host website content without conflicting
# with SELinux
semanage fcontext -a -t httpd_sys_content_t '/var/share/repo(/.*)?'

YUM Repositories

Now we’ll want to update our servers around our office, or maybe just on this PC we’re using to point to our new repositories. Here is probably the easiest way:

# Disable any existing repository setup (the following has
# to be ran as root):
pushd /etc/yum.repo.d/
sed -e 's/^\(enabled\)=.*/\1=0/g' \
   fedora*.repo centos*.repo epel*.repo &>/dev/null
popd

# Set this variable to the ip/host of the machine you set up your
# repository on:
MYREPOADDR=localhost

# Now install our new file:
cat << _EOF > /etc/yum.repo.d/centos.internal.repo
[internal-base]
name=CentOS \$releasever - \$basearch - Base
baseurl=http://$MYREPOADDR/repo/centos/\$releasever/\$basearch/os/
enabled=1
priority=1
gpgcheck=0
skip_if_unavailable=False

[internal-updates]
name=CentOS \$releasever - \$basearch - Updates
failovermethod=priority
baseurl=http://$MYREPOADDR/repo/centos/\$releasever/\$basearch/updates/
enabled=1
priority=1
gpgcheck=0
skip_if_unavailable=True

[internal-extras]
name=CentOS \$releasever - \$basearch - Extras
failovermethod=priority
baseurl=http://$MYREPOADDR/repo/centos/\$releasever/\$basearch/extras/
enabled=1
priority=1
gpgcheck=0
skip_if_unavailable=True

[internal-epel]
name=CentOS \$releasever - \$basearch - Internal EPEL
failovermethod=priority
baseurl=http://$MYREPOADDR/repo/centos/\$releasever/\$basearch/epel/
enabled=1
priority=1
gpgcheck=0
skip_if_unavailable=True
_EOF

You can easily take the examples provided here and alter them for other repositories you wish to host.

Custom Repositories

Perhaps you’re picking and choosing RPMs from a ton of different sources, or you’re building your own. It’s a good idea not to touch the repositories we’re already mirroring. Leave them exactly the way they are. However, you can create your own repository instead that you can place your personal collection of rpms in:

# Lets presume we want to start a custom CentOS 7 (64-bit) repository.
# we'll call it 'custom' for now, but you can call it whatever you want

[ ! -d /var/share/repo/centos/7/x86_64/custom ] && \
    mkdir -p /var/share/repo/centos/7/x86_64/custom

# Create a fixed link to our repo
pushd /var/share/repo/centos/7.2/x86_64
ln -snf ../../7/x86_64/custom custom
popd

# Okay... now you can drop all the RPMs you want into this custom repository.
# just copy them in; no strings attached.
cp my.awesome.application-1.0.0-1.x86_64.rpm

# The next step is to make the repository accessible by yum
# If you're using dnf, there is no problem, these commands still apply:

# First we want to reset our repodata folder.  This probably won't
# exist the first time you do this, but in the future, when you add
# more rpms into this directory, this will be an essential step:
[ -d /var/share/repo/centos/7/x86_64/custom/repodata ] && \
    rm -rf /var/share/repo/centos/7/x86_64/custom/repodata

# Now lets clean out any duplicate/old RPM entries
# This is optional; it basically looks for 2 RPMs of the same name
# but of different versions and only keeps the newest.  If you like
# hanging on to all of the versions of your software, you don't
# need to run the below command.
repomanage -o /var/share/repo/centos/7/x86_64/custom/repodata | \
   xargs rm -f

The next step requires us to generate a comps file. These are XML files that define details of our repository. You can get complicated and define it yourself if you like, but to make things simple, you can just use this script i wrote which will generate one on the fly based on the directory you specify it to parse:

#!/bin/sh
# Name: updaterepo
# Author: Chris Caron <lead2gold@gmail.com>
# URL: See https://nuxref.com for details
# Description:
#   A simple script for creating a repository directory.
#
# Syntax: updaterepo <path_to_repository_hosting_rpms>
#
REPODIR=$1
if [ -z "$REPODIR" ]; then
   echo "You must specify a repository directory to scan (containing rpms)."
   exit 1
fi

if [ ! -d "$REPODIR" ]; then
   echo "You must specify a repository directory to scan (containing rpms)."
   exit 1
fi

# Ensure we're dealing with absolute paths
pushd $REPODIR &>/dev/null && REPODIR=$(pwd) && popd &>/dev/null

# Some Default Variables; please feel free to change these to
# suit your own needs and package identification. Whatever you
# specify here will become a 'group' as far as yum and dnf are
# concerned allowing you to run (set $ID according):
#   yum groupinstall custom
ID="custom"
NAME="Core"
DESC="Custom Built Packages"

# Generate a comps file for a repository dynamically
COMPSFILE=comps-$ID.xml
COMPSDIR=$REPODIR/repodata
COMPS=$COMPSDIR/$COMPSFILE

# Directory Management
[ -d $COMPSDIR ] && rm -rf $COMPSDIR
[ ! -d $COMPSDIR ] && mkdir -p $COMPSDIR

FILES=$(find -L $REPODIR -mindepth 1 -maxdepth 1 -type f -name "*.rpm" -exec basename {} \;)

   cat << _EOF > "$COMPS"
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//REDHAT//DTD Comps info//EN" "comps.dtd">
<comps>
  <group>
    <id>$ID</id>
    <name>$NAME</name>
    <description>$DESC</description>
    <default>True</default>
    <uservisible>True</uservisible>
    <packagelist>
_EOF
for FILE in $FILES; do
   PFILE=$(rpm -qp "$REPODIR/$FILE" --nosignature --queryformat="%{NAME}" 2>/dev/null)
   cat << _EOF >> "$COMPS"
         <packagereq type="default">$PFILE</packagereq>
_EOF
done
cat << _EOF >> "$COMPS"
    </packagelist>
  </group>
  <category>
    <id>$ID</id>
    <name>$NAME</name>
    <display_order>99</display_order>
    <grouplist>
      <groupid>$ID</groupid>
    </grouplist>
  </category>
</comps>
_EOF

# Now rebuild our repo information
createrepo -d -q -g $COMPS $REPODIR

Place the script file above into your /usr/bin directory if you like (called updaterepo) and call it on any directory you want to turn into a repository from then on:

# Make sure our new script is executable
chmod 775 /bin/updaterepo

# Now call it against any directory we want to turn into a repository
[ ! -d /var/share/repo/centos/7/x86_64/custom ] && 
    mkdir -p  /var/share/repo/centos/7/x86_64/custom

# Now we can copy our rpms into this directory at will.
# Then we just need to run this command when we're done to apply
# our changes. This will allow people who are pointing to our
# repository to see these new changes we made and access it via yum/dnf
updaterepo  /var/share/repo/centos/7/x86_64/custom
[code]

You're done! Now you can update your <strong>/etc/yum.repos.d/</strong> to include this new location with each yum call you make! Just use the other examples already provided in this blog as a template!

<h1>Mock</h1>
<a href="https://github.com/rpm-software-management/mock/wiki" target="_blank">Mock</a> is a fantastic tool for RPM management.  It's also a great tool for someone who just wants to test and see if they're code will run on another platform.  You could almost think of Mock as a poor man's <a href="https://linuxcontainers.org/" target="_blank">Linux Container</a> which are pretty popular these days.  Mock isn't as contained, but it can accomplish the same feat and is even faster since it doesn't have the (Linux) container overhead.

Mock is set up as follows:
[code lang="bash"]
# first install it (as root)
yum install -y mock

# If you followed all of the instructions above, then this will pull mock
# out of your internal epel repository! Nice!

# Now just add your user account you usually use on your system
# (a non-root user).  This user will be granted mock privileges.
# substitute [User name] below with the username you usually use:
usermod -a -G mock [User name]

Just like that’ you’re now ready to create mock environments. Now by default, mock is configured to fetch it’s information from the external repositories on the internet. But it’s configuration is really easy to get ahold of and update. Check out the /etc/mock directory.

You can now edit the mock environment you wish to host and optionally update the repositories to point to your own. For example, pick a file like /etc/mock/epel-7-x86_64.cfg which is used to generate a 64-bit Enterprise Linux (RedHat/CentOS) 7.x environment.

You’ll see entries like this which will look very familiar (it’s a yum/dnf configuration file entry just like the ones identified above). You can comment out the mirrorlist entry and swap it with your own local repository.

...  ...
[base]
name=BaseOS
# comment out the mirrorlist reference:
# mirrorlist=http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os
# point to our own local repository instead:
baseurl=http://localhost/repo/centos/7/x86_64/os/
failovermethod=priority
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-7
gpgcheck=1

[updates]
name=updates
enabled=1
# comment out the mirrorlist reference:
# mirrorlist=http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates
# point to our own local repository instead:
baseurl=http://localhost/repo/centos/7/x86_64/updates/
failovermethod=priority
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-7
gpgcheck=1

[extras]
name=extras
enabled=1
# comment out the mirrorlist reference:
# mirrorlist=http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=extras
# point to our own local repository instead:
baseurl=http://localhost/repo/centos/7/x86_64/extras/
failovermethod=priority
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-7
gpgcheck=1

[epel]
name=epel
# comment out the mirrorlist reference:
# mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-7&arch=x86_64
# point to our own local repository instead:
baseurl=http://localhost/repo/centos/7/x86_64/updates/
failovermethod=priority
gpgkey=file:///usr/share/distribution-gpg-keys/epel/RPM-GPG-KEY-EPEL-7
gpgcheck=1

...  ...

You can also just copy and paste one of the config files to another and change it around. Add repositories, remove some; the config file for mock is really straight forward if you use another as a template.

Mock is pretty straight forward to use. You just need to pass it in the environment you’re using with each call you make to it.

# Initialize a Mock Environment of Enterprise Linux 7 (64-bit):
mock -r epel-7-x86_64 --init

# The above command works because a /etc/mock/epel-7-x86_64.cfg file exists.
# mock is relatively quiet and doesn't output a lot of information which can
# sometimes make you unsure if it's actually doing anything.  So it doesn't
# hurt to also get in the habit of adding -v to every single mock action you
# perform.

# You can install RPMs into your mock environment easily with:
mock -v -r epel-7-x86_64 install hostname vi

# The following command will cause us to leave our current native environment
# and access the mock environment (an advanced chroot basically):
mock -v -r epel-7-x86_64 --shell

# Once inside, you can do whatever you want; just press Ctrl-D or type 'exit'
# to return back to your normal environment.

# You can even rebuild a source rpm package from another distribution
# into one you might use.  For example, perhaps you visited pkgs.org and saw
# a newer version of an awesome application available for Fedora 24, but
# can't get it for CentOS 7.  Well you can download the src.rpm file and
# rebuild it using mock:
mock -v -r epel-7-x86_64 --rebuild awesome.application-v2.0.0-1.fc24.src.rpm \
    --resultdir=awesome.app

# I added the --resultdir= on the line above to tell mock that after it rebuilds
# our new RPMs for CentOS 7, it places them into a directory called 'awesome.app'.
# This isn't necessary, but makes it easier to find the RPMs when the build is
# complete.

Credit

If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

Sources

  • Mock: A fantastic development tool that allows you to dynamically generate a fully functional development environment for a CentOS/Red Hat/Fedora distribution of your choice. It allows you to keep development libraries out of your native (working) environment keeping things clean.
  • Some good repositories worth mirroring (in no particular order):
    • CentOS 5.x, 6.x and 7.x: Here is a list of all of the mirrors you can sync from.
    • Extra Packages for Enterprise Linux (EPEL): CentOS/Red Hat users shouldn’t be without this one. It has tons of the building blocks you might need to reconstruct things with.
    • NuxRef: Yes… my repository; I mean why not, right?
    • Fedora: Bleeding edge distributions don’t stick around long. After a year or so, you’ll be hard-pressed to find RPMs. It might be wise to mirror the distribution you’re using if you don’t plan on upgrading anytime soon. There is enough information in this blog to mirror a Fedora repository if you want to do so.
Linux HOWTO, Missing Linux Packages

NZBGet v17 for CentOS 7

Leave a comment

Introduction

Recently NZBGet v17 was released which involved a massive re-factoring and over-haul of most of it’s back-end. Here is a quote taken from the ticket (#88) that makes up the bulk of this release:

Due to a wide variety of platforms we could not control which version of compiler is used. To make it compatible even with old compilers NZBGet was targeting old C++ versions.

In last years C++ has been evolved dramatically. New versions were standardized by ISO as C++11 and lately C++14. These new standards bring many interesting features to the language making it much more pleasure to work with.

The new (C++) standards are amazing, there is no doubt about that. But the problem is that not all (Stable) Linux distributions support these new standards. To put it in perspective, here is a chart of some Linux Distributions and the GCC standards they support (without any extra hacking/effort):

Compatibility C++98 C++03 C++11 C++14 C++17
Red Hat 5 / CentOS 5 Yes Yes No No No
Red Hat 6 / CentOS 6 Yes Yes No No No
Red Hat 7 / CentOS 7 Yes Yes Yes No No
Ubuntu 12.x Yes Yes No No No
Ubuntu 14.x Yes Yes Yes No No
Ubuntu 16.x Yes Yes Yes Yes No
Fedora 22 Yes Yes Yes Yes no
Fedora 23 Yes Yes Yes Yes Yes
Fedora 24 Yes Yes Yes Yes Yes

The point is: By making NZBGet require the C++14 standard, the list of supported (Linux) distributions drops significantly.

Just use the NZBGet Binaries Already Provided

Yes, NZBGet allows you to install an already pre-compiled version of it, and if this satisfies your needs, then I’m not here to tell you to change.

In my personal opinion, there are several problems with going with this approach:

  1. Your program is statically linked to everything it uses. What does this mean? It means that any vulnerabilities discovered and patched on your system (via distributed system updates) will NOT patch NZBGet. These patches that are released don’t just resolve vulnerabilities, but they also add stability too.

    This isn’t to say NZBGet isn’t stable, it’s to point out that it can’t reap benefits of added stability when it becomes available.
  2. There is no package management using the binary provided on NZBGet’s website and therefore no version control (unless you’re using Microsoft). Thus, you can’t automate it’s deployment through tools like yum, dnf or apt-get (depending on your distribution).

Just to give you an idea of how many libraries NZBGet (v17) links to; here they are. I also identify the likelihood of a patch to be released between now and the next 3 months (purely speculation based on recent releases):

Package Risk Description
glibc, libgcc & libstdc++ MED The systems standard libraries (what this whole blog is about really). Ideally you don’t want to use programs that aren’t linked to a different system core then the one your system depends on. This core is managed by your distribution and is key to it’s stability. It only makes sense to use it for everything else too!
gmp LOW Just a really powerful library for doing mathematical calculations. There are rarely exploits found here; but it’s worth noting that this is a dependency of NZBGet.
keyutils-libs MED A wrapper library for the key management facility system
calls
krb5-libs MED A network authentication system.
libselinux LOW An API for SELinux applications to get and set process and file security contexts and to obtain security policy decisions. This is very customized for each distribution. Ideally you’ll always want to link to you’re managed library.
libxml2 LOW XML Parser which is used for reading through your NZB Files and RSS feeds.
nettle, trousers & openssl-libs HIGH The encryption between you an your Usenet provider is here. This is also the encryption between you and your indexer. There are constant exploits an vulnerabilities detected in these encryption libraries. These libraries alone are the main reason you should not use the NZBGet binaries and link to the ones you can keep current instead.
xz-libs, & zlib LOW Compression libraries for handling zip files (like those zipped NZB files for example when bundled).
p11-kit, pcre, ncurses-libs, libtasn1, libffi, & libcom_err LOW Some important libraries, but nothing with a high risk level associated with them. They’re still worth noting as they’re part of its requirements.

The point is that there are a lot of moving parts that make up NZBGet and you should make sure that it links to shared actively maintained libraries that you have control over.

You Talk To Much, Just Hand Over Whatever It Is You’re Selling!

Sure, here they are:

  • CentOS 7: rpm / src
    Note: This is the true bread and butter of this blog because this rpm contains the NZBGet v17 installable package completely modified to work with the C++11 standard. Thus you can take advantage of this beautiful piece of software while still using your shared libraries.
  • Fedora 22: rpm / src
  • Fedora 23: rpm / src
  • Fedora 24: rpm / src

The Fedora packages were all compiled using C++14 standard; there was no patch required for them.

You can also feel free to connect to my repository by following the information provided here and then download them using yum or dnf.

A worthy note about these packages:

  • It’ll automatically create a nzbget user (an group) for you so that you can control access to your downloads better.
    Adding users to it’s access is as simple as just adding them to the nzbget group:

    # Add a user to the nzbget group
usermod -a -G nzbget myuser
  • Provides a SystemD service ready to go for those who want to tie it to their systems startup and shutdown. 
    # Add NZBGet to the startup of your system
systemctl enable nzbget.service

# Start it up now if you like too!
systemctl start nzbget.service

# Now you can go ahead and access http://localhost:6789 with your browser
# The default login/password is nzbget/tegbzn6789
  • A small separate patch in these RPMs change:
    • All of the ~/downloads references to ~/Downloads.
    • All references to /usr/local/bin to /usr/bin.
    • Added ~/.config/nzbget/ to the default paths list (for those who just want to run nzbget -d as they’re user). I personally prefer to tuck all my configuration files in .config/ instead of the root folder of my home directory (which still works too if you want).
  • A default configuration (/etc/nzbget.conf) is pre-prepared for you which
    • configures most of the basic settings out of the box such as unrar and 7za paths.
    • sets up the logging directory to be /var/log/nzbget/nzbget.log (with log rotation on).
    • sets up the $MainDir directory to /var/lib/nzbget which is pre-created already for you and protected by the nzbget user and group.

What About Everybody Else?

For those using other Linux Distributions like Ubuntu, you can download my patch file which back-ports all of the C++14 code to C++11 here.

Alternatively, you can follow these instructions which should get it working for you (using share libraries instead):

# Note: The below is only applicable to v17.0.
# Download the NZBGet v17 Source
wget https://github.com/nzbget/nzbget/releases/download/v17.0/nzbget-17.0-src.tar.gz

# Download the patch:
wget https://repo.nuxref.com/pub/patches/nzbget/nzbget-use.c++11.patch

# Extract the tarball
tar xfz nzbget-17.0-src.tar.gz

# Change into the directory it extracts to
pushd nzbget-17.0

# Apply the patch
patch -p1 < ../nzbget-use.c++11.patch

You’ll see output like this:

patching file config.h.in
patching file configure.ac
patching file daemon/connect/Connection.cpp
patching file daemon/connect/TlsSocket.cpp
patching file daemon/main/Scheduler.cpp
patching file daemon/queue/QueueEditor.cpp
patching file daemon/util/Observer.h
patching file daemon/util/Thread.cpp
patching file daemon/util/Thread.h
patching file daemon/util/Util.h
patching file linux/build-nzbget
patching file Makefile.in

You can now build NZBGet (assuming you have all the right libraries):

# first rebuild the configure script (part of what the patch prepares)
autoreconf -i
automake

# You may need to install the following packages
# Red Hat / CentOS 7:
#  sudo yum install libxml2-devel libpar2-devel \
#      ncurses-devel dos2unix \
#      openssl-devel autoconf, libtool, automake
#
# Ubuntu:
#  sudo apt-get install libxml2-dev libpar2-dev \
#      ncurses-dev dos2unix \
#      libssl-dev autoconf, libtool, automake

# Now you can run the rest of the setup:
./configure

# correct any problems you have here. The configure
# script will look after telling you if you're missing
# any libraries or not. If there are no problems then
# go ahead and type:
make

# Now you can install it into your system
make install

Now the last commands don’t provide you any kind of package management like RPMs or DEB packages do. But there is enough information in this blog create your own installable package here if you wanted. The main thing you accomplish is that you link a working copy of NZBGet to your stable (upgradable) system libraries.

Credit

Please note that this information took me several days to put together and test thoroughly. The C++11 patch alone took me several hours to test an create. I may not blog often; but I want to re-assure the stability and testing I put into everything I intend share.

If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

Aug 5th, 2016 – Update

Since writing this blog, the amazing developer (hugbug) of NZBGet reached out to me to incorporate my patch (a much simplified version of it) into the next release of NZBGet (v17.1)! Here is a link to the pull request. So this pretty much renders this blog entry useless but accomplishing (at the end) none the less! 🙂 I still intend to package NZBGet (and all of it’s future releases) on my repository for those still interested.

Sept 10th, 2016 – Update

NZBGet v17.1 came out and the packages in this blog and the repository have been updated to reflect this. Those that still wish to build it manually can do so as follows:

# Download the latest version of NZBGet
wget https://github.com/nzbget/nzbget/releases/download/v17.1/nzbget-17.1-src.tar.gz

# You may need to install the following packages
# Red Hat / CentOS 7:
#  sudo yum install libxml2-devel libpar2-devel \
#      ncurses-devel dos2unix \
#      openssl-devel autoconf, libtool, automake
#
# Ubuntu:
#  sudo apt-get install libxml2-dev libpar2-dev \
#      ncurses-dev dos2unix \
#      libssl-dev autoconf, libtool, automake

# Now you can run the rest of the setup:
CXXFLAGS="-std=c++11 -O2 -s" ./configure --disable-cpp-check

# correct any problems you have here. The configure
# script will look after telling you if you're missing
# any libraries or not. If there are no problems then
# go ahead and type:
make

# Now you can install it into your system
make install

Sources

Linux HOWTO, Missing Linux Packages

Free Home Security Monitoring using Linux

3 Comments

Introduction

Home security is important to all of us but why should we pay someone to monitor our home when we can do it ourselves (for free)? I mean, imagine if it were possible to just receive a text message on your phone when your alarm has been set off. Imagine if you could check to see if you armed the alarm this morning before you left for work. If you had forgotten, wouldn’t it offer you a peace of mind to be able to arm it from wherever you are?

Some great reasons why one would choose to monitor their home themselves:

  1. Hundreds of Dollars in Yearly Savings: Think of how much money you invest in a 3rd party company that does nothing unless your alarm goes off. Even if you’re just paying $25 (before tax) a month for home security, we’re talking about a $300+ in yearly savings that you can pocket!
  2. Faster Emergency Response Time: Consider the scenario where a thief has broken into your home and has begun loading up his van with all of your valuables.

    Regardless of whether you’re paying for home monitoring or doing it yourself, the first thing that will happen is: Your alarm will sound within a 30+ seconds of the perpetrator’s entry; a very loud ear piercing siren I might add.

    • If you’re paying for a monitoring station service: They will call your home line to see if the threat is real or not (meanwhile your home is still being robbed). If they don’t receive an answer or get a busy signal, only then will they call the police. We’re easily a few minutes into the robbery before the authorities are even notified. The police will eventually arrive, they’ll file a report and you’re left to deal with insurance because the crime is long over and the criminals are long gone.
    • If you’re monitoring the home yourself: Well then your cell phone has already alerted you the second the alarm was set off. It’s a no brainer at this point, you can take action right then and there and call the police or 911 depending on if you’re home or not. You just immediately reduced the time the thief has to accomplish his crime in.
  3. Remote Control: Ensure the alarm is armed at any time. Change the alarm keycode remotely or disarm it from a distance to let a construction worker in. You can even arm/disarm your alarm from the comfort of your bed via your tablet, laptop or cell phone.

This blog focuses on the amazing hard work of Alarm Decoder and their team. It specifically focuses on their AD2USB device which allows you to connect a virtual keypad to your existing alarm system.

It’s also worth noting that I specifically focus on the Honeywell Vista 15p/20p models because that’s what I have. But I’m sure you could use the content of this blog and the other supported alarm systems Alarm Decoders AD2USB supports and follow along.

The Goal

The goal of this blog is to share with you my success, but more importantly:

  • To show you how easy it is to tap into an existing (pre-wired) alarm system without paying for a monitoring service.
  • Receive SMS (text) messages, emails or other notification services when your alarm is ever set off.
  • To be able to control the (security) keypad in your house from anywhere and at anytime.
  • To provide you all of this in a cookie cutter solution that works out of the box. I spent several hours packaging everything into self installing RPMs residing on my repository.

This blog will assume you’re both familiar with and have access to either CentOS (or Red Hat) 6.x or 7.x.

I’ll also assume you don’t have much experience with alarm systems (as I certainly didn’t at the start). I’ll try to save you as much research as I can and provide everything to you here. As a result, this blog is probably a bit longer then it needs to be since I get wordy trying to explain it.

Prerequisites

There are a few things you’ll need to be able to pull this feat off:

  1. Honeywell Vista 15p / 20pAn Ademco/Honeywell Vista 15p or 20p alarm system, but it doesn’t mean you can’t keep reading (just as long as your alarm panel is listed here as being supported).

    If you don’t have an alarm system pre-wired in your home already, it might be worth spending the money to have one installed (sensors and keypad included). It will be the last cost you make since the monitoring will be free from that point forward. If you don’t intend do this, then the this blog won’t be of much use to you.

    This blog assumes that the zones (sensors/alarms) are already pre-configured too. Thus, your sensors are all hooked up to the alarm system already.

    Note: If you’re currently paying a 3rd party company to monitor your home, then you MUST cancel your contract with them before you proceed any further!

  2. ad2usb DeviceYou’ll need full access to your Honeywell alarm system (usually in the basement) and at least one keypad (usually near the front entrance). Most alarm systems are (and should be) locked up in a small box accessible with a key. You will want this key because you’ll want to lock everything up when we’re done too!
  3. AD2USB Security Big PictureYou’ll need one NuTech AD2USB device which is what will bridge/connect your security system to your Linux Server. The AD2USB device effectively just becomes a second keypad attached to your alarm system.

    At the time this blog was written, the NuTech AD2USB devices cost about $88USD ($119.99 CAN) ; a one time fee. The cost is well worth it since it’ll pay itself off in 2-4 months (if you compare it to the costs of a 3rd party home monitoring solution).

  4. Mini USB Cable
    A Mini USB to USB Cable
    You’ll need one Linux server pre-configured with CentOS or Red Hat 6.x (or 7.x).
  5. Enough wiring to get from the locked up alarm system to your Linux server. In my case, I am hooked up just below the panel, so I just needed about 4 feet. You can order this cable with your purchase;it’s only about .50 cents a foot (very inexpensive).
  6. Wire strippers.
  7. A small (1/4″) Philips screw driver.
  8. A Mini-USB to USB cable (no longer then 10ft).

Alarm Installer Code

Before we can even look or use the NuTech AD2USB device, we need to first alert our current alarm system of it’s presence. To do this, you need to be able to re-program a tiny portion of the alarm system. Don’t worry, it’s not as complicated as it sounds! This step is done through the keypad already installed in your house (usually at one of your front entrance ways).

The installer code grants us unrestricted access to our entire alarm system (through it’s programming mode). The installer code is a 4 digit (numeric code) and won’t be the same one you’re used to keying in when activating/deactivate your alarm.

If you already know your installer code you can access the programming mode by keying in the following and skipping to the next section (substitute [ABCD] with the installer code):
[ABCD] + [8] + [00]

If you don’t know what your installer code is then don’t worry, just keep reading…

Again, I can’t stress enough that you make sure you are ‘NOT’ still hooked up with an external monitoring service at this point because these next steps will disconnect them from your home (and could cause them to treat the situation as a crime in progress):

  1. Unplug your alarm system’s main power by simply unplugging the transformer nearby from the wall.
  2. Disconnect the battery backup from our alarm system. No need to unplug both terminals, just unplug one for the time being. We’ll be restoring the power back soon enough.

    Note: At this point you’ve effectively powered off your alarm system completely.

  3. To gain access to the programming mode and ‘take back’ the alarm system for your own, you need to just plug the transformer back in (leave the battery unplugged now).You now have 30 seconds to locate your alarm keypad and press and hold both the asterisks [*] and the hash button [#] down at the same time (for about ~1 to 2 seconds).
  4. If the above step worked correctly for you, then you should see the value of 20 written on your keypad’s display. If not, then you’ll need to repeat all of the steps above again.

Note: If your battery backup for your alarm system is still unplugged, now would be a good time to just plug it back in (so we don’t forget about it later).

Keypad Configuration

This step requires you to be in-front of the same keypad described in the last section. More importantly, you must be in it’s programming mode. This means you either used your known installer key code or you booted the alarm up in such a away that granted you access to the programming mode (without a code). If you’re unsure what I’m talking about here, make sure to re-read the last section before proceeding!

Your alarm system is effectively organized as a series of registers (think of them as mailboxes organized by 2-3 digit addresses). Registers are basically a spot you can store and retrieve information from. You’ll always press the [*] followed by the register you want to modify (using it’s address), followed by the new value you want to store in it. Alternatively you can always press the hash [#] key followed by the register just to display it’s contents instead.

So with respect to the Mailbox analogy, programming our alarm system through our keypad works like so:

  • [*] + [mailbox] + [new value]
    Save content into a specific mailbox (by address) and then print it to the display.
  • [#] + [mailbox]
    Just print the content of the mailbox (by address) to the display.

Simple enough right?

Just to avoid any confusion, when I group numbers below as [20] and [1234], they can still be interpreted as pressing (sequentially / one after the other) [2] + [0] + [1] + [2] + [3] + [4]. It’s easier to group the actions separately for readability; this way you can think of it with respect to the mailbox/content analogy.

While in programming mode, execute the following commands:

Command Details
[*]+ [20] + [ABCD] Substitute ABCD with a 4 digit code you want to become the new programmer code. This is what you can use in the future to get into programming mode (without having to power off the entire system again).
[*]+ [41] + [*] Remove the primary phone #’s of any external monitoring station that may or may not be hooked up. Even if this register is already empty; it doesn’t hurt to run this again just to ensure it really is empty. We don’t want our alarm system calling out to anyone.
[*]+ [42] + [*] Remove the secondary phone #’s of any external monitoring station (if any is set). Again…even if this register is already empty; it doesn’t hurt to run this command again just to ensure it really is empty.. We don’t want our alarm system calling out to anyone.
[*] + [191] + [10] Enabled keypad slot at Address 18; this is what the AD2USB device defaults to using. This default address should work for everyone.
[*] + [99] Exit the installer mode returning the keypad to it’s regular state

We’re not done yet though, if this is you first time using your alarm system, it might be wise to set a new Master Code and clear all existing User Codes. In case you didn’t already know, there are actually 3 types of users that are associated with our alarm system:

  1. The Installer: A key code that allows us to add/remove new keypads, acknowledge new sensors, configure zones, etc. In our case, our alarm system will only allows for 1 designated Installer Code.
  2. The Master: A key code that allows us to to create/delete Users of the system. In our case, our alarm system only allows us to have 1 designated Master Code.
  3. The User: Our alarm system allows us to program up to 31 of them (32 if you count the master). Residential owners will probably only need to set one user while a commercial business owner might set several for each of their employees. It will be our User Code that we’ll be using when interfacing with our alarm system 99% of the time. This blog will focus on just dealing with 1 main user.

So if you inherited this alarm, you may not know what was previously configured into it. You ideally should set a new Master Code (different then the Installer Code) and make sure there are no lingering User Codes active that we don’t know about.
Note: The commands identified above are called outside of programming mode. They can be issued at any time as long as your alarm is not armed. If you’re still in programming mode, just type [*] + [99] to exit it.

Command Details
[Installer Code] + [8] + [02] + [DEFG] Substitute [DEFG] with your new 4 digit Master Code using the Installer Code we setup above.
[Master Code] + [8] + [03] + [WXYZ] Substitute [WXYZ] with your new 4 digit User Code using the Master Code. This will become our main User account; the [03] identifies us as user #1. This is a code you’ll want to keep secret but can safely tell those whom you trust with access to and from your home.
[Master Code] + [8] + [04] + [#] + [0]

[Master Code] + [8] + [05] + [#] + [0]

[Master Code] + [8] + [06] + [#] + [0]

…

[Master Code] + [8] + [33] + [#] + [0]

Delete User #2 ([04]), #3 ([05]) and , #4 ([06]) in our alarm system (in case they exists). This step is entirely optional, but worth doing since you really don’t know how many users are actually set up from whomever had it before you.

If you’re really paranoid, you’ll iterate through all of the combinations identified here to cover of all 31 possible users where the last user (#31) is identified by [33]. In most cases your system will probably have just a few users (worst case) in it. Thus it’s probably safe to stop at [06] (user #4). I’ll let you use your own paranoia and discretion here as to how far you go.

You now have a security system and 3 Codes. The Installer and Master codes are useful to keep documented (but hidden) somewhere. The User Code will be the main code we use from now on; this code is safe to share with those you share your home with; obviously keep it safe from others.

Hook Up our NuTech AD2USB Device

First of all; let me point out to you that the Alarm Decoder’s Installation videos are a much better reference then anything I’ll identify in this section. I strongly encourage you to check the videos out (they’re only a few minutes) and get properly set up.

But in short, it really just boils down to connecting the AD2USB Device to our Home Alarm System and getting the wires correctly matched up and fastened tightly. From there you just connect the AD2USB device to your Linux Server via the mini-usb cable.

NuTech AD2USB Wiring In a Nutshell
NuTech AD2USB Wiring In a Nutshell

Your goal is to see the little green light blink; this is the devices internal heartbeat. It’s a good thing when this starts to flash on and off; if it isn’t, you’ll want to revisit your wiring!

Here is what my AD2USB device looked like after it was all hooked up. Obviously I can’t capture a pulsating green light in a single photo, so you’ll just to have to take my word for it that the light was blinking on and off. 🙂

NuTech AD2USB Device
NuTech AD2USB Device

At the end of the day, the AD2USB device is nothing but ‘another’ alarm keypad in your house. As far as the Alarm System is concerned, there is no distinguishable difference at all. What makes the device so powerful is the fact that now we can interface with it using software from anywhere!

Software Installation

Now there are many ways to go about installing the software; specifically our core components: alarmdecoder, ser2sock, and alarmdecoder-webapp. One way is to use the instructions right on their GitHub page. If you go this route, then you’re done with this section of the blog.

However, I’ve spent hours (sadly) making it possible for these tools to just work in both CentOS 6.x and 7.x via installable RPMs. I strongly encourage you to go this route for several reasons:

  • You won’t be required to compile anything.
  • I handle SELinux for you so you won’t have to to disable it!
  • I handle all of the initial (and complicated configuration) out of the box for you.
  • I handled all of the system dependencies for you.
  • It’s easier to provide safe upgrades and version control through RPMs. Relying on pypi (pip) instead will make it difficult to reproduce your working setup since the packages there change constantly (for better of for worse).

If you want the RPM approach, then keep reading…

Install Core Components

You must be connected to my repository for this to work:

# Assuming you're correctly hooked up to the nuxref repositories:
yum --enablerepo=nuxref --enablerepo=nuxref-shared \
    alarmdecoder ser2sock \
    alarmdecoder-webapp alarmdecoder-webapp-selinux

Note: the SElinux package has to interact with the kernel to enable 4 web ports Alarm Decoder (WebApp) uses. During this time, it will appear as though your installation froze and/or hung. Do not abort this task, it’s completely normal to take up to a minute or more to do this. If you’re not using SELinux then you don’t need this package.

Enable ser2sock; this will convert your AD2USB device (in /dev/) into a TCP/IP stream. This is much easier to work with (and debug)

# Enable ser2sock and start it
chkconfig ser2sock on
service ser2sock start

If your ser2sock doesn’t start, then you have a problem with your AD2USB configuration and/or hookup. Feel free to have a look at /etc/ser2sock/ser2sock.conf to see the configuration it’s using. Alternatively, the next section discusses some troubleshooting steps you can do.

Start up our Alarm Decoder Webapp

Even if you’re using systemd, these commands will work (it’ll call the proper command for you).

# Enable gunicorn (back-end webserver)
chkconfig ad2web on
service ad2web start

# Enable NginX (front-end web hosting and static file control)
chkconfig nginx on
service nginx start

# If it was already running, then:
service nginx reload

You’ll need to open ports 5080 and 5443 on your firewall. CentOS/Red Hat 6.x users should be able to apply something like this in their /etc/sysconfig/iptables file:

# add (something like) this to your /etc/sysconfig/iptables
#---------------------------------------------------------------
# Home Security Monitoring
#---------------------------------------------------------------
# 192.168.1.0/24 is the subnet of my local home network and eth0
# is the device it access.
-A INPUT -i eth0 -s 192.168.1.0/24  -p tcp --dport 5080 -j ACCEPT
-A INPUT -i eth0 -s 192.168.1.0/24  -p tcp --dport 5443 -j ACCEPT

If you’re using CentOS 7.x, then you’ll need to use firewalld to accomplish the same task:

# Open the port in your home network
firewall-cmd --zone=home --add-port=5080/tcp --permanent
firewall-cmd --zone=home --add-port=5443/tcp --permanent
firewall-cmd --reload

I do not recommend opening the ports to the public (zone) network unless you strap on a bit of extra security such as fail2ban. This blog is already long enough, so I won’t discuss that here. Although, you can check out a blog I did on security that can get you started with fail2ban (6.x users).

It’s worth confirming at this point that everything is up and running too:

# This command just checks that we're properly listening on the
# ports we care about (with respect to Alarm Decoder):
netstat -nat | egrep LISTEN | egrep ':(10000|5(00[01]|080|443))'

The command above should produce (or similar to):

tcp        0      0 0.0.0.0:5443                0.0.0.0:*                   LISTEN
tcp        0      0 0.0.0.0:5000                0.0.0.0:*                   LISTEN
tcp        0      0 127.0.0.0:5001              0.0.0.0:*                   LISTEN
tcp        0      0 0.0.0.0:10000               0.0.0.0:*                   LISTEN
tcp        0      0 0.0.0.0:5080                0.0.0.0:*                   LISTEN

The TCP ports break down as follows:

  • 5080: NginX non-encrypted hosting of the Alarm Decoder WeAapp.
  • 5443: NginX secure hosting of the Alarm Decoder WeAapp.
  • 10000: Ser2Sock (this is converting you’re AD2USB to a TCP/IP stream that the Alarm Decoder WebApp can read and display to you).
  • 5000: This is forked from the gunicorn instance to handle all incoming requests from NginX
  • 5001: This is the gunicorn application which helps with handling some of the backend tasks such as the detection of your AD2USB device.

You should be able to access the keypad website now; simply visit http://localhost:5080/ or it’s secure version https://localhost:5443/. If you’re configuring this on another server, we’ll then you’ll need to access the host/ip accordingly.

Alarm Decoder WebApp Installation
Alarm Decoder WebApp Installation

You can accept the defaults for everything, just keep pressing next.

Next 3 Setup Screens
Next 3 Setup Screens

When you’re all done with the setup and have created your administrative account, you’ll be able to sign in an access your new virtual keypad.

Alarm Decoder Keypad
Alarm Decoder Keypad

You can explore the software and take advantage of all of it’s features. You can setup a twilio account and have the Alarm Decoder WebApp send you a text message whenever an even occurs. You can have it email you, and/or use a variety of other messaging services too!

If the page still isn’t coming up, it might be because you have SELinux enabled. Unlike other bloggers, I will NOT tell you to disable it (although that’s an option). Instead I’d advise that you install the SELinux package I put together:

yum --enablerepo=nuxref alarmdecoder-webapp-selinux

Some other useful things worth knowing:

  • /etc/nginx/conf.d/alarmdecoder-webapp.conf: Here is your NginX configuration file. This is the front end of your keypad.
  • /etc/alarmdecoder-webapp/alembic.ini: This is another configuration file used by the website. It defines information such as the database type and location. By default, it’s just a simple sqlite database. But feel free to point it to a PostgreSQL or MySQL database too (see the sqlalchemy.url directive).
  • /etc/ser2sock/ser2sock.conf: This is ser2sock configuration file.
  • /etc/alarmdecoder-webapp/ad2web.conf.py: This is where your gunicorn configuration is defined.
  • /var/lib/alarmdecoder-webapp/instance/db.sqlite: Here is the database the website uses and interfaces with.
  • /var/lib/alarmdecoder-webapp/instance/logs/: Here are where you’re logs will appear from the webapp.

Troubleshooting

  1. First make sure CentOS (or Red Hat) can see your NuTech AD2USB device properly. 
    # This ideally should return something like /dev/ttyUSB0
find /dev -mindepth 1 -maxdepth 1 -type c -name 'ttyUSB*'

# If you're using my RPM files, you'll also have another
# entry you can look for. The following should output 
# something like /dev/alarm-XXXXXXXX where the X's represent
# the Serial # of your NuTech AD2USB
find /dev -mindepth 1 -maxdepth 1 -type l -name 'alarm-*'

    If you can’t find your device, then:

    • Double check that the USB connections are firmly plugged in to both your PC and the AD2USB device.
    • You’ll want to see a pulsating green light on the AD2USB device signifying it’s getting power and running.
    • NuTech also make mention that the USB cable you use can not be longer then 10ft. Ideally the shorter the cable the better though (less noise/interference).

    If your device does exist, then read on to the next step…

  2. Ensure that the Honeywell Alarm System setup has been correctly configured to support a second keypad. This was discussed above in the Keypad Configuration section of the blog. Specifically you need to be in Programming Mode and enable keypad address 18 (done via the command: [*] + [191] + [10]).

    You can verify that you’ve configured this correctly by accessing the Programming mode of your alarm keypad and typing: [#] + [191]. The device should output either a 10 or 0A (both are valid responses).

  3. Make sure your NuTech AD2USB can properly communicate with your Alarm system.
    I took extra care and time packaging everything, so I actually have another RPM you can install that may help you out:

    # Assuming you're correctly hooked up to the nuxref repositories:
yum --enablerepo=nuxref alarmdecoder-cli

    This will grant you access to a few scripts I created based on a couple of great examples the Alarm Decoder team put on their GitHub page:

    # These commands require you to be the superuser (root)
# or a user that is as a member of the 'alarm' group.

# Test your data feed (through ser2sock):
ad2-disp-sock

# Or test your block device directly:
ad2-disp-dev

    Note: If ser2sock is running properly, you will not be able to test the block device (using ad2-disp-dev) until you stop it first. So don’t panic if the ad2-disp-sock works and ad2-disp-dev doesn’t!

    You can adjust these test tools by editing /etc/ad2cli.conf (if the defaults
    aren’t working for you).

    If you can’t communicate properly, then you may need to recheck your wiring. Review the NuTech YouTube videos again if you need to.

    If you still can’t communicate with your alarm system after confirming all of the suggestions above, then you may have a faulty device.

Don’t give up if you’re having problems. It’s worth searching the Alarm Decoder Forums for an answer. Post a question there too if you like. The Alarm Decoder team are a great bunch of people always willing to help.

What If

  • What if the thief cuts the power?
    In my case, my alarm system (yours should too) has a battery backup locked up tight with it. So it’ll keep monitoring in this situation no matter what. It wouldn’t hurt to connect both your server internet modem into a Uninterruptible Power Supply (UPS). These things are relatively cheap for what they offer. A UPS would certainly handle this situation.
  • What if the thief powers off the computer within the 30 second deactivation window?
    Well, if they are this clever then yes, they cut you off from remote monitoring. But that’s all they do. Keep in mind that the 30 second deactivation timer is still counting down. It will reach zero… and when it does…

    A properly installed alarm system will come with a siren (I know mine does) and they are quite loud and irritating! The siren will sound regardless of the NuTech Alarm Decoder, Internet or PC get shut off or unplugged. It might be a good idea to make sure your siren is installed in the attic where it is difficult to get to (and disable). Most people and installations place these devices in the basement. This is silly because a thief can just smash it or yank the wire from it to kill the sound. Either way, unless the thief knows your setup, he’s certainly playing with fire if he thinks silencing the siren will keep him safe.

    Consider a backup plan; I mean what would the alarm company do if they could no longer suddenly monitor you’re home. They’d probably immediately attempt to contact you and then call the police. You could do the same. Call a neighbor, call a friend. The fact you even just have a heads up on the situation is more then the guy who doesn’t have any monitoring solution at all.

    Another great tool you can use is Nagios; I wrote a blog on it a while ago. Pair that up with aNag (for Google Devices) or iNag (for Apple Devices), and you’ll be monitoring your ability to monitor. You can be notified immediately if your monitoring capabilities diminish.

    On a side note, consider notifying your (trustworthy) neighbor(s) too of weeks you intend to be out of town, I’m sure they can alert authorities for you if they hear your house alarm going off.

Home Security Tips

Remember, since you’re doing the monitoring yourself, you need to be prepared for when or if a situation ever arises. There is no longer a remote monitoring station looking out for you. But remember that even they can fail to pull through, so the following information is useful regardless of what your monitoring situation is.

I am not a trained professional when it comes the safety and responsibilities (nor will I claim to be). But there are clear cut obvious things worth stating up front that you must always consider:

  • If you’re at home at night and the alarm goes off:
    1. It’s our human nature to panic, but just don’t let the panic take over the situation.
    2. Don’t ever second guess the potential danger of the situation!
    3. Do not ever decide to be brave and investigate the situation unless you’re absolutely sure!!
    4. Call 911 immediately. If the phone line is dead, then use a cell phone if you have one.
  • If you’re at work or on vacation and the alarm goes off:
    1. Ask Yourself? Could it be your spouse? A Child? A Pet? Does the timing of this alarm jive with a notable time of day such as after school (maybe the kids got home)? Perhaps it’s around the time your spouse returns from work or other family member?
    2. Can you contact a specific person in mind directly based on the time of day (with respect to the alarm going off)?

      Remember though: if this truly is a robbery, then time is not on your side! The longer you wait, the more your home is exposed.

    3. If you can’t be sure, then call the local police immediately!. If you fear someone else could be home during this time and at risk then call 911! Don’t think twice!

Make sure you have your local police contact information programmed in your cell phone. Alternatively, keep it on a sheet of paper in your wallet or purse.

If you have children, make sure they know what to do if they hear the alarm go off. If they have access to a phone, they should know how to call 911 during these kind of emergencies. Obviously their safety becomes your number one concern, so be sure to educate them to hide someplace close by. Having them leaving the room their in when the alarm is going off could put them in serious jeopardy depending on the situation.

Credit

This blog took me a very (,very) long time to put together and test! The repository hosting alone accommodates all my blog entries up to this date. If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

Special thanks to the Alarm Decoder team; their support made it possible for me to write this blog. In return I was able to make it easier for everyone else!

Sources