Tag Archives: RHEL

NRDP for Nagios Core on CentOS 7.x

Introduction

In continuation to part 1 of this blog series… NRDP (Nagios Remote Data Processor) is a Client/Server plugin for Nagios. With the NRDP model, it is the Application (or the server hosting it) that is responsible for reporting in it’s status. The beauty of this is you can have applications installed all throughout your business, your home, across data centers, overseas, etc all reporting to 1 single Nagios instance. You can monitor your entire infrastructure with this tool.

NRDP Overview

NRDP Overview


The diagram above illustrates how the paradigm works.

  • The A represents the Application Server. There is no limit to the number of these guys.
  • The N represents the Nagios Server. You’ll only ever need one Nagios server.
  • The Application (and/or the server hosting it) will make periodic status reports to the Nagios server. It should report that everything is okay, or report that it isn’t. Out of the box (using my rpms) Nagios will wait for up to X minutes (configurable) for a message to be received before reporting the service/host as being offline. Ideally you should send at least 2 or 3 notifications within this grace period. Currently X is set to 5 minutes with the templates in the RPMs I provide. But you can change this.
  • If Nagios doesn’t hear from the Application for an extended period of time, then it assumes the worst and reports to the user that there is a problem.

NRDP is a newer chapter in the Nagios world which allows you to monitor more applications you wouldn’t have otherwise been able to do. NRDP replaces it’s predecessor NSCA which performs the exact same task/function. One of the key differences is that NRDP is much easier to set up and use than NSCA is. NRDP is also much less lines of code too! The RPMs I provide make the setup even easier. That said, if you’re interested in using NSCA instead, I still package it here (and client here). But lets get back to talking about NRDP….

NRDP Clients pushing to Central NRDP Server

NRDP Clients pushing to Central NRDP Server


Here is what you’re getting with this blog and the packaged rpms that go with it:

  • Nagios Core v4.x: updated RPM packaging which I continued to maintain and carry forward from my previous blogs.
    Note: You will need this installed in order to use NRDP. See part 1 of this blog series for more information if you don’t already have it set up.
  • NRDP (Nagios Remote Data Processor) v1.x: My custom RPM packaging to just work for everyone right out of the box. Here are some modifications I made to the package to make our lives easier:
    • The default insecure port is configured to use 5668.
    • The default secure port is configured to use 5669.
    • I included my own Apache configuration so that the service would run right out of the box.
    • There are a several patches I had to make in order for the NRDP package to even work in our CentOS environment.
    • There is firewall configuration all ready to use with FirewallD.

Best of all, with my RPMs, you can run SELinux in full Enforcing mode for that extra piece of mind from a security standpoint!

The Goods

For those who really don’t care and want to just jump right in with the product. Here you go!

You can download the packages manually if you choose, or reference them using my repository:

Package Download Description
nrdp el7.rpm The NRDP Server: This must be installed on the same server running Nagios. It is in charge of listening to requests sent by nrdp-clients and reporting their status directly to Nagios. Don’t forget to change your tokens which we’ll discuss further down if you don’t know what I’m talking about!
nrdp-selinux el7.rpm An SELinux add-on package that allows the NRDP Server to operate under Enforcing Mode.

Note: This RPM is not required by the NRDP server to run correctly.
nrdp-client el7.rpm The NRDP Client: You’ll only need to optionally install this on to the remote servers that will be sending their status along to your NRDP Server. Since the protocol is so simple to adapt, you can also just choose to build the client aspect into the program you want to monitor with. If you do choose to install this RPM, you get access to a small program called send_nrdp.php which allows you to post status messages to the Nagios Server.
nrdp-doc el7.rpm Just some documentation that is already publicly available on NRDP’s website.
Note: This RPM is not required by Nagios to run correctly.

Note: The source rpm can be obtained here which builds everything you see in the table above. It’s not required for the application to run, but might be useful for developers or those who want to inspect how I put the package together.

NRDP Server Side Configuration

The NRDP Server is only going to work if it’s installed on the same server as Nagios.

Assuming you hooked up to my repository here, the NRDP server can can be easily installed with the following command:

# Install NRDP (Server) on the same server as the one running Nagios
yum install nrdp nrdp-selinux

NRDP was written in PHP and therefore depends on a few small tweaks on your part. For one, you’ll want to make sure you have a timezone configured too to silence any potential errors and keep the reporting consistent. You’ll want to do this from your /etc/php.ini file; look for a directive called date.timezone. Here is a list of supported timezones by PHP. But you can also just do this which works too:

# This clever trick (a 1 liner) lets the timezone in the /etc/php.ini
# to that of whatever your current system is set to!
sed -i -e "s/^[; \t]\?\(date\.timezone\)[ \t]*=.*/\1 = \"$(date +%Z)\"/g" \
   /etc/php.ini

If Apache isn’t running; now’s the time to start it up.

# This simple command should be ran as root and will
# start Apache if it isn't running already, otherwise it
# will send a reload signal if it is.  Regardless,
# after you finish this command, the new NRDP Configuration
# will be running and ready to go.
systemctl status httpd.service && \
   systemctl reload httpd.service || \
   systemctl start httpd.service

# Make sure we start up on reboots (if we haven't done so already)
systemctl enable httpd.service

NRDP/Nagios Configuration

You need to let Nagios know what services it should expect notifications from. The key thing that separates these entries from any other Nagios entries you specified is the ‘use‘ directive. For hosts reporting in, you’ll want to use the generic-nrdp-host directive. For services reporting in, you’ll want to use the generic-nrdp-service entry.

A Configuration file on your Nagios Server (the same server running your NRDP Server) might look like this:

cat << _EOF > /etc/nagios/conf.d/somehost.cfg
# We might have a test server defined like this
define host {
    # Name of host template to use (notice we're using the
    # 'generic-nrdp-host' template).  If you want to actively ping
    # this server and not await to hear from it, you can change this
    # to use the 'linux-server' template instead.
    use                         generic-nrdp-host
    # Hostname
    host_name                   somehost
    # Human Readable
    alias                       My Test Server
}

# Now we might identify a component to associate with our test server called <em>somehost</em>.
define service{
   use                 generic-nrdp-service
   service_description someservice
   host_name           somehost
}
_EOF

With the templates I provide, Nagios will wait for up to 5 minutes for a notification before assuming nothing is coming (and changing the state to CRITICAL). If this is too long (or too short) of a wait time for you, you can change it by modifying the NRDP templates (or creating your own). The provided NRDP templates can be found here: /etc/nagios/conf.d/nrdp.conf. You’ll want to focus on the entry entitled freshness_threshold which is currently set to 300 (300 seconds is equal to 5 minutes) and change it to your liking.

Alternatively, you can over-ride the template value by just adding the freshness_threshold to the service configuration you create like so:

cat << _EOF > /etc/nagios/conf.d/somehost.10min.service.cfg
# Over-ride our template timeout with a new one specified
# in our service declaration
define service{
   use                 generic-nrdp-service
   service_description someservice
   host_name           somehost
   # Over-ride the defaults in our template and only go stale
   # if 10 minutes elapses (600 seconds = 10 min)
   freshness_threshold 600
}
_EOF

If you’ve modified your configuration in anyway; be sure to reload Nagios so that it can take in our new configuration:

# Reload our service (if it's running)
systemctl reload nagios.service

# Be sure to start it if it isn't already:
systemctl start nagios.service

Testing your NRDP Server Setup

Once NRDP is up and running it will provide you a simple website you can use to interact with it. This interface allows you to manually send passive checks through to Nagios which is really useful for testing if everything is working okay!

Here is what the test page looks like:

NRDP Server Side

NRDP Server Side – use nuxref as a token to test with.

Note: The blog does not touch on the Submit Nagios Command portion of this page; but that’s another topic for another day. You want to focus on the Submit Check Data section.

To use NRDP you will be required to provide a token with every submission you ever make to it. This is just a security feature to prevent people from sending messages into your system who shouldn’t be. By default (if you’re using my rpms), the token is nuxref. You’ll notice in the above example, I’ve gone ahead and filed this field in to show you where it goes.

To change this token (which you really should do) just open up the php file /etc/nrdp/config.php. You can specify as many tokens as you like that you wish to accept. Here is what you’re looking for:

// ...
// look for the authorized_tokens directive
// and add your tokens here:
$cfg['authorized_tokens'] = array(
   "the-longer-and-more-encrypted-your-token-is-the-better",
   "specify-as-many-tokens-as.-you-want",
);

If you followed the blog so far and created the test examples I provided (above), then you can now submit a passive check using the default settings; don’t forget to provide your token!

If everything goes according to plan, you should be able to check out your message safely being acknowledged:

NRDP Test Submission Results (using defaults)

NRDP Test Submission Results (using defaults)

NRDP Server Security

Since you can access a very simple NRDP webpage on the same server hosting Nagios via http://localhost:5668/ (insecure) or https://localhost:5669/ (secure), you’ll want to protect it from those who shouldn’t be visiting it.

The below commands open up the protocol to everyone on your network. You should only perform this command if your Nagios Server will be running on a local private network:

# Enable insecure NRDP port (5668)
firewall-cmd --permanent --add-service=nrdp
firewall-cmd --add-service=nrdp

# Enable secure NRDP port (5669)
firewall-cmd --permanent --add-service=nrdps
firewall-cmd --add-service=nrdps

If you’re opening this up to the internet, then you might want to just open the (NRDP) port(s) exclusively used by the remote applications you intend to allow reporting from. It might be worth investing time into fail2ban as well as an additional precaution. The following presumes you know the IP of the application server and will open access to ‘JUST’ that system:

# Assuming 1.2.3.4 belongs to a system you trust who will be sending you
# Nagios status reports via NRDP:
firewall-cmd --permanent --zone=public \
   --add-rich-rule='\
       rule family="ipv4" \
       source address="1.2.3.4/32" \
       port protocol="tcp" \
       port="5668" \
       accept'

# Here is the Secure version of the same command
firewall-cmd --permanent --zone=public \
   --add-rich-rule='\
       rule family="ipv4" \
       source address="1.2.3.4/32" \
       port protocol="tcp" \
       port="5669" \
       accept'

Note: Again, I want to stress you should only open ports you intend to use and to the Application Servers who intend to use it. If you don’t intend on using the insecure port, then don’t even open it on your firewall!

Once you’re satisfied that everything is working. You might want to disable GET requests on your NRDP Server. This will prevent people from using the test website and only accept POST requests. To do this, just have a look in /etc/httpd/conf.d/nrdp.conf and uncomment (hence, remove the #) the line that reads:

RewriteCond %{REQUEST_METHOD} !^(POST) [NC]

You’ll need to reload your Apache server for the changes to take effect. If ever you want to test again, just comment the line back out again:

# This simple command should be ran as root and will
# start Apache if it isn't running already, otherwise it
# will send a reload signal if it is.  Regardless,
# after you finish this command, the new NRDP Configuration
# will be running and ready to go.
systemctl status httpd.service && \
   systemctl reload httpd.service || \
   systemctl start httpd.service

# Make sure we start up on reboots (if we haven't done so already)
systemctl enable httpd.service

NRDP Client Side Configuration

The protocol is so simple, that you can easily adapt this into programs you write or tool-sets. You can also just simple use the tool provided

# Install Nagios Core using NuxRef
# See: http://nuxref.com/repo for more information
# Install NRDP Client (CLI) on any server you want to
# be able to report it's status to (onto Nagios)
yum install nrdp-client

Now… from our application server we would want to install our nrdp-client package. so that we can access our send_nrdp.php tool.

The tool is very simple to use; here is an example how you can send a simple status message to our Nagios server:

# The below sends a host acknowledgment to our NRDP server
# url:   web address to our Nagios server
# token: the security token (this is to prevent anyone from
#        sending status requests to your Nagios server.
#        The default is 'nuxref' until you change it (which you
#        really should consider doing)
# state: This will work the color coding of Nagios, you can
#        specify one of the following:
#           0 - OKAY (green)
#           1 - WARNING (yellow/orange)
#           2 - CRITICAL (red)
#           3 - UNKNOWN (blue)
# output: Attach a message with our state; this will be visible
#         as well from the Nagios display screen:

# Here is a NRDP notification:
send_nrdp.php --url=http://nagios.server.addr:5668 --token=nuxref \
   --host=somehost \
   --state=0 \
   --output="Test is is working great."

# This will send a warning status to our component we identified
send_nrdp.php --url=http://nagios.server.addr:5668 --token=nuxref \
   --host=somehost \
   --state=0 \
   --output="Test is is working great."

Note: It’s worth noting that there are no WARNING states for hosts. A zero (0) will report that it is online, and anything else will report it as CRITICAL.

NRDP Protocol

Are you a developer? If you are, you may want to avoid using the script and just talk directly to the NRDP Server from your application. It’s incredibly easy to do. If you’re not, don’t worry; the send_nrdp.php script looks after all of this for you. You can still read this section if you’re interested none the less.

NRDP is literally just a small PHP website that relays XML it receives via it’s restful API into a language that Nagios can interpret. In fact NRDP is just a website that writes specially formatted files into the /var/nagios/spool/checkresults/ directory to which Nagios scans for processing.

The NRDP Payload

The payload is XML; the schema looks like this:

<?xml version='1.0'?> 
<!-- For checking a host -->
<checkresults>
    <!-- The type can be either 'host' or 'service' -->
    <!-- The checktype is always 1 for relaying passive status updates -->
  <checkresult type='host' checktype='1'>
    <hostname>somehost</hostname>
    <!-- 0 = Success, 1 = Warning, 2 = Critical -->
    <state>0</state>
    <!-- A message we might want to associate  -->
    <output>Our somehost is behaving perfectly!</output>
  </checkresult>
</checkresults>

Easy peasy right? A Service message looks like this:

<?xml version='1.0'?> 
<!-- For checking a service -->
<checkresults>
    <!-- The type can be either 'host' or 'service' -->
    <!-- The checktype is always 1 for relaying passive status updates -->
  <checkresult type='service' checktype='1'>
    <!-- This has to be the hostname associated with the service -->
    <hostname>somehost</hostname>
    <!-- This has to be the service we configured using our NRDP
         template.  It's identified by the `service_description`
         field as part of our nagios configuration. -->
    <servicename>someservice</servicename>
    <!-- 0 = Success, 1 = Warning, 2 = Critical -->
    <state>2</state>
    <!-- A message we might want to associate  -->
    <output>Danger Will Robinson! Danger!</output>
  </checkresult>
</checkresults>

You can stack as many messages as you want in a single payload which really adds to NRDPs flexibility:

<?xml version='1.0'?> 
<!-- A mix of status updates -->
<checkresults>
  <checkresult type='host' checktype='1'>
    <hostname>somehost</hostname>
    <state>0</state>
    <output>Our testserver is behaving perfectly!</output>
  </checkresult>

  <checkresult type='service' checktype='1'>
    <hostname>somehost</hostname>
    <servicename>someservice</servicename>
    <state>2</state>
    <output>Houston, we have a problem.</output>
  </checkresult>

  <checkresult type='service' checktype='1'>
    <hostname>somehost</hostname>
    <servicename>my-other-service</servicename>
    <state>1</state>
    <output>Let's hope the problem goes away.</output>
  </checkresult>

</checkresults>

The Transaction

Since it’s a PHP website, it’s hosted via Apache, NginX, etc. This blog sets up Apache but nothing is stopping you from using another service.

This means however that all interactions are web requests. A transaction must be an URL encoded HTTP POST and it must include a special token as part of its payload (part of the verification process). If the token specified is invalid or missing, then the NRDP server will just reject the message. The same rules apply if you don’t POST the message.

You could accomplish this with Python like so:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Site: http://nuxref.com
# Desc: A simply python snip-it to show you how easy it is
#       to interface with NRDP server:
#
# Note: 'requests' (which is required) can be installed 
#       like so (if you don't already have it):
#  #> yum install python-requests
import requests

xml = """<?xml version='1.0'?> 
<checkresults>
  <checkresult type='service' checktype='0'>
    <hostname>somehost</hostname>
    <servicename>someservice</servicename>
    <state>0</state>
    <output>Everything is running great!</output>
  </checkresult>
</checkresults>"""

params = {
   'token': 'nuxref',
   'cmd': 'submitcheck',
   'XMLDATA': xml,
}
r = requests.post(
    'https://nagios.server.addr:5669',
    params=params,
    headers={'Content-Type': 'application/xml'},
    # This is only in place for those who have self signed secure
    # certificates for private networks; you may wish to change
    # this to True in a real environment if you have the ability
    # to verify the host
    verify=False,
)

# Ideally you want to see: 200
print(r.status_code)

# Print our data
print(r.text)

Credit

This blog took me a very (,very) long time to put together and test! The repository hosting alone accommodates all my blog entries up to this date. All of the custom packaging described here was done by me personally. I took the open source available to me and rebuilt it to make it an easier solution and decided to share it.

NRDP certainly does not work like this out of the box; here were the patches I created (see pull request #12 on NRDPs Repository for upstream pushes of this):

  1. nrdp-php_headerfix.patch: Fix PHP header so tool can be executed on the shell.
  2. nrdp-basic_authorized_token.patch:Default (nuxref) token for an out-of-the-box working solution.
  3. nrdp-date.timezone.warning.patch: Silence potential Timezone Warning Message.
  4. nrdp-permissions.patch: Better passive check permission handling for NRDP/Nagios
  5. nrdp-client-support_ports.patch: Support more then just port http 80 (as this blog sets up our NRDP server using ports 5669 and/or 5668).

If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

Sources

SABnzbd Installation for CentOS 7.x

Introduction

SABnzbd is a versatile tool written in Python specifically designed to take an NZB-File as input and then retrieve all of the content defined inside of it for you automatically. Recently there have been new improvements with SABnzbd in the past few releases. So I thought I’d share a working out of the box solution using RPMs. In addition to this, I’ve packaged up my notification script I wrote which can further enhance this great product.

SABnzbd and Usenet

First off; lets summarize Usenet for those who don’t use it or aren’t already familiar with it. In the simplest terms: Usenet is a great big fucking mess; a total disaster. It’s basically a location where literally anyone with access to it can post/upload all they want at their free will anonymously. Don’t get me wrong; this is cool too because it’s basically one very large hard drive without anyone telling you what you can and can’t place onto it. Seriously though, just consider for a second what your computer would look like if you allowed absolutely anyone who has access to the internet to upload content freely and without rules to it. You’d probably run out of hard disk space quickly; and you’ll be left with a lot of content everywhere. Well; that’s exactly what Usenet is (greatly simplified); it’s one large centralized location filled with petabytes of data. The thing with Usenet though is… it isn’t, and won’t be running out of disk space anytime soon.

The good news is, (thankfully) people sift through the heaps of information constantly being posted (onto Usenet) using automated tools regularly. These tools that do the sifting are generally known as Indexers. I talk more about this in another blog I wrote here a while back. But basically whenever an Indexer finds some useful data, it records the location as to where it was found. Indexers record this information in a special file call an NZB-File. NZB-Files are effectively treasure maps containing the coordinates on Usenet to which a specific piece of data can be located at.

NZB-Files are like Treasure Maps

NZB-Files are like Treasure Maps

It should be known that if you’re planning on posting/uploading stuff to Usenet yourself (for backup purposes or what have you), most (good) software will generate you an NZB-File afterwards allowing you to retrieve your data back again later on.

Now lets throw SABnzbd into the picture because it specifically is designed to retrieve content based on an NZB-File it’s provided. The process looks a little like this:

SABnzbd Download Process

SABnzbd Download Process

  1. Acquire NZB-File: Presumably you already have it because you’re just trying to retrieve data you posted in the past. But alternatively, the NZB-File could have also been acquired from and Indexer too.
  2. Give SABnzbd Access to NZB-File: You simply hand off your treasure map to the application who’s actually going to go out there and get the content for you.

    This is truly the bread and butter of SABnzbd. The next set of steps identified below are all fully automated (and done behind the scenes) for you. They require nothing from you, but it’s worth explaining it for those who are interested.

  3. SABnzbd Establishes a Connection to Usenet: The catch with SABnzbd is it’s merely a vessel for fetching the data. You need to have provided it access to Usenet to which it will fetch this data from. Thus SABnzbd must (obviously) connect to a Usenet provider in order to retrieve data from Usenet itself. This step is put in place to make sure you’ve got yourself an account with a provider!
  4. SABnzbd Downloads Content: Segment by segment, each portion of your data is downloaded and re-assembled. There are lots of additional things that go on too such as making sure the data isn’t damaged and attempting to correct it (all automatic) if it is.
  5. SABnzbd Saves Data:If all goes well; and in most circumstances it will. You will have all of the content successfully retrieved at this point!

SABnzbd has tons of automation built into it; I’m really just focusing on the basics here to get you going.

I Want To Try It

Of course you do! So here’s the thing; if you’ve already connected to my repository here, you can get it by simply typing the following:

# Download and install SABnzbd plus the Notification Addon
# on CentOS 7.x using the nuxref repositories located
# here: http://nuxref.com/repo/
yum install -y --enablerepo=nuxref \
               --enablerepo=nuxref-shared \
               sabnzbd sabnzbd-script-notify

You can also just visit the location I host these packages directly (via my repository) here and download the RPMs for yourself:

Note: I provided the source rpms optionally; they are not required unless you want to build this for yourself from scratch.

SABnzbd Environment

It’s worth giving you a quick rundown of how the RPM installs itself upon your computer: First off, it creates a general user/group called sabnzbd. By default this is the user/group it will run as. To grant a user access to all of the content retrieved by SABnzbd, you can just add yourself to the sabnzbd group:

# as root; we can add ourselves to the sabnzbd group
usermod -a -G sabnzbd myuserid

# Note: if this user is logged in, they will have to log out and log
#       back in to have these new security credentials noticed. 

You can start it up with the command:

# as root; start up SABnzbd:
systemctl start sabnzbd.service

You’ll be able to access the web page through your browser by punching in http://localhost:9080.

You can set it up to survive reboots with the following command:

# as root; start up SABnzbd when the machine is first
# powered on:
systemctl enable sabnzbd.service
  • All of your log files will show up in /var/log/sabnzbd/sabnzbd.log
  • All of configuration will get written to /etc/sabnzbd/sabnzbd.conf
  • All of the variable data (file processing, etc) will be located in /var/lib/sabnzbd/*. In fact this is a very important directory because unless you configure things differently, all downloaded content will appear in /var/lib/sabnzbd/complete.

Firewall Configuration

The package will provide you the files needed to set up the firewall and make SABnzbd available to you from other stations by simply doing the following:

# as root; start up SABnzbd when the machine is first
# powered on (assuming your network is set up to the `home` zone
firewall-cmd --zone=home --add-service sabnzbd --permanent

# Now reload your firewall to take on the new change:
firewall-cmd --reload

SABnzbd Notifications

SABnzbd can keep you posted on what it’s doing by sending you emails when a download completes (or fails). It can send you an notification on Pushbullet, and a few others too.

If you want to use the Notify script I wrote, you’re already almost set up and ready to go because it’s in the sabnzbd-script-notify rpm you already installed.

To enable it, you simply need to need to access the Notifications tab from within the SABnzbd Configuration section.

SABnzbd Notification Setup

SABnzbd Notification Setup

  1. Select the Enable notification script checkbox
  2. Select the Notify.py script from the dropdown list next to the Script category
  3. Next to the Parameter category, you must specify the URL(s) identifying which service(s) you want to notify.

    Depending on what you want plan on alerting, the URL(s) you specify in the Parameter field will vary. You can get a better understanding of the URL options supported here.

Credit

This blog took me a very long time to put together and test! The repository hosting alone accommodates all my blog entries up to this date. If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

Special thanks to Safihre for reaching out to me and allowing me to contribute to their product with the notification add-on.

Sources

Repository Hosting

Hosting Your Own RPM Repository

Introduction

There are lots of reasons why you might want to host your own RPM repository. A few reasons up front:

  • Speed: Performing a yum (or dnf) update from one of your locally hosted repositories is much faster than checking the ones on the internet!
  • Cost: If you run a business, then you know bandwidth can be expensive. If Linux is the operating system your developers and/or employees use, then hosting the repository internally means they won’t waste bandwidth each time one of them performs a system update. You’ll have downloaded everything once, and provided a central controlled share point they can apply all of their updates from.
  • Retention: Over time, your successfully deployed systems will age and the packages that were once readily available for it may be a lot harder to acquire! Sometimes it’s a good thing to create and manage your own internal repository so that you always have access to the exact RPMs you used to deploy/create your development environment with. This is especially the case for that one rare day that arises requiring you to go back and debug a legacy system of yours. If the packages are already available to you locally; you should have no problem reproducing any detected problems since you’ll be able to reconstruct their environment exactly.
  • Mock: I love this tool; mock allows you to dynamically generate different distributions of CentOS/Fedora (new and old) allowing you to build your applications to test them in it. You simply create a mock environment of the distribution of interest. In this new environment, you can do whatever it was you wanted to, then you can easily blow it away when you’re done. The best part about mock is that it allows you to build/test things without hauling in all of the development libraries into your native working environment. Quite frankly, I couldn’t have hosted or tested half of the things I do in all my blogs and my repository without it. Mock generates it’s ‘throw/away’ environments by connection to a yum repository and setting itself up. If you host your own repositories, it can greatly speed up this process.
  • Consistency: If everyone in your department were to reference the same rpm repository instead of one of the hundreds of mirrors available on the internet, you would consistently be hosting and sharing the same packages internally with your team. This is fantastic in a software development environment where everyone should be using the same packages anyway. This isn’t to say that all public CentOS mirrors are different, but they do go up and down from time to time. They also all synchronize themselves with whatever the latest and greatest at different times too.
  • Mirrors: Even if you just host your own mirror publicly, you’d be doing the Linux community good! You’d become another server of the hundreds already out there providing a source for free software! Your efforts would offload network congestion others face and speed up everyone’s Linux experience.
  • Custom Media: Hosting your own RPM packages can grant you the power to build your own custom media. You can accomplish the same task without hosting internally, but it’s much (,much) slower! This will make for a great topic in another blog though since this process is a topic of it’s own. I do however explain how to host your own custom repositories in this blog though!

Local Hosting Environment

The whole hosting process will take up some disk-space… 150GB or so if you decide to host everything I identify here. Nothing major, but worth noting for those with smaller disk drives.

The first thing you need to do is decide where you’ll host everything from. I will use the following directories for this blog entry:

Directory Details
/var/share/repo The location we’ll host the repository from.
/var/share/isos The location we’ll store our ISO image files in.

First let’s make sure our directories exist:

# Repository data will go here: /var/share/repo
[ ! -d /var/share/repo ] && \
   mkdir -p var/share/repo

# ISOs downloaded/kept will go here: /var/share/isos
[ ! -d /var/share/isos ] && \
   mkdir -p var/share/isos

You’ll want to have the following tools on hand as well if you plan on hosting your own repositories:

# Mirroring tools
yum install -y rsync lftp

# Repository management tools
yum install -y createrepo yum-utils find

Repository Hosting

The next few sections focus on the following repositories; you may or may not need them all. So feel free to just use what you need.

Distribution 64-Bit 32-Bit
CentOS 5.11 x86_64 i386
CentOS 6.8 x86_64 i386
CentOS 7.2 x86_64 n/a

Note: The blog utilizes the lftp tool for all of the synchronization, but the rsync command is also documented in the comments if you prefer that route as well. It’s also worth noting that you can re-run these mirror commands again and again to keep your local repository updated with the latest.

CentOS 5.11

64-bit

# Change to our ISO directory
pushd /var/share/isos

wget http://mirror.csclub.uwaterloo.ca/centos/5.11/isos/x86_64/CentOS-5.11-x86_64-bin-DVD-1of2.iso \
   -O /var/share/isos/CentOS-5.11-x86_64-DVD1.iso
wget http://mirror.csclub.uwaterloo.ca/centos/5.11/isos/x86_64/CentOS-5.11-x86_64-bin-DVD-2of2.iso \
   -O /var/share/isos/CentOS-5.11-x86_64-DVD2.iso

# Create temporary directories to work in
mkdir dvd1
mkdir dvd2

# Mount it
mount -o loop,ro CentOS-5.11-x86_64-DVD1.iso dvd1
mount -o loop,ro CentOS-5.11-x86_64-DVD2.iso dvd2

# CentOS Base Mirror
[ ! -d /var/share/repo/centos/5.11/x86_64/os ] && \
    mkdir -p /var/share/repo/centos/5.11/x86_64/os

# CentOS Updates Mirror
[ ! -d /var/share/repo/centos/5.11/x86_64/updates ] && \
    mkdir -p /var/share/repo/centos/5.11/x86_64/updates

# CentOS Extras Mirror
[ ! -d /var/share/repo/centos/5.11/x86_64/extras ] && \
    mkdir -p /var/share/repo/centos/5.11/x86_64/extras

# Setup EPEL location
[ ! -d /var/share/repo/centos/5/x86_64/epel ] && \
    mkdir -p /var/share/repo/centos/5/x86_64/epel

# Create a fixed link to newest repo
pushd /var/share/repo/centos/5/x86_64
ln -snf ../../5.11/x86_64/os os
ln -snf ../../5.11/x86_64/updates updates
ln -snf ../../5.11/x86_64/extras extras
popd

# Set up other repositories
pushd /var/share/repo/centos/5.11/x86_64
ln -snf ../../5/x86_64/epel epel
popd

# Mirror repository
rsync -av --ignore-existing dvd1/ /var/share/repo/centos/5.11/x86_64/os/
rsync -av --ignore-existing dvd2/CentOS/ /var/share/repo/centos/5.11/x86_64/os/Packages/
 
# Cleanup
umount dvd1
umount dvd2
rmdir dvd1
rmdir dvd2
popd

# Mirror (5.11) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/5.11/updates/x86_64/ /var/share/repo/centos/5.11/x86_64/updates/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/5.11/updates/x86_64/ /var/share/repo/centos/5.11/x86_64/updates"

# Mirror (5.11) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/5.11/extras/x86_64/ /var/share/repo/centos/5.11/x86_64/extras/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/5.11/extras/x86_64/ /var/share/repo/centos/5.11/x86_64/extras"

# EPEL: (see https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/5 for list of servers)
# In the future you can run these commands again and again to update your repository
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/fedora-enchilada/epel/6/x86_64/ /var/share/repo/centos/5/x86_64/epel/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /fedora/epel/5/x86_64/ /var/share/repo/centos/5/x86_64/epel"

32-bit

# Change to our ISO directory
pushd /var/share/isos

wget http://mirror.csclub.uwaterloo.ca/centos/5.11/isos/i386/CentOS-5.11-i386-bin-DVD-1of2.iso \
   -O /var/share/isos/CentOS-5.11-i386-DVD1.iso
wget http://mirror.csclub.uwaterloo.ca/centos/5.11/isos/i386/CentOS-5.11-i386-bin-DVD-2of2.iso \
   -O /var/share/isos/CentOS-5.11-i386-DVD2.iso

# Create temporary directories to work in
mkdir dvd1
mkdir dvd2

# Mount it
mount -o loop,ro CentOS-5.11-i386-DVD1.iso dvd1
mount -o loop,ro CentOS-5.11-i386-DVD2.iso dvd2

# CentOS Base Mirror
[ ! -d /var/share/repo/centos/5.11/i386/os ] && \
    mkdir -p /var/share/repo/centos/5.11/i386/os

# CentOS Updates Mirror
[ ! -d /var/share/repo/centos/5.11/i386/updates ] && \
    mkdir -p /var/share/repo/centos/5.11/i386/updates

# CentOS Extras Mirror
[ ! -d /var/share/repo/centos/5.11/i386/extras ] && \
    mkdir -p /var/share/repo/centos/5.11/i386/extras

# Setup EPEL Mirror
[ ! -d /var/share/repo/centos/5/i386/epel ] && \
    mkdir -p /var/share/repo/centos/5/i386/epel

# Create a fixed link to newest repo
pushd /var/share/repo/centos/5/i386
ln -snf ../../5.11/i386/os os
ln -snf ../../5.11/i386/updates updates
ln -snf ../../5.11/i386/extras extras
popd

# Set up other repositories
pushd /var/share/repo/centos/5.11/i386
ln -snf ../../5/i386/epel epel
popd

# Mirror repository
rsync -av --ignore-existing dvd1/ /var/share/repo/centos/5.11/i386/os/
rsync -av --ignore-existing dvd2/CentOS/ /var/share/repo/centos/5.11/i386/os/Packages/
 
# Cleanup
umount dvd1
umount dvd2
rmdir dvd1
rmdir dvd2
popd

# Mirror (5.11) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/5.11/updates/i386/ /var/share/repo/centos/5.11/i386/updates/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/5.11/updates/i386/ /var/share/repo/centos/5.11/i386/updates"

# Mirror (5.11) Extras
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/5.11/extras/i386/ /var/share/repo/centos/5.11/i386/extras/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/5.11/extras/i386/ /var/share/repo/centos/5.11/i386/extras"

# EPEL: (see https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/5 for list of servers)
# In the future you can run these commands again and again to update your repository
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/fedora-enchilada/epel/6/i386/ /var/share/repo/centos/5/i386/epel/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /fedora/epel/5/i386/ /var/share/repo/centos/5/i386/epel"

CentOS 6.8

64-bit

# Change to our ISO directory
pushd /var/share/isos

wget http://mirror.its.dal.ca/centos/6.8/isos/x86_64/CentOS-6.8-x86_64-bin-DVD1.iso \
   -O /var/share/isos/CentOS-6.8-x86_64-DVD1.iso
wget http://mirror.its.dal.ca/centos/6.8/isos/x86_64/CentOS-6.8-x86_64-bin-DVD2.iso \
   -O /var/share/isos/CentOS-6.8-x86_64-DVD2.iso

# Create temporary directories to work in
mkdir dvd1
mkdir dvd2

# Mount it
mount -o loop,ro CentOS-6.8-x86_64-DVD1.iso dvd1
mount -o loop,ro CentOS-6.8-x86_64-DVD2.iso dvd2

# CentOS Base Mirror
[ ! -d /var/share/repo/centos/6.8/x86_64/os ] && \
    mkdir -p /var/share/repo/centos/6.8/x86_64/os

# CentOS Updates Mirror
[ ! -d /var/share/repo/centos/6.8/x86_64/updates ] && \
    mkdir -p /var/share/repo/centos/6.8/x86_64/updates

# CentOS Extras Mirror
[ ! -d /var/share/repo/centos/6.8/x86_64/extras ] && \
    mkdir -p /var/share/repo/centos/6.8/x86_64/extras

# Setup EPEL location
[ ! -d /var/share/repo/centos/6/x86_64/epel ] && \
    mkdir -p /var/share/repo/centos/6/x86_64/epel

# Create a fixed link to newest repo
pushd /var/share/repo/centos/6/x86_64
ln -snf ../../6.8/x86_64/os os
ln -snf ../../6.8/x86_64/updates updates
ln -snf ../../6.8/x86_64/extras extras
popd

# EPEL Repository Mirror
pushd /var/share/repo/centos/6.8/x86_64
ln -snf ../../6/x86_64/epel epel
popd

# Mirror repository
rsync -av --ignore-existing dvd1/ /var/share/repo/centos/6.8/x86_64/os/
rsync -av --ignore-existing dvd2/Packages/ /var/share/repo/centos/6.8/x86_64/os/Packages/

# Cleanup
umount dvd1
umount dvd2
rmdir dvd1
rmdir dvd2
popd

# Mirror (6.8) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/6.8/updates/x86_64/ /var/share/repo/centos/6.8/x86_64/updates/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/6.8/updates/x86_64/ /var/share/repo/centos/6.8/x86_64/updates"

# Mirror (6.8) Extras
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/6.8/extras/x86_64/ /var/share/repo/centos/6.8/x86_64/extras/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/6.8/extras/x86_64/ /var/share/repo/centos/6.8/x86_64/extras"

# EPEL: (see https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/6 for list of servers)
# In the future you can run these commands again and again to update your repository
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/fedora-enchilada/epel/6/x86_64/ /var/share/repo/centos/6/x86_64/epel/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /fedora/epel/6/x86_64/ /var/share/repo/centos/6/x86_64/epel"

32-bit

# Change to our ISO directory
pushd /var/share/isos

wget http://mirror.its.dal.ca/centos/6.8/isos/i386/CentOS-6.8-i386-bin-DVD1.iso \
   -O /var/share/isos/CentOS-6.8-i386-DVD1.iso
wget http://mirror.its.dal.ca/centos/6.8/isos/i386/CentOS-6.8-i386-bin-DVD2.iso \
   -O /var/share/isos/CentOS-6.8-i386-DVD2.iso

# Create temporary directories to work in
mkdir dvd1
mkdir dvd2

# Mount it
mount -o loop,ro CentOS-6.8-i386-DVD1.iso dvd1
mount -o loop,ro CentOS-6.8-i386-DVD2.iso dvd2

# CentOS Base Mirror
[ ! -d /var/share/repo/centos/6.8/i386/os ] && \
    mkdir -p /var/share/repo/centos/6.8/i386/os

# CentOS Updates Mirror
[ ! -d /var/share/repo/centos/6.8/i386/updates ] && \
    mkdir -p /var/share/repo/centos/6.8/i386/updates

# CentOS Extras Mirror
[ ! -d /var/share/repo/centos/6.8/i386/extras ] && \
    mkdir -p /var/share/repo/centos/6.8/i386/extras

# EPEL Repository Mirror
[ ! -d /var/share/repo/centos/6/i386/epel ] && \
    mkdir -p /var/share/repo/centos/6/i386/epel

# Create a fixed link to newest repo
pushd /var/share/repo/centos/6/i386
ln -snf ../../6.8/i386/os os
ln -snf ../../6.8/i386/updates updates
ln -snf ../../6.8/i386/extras extras
popd

# Set up other repositories
pushd /var/share/repo/centos/6.8/i386
ln -snf ../../6/i386/epel epel
popd

# Mirror repository
rsync -av --ignore-existing dvd1/ /var/share/repo/centos/6.8/i386/os/
rsync -av --ignore-existing dvd2/Packages/ /var/share/repo/centos/6.8/i386/os/Packages/

# Cleanup
umount dvd1
umount dvd2
rmdir dvd1
rmdir dvd2
popd

# Mirror (6.8) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/6.8/updates/i386/ /var/share/repo/centos/6.8/i386/updates/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/6.8/updates/i386/ /var/share/repo/centos/6.8/i386/updates"

# Mirror (6.8) Extras
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/6.8/extras/i386/ /var/share/repo/centos/6.8/i386/extras/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/6.8/extras/i386/ /var/share/repo/centos/6.8/i386/extras"

# EPEL: (see https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/6 for list of servers)
# In the future you can run these commands again and again to update your repository
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/fedora-enchilada/epel/6/i386/ /var/share/repo/centos/6/i386/epel/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /fedora/epel/6/i386/ /var/share/repo/centos/6/i386/epel"

CentOS 7.2

64-bit

# Change to our ISO directory
pushd /var/share/isos

# CentOS 7.2
[ ! -d /var/share/isos/ ] && mkdir -p /var/share/isos/
wget http://mirror.its.dal.ca/centos/7.2.1511/isos/x86_64/CentOS-7-x86_64-Everything-1511.iso \
   -O /var/share/isos/CentOS-7.2.1511-x86_64-Everything.iso

# CentOS Base Mirror
[ ! -d /var/share/repo/centos/7.2/x86_64/os ] && \
    mkdir -p /var/share/repo/centos/7.2/x86_64/os

# CentOS Updates Mirror
[ ! -d /var/share/repo/centos/7.2/x86_64/updates ] && \
    mkdir -p /var/share/repo/centos/7.2/x86_64/updates

# CentOS Extras Mirror
[ ! -d /var/share/repo/centos/7.2/x86_64/extras ] && \
    mkdir -p /var/share/repo/centos/7.2/x86_64/extras

# EPEL Repository Mirror
[ ! -d /var/share/repo/centos/7/x86_64/epel ] && \
    mkdir -p /var/share/repo/centos/7/x86_64/epel

# Create a fixed link to newest repo
pushd /var/share/repo/centos/7/x86_64
ln -snf ../../7.2/x86_64/os os
ln -snf ../../7.2/x86_64/updates updates
ln -snf ../../7.2/x86_64/extras extras
popd

# Set up other repositories
pushd /var/share/repo/centos/7.2/x86_64
ln -snf ../../7/x86_64/epel epel
popd

# Mount it
mount -o loop,ro /var/share/isos/CentOS-7.2.1511-x86_64-Everything.iso \
    /var/share/repo/centos/7.2/x86_64/os

# Update our fstab file so we persistently do this on reboots; This saves disk space
sed -i -e '/CentOS-7\.2\.1511-x86_64-Everything\.iso/d' /etc/fstab
echo '/var/share/isos/CentOS-7.2.1511-x86_64-Everything.iso /var/share/repo/centos/7.2/x86_64/os iso9660 loop,ro,auto 0 0' >> /etc/fstab
# If you prefer not to run the above command, you can follow through with the same rsync
# commands identified above and mirror the contents of the iso to disk

# Mirror (7.2) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/7.2.1511/updates/x86_64/ /var/share/repo/centos/7.2/x86_64/updates/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/7.2.1511/updates/x86_64/ /var/share/repo/centos/7.2/x86_64/updates"

# Mirror (7.2) Extras
# In the future you can run these commands again and again to update your repository
# EPEL: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/7.2.1511/extras/x86_64/ /var/share/repo/centos/7.2/x86_64/extras/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/7.2.1511/extras/x86_64/ /var/share/repo/centos/7.2/x86_64/extras"

# Mirror EPEL
# In the future you can run these commands again and again to update your repository
# EPEL: (see https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/7 for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/fedora-enchilada/epel/7/x86_64/ /var/share/repo/centos/7/x86_64/epel/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /fedora/epel/7/x86_64/ /var/share/repo/centos/7/x86_64/epel"

Web Hosting

Once you’ve got your repositories mirrored, you need to host them. Here is the easiest way to do so:
If you’re (planning on) using NginX, then the following will get you going:

# Install nginx (if it's not there already)
yum install nginx -y

# Enable for future reboots
systemctl enable nginx.service

# Start it up
systemctl start nginx.service

cat << _EOF > /etc/nginx/default.d/repo.conf
location /repo/ {
   alias         /var/share/repo/;
   autoindex on;
}
_EOF
# Reload
systemctl reload nginx.service

Now you should be able to access your website by visiting the server you set this up on with /repo as the path. ie: http://localhost/repo

SELinux

Users running SELinux in enforcing mode will want to do the following so that they’re repository can be hosted properly:

# make sure /var/share/repo can host website content without conflicting
# with SELinux
semanage fcontext -a -t httpd_sys_content_t '/var/share/repo(/.*)?'

YUM Repositories

Now we’ll want to update our servers around our office, or maybe just on this PC we’re using to point to our new repositories. Here is probably the easiest way:

# Disable any existing repository setup (the following has
# to be ran as root):
pushd /etc/yum.repo.d/
sed -e 's/^\(enabled\)=.*/\1=0/g' \
   fedora*.repo centos*.repo epel*.repo &>/dev/null
popd

# Set this variable to the ip/host of the machine you set up your
# repository on:
MYREPOADDR=localhost

# Now install our new file:
cat << _EOF > /etc/yum.repo.d/centos.internal.repo
[internal-base]
name=CentOS \$releasever - \$basearch - Base
baseurl=http://$MYREPOADDR/repo/centos/\$releasever/\$basearch/os/
enabled=1
priority=1
gpgcheck=0
skip_if_unavailable=False

[internal-updates]
name=CentOS \$releasever - \$basearch - Updates
failovermethod=priority
baseurl=http://$MYREPOADDR/repo/centos/\$releasever/\$basearch/updates/
enabled=1
priority=1
gpgcheck=0
skip_if_unavailable=True

[internal-extras]
name=CentOS \$releasever - \$basearch - Extras
failovermethod=priority
baseurl=http://$MYREPOADDR/repo/centos/\$releasever/\$basearch/extras/
enabled=1
priority=1
gpgcheck=0
skip_if_unavailable=True

[internal-epel]
name=CentOS \$releasever - \$basearch - Internal EPEL
failovermethod=priority
baseurl=http://$MYREPOADDR/repo/centos/\$releasever/\$basearch/epel/
enabled=1
priority=1
gpgcheck=0
skip_if_unavailable=True
_EOF

You can easily take the examples provided here and alter them for other repositories you wish to host.

Custom Repositories

Perhaps you’re picking and choosing RPMs from a ton of different sources, or you’re building your own. It’s a good idea not to touch the repositories we’re already mirroring. Leave them exactly the way they are. However, you can create your own repository instead that you can place your personal collection of rpms in:

# Lets presume we want to start a custom CentOS 7 (64-bit) repository.
# we'll call it 'custom' for now, but you can call it whatever you want

[ ! -d /var/share/repo/centos/7/x86_64/custom ] && \
    mkdir -p /var/share/repo/centos/7/x86_64/custom

# Create a fixed link to our repo
pushd /var/share/repo/centos/7.2/x86_64
ln -snf ../../7/x86_64/custom custom
popd

# Okay... now you can drop all the RPMs you want into this custom repository.
# just copy them in; no strings attached.
cp my.awesome.application-1.0.0-1.x86_64.rpm

# The next step is to make the repository accessible by yum
# If you're using dnf, there is no problem, these commands still apply:

# First we want to reset our repodata folder.  This probably won't
# exist the first time you do this, but in the future, when you add
# more rpms into this directory, this will be an essential step:
[ -d /var/share/repo/centos/7/x86_64/custom/repodata ] && \
    rm -rf /var/share/repo/centos/7/x86_64/custom/repodata

# Now lets clean out any duplicate/old RPM entries
# This is optional; it basically looks for 2 RPMs of the same name
# but of different versions and only keeps the newest.  If you like
# hanging on to all of the versions of your software, you don't
# need to run the below command.
repomanage -o /var/share/repo/centos/7/x86_64/custom/repodata | \
   xargs rm -f

The next step requires us to generate a comps file. These are XML files that define details of our repository. You can get complicated and define it yourself if you like, but to make things simple, you can just use this script i wrote which will generate one on the fly based on the directory you specify it to parse:

#!/bin/sh
# Name: updaterepo
# Author: Chris Caron <lead2gold@gmail.com>
# URL: See http://nuxref.com for details
# Description:
#   A simple script for creating a repository directory.
#
# Syntax: updaterepo <path_to_repository_hosting_rpms>
#
REPODIR=$1
if [ -z "$REPODIR" ]; then
   echo "You must specify a repository directory to scan (containing rpms)."
   exit 1
fi

if [ ! -d "$REPODIR" ]; then
   echo "You must specify a repository directory to scan (containing rpms)."
   exit 1
fi

# Ensure we're dealing with absolute paths
pushd $REPODIR &>/dev/null && REPODIR=$(pwd) && popd &>/dev/null

# Some Default Variables; please feel free to change these to
# suit your own needs and package identification. Whatever you
# specify here will become a 'group' as far as yum and dnf are
# concerned allowing you to run (set $ID according):
#   yum groupinstall custom
ID="custom"
NAME="Core"
DESC="Custom Built Packages"

# Generate a comps file for a repository dynamically
COMPSFILE=comps-$ID.xml
COMPSDIR=$REPODIR/repodata
COMPS=$COMPSDIR/$COMPSFILE

# Directory Management
[ -d $COMPSDIR ] && rm -rf $COMPSDIR
[ ! -d $COMPSDIR ] && mkdir -p $COMPSDIR

FILES=$(find -L $REPODIR -mindepth 1 -maxdepth 1 -type f -name "*.rpm" -exec basename {} \;)

   cat << _EOF > "$COMPS"
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//REDHAT//DTD Comps info//EN" "comps.dtd">
<comps>
  <group>
    <id>$ID</id>
    <name>$NAME</name>
    <description>$DESC</description>
    <default>True</default>
    <uservisible>True</uservisible>
    <packagelist>
_EOF
for FILE in $FILES; do
   PFILE=$(rpm -qp "$REPODIR/$FILE" --nosignature --queryformat="%{NAME}" 2>/dev/null)
   cat << _EOF >> "$COMPS"
         <packagereq type="default">$PFILE</packagereq>
_EOF
done
cat << _EOF >> "$COMPS"
    </packagelist>
  </group>
  <category>
    <id>$ID</id>
    <name>$NAME</name>
    <display_order>99</display_order>
    <grouplist>
      <groupid>$ID</groupid>
    </grouplist>
  </category>
</comps>
_EOF

# Now rebuild our repo information
createrepo -d -q -g $COMPS $REPODIR

Place the script file above into your /usr/bin directory if you like (called updaterepo) and call it on any directory you want to turn into a repository from then on:

# Make sure our new script is executable
chmod 775 /bin/updaterepo

# Now call it against any directory we want to turn into a repository
[ ! -d /var/share/repo/centos/7/x86_64/custom ] && 
    mkdir -p  /var/share/repo/centos/7/x86_64/custom

# Now we can copy our rpms into this directory at will.
# Then we just need to run this command when we're done to apply
# our changes. This will allow people who are pointing to our
# repository to see these new changes we made and access it via yum/dnf
updaterepo  /var/share/repo/centos/7/x86_64/custom


You're done! Now you can update your <strong>/etc/yum.repos.d/</strong> to include this new location with each yum call you make! Just use the other examples already provided in this blog as a template!

<h1>Mock</h1>
<a href="https://github.com/rpm-software-management/mock/wiki" target="_blank">Mock</a> is a fantastic tool for RPM management.  It's also a great tool for someone who just wants to test and see if they're code will run on another platform.  You could almost think of Mock as a poor man's <a href="https://linuxcontainers.org/" target="_blank">Linux Container</a> which are pretty popular these days.  Mock isn't as contained, but it can accomplish the same feat and is even faster since it doesn't have the (Linux) container overhead.

Mock is set up as follows:

# first install it (as root)
yum install -y mock

# If you followed all of the instructions above, then this will pull mock
# out of your internal epel repository! Nice!

# Now just add your user account you usually use on your system
# (a non-root user).  This user will be granted mock privileges.
# substitute [User name] below with the username you usually use:
usermod -a -G mock [User name]

Just like that' you're now ready to create mock environments. Now by default, mock is configured to fetch it's information from the external repositories on the internet. But it's configuration is really easy to get ahold of and update. Check out the /etc/mock directory.

You can now edit the mock environment you wish to host and optionally update the repositories to point to your own. For example, pick a file like /etc/mock/epel-7-x86_64.cfg which is used to generate a 64-bit Enterprise Linux (RedHat/CentOS) 7.x environment.

You'll see entries like this which will look very familiar (it's a yum/dnf configuration file entry just like the ones identified above). You can comment out the mirrorlist entry and swap it with your own local repository.

...  ...
[base]
name=BaseOS
# comment out the mirrorlist reference:
# mirrorlist=http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os
# point to our own local repository instead:
baseurl=http://localhost/repo/centos/7/x86_64/os/
failovermethod=priority
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-7
gpgcheck=1

[updates]
name=updates
enabled=1
# comment out the mirrorlist reference:
# mirrorlist=http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates
# point to our own local repository instead:
baseurl=http://localhost/repo/centos/7/x86_64/updates/
failovermethod=priority
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-7
gpgcheck=1

[extras]
name=extras
enabled=1
# comment out the mirrorlist reference:
# mirrorlist=http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=extras
# point to our own local repository instead:
baseurl=http://localhost/repo/centos/7/x86_64/extras/
failovermethod=priority
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-7
gpgcheck=1

[epel]
name=epel
# comment out the mirrorlist reference:
# mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-7&arch=x86_64
# point to our own local repository instead:
baseurl=http://localhost/repo/centos/7/x86_64/updates/
failovermethod=priority
gpgkey=file:///usr/share/distribution-gpg-keys/epel/RPM-GPG-KEY-EPEL-7
gpgcheck=1

...  ...

You can also just copy and paste one of the config files to another and change it around. Add repositories, remove some; the config file for mock is really straight forward if you use another as a template.

Mock is pretty straight forward to use. You just need to pass it in the environment you're using with each call you make to it.

# Initialize a Mock Environment of Enterprise Linux 7 (64-bit):
mock -r epel-7-x86_64 --init

# The above command works because a /etc/mock/epel-7-x86_64.cfg file exists.
# mock is relatively quiet and doesn't output a lot of information which can
# sometimes make you unsure if it's actually doing anything.  So it doesn't
# hurt to also get in the habit of adding -v to every single mock action you
# perform.

# You can install RPMs into your mock environment easily with:
mock -v -r epel-7-x86_64 install hostname vi

# The following command will cause us to leave our current native environment
# and access the mock environment (an advanced chroot basically):
mock -v -r epel-7-x86_64 --shell

# Once inside, you can do whatever you want; just press Ctrl-D or type 'exit'
# to return back to your normal environment.

# You can even rebuild a source rpm package from another distribution
# into one you might use.  For example, perhaps you visited pkgs.org and saw
# a newer version of an awesome application available for Fedora 24, but
# can't get it for CentOS 7.  Well you can download the src.rpm file and
# rebuild it using mock:
mock -v -r epel-7-x86_64 --rebuild awesome.application-v2.0.0-1.fc24.src.rpm \
    --resultdir=awesome.app

# I added the --resultdir= on the line above to tell mock that after it rebuilds
# our new RPMs for CentOS 7, it places them into a directory called 'awesome.app'.
# This isn't necessary, but makes it easier to find the RPMs when the build is
# complete.

Credit

If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

Sources

  • Mock: A fantastic development tool that allows you to dynamically generate a fully functional development environment for a CentOS/Red Hat/Fedora distribution of your choice. It allows you to keep development libraries out of your native (working) environment keeping things clean.
  • Some good repositories worth mirroring (in no particular order):
    • CentOS 5.x, 6.x and 7.x: Here is a list of all of the mirrors you can sync from.
    • Extra Packages for Enterprise Linux (EPEL): CentOS/Red Hat users shouldn't be without this one. It has tons of the building blocks you might need to reconstruct things with.
    • NuxRef: Yes... my repository; I mean why not, right?
    • Fedora: Bleeding edge distributions don't stick around long. After a year or so, you'll be hard-pressed to find RPMs. It might be wise to mirror the distribution you're using if you don't plan on upgrading anytime soon. There is enough information in this blog to mirror a Fedora repository if you want to do so.