Tag Archives: Fedora

Repository Hosting

Hosting Your Own RPM Repository

Introduction

There are lots of reasons why you might want to host your own RPM repository. A few reasons up front:

  • Speed: Performing a yum (or dnf) update from one of your locally hosted repositories is much faster than checking the ones on the internet!
  • Cost: If you run a business, then you know bandwidth can be expensive. If Linux is the operating system your developers and/or employees use, then hosting the repository internally means they won’t waste bandwidth each time one of them performs a system update. You’ll have downloaded everything once, and provided a central controlled share point they can apply all of their updates from.
  • Retention: Over time, your successfully deployed systems will age and the packages that were once readily available for it may be a lot harder to acquire! Sometimes it’s a good thing to create and manage your own internal repository so that you always have access to the exact RPMs you used to deploy/create your development environment with. This is especially the case for that one rare day that arises requiring you to go back and debug a legacy system of yours. If the packages are already available to you locally; you should have no problem reproducing any detected problems since you’ll be able to reconstruct their environment exactly.
  • Mock: I love this tool; mock allows you to dynamically generate different distributions of CentOS/Fedora (new and old) allowing you to build your applications to test them in it. You simply create a mock environment of the distribution of interest. In this new environment, you can do whatever it was you wanted to, then you can easily blow it away when you’re done. The best part about mock is that it allows you to build/test things without hauling in all of the development libraries into your native working environment. Quite frankly, I couldn’t have hosted or tested half of the things I do in all my blogs and my repository without it. Mock generates it’s ‘throw/away’ environments by connection to a yum repository and setting itself up. If you host your own repositories, it can greatly speed up this process.
  • Consistency: If everyone in your department were to reference the same rpm repository instead of one of the hundreds of mirrors available on the internet, you would consistently be hosting and sharing the same packages internally with your team. This is fantastic in a software development environment where everyone should be using the same packages anyway. This isn’t to say that all public CentOS mirrors are different, but they do go up and down from time to time. They also all synchronize themselves with whatever the latest and greatest at different times too.
  • Mirrors: Even if you just host your own mirror publicly, you’d be doing the Linux community good! You’d become another server of the hundreds already out there providing a source for free software! Your efforts would offload network congestion others face and speed up everyone’s Linux experience.
  • Custom Media: Hosting your own RPM packages can grant you the power to build your own custom media. You can accomplish the same task without hosting internally, but it’s much (,much) slower! This will make for a great topic in another blog though since this process is a topic of it’s own. I do however explain how to host your own custom repositories in this blog though!

Local Hosting Environment

The whole hosting process will take up some disk-space… 150GB or so if you decide to host everything I identify here. Nothing major, but worth noting for those with smaller disk drives.

The first thing you need to do is decide where you’ll host everything from. I will use the following directories for this blog entry:

Directory Details
/var/share/repo The location we’ll host the repository from.
/var/share/isos The location we’ll store our ISO image files in.

First let’s make sure our directories exist:

# Repository data will go here: /var/share/repo
[ ! -d /var/share/repo ] && \
   mkdir -p var/share/repo

# ISOs downloaded/kept will go here: /var/share/isos
[ ! -d /var/share/isos ] && \
   mkdir -p var/share/isos

You’ll want to have the following tools on hand as well if you plan on hosting your own repositories:

# Mirroring tools
yum install -y rsync lftp

# Repository management tools
yum install -y createrepo yum-utils find

Repository Hosting

The next few sections focus on the following repositories; you may or may not need them all. So feel free to just use what you need.

Distribution 64-Bit 32-Bit
CentOS 5.11 x86_64 i386
CentOS 6.8 x86_64 i386
CentOS 7.2 x86_64 n/a

Note: The blog utilizes the lftp tool for all of the synchronization, but the rsync command is also documented in the comments if you prefer that route as well. It’s also worth noting that you can re-run these mirror commands again and again to keep your local repository updated with the latest.

CentOS 5.11

64-bit

# Change to our ISO directory
pushd /var/share/isos

wget http://mirror.csclub.uwaterloo.ca/centos/5.11/isos/x86_64/CentOS-5.11-x86_64-bin-DVD-1of2.iso \
   -O /var/share/isos/CentOS-5.11-x86_64-DVD1.iso
wget http://mirror.csclub.uwaterloo.ca/centos/5.11/isos/x86_64/CentOS-5.11-x86_64-bin-DVD-2of2.iso \
   -O /var/share/isos/CentOS-5.11-x86_64-DVD2.iso

# Create temporary directories to work in
mkdir dvd1
mkdir dvd2

# Mount it
mount -o loop,ro CentOS-5.11-x86_64-DVD1.iso dvd1
mount -o loop,ro CentOS-5.11-x86_64-DVD2.iso dvd2

# CentOS Base Mirror
[ ! -d /var/share/repo/centos/5.11/x86_64/os ] && \
    mkdir -p /var/share/repo/centos/5.11/x86_64/os

# CentOS Updates Mirror
[ ! -d /var/share/repo/centos/5.11/x86_64/updates ] && \
    mkdir -p /var/share/repo/centos/5.11/x86_64/updates

# CentOS Extras Mirror
[ ! -d /var/share/repo/centos/5.11/x86_64/extras ] && \
    mkdir -p /var/share/repo/centos/5.11/x86_64/extras

# Setup EPEL location
[ ! -d /var/share/repo/centos/5/x86_64/epel ] && \
    mkdir -p /var/share/repo/centos/5/x86_64/epel

# Create a fixed link to newest repo
pushd /var/share/repo/centos/5/x86_64
ln -snf ../../5.11/x86_64/os os
ln -snf ../../5.11/x86_64/updates updates
ln -snf ../../5.11/x86_64/extras extras
popd

# Set up other repositories
pushd /var/share/repo/centos/5.11/x86_64
ln -snf ../../5/x86_64/epel epel
popd

# Mirror repository
rsync -av --ignore-existing dvd1/ /var/share/repo/centos/5.11/x86_64/os/
rsync -av --ignore-existing dvd2/CentOS/ /var/share/repo/centos/5.11/x86_64/os/Packages/
 
# Cleanup
umount dvd1
umount dvd2
rmdir dvd1
rmdir dvd2
popd

# Mirror (5.11) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/5.11/updates/x86_64/ /var/share/repo/centos/5.11/x86_64/updates/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/5.11/updates/x86_64/ /var/share/repo/centos/5.11/x86_64/updates"

# Mirror (5.11) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/5.11/extras/x86_64/ /var/share/repo/centos/5.11/x86_64/extras/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/5.11/extras/x86_64/ /var/share/repo/centos/5.11/x86_64/extras"

# EPEL: (see https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/5 for list of servers)
# In the future you can run these commands again and again to update your repository
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/fedora-enchilada/epel/6/x86_64/ /var/share/repo/centos/5/x86_64/epel/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /fedora/epel/5/x86_64/ /var/share/repo/centos/5/x86_64/epel"

32-bit

# Change to our ISO directory
pushd /var/share/isos

wget http://mirror.csclub.uwaterloo.ca/centos/5.11/isos/i386/CentOS-5.11-i386-bin-DVD-1of2.iso \
   -O /var/share/isos/CentOS-5.11-i386-DVD1.iso
wget http://mirror.csclub.uwaterloo.ca/centos/5.11/isos/i386/CentOS-5.11-i386-bin-DVD-2of2.iso \
   -O /var/share/isos/CentOS-5.11-i386-DVD2.iso

# Create temporary directories to work in
mkdir dvd1
mkdir dvd2

# Mount it
mount -o loop,ro CentOS-5.11-i386-DVD1.iso dvd1
mount -o loop,ro CentOS-5.11-i386-DVD2.iso dvd2

# CentOS Base Mirror
[ ! -d /var/share/repo/centos/5.11/i386/os ] && \
    mkdir -p /var/share/repo/centos/5.11/i386/os

# CentOS Updates Mirror
[ ! -d /var/share/repo/centos/5.11/i386/updates ] && \
    mkdir -p /var/share/repo/centos/5.11/i386/updates

# CentOS Extras Mirror
[ ! -d /var/share/repo/centos/5.11/i386/extras ] && \
    mkdir -p /var/share/repo/centos/5.11/i386/extras

# Setup EPEL Mirror
[ ! -d /var/share/repo/centos/5/i386/epel ] && \
    mkdir -p /var/share/repo/centos/5/i386/epel

# Create a fixed link to newest repo
pushd /var/share/repo/centos/5/i386
ln -snf ../../5.11/i386/os os
ln -snf ../../5.11/i386/updates updates
ln -snf ../../5.11/i386/extras extras
popd

# Set up other repositories
pushd /var/share/repo/centos/5.11/i386
ln -snf ../../5/i386/epel epel
popd

# Mirror repository
rsync -av --ignore-existing dvd1/ /var/share/repo/centos/5.11/i386/os/
rsync -av --ignore-existing dvd2/CentOS/ /var/share/repo/centos/5.11/i386/os/Packages/
 
# Cleanup
umount dvd1
umount dvd2
rmdir dvd1
rmdir dvd2
popd

# Mirror (5.11) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/5.11/updates/i386/ /var/share/repo/centos/5.11/i386/updates/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/5.11/updates/i386/ /var/share/repo/centos/5.11/i386/updates"

# Mirror (5.11) Extras
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/5.11/extras/i386/ /var/share/repo/centos/5.11/i386/extras/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/5.11/extras/i386/ /var/share/repo/centos/5.11/i386/extras"

# EPEL: (see https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/5 for list of servers)
# In the future you can run these commands again and again to update your repository
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/fedora-enchilada/epel/6/i386/ /var/share/repo/centos/5/i386/epel/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /fedora/epel/5/i386/ /var/share/repo/centos/5/i386/epel"

CentOS 6.8

64-bit

# Change to our ISO directory
pushd /var/share/isos

wget http://mirror.its.dal.ca/centos/6.8/isos/x86_64/CentOS-6.8-x86_64-bin-DVD1.iso \
   -O /var/share/isos/CentOS-6.8-x86_64-DVD1.iso
wget http://mirror.its.dal.ca/centos/6.8/isos/x86_64/CentOS-6.8-x86_64-bin-DVD2.iso \
   -O /var/share/isos/CentOS-6.8-x86_64-DVD2.iso

# Create temporary directories to work in
mkdir dvd1
mkdir dvd2

# Mount it
mount -o loop,ro CentOS-6.8-x86_64-DVD1.iso dvd1
mount -o loop,ro CentOS-6.8-x86_64-DVD2.iso dvd2

# CentOS Base Mirror
[ ! -d /var/share/repo/centos/6.8/x86_64/os ] && \
    mkdir -p /var/share/repo/centos/6.8/x86_64/os

# CentOS Updates Mirror
[ ! -d /var/share/repo/centos/6.8/x86_64/updates ] && \
    mkdir -p /var/share/repo/centos/6.8/x86_64/updates

# CentOS Extras Mirror
[ ! -d /var/share/repo/centos/6.8/x86_64/extras ] && \
    mkdir -p /var/share/repo/centos/6.8/x86_64/extras

# Setup EPEL location
[ ! -d /var/share/repo/centos/6/x86_64/epel ] && \
    mkdir -p /var/share/repo/centos/6/x86_64/epel

# Create a fixed link to newest repo
pushd /var/share/repo/centos/6/x86_64
ln -snf ../../6.8/x86_64/os os
ln -snf ../../6.8/x86_64/updates updates
ln -snf ../../6.8/x86_64/extras extras
popd

# EPEL Repository Mirror
pushd /var/share/repo/centos/6.8/x86_64
ln -snf ../../6/x86_64/epel epel
popd

# Mirror repository
rsync -av --ignore-existing dvd1/ /var/share/repo/centos/6.8/x86_64/os/
rsync -av --ignore-existing dvd2/Packages/ /var/share/repo/centos/6.8/x86_64/os/Packages/

# Cleanup
umount dvd1
umount dvd2
rmdir dvd1
rmdir dvd2
popd

# Mirror (6.8) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/6.8/updates/x86_64/ /var/share/repo/centos/6.8/x86_64/updates/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/6.8/updates/x86_64/ /var/share/repo/centos/6.8/x86_64/updates"

# Mirror (6.8) Extras
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/6.8/extras/x86_64/ /var/share/repo/centos/6.8/x86_64/extras/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/6.8/extras/x86_64/ /var/share/repo/centos/6.8/x86_64/extras"

# EPEL: (see https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/6 for list of servers)
# In the future you can run these commands again and again to update your repository
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/fedora-enchilada/epel/6/x86_64/ /var/share/repo/centos/6/x86_64/epel/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /fedora/epel/6/x86_64/ /var/share/repo/centos/6/x86_64/epel"

32-bit

# Change to our ISO directory
pushd /var/share/isos

wget http://mirror.its.dal.ca/centos/6.8/isos/i386/CentOS-6.8-i386-bin-DVD1.iso \
   -O /var/share/isos/CentOS-6.8-i386-DVD1.iso
wget http://mirror.its.dal.ca/centos/6.8/isos/i386/CentOS-6.8-i386-bin-DVD2.iso \
   -O /var/share/isos/CentOS-6.8-i386-DVD2.iso

# Create temporary directories to work in
mkdir dvd1
mkdir dvd2

# Mount it
mount -o loop,ro CentOS-6.8-i386-DVD1.iso dvd1
mount -o loop,ro CentOS-6.8-i386-DVD2.iso dvd2

# CentOS Base Mirror
[ ! -d /var/share/repo/centos/6.8/i386/os ] && \
    mkdir -p /var/share/repo/centos/6.8/i386/os

# CentOS Updates Mirror
[ ! -d /var/share/repo/centos/6.8/i386/updates ] && \
    mkdir -p /var/share/repo/centos/6.8/i386/updates

# CentOS Extras Mirror
[ ! -d /var/share/repo/centos/6.8/i386/extras ] && \
    mkdir -p /var/share/repo/centos/6.8/i386/extras

# EPEL Repository Mirror
[ ! -d /var/share/repo/centos/6/i386/epel ] && \
    mkdir -p /var/share/repo/centos/6/i386/epel

# Create a fixed link to newest repo
pushd /var/share/repo/centos/6/i386
ln -snf ../../6.8/i386/os os
ln -snf ../../6.8/i386/updates updates
ln -snf ../../6.8/i386/extras extras
popd

# Set up other repositories
pushd /var/share/repo/centos/6.8/i386
ln -snf ../../6/i386/epel epel
popd

# Mirror repository
rsync -av --ignore-existing dvd1/ /var/share/repo/centos/6.8/i386/os/
rsync -av --ignore-existing dvd2/Packages/ /var/share/repo/centos/6.8/i386/os/Packages/

# Cleanup
umount dvd1
umount dvd2
rmdir dvd1
rmdir dvd2
popd

# Mirror (6.8) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/6.8/updates/i386/ /var/share/repo/centos/6.8/i386/updates/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/6.8/updates/i386/ /var/share/repo/centos/6.8/i386/updates"

# Mirror (6.8) Extras
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/6.8/extras/i386/ /var/share/repo/centos/6.8/i386/extras/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/6.8/extras/i386/ /var/share/repo/centos/6.8/i386/extras"

# EPEL: (see https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/6 for list of servers)
# In the future you can run these commands again and again to update your repository
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/fedora-enchilada/epel/6/i386/ /var/share/repo/centos/6/i386/epel/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /fedora/epel/6/i386/ /var/share/repo/centos/6/i386/epel"

CentOS 7.2

64-bit

# Change to our ISO directory
pushd /var/share/isos

# CentOS 7.2
[ ! -d /var/share/isos/ ] && mkdir -p /var/share/isos/
wget http://mirror.its.dal.ca/centos/7.2.1511/isos/x86_64/CentOS-7-x86_64-Everything-1511.iso \
   -O /var/share/isos/CentOS-7.2.1511-x86_64-Everything.iso

# CentOS Base Mirror
[ ! -d /var/share/repo/centos/7.2/x86_64/os ] && \
    mkdir -p /var/share/repo/centos/7.2/x86_64/os

# CentOS Updates Mirror
[ ! -d /var/share/repo/centos/7.2/x86_64/updates ] && \
    mkdir -p /var/share/repo/centos/7.2/x86_64/updates

# CentOS Extras Mirror
[ ! -d /var/share/repo/centos/7.2/x86_64/extras ] && \
    mkdir -p /var/share/repo/centos/7.2/x86_64/extras

# EPEL Repository Mirror
[ ! -d /var/share/repo/centos/7/x86_64/epel ] && \
    mkdir -p /var/share/repo/centos/7/x86_64/epel

# Create a fixed link to newest repo
pushd /var/share/repo/centos/7/x86_64
ln -snf ../../7.2/x86_64/os os
ln -snf ../../7.2/x86_64/updates updates
ln -snf ../../7.2/x86_64/extras extras
popd

# Set up other repositories
pushd /var/share/repo/centos/7.2/x86_64
ln -snf ../../7/x86_64/epel epel
popd

# Mount it
mount -o loop,ro /var/share/isos/CentOS-7.2.1511-x86_64-Everything.iso \
    /var/share/repo/centos/7.2/x86_64/os

# Update our fstab file so we persistently do this on reboots; This saves disk space
sed -i -e '/CentOS-7\.2\.1511-x86_64-Everything\.iso/d' /etc/fstab
echo '/var/share/isos/CentOS-7.2.1511-x86_64-Everything.iso /var/share/repo/centos/7.2/x86_64/os iso9660 loop,ro,auto 0 0' >> /etc/fstab
# If you prefer not to run the above command, you can follow through with the same rsync
# commands identified above and mirror the contents of the iso to disk

# Mirror (7.2) Updates
# In the future you can run these commands again and again to update your repository
# Updates: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/7.2.1511/updates/x86_64/ /var/share/repo/centos/7.2/x86_64/updates/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/7.2.1511/updates/x86_64/ /var/share/repo/centos/7.2/x86_64/updates"

# Mirror (7.2) Extras
# In the future you can run these commands again and again to update your repository
# EPEL: (see https://www.centos.org/download/mirrors/ for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/centos/7.2.1511/extras/x86_64/ /var/share/repo/centos/7.2/x86_64/extras/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /centos/7.2.1511/extras/x86_64/ /var/share/repo/centos/7.2/x86_64/extras"

# Mirror EPEL
# In the future you can run these commands again and again to update your repository
# EPEL: (see https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/7 for list of servers)
# rsync -av --ignore-existing rsync://mirror.csclub.uwaterloo.ca/fedora-enchilada/epel/7/x86_64/ /var/share/repo/centos/7/x86_64/epel/
# or via lftp:
lftp mirror.csclub.uwaterloo.ca -e "mirror --verbose /fedora/epel/7/x86_64/ /var/share/repo/centos/7/x86_64/epel"

Web Hosting

Once you’ve got your repositories mirrored, you need to host them. Here is the easiest way to do so:
If you’re (planning on) using NginX, then the following will get you going:

# Install nginx (if it's not there already)
yum install nginx -y

# Enable for future reboots
systemctl enable nginx.service

# Start it up
systemctl start nginx.service

cat << _EOF > /etc/nginx/default.d/repo.conf
location /repo/ {
   alias         /var/share/repo/;
   autoindex on;
}
_EOF
# Reload
systemctl reload nginx.service

Now you should be able to access your website by visiting the server you set this up on with /repo as the path. ie: http://localhost/repo

SELinux

Users running SELinux in enforcing mode will want to do the following so that they’re repository can be hosted properly:

# make sure /var/share/repo can host website content without conflicting
# with SELinux
semanage fcontext -a -t httpd_sys_content_t '/var/share/repo(/.*)?'

YUM Repositories

Now we’ll want to update our servers around our office, or maybe just on this PC we’re using to point to our new repositories. Here is probably the easiest way:

# Disable any existing repository setup (the following has
# to be ran as root):
pushd /etc/yum.repo.d/
sed -e 's/^\(enabled\)=.*/\1=0/g' \
   fedora*.repo centos*.repo epel*.repo &>/dev/null
popd

# Set this variable to the ip/host of the machine you set up your
# repository on:
MYREPOADDR=localhost

# Now install our new file:
cat << _EOF > /etc/yum.repo.d/centos.internal.repo
[internal-base]
name=CentOS \$releasever - \$basearch - Base
baseurl=http://$MYREPOADDR/repo/centos/\$releasever/\$basearch/os/
enabled=1
priority=1
gpgcheck=0
skip_if_unavailable=False

[internal-updates]
name=CentOS \$releasever - \$basearch - Updates
failovermethod=priority
baseurl=http://$MYREPOADDR/repo/centos/\$releasever/\$basearch/updates/
enabled=1
priority=1
gpgcheck=0
skip_if_unavailable=True

[internal-extras]
name=CentOS \$releasever - \$basearch - Extras
failovermethod=priority
baseurl=http://$MYREPOADDR/repo/centos/\$releasever/\$basearch/extras/
enabled=1
priority=1
gpgcheck=0
skip_if_unavailable=True

[internal-epel]
name=CentOS \$releasever - \$basearch - Internal EPEL
failovermethod=priority
baseurl=http://$MYREPOADDR/repo/centos/\$releasever/\$basearch/epel/
enabled=1
priority=1
gpgcheck=0
skip_if_unavailable=True
_EOF

You can easily take the examples provided here and alter them for other repositories you wish to host.

Custom Repositories

Perhaps you’re picking and choosing RPMs from a ton of different sources, or you’re building your own. It’s a good idea not to touch the repositories we’re already mirroring. Leave them exactly the way they are. However, you can create your own repository instead that you can place your personal collection of rpms in:

# Lets presume we want to start a custom CentOS 7 (64-bit) repository.
# we'll call it 'custom' for now, but you can call it whatever you want

[ ! -d /var/share/repo/centos/7/x86_64/custom ] && \
    mkdir -p /var/share/repo/centos/7/x86_64/custom

# Create a fixed link to our repo
pushd /var/share/repo/centos/7.2/x86_64
ln -snf ../../7/x86_64/custom custom
popd

# Okay... now you can drop all the RPMs you want into this custom repository.
# just copy them in; no strings attached.
cp my.awesome.application-1.0.0-1.x86_64.rpm

# The next step is to make the repository accessible by yum
# If you're using dnf, there is no problem, these commands still apply:

# First we want to reset our repodata folder.  This probably won't
# exist the first time you do this, but in the future, when you add
# more rpms into this directory, this will be an essential step:
[ -d /var/share/repo/centos/7/x86_64/custom/repodata ] && \
    rm -rf /var/share/repo/centos/7/x86_64/custom/repodata

# Now lets clean out any duplicate/old RPM entries
# This is optional; it basically looks for 2 RPMs of the same name
# but of different versions and only keeps the newest.  If you like
# hanging on to all of the versions of your software, you don't
# need to run the below command.
repomanage -o /var/share/repo/centos/7/x86_64/custom/repodata | \
   xargs rm -f

The next step requires us to generate a comps file. These are XML files that define details of our repository. You can get complicated and define it yourself if you like, but to make things simple, you can just use this script i wrote which will generate one on the fly based on the directory you specify it to parse:

#!/bin/sh
# Name: updaterepo
# Author: Chris Caron <lead2gold@gmail.com>
# URL: See http://nuxref.com for details
# Description:
#   A simple script for creating a repository directory.
#
# Syntax: updaterepo <path_to_repository_hosting_rpms>
#
REPODIR=$1
if [ -z "$REPODIR" ]; then
   echo "You must specify a repository directory to scan (containing rpms)."
   exit 1
fi

if [ ! -d "$REPODIR" ]; then
   echo "You must specify a repository directory to scan (containing rpms)."
   exit 1
fi

# Ensure we're dealing with absolute paths
pushd $REPODIR &>/dev/null && REPODIR=$(pwd) && popd &>/dev/null

# Some Default Variables; please feel free to change these to
# suit your own needs and package identification. Whatever you
# specify here will become a 'group' as far as yum and dnf are
# concerned allowing you to run (set $ID according):
#   yum groupinstall custom
ID="custom"
NAME="Core"
DESC="Custom Built Packages"

# Generate a comps file for a repository dynamically
COMPSFILE=comps-$ID.xml
COMPSDIR=$REPODIR/repodata
COMPS=$COMPSDIR/$COMPSFILE

# Directory Management
[ -d $COMPSDIR ] && rm -rf $COMPSDIR
[ ! -d $COMPSDIR ] && mkdir -p $COMPSDIR

FILES=$(find -L $REPODIR -mindepth 1 -maxdepth 1 -type f -name "*.rpm" -exec basename {} \;)

   cat << _EOF > "$COMPS"
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//REDHAT//DTD Comps info//EN" "comps.dtd">
<comps>
  <group>
    <id>$ID</id>
    <name>$NAME</name>
    <description>$DESC</description>
    <default>True</default>
    <uservisible>True</uservisible>
    <packagelist>
_EOF
for FILE in $FILES; do
   PFILE=$(rpm -qp "$REPODIR/$FILE" --nosignature --queryformat="%{NAME}" 2>/dev/null)
   cat << _EOF >> "$COMPS"
         <packagereq type="default">$PFILE</packagereq>
_EOF
done
cat << _EOF >> "$COMPS"
    </packagelist>
  </group>
  <category>
    <id>$ID</id>
    <name>$NAME</name>
    <display_order>99</display_order>
    <grouplist>
      <groupid>$ID</groupid>
    </grouplist>
  </category>
</comps>
_EOF

# Now rebuild our repo information
createrepo -d -q -g $COMPS $REPODIR

Place the script file above into your /usr/bin directory if you like (called updaterepo) and call it on any directory you want to turn into a repository from then on:

# Make sure our new script is executable
chmod 775 /bin/updaterepo

# Now call it against any directory we want to turn into a repository
[ ! -d /var/share/repo/centos/7/x86_64/custom ] && 
    mkdir -p  /var/share/repo/centos/7/x86_64/custom

# Now we can copy our rpms into this directory at will.
# Then we just need to run this command when we're done to apply
# our changes. This will allow people who are pointing to our
# repository to see these new changes we made and access it via yum/dnf
updaterepo  /var/share/repo/centos/7/x86_64/custom


You're done! Now you can update your <strong>/etc/yum.repos.d/</strong> to include this new location with each yum call you make! Just use the other examples already provided in this blog as a template!

<h1>Mock</h1>
<a href="https://github.com/rpm-software-management/mock/wiki" target="_blank">Mock</a> is a fantastic tool for RPM management.  It's also a great tool for someone who just wants to test and see if they're code will run on another platform.  You could almost think of Mock as a poor man's <a href="https://linuxcontainers.org/" target="_blank">Linux Container</a> which are pretty popular these days.  Mock isn't as contained, but it can accomplish the same feat and is even faster since it doesn't have the (Linux) container overhead.

Mock is set up as follows:

# first install it (as root)
yum install -y mock

# If you followed all of the instructions above, then this will pull mock
# out of your internal epel repository! Nice!

# Now just add your user account you usually use on your system
# (a non-root user).  This user will be granted mock privileges.
# substitute [User name] below with the username you usually use:
usermod -a -G mock [User name]

Just like that' you're now ready to create mock environments. Now by default, mock is configured to fetch it's information from the external repositories on the internet. But it's configuration is really easy to get ahold of and update. Check out the /etc/mock directory.

You can now edit the mock environment you wish to host and optionally update the repositories to point to your own. For example, pick a file like /etc/mock/epel-7-x86_64.cfg which is used to generate a 64-bit Enterprise Linux (RedHat/CentOS) 7.x environment.

You'll see entries like this which will look very familiar (it's a yum/dnf configuration file entry just like the ones identified above). You can comment out the mirrorlist entry and swap it with your own local repository.

...  ...
[base]
name=BaseOS
# comment out the mirrorlist reference:
# mirrorlist=http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os
# point to our own local repository instead:
baseurl=http://localhost/repo/centos/7/x86_64/os/
failovermethod=priority
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-7
gpgcheck=1

[updates]
name=updates
enabled=1
# comment out the mirrorlist reference:
# mirrorlist=http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates
# point to our own local repository instead:
baseurl=http://localhost/repo/centos/7/x86_64/updates/
failovermethod=priority
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-7
gpgcheck=1

[extras]
name=extras
enabled=1
# comment out the mirrorlist reference:
# mirrorlist=http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=extras
# point to our own local repository instead:
baseurl=http://localhost/repo/centos/7/x86_64/extras/
failovermethod=priority
gpgkey=file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-7
gpgcheck=1

[epel]
name=epel
# comment out the mirrorlist reference:
# mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-7&arch=x86_64
# point to our own local repository instead:
baseurl=http://localhost/repo/centos/7/x86_64/updates/
failovermethod=priority
gpgkey=file:///usr/share/distribution-gpg-keys/epel/RPM-GPG-KEY-EPEL-7
gpgcheck=1

...  ...

You can also just copy and paste one of the config files to another and change it around. Add repositories, remove some; the config file for mock is really straight forward if you use another as a template.

Mock is pretty straight forward to use. You just need to pass it in the environment you're using with each call you make to it.

# Initialize a Mock Environment of Enterprise Linux 7 (64-bit):
mock -r epel-7-x86_64 --init

# The above command works because a /etc/mock/epel-7-x86_64.cfg file exists.
# mock is relatively quiet and doesn't output a lot of information which can
# sometimes make you unsure if it's actually doing anything.  So it doesn't
# hurt to also get in the habit of adding -v to every single mock action you
# perform.

# You can install RPMs into your mock environment easily with:
mock -v -r epel-7-x86_64 install hostname vi

# The following command will cause us to leave our current native environment
# and access the mock environment (an advanced chroot basically):
mock -v -r epel-7-x86_64 --shell

# Once inside, you can do whatever you want; just press Ctrl-D or type 'exit'
# to return back to your normal environment.

# You can even rebuild a source rpm package from another distribution
# into one you might use.  For example, perhaps you visited pkgs.org and saw
# a newer version of an awesome application available for Fedora 24, but
# can't get it for CentOS 7.  Well you can download the src.rpm file and
# rebuild it using mock:
mock -v -r epel-7-x86_64 --rebuild awesome.application-v2.0.0-1.fc24.src.rpm \
    --resultdir=awesome.app

# I added the --resultdir= on the line above to tell mock that after it rebuilds
# our new RPMs for CentOS 7, it places them into a directory called 'awesome.app'.
# This isn't necessary, but makes it easier to find the RPMs when the build is
# complete.

Credit

If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

Sources

  • Mock: A fantastic development tool that allows you to dynamically generate a fully functional development environment for a CentOS/Red Hat/Fedora distribution of your choice. It allows you to keep development libraries out of your native (working) environment keeping things clean.
  • Some good repositories worth mirroring (in no particular order):
    • CentOS 5.x, 6.x and 7.x: Here is a list of all of the mirrors you can sync from.
    • Extra Packages for Enterprise Linux (EPEL): CentOS/Red Hat users shouldn't be without this one. It has tons of the building blocks you might need to reconstruct things with.
    • NuxRef: Yes... my repository; I mean why not, right?
    • Fedora: Bleeding edge distributions don't stick around long. After a year or so, you'll be hard-pressed to find RPMs. It might be wise to mirror the distribution you're using if you don't plan on upgrading anytime soon. There is enough information in this blog to mirror a Fedora repository if you want to do so.
Google Play Music Desktop Player

Google Play Music Desktop Player for Linux

Introduction

Google Play Music Desktop Player (GPMDP) is an amazing application. Instead of opening up a browser every time to access your Google Music account (and songs), you can integrate it into your desktop experience instead. One of the best features GPMDP offers is the fact it becomes a standalone application and no longer requires you to have to deal with the memory hogging web browsers (especially Chrome) uses to stream your favorite music!

GPMDP doesn’t stop there though, there are tons of great features it offers that extend the already great functionality Google Music offers! Just to quote directly from their README file:

  • Supports media keys (play, pause, stop, next, previous)
  • last.fm Scrobbling and Now Playing support!
  • Hands free Voice Controls!
  • Desktop notifications on track change
  • A simplistic mini player
  • Customizable Dark theme
  • Smooth scrolling and overlay scrollbars for a nicer user experience.
  • HTML5 Audio Support – No more Adobe Flash Player!
  • Minimize to task bar for background music playing
  • Customizable hotkeys for those who don’t have media keys.
  • Choose your audio output device from within the player.
Google Music Dark Theme

Google Music Dark Theme

I set up this blog because the RPMs distributed from GPMDP’s official website didn’t work for me. So I just ended up repackaging the whole thing and thought I would share my success.

It’s also important to note that this blog assumes you have a subscription to Google Music. If you don’t have one (and don’t intend to get one), then there is no need to read any further. 🙂

Installation

If you are hooked up to my repository, you can upgrade to the latest packages through the following simple command:

  • CentOS (or Red Hat) 7.x:
    # Run as root (or a user with sudoers permission)
    sudo yum -y install --enablrepo=nuxref google-play-music-desktop-player
    
    # Or you can do it manually:
    # I sign everything; so it wouldn't hurt to just grab my key
    rpm --import http://repo.nuxref.com/pub/NUXREF-GPG-KEY
    # Now you can install the player
    rpm -Uhi http://repo.nuxref.com/centos/7/en/x86_64/custom/google-play-music-desktop-player-3.2.1-1.el7.nuxref.x86_64.rpm
    
  • Fedora 22:
    # I sign everything; so it wouldn't hurt to just grab my key
    rpm --import http://repo.nuxref.com/pub/NUXREF-GPG-KEY
    # Now you can install the player
    rpm -Uhi http://repo.nuxref.com/fedora/fc22/en/x86_64/custom/google-play-music-desktop-player-3.2.1-1.fc22.nuxref.x86_64.rpm
    
    
  • Fedora 23:
    # I sign everything; so it wouldn't hurt to just grab my key
    rpm --import http://repo.nuxref.com/pub/NUXREF-GPG-KEY
    # Now you can install the player
    rpm -Uhi http://repo.nuxref.com/fedora/fc23/en/x86_64/custom/google-play-music-desktop-player-3.2.1-1.fc23.nuxref.x86_64.rpm
    
    
  • Fedora 24:
    # I sign everything; so it wouldn't hurt to just grab my key
    rpm --import http://repo.nuxref.com/pub/NUXREF-GPG-KEY
    # Now you can install the player
    rpm -Uhi http://repo.nuxref.com/fedora/fc24/en/x86_64/custom/google-play-music-desktop-player-3.2.1-1.fc24.nuxref.x86_64.rpm
    
    
  • Fedora 25:
    # I sign everything; so it wouldn't hurt to just grab my key
    rpm --import http://repo.nuxref.com/pub/NUXREF-GPG-KEY
    # Now you can install the player
    rpm -Uhi http://repo.nuxref.com/fedora/fc25/en/x86_64/custom/google-play-music-desktop-player-3.2.1-1.fc25.nuxref.x86_64.rpm
    
    

Gnome Media Player Shell Extension

Pair this application with the Gnome Media Player Shell Extension and you’ll allow even get desktop notifications when songs change.

Media Player Gnome Extension

Media Player Gnome Extension

For CentOS / Red Hat 7 users, this extension sometimes does not properly install itself. So to make your life easier, I went ahead and packaged it into it’s own RPM as well. You can install it like so:

# Assuming you're hooked up to my repository, you can
# just easily install it:
sudo yum install --enablerepo=nuxref gnome-shell-extension-mediaplayer

I didn’t forget about the Fedora users either. It’s also in the repository (if you’re hooked up):

# Assuming you're hooked up to my repository, you can
# just easily install it:
sudo dnf install --enablerepo=nuxref gnome-shell-extension-mediaplayer

RPMs

You can also just visit the locations where I posted these RPMs and fetch them manually:

  Google Play Music Desktop Player Gnome Media Player Shell Extension
CentOS/Red Hat 7.x rpm / src rpm / src
Fedora 22 rpm / src rpm / src
Fedora 23 rpm / src rpm / src
Fedora 24 rpm / src rpm / src
Fedora 25 rpm / src rpm / src

Credit

All of the custom packaging in this blog was done by me personally. I took the open source available to me and rebuilt it to make it an easier solution and decided to share it. If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

Sources

Unlock and Root Your Nexus Using Linux

Introduction

I’ve had a lot of trouble gathering information on how to unlock and root my Nexus 6 phone (having never done anything like this before). The information on how to do this is definitely out there, but it’s all over the place in bits and pieces. So that’s where this blog comes in; I tried to group everything I learned after successfully rooting my phone and installing TWRP on it.

This diagram explains very loosely how the upgrades work; the diagram will make more sense as you read on.
Nexus Upgrade In Nutshell
Here are the topics of interest:

  • Installing the Android SDK and preparing a useful (Linux) Environment: This step is important in order to be able to interact with your device and update it via a USB cable attached to your PC. It specifically focuses on enabling the use of the adb and fastboot tools. This entire blog is useless to you if you don’t have this part set up.
  • Enabling Debug (Developer) Mode on your Android device: This step pairs with the SDK. The SDK allows your PC to communicate with your Android device while this step allows your device to be communicated with. Each device you plan on unlocking or rooting will have to have Debug Mode turned on.
  • Unlock Your Bootloader: You can not root your device unless your bootloader is unlocked. This action opens your device up to a world of tweaks and hacks. Note: this step can also can void your warranty.
  • Root Your Device: You can not root your device unless your bootloader is unlocked. This step focuses on using the excellent work of Chainfire‘s (CF-)Auto-Root boot.img and SuperSU application.
  • Upgrade Your Rooted Device: When Google puts out their new Over The Air (OTA) upgrade. You’ll notice that rooted devices won’t successfully get upgraded. Don’t worry; nothing breaks, the update simply fails. I identify the steps here how you can play along and still upgrade your rooted device.

A few other things worth sharing:

  • Fantastic Android (Root Required) Apps: Got your Android device rooted? Here are some amazing applications worth installing on it.
  • TWRP Setup: TWRP allows you to try out custom firmware. This is one of the fantastic apps i had to make a separate section for since it’s setup can be a bit confusing if you’ve never done it before.
  • Backups and Restores: Just some tips I found on creating backups and their accompanied restoring operations (in a time of need).

Getting Started

I performed all of the steps below on a Fedora 20 Linux box, but it should work on any variation of Linux (the instructions that is). Please note that these steps are specifically geared towards a Nexus 6, but I’m sure any competent person can adjust what is identified here to work with their own Android device (phone/tablet) too. 🙂

Android SDK Preparation

This step is vital for the two-way communication to your device using your PC. I did the following using Fedora 20; but the rules don’t change for most distributions.

  1. As root, make sure the proper libraries are installed and available to your environment:
    # You'll need the 32-bit version of libstdc and the JDK 
    yum install libstdc++.i686 java-1.7.0-openjdk -y
    
  2. Download the full Android SDK here (scroll to the bottom of the page>DOWNLOAD FOR OTHER PLATFORMS>SDK Tools Only). At the time I did this, the file was called: android-sdk_r24.0.2-linux.tgz.
    # Alternatively, you can fetch it this way
    # (as long as the site doesn't change their paths around):
    wget http://dl.google.com/android/android-sdk_r24.0.2-linux.tgz
    
  3. Install it’s contents into a workable directory; the following prepared my development environment placing everything in ”’~/Development/Android/SDK”’:
    mkdir -p ~/Development/Android/SDK
    # Extract contents of the SDK into our newly created directory
    # the --strip 1 eliminates the parent directory (called android-sdk-linux)
    # we just want the root of this to go into the SDK directory instead
    tar xvfz android-sdk_r24.0.2-linux.tgz -C ~/Development/Android/SDK --strip 1
    
  4. The following will allow regular users to utilize the fastboot command. Without this step, you must switch to root to utilize the command.

    cat < /etc/udev/rules.d/51-android.rules
    # Nexus 6 ID: 18d1
    SUBSYSTEM=="usb", ATTR{idVendor}=="18d1", MODE="0666", GROUP="plugdev" 
    _EOF
    chmod 644 /etc/udev/rules.d/51-android.rules
    
  5. Now we need to add a few more tools to complete our setup. This can be done through the tools/android binary.

    # Create an environment variable to simplify the commands
    ASDK="$HOME/Development/Android/SDK"
    # Change to our development directory
    pushd $ASDK
    # Update our path
    export PATH="$PATH$ASDK/tools:$ASDK/platform-tools"
    
    # Install (the following were in the SDK r24.0.2 package)
    # they may be different in yours.
    #  1- Android SDK Platform-tools
    #  2- Android SDK Build-tools
    # 57- Android Support Library
    #
    # The following command lists all of the packages you can install:
    #    > android list sdk
    #
    # If you want to be certain you grab the right packages and
    # are using a different release, here is a cheat that will
    # fetch this information for you:
    PACKAGES=$(tools/android list sdk --extended | 
                egrep -B2 -i 'Android (Support Library|SDK Platform-tools|SDK Build-tools)' | 
                egrep '^id:' | 
                sed -e 's/[^"]+"([^"]+)"/1/g' | 
                cut -d- -f1 | tr -s '[:space:]' , | 
                sed -e 's/,+$//g')
    
    # Now install our packages! (if there are any to install)
    android update sdk -u -a -t $PACKAGES
    
    # if you get the following error, it's because you didn't install
    # libstdc++ as identified in an earlier setup:
    #   ...
    #   Stopping ADB server failed (code 127).
    #   Starting ADB server failed (code 127).
    #   Done. 3 packages installed.
    #
    # To fix this, make sure you have installed the 32-bit libstd++ (as root):
    #    yum install libstdc++.i686
    #
    # After it's installed, you can restart the adb server with the
    # following command:
    #     adb start-server
    

    Alternatively, you can just launch the tools/android binary and use the GUI to set things up. You will need to install the following (3) packages (minimally):

    1. Tools >> Android SDK Tools
    2. Tools >> Android SDK Platform-tools
    3. Extras >> Android Support Library
  6. You can test to see if you’ve got everything working by preforming the following:
    # Tests if the adb-server is successfully running
    adb version
    
    # NOTE:
    # If it doesn't work, don't panic (yet); try killing the server
    # (it's possible it's not even running) and restarting it:
    adb kill-server
    adb start-server
    
    
  7. Now try testing the command identified earlier again:
    adb version

    If it displays Android Debug Bridge version x.x.x, then you’ve completed the first part successfully.

Enable Development / Debugging Mode on Device

You must log into your device and access the settings to enable Development / Debugging mode:

  1. Go into Settings.
  2. Under About, you’ll be able to locate your Build Number.
  3. Tap Build Number 7 times until you are notified that you have activated Developer options.
  4. Go into Developer Options, ensure it is enabled and check the Enable OEM Unlock checkbox.
  5. While in Developer Options, ensure the USB Debugging checkbox is checked too.

Unlock the Bootloader

Once the bootloader is unlocked, your Android device will prompt you that it will erase everything.
Before proceeding with this step, make sure you’ve backed up everything important to you. Preferably you’re doing this step with a brand new device so you have nothing to lose at this time.

  1. Turn the device off (if it’s on already) and turn it back on while holding the volume down and the power button at the same time. This will start your device into the bootloader/fastboot mode.
  2. Plug the device into your PC (via a USB cable), then open a command prompt window window and type the following:
    # Create an environment variable to simplify the commands
    ASDK="$HOME/Development/Android/SDK"
    # Change to our development directory
    pushd $ASDK
    # Update our path
    export PATH="$PATH:$ASDK/tools:$ASDK/platform-tools"
    
    # list the devices detected in fastboot mode:
    fastboot devices
    

    If your device’s serial number shows up, you are good to go! If not, you’ll need to figure this part out before continuing.

  3. This next step will completely wipe EVERYTHING off of your device. Make sure to back up anything worth saving before proceeding. If you’re having second thoughts about this; now is the time to back out. You can reboot your device again and it will return how it was.

    Otherwise, for all intent and purposes, unlock the bootloader with the following command:

    fastboot oem unlock

    On the device (still attached to the computer), a screen should pop up asking whether or not you would like to unlock the bootloader. Use the volume rockers to highlight “Yes” then press power to confirm the action.
    After the above command has finished executing, run the following to restart your device:

    fastboot reboot

    The device will reboot at this time.

    Next you will be presented with a screen containing an Android logo as well as a progress bar. It can take about 8 minutes or so to finish.

    When it’s complete, it will restart and act as though it was the first time you’ve ever turned on the device. There is no sense in doing anything yet, we still have a few more steps to do. Just power off the device at this time and keep reading.

Root Your Android Device

Your bootloader must be unlocked for this step to work correctly. Otherwise, you can root your device with the following actions:

  1. Download the Chainfire‘s (CF-)Auto-Root tool to automate everything for you downloadable from here.
  2. Extract the ZIP file to your Desktop; this makes it easy to access.
  3. Turn the device off (if it’s on already) and turn it back on while holding the volume down + power button at the same time. This will start your device into the bootloader/fastboot mode.
  4. Plug the device into your PC (via a USB cable), then open a command prompt window window and type the following:
    # Create an environment variable to simplify the commands
    ASDK="$HOME/Development/Android/SDK"
    # Change to our development directory
    pushd $ASDK
    # Update our path
    export PATH="$PATH:$ASDK/tools:$ASDK/platform-tools"
    
    # Create a directory to place the Auto-Root contents in:
    mkdir -p "$ASDK/autoroot"
    
    # Uncompress contents to this directory:
    unzip CF-Auto-Root-shamu-shamu-nexus6.zip -d "$ASDK/autoroot"
    
    # Reboot your device into bootloader mode (power+volume down or this option):
    adb reboot bootloader
    
    # Now root your device with the following commands:
    IMAGE=$(find autoroot/image -type f -name '*.img')
    chmod ug+x autoroot/tools/fastboot-linux
    autoroot/tools/fastboot-linux boot "$IMAGE"
    

Upgrade Your Rooted Device

When Android v5.1 came out; the OTA failed (rebooted to an ‘error’). I had to fetch the stock version I was currently running so I could specifically just flash 2 images (system and recovery).

  1. Setup your development environment
    # Create an environment variable to simplify the commands
    ASDK="$HOME/Development/Android/SDK"
    # Change to our development directory
    pushd $ASDK
    # Update our path
    export PATH="$PATH:$ASDK/tools:$ASDK/platform-tools"
    
  2. The point of this section is you want to update your Android ‘without’ losing your data.Nexus 6 Settings Build Number

    Download the stock firmware associated with the (rooted) version currently installed on your device here. The steps below will show you how you can access what is important from what you just downloaded.

    The best way to be sure you’re downloading the correct version is to go into the About settings of your device and pay attention to Build number identified at the bottom. You just need to download and work with this version as your base.

    # 5.0.1 shamu Stock (Nexus 6); You will need to visit the following
    # website to get the right firmware for your device if it's not the
    # same version I'm using: 
    # - https://developers.google.com/android/nexus/images
    #
    wget https://dl.google.com/dl/android/aosp/shamu-lrx22c-factory-ff173fc6.tgz
    mkdir -p "$ASDK/firmware.nexus.6-shamu.5.0.1"
    tar xvfz shamu-lrx22c-factory-ff173fc6.tgz -C "$ASDK/firmware.nexus.6-shamu.5.0.1"  --strip 1
    
    # Change into our Firmware Directory
    pushd "$ASDK/firmware.nexus.6-shamu.5.0.1"
    
    ######################################
    # After previously running the update from 5.0.1 to 5.1.0
    # My Nexus was updated to LMY47D (seen from options)
    # So this can be set up as follows:
    wget https://dl.google.com/dl/android/aosp/shamu-lmy47d-factory-6c44d402.tgz
    mkdir -p "$ASDK/firmware.nexus.6-shamu.5.1.0"
    tar xvfz shamu-lmy47d-factory-6c44d402.tgz -C "$ASDK/firmware.nexus.6-shamu.5.1.0"  --strip 1
    
    # Change into our Firmware Directory
    pushd "$ASDK/firmware.nexus.6-shamu.5.1.0"
    
    ######################################
    # After the update from 5.1.0 to 5.1.1 was successful
    # I checked for the version i was running for a recovery
    # point (in the future)
    # My Nexus was updated to LMY47Z (seen from options)
    # So this can be set up as follows:
    wget https://dl.google.com/dl/android/aosp/shamu-lmy47z-factory-33951732.tgz
    mkdir -p "$ASDK/firmware.nexus.6-shamu.5.1.1"
    tar xvfz shamu-lmy47z-factory-33951732.tgz -C "$ASDK/firmware.nexus.6-shamu.5.1.1"  --strip 1
    
    # Change into our Firmware Directory
    pushd "$ASDK/firmware.nexus.6-shamu.5.1.1"
    
    ######################################
    # The below applies to any firmware previously downloaded above
    
    # Extract the system images next
    unzip *.zip -d image
    
    # Reboot your device into bootloader mode (power+volume down or this option):
    adb reboot bootloader
    
    # Flash the 2 critical images that will allow your OTA to properly
    # update itself. You will loose 'root' if you've done so.
    # Don't worry; you won't lose your data when you do this.
    fastboot flash system image/system.img
    fastboot flash recovery image/recovery.img
    
    # If you've got other boot modifications (mods) installed (such as TWRP
    # or BusyBox) running your Android device in an unencrypted mode,
    # you should additionally run the following:
    fastboot flash boot image/boot.img
    
    # If you have nothing to lose and just want to completely reset
    # your device to the version you downloaded, you can run the command:
    #    ./flash-all.sh
    # Otherwise, don't do this (to save your data)!! Just move on
    # to the next command identified below to reboot.
    
    # Your device no longer has root access at this point
    # you can reboot it now
    fastboot reboot
    
    popd
    
  3. Nexus 6 OTA UpdateAfter the reboot, you will have lost your root privileges. However, you can now upgrade using the OTA update Google provides. Once the OTA upgrade has complete, you can re-root your device using the steps already identified above.

Fantastic (Root Requiring) Apps to Install

Here are some great apps worth installing on your rooted devic:

If you’re feeling brave and want to start playing with custom firmware:

If I forgot one, or you have one to share; please feel free to let me know and I’ll add it!

TWRP Setup

TWRP allows you to backup your firmware as well as flash and install/uninstall custom firmware. It requires (Stericson) BusyBox or BusyBox Pro to work correctly. Naturally it also requires that your device is rooted already. Here are the devices TWRP support. For the Nexus 6; here is the direct link.

You can think of this application has installing 2 key components onto your device:

  • The .apk (front end application) which provides you with some rudimentary control to it’s backend. This is a simple app you just download from the Google Play store here.

    It provides you with the ability to flash the backend for you. I’ll explain how to flash it manually below; the only advantage of doing it manually is you can use some of the newer firmware that hasn’t made it back to the .apk yet. That and you know for certain you’ll flash the right ROM as using the .apk gives you the ‘chance’ to brick your device through accidental choice selections.

  • The backend itself. This is the beauty behind TWRP. It’s an image file you need to flash directly to your device (overwrite the stock version). Through this you can:
    1. Backup your entire device’s state (everything; including the firmware you’re running)
    2. Restore any previous backup you already created.
    3. Flash a new custom firmware you feel is worth trying without worry for if something goes wrong, you can just restore a backup. Note: It’s presumed that the first thing you do once you get this software installed is backup your current system so you have a good restore point.
  1. Setup your development environment
    # Create an environment variable to simplify the commands
    ASDK="$HOME/Development/Android/SDK"
    # Change to our development directory
    pushd $ASDK
    # Update our path
    export PATH="$PATH:$ASDK/tools:$ASDK/platform-tools"
    
  2. The point of this section is you want to update your Android ‘without’ losing your data.

    Download the stock firmware associated with the (rooted) version currently installed on your device here and prepare it’s content. If you have nothing to loose; the easiest way is to just flash everything using the latest stock firmware. But if you’re like me; here is the custom steps i took to upgrade from 5.0.1 to 5.1.0.

    # 2.8.6.0 shamu Stock (Nexus 6); You will need to visit the following
    # website to get the right firmware for your device if it's not the
    # same version I'm using:
    # - http://dl.twrp.me/shamu/
    
    [ ! -d "$ASDK/twrp/" ] && mkdir -p "$ASDK/twrp/"
    pushd "$ASDK/twrp/"
    # At the time; the latest image was twrp-2.8.6.0-shamu.img
    curl --referer http://dl.twrp.me/shamu/twrp-2.8.6.0-shamu.img 
         -O http://dl.twrp.me/shamu/twrp-2.8.6.0-shamu.img
    
    # Reboot your device into bootloader mode (power+volume down or this option):
    adb reboot bootloader
    
    # Flash the recovery image with TWRP's version
    fastboot flash recovery twrp-2.8.6.0-shamu.img
    
    # You now have TWRP installed
    fastboot reboot
    
    popd
    

Backup / Restore

  • TWRP allows you to preform full backups and restores of your device without the need of a PC connected to it. It doesn’t hurt to use this option. In order to do this; it needs to modify the recovery ROM of your device. But this isn’t really a big deal since you can easily flash it back to where it was using the firmware provided on Google’s website. Note: This application requires that your device is already rooted.
  • Titanimum Backup allows you to backup your applications and even clean out the ones you don’t want. This application is an absolutely MUST for all rooted devices!
  • Android SDK supports backups and restores too.Here is a great forum post explaining the whole operation in great detail; but in short the following commands will do the trick (assuming your device is powered on).
    # Create an environment variable to simplify the commands
    ASDK="$HOME/Development/Android/SDK"
    # Update our path
    export PATH="$PATH:$ASDK/tools:$ASDK/platform-tools"
    
    # Create Backup
    adb backup -f $(date +'%Y.%m.%d-nexus6.backup') -apk -shared -all
    
    # You can later restore this file with the command:
    adb restore YYYY.MM.DD-nexus6.backup
    

    The nice thing about the built in Android backup and restore commands is that they don’t require a rooted device to use. Your device has to be unlocked and connected to your PC (via USB) for this option to work though! Honestly, if your device is already rooted, then TWRP and Titanium Backup are the best solutions.

Reverting Everything Back

Sometimes you may want to revert everything back to how it was. This is especially an essential step if you have to send your device back to Google (or another carrier) to full fill it’s warranty.

  1. Setup your development environment
    # Create an environment variable to simplify the commands
    ASDK="$HOME/Development/Android/SDK"
    # Change to our development directory
    pushd $ASDK
    # Update our path
    export PATH="$PATH:$ASDK/tools:$ASDK/platform-tools"
    
  2. Download the stock firmware associated with your device here and prepare it’s content.

    # 5.0.1 shamu Stock (Nexus 6); You will need to visit the following
    # website to get the right firmware for your device if it's not the
    # same version I'm using: 
    # - https://developers.google.com/android/nexus/images
    # 2015 Jan - Updated Nexus 6 to v5.1.0; Future flashes for myself
    #            will involve treating that version as my stock instead.
    wget https://dl.google.com/dl/android/aosp/shamu-lrx22c-factory-ff173fc6.tgz
    mkdir -p "$ASDK/firmware.nexus.6-shamu.5.0.1"
    tar xvfz shamu-lrx22c-factory-ff173fc6.tgz -C "$ASDK/firmware.nexus.6-shamu.5.0.1"  --strip 1
    
    # now run the firmware restore
    pushd "$ASDK/firmware.nexus.6-shamu.5.0.1"
    # This command does it all and restarts the device when your done.
    ./flash-all.sh
    popd
    
  3. Now you need to re-lock the device. The easiest way to do this is power on the device with the power and volume down button held down.

    Note: if you are in a unique position where the volume down button doesn’t work (making it impossible to do the bootloader combination; do the following:

    • Log back into your device using any user (it really doesn’t matter); Don’t bother putting your real information in otherwise you’ll have to wait while your device downloads and re-installs all of your apps. The main thing is you need to enable the developer mode again and Debugging Mode. You will be prompted to authorize the connection within your device (make sure to do so).
    • Once Debugging Mode is enabled, you can connect your device up to your laptop and type the following:
      adb reboot bootloader
    • You can safely relock your device with the following command:
      fastboot oem lock
      

      After the above command has finished executing, run the following to restart your device:

      fastboot reboot

      The device will reboot at this time.

    Next you will be presented with a screen containing an Android logo as well as a progress bar. It can take about 8 minutes or so to finish.

    When it’s complete, it will restart and act as though it was the first time you’ve ever turned on the device. You can safely power it off now. It’s good to be shipped back to Google!

Sources

Credit

This blog took me a very long time to put together and test! If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.