All posts by l2g

Datetools: Date Manipulation and Cron Alternative For Linux

Introduction to Datetools

I wrote Datetools (in C++) to allow the manipulation of date time from the command line. It greatly simplified my life and maybe it will help yours out too!. It comprises of two core applications:

  • Dateblock: allows you to block until a scheduled period of time arrives unlike sleep which blocks for a set period of time. I found this so helpful, I ended up additionally building in an python extension for it.
  • Datemath: This application is just a simple way of preforming simple math on the system date.

The source code can be found here on GitHub if you’re interested in compiling it yourself. Or you can just scroll to the bottom of this blog where I provided the packaged goods.

Dateblock

The tool works very similarly to cron and sleep (a combination of the two); you can pass it a crontab string if that’s what you’re used too, or you can simply pass it in variables as arguments as well (as all other commands work):

Here’s is an example of what I mean:

# block until a minute divisible by 10 is reached:
# ex: HH:00, HH:10, HH:20, HH:30, HH:40, and HH:50
dateblock --minute=/10
# We will only reach this line when the above scheduled time has
# been met.
echo "Scheduled time reached; current time is: $(date)"

An equivalent crontab entry would look like this:

# block until a minute divisible by 10 is reached:
/10 * * * * echo "Scheduled time reached; current time is: $(date)"

Dateblock can also do another cool feature called ‘drifting’ which allows you to schedule processes on delayed cycles… Note that drifting is always specified in seconds. For example:

# Unblock on 5 minute cycles, but 2 minutes (120 seconds) into them:
# ex: HH:02, HH:07, HH:12, HH:17, HH:22, etc..
dateblock --minute=/5 --drift=120
# We will only reach this line when the above scheduled time has
# been met.
echo "Scheduled time reached; current time is: $(date)"

An equivalent crontab entry would look like this:

# block until a minute divisible by 10 is reached:
/5 * * * * sleep 120; echo "Scheduled time reached; current time is: $(date)"

The complexity of the tool can be as powerful as you want it to be:

# Unblock only on hours divisible by 5 on the 1st through to the 14th
# of every month (as well as the 20th). Unblock only when 30 seconds
# of that minute has elapsed.
 dateblock -o /5 -d 1-14,20 -s 30
# We will only reach this line when the above scheduled time has
# been met.
echo "Scheduled time reached; current time is: $(date)"

There is no way to reproduce this in a crontab unless the 30 second reference at the end is unnecessary… in that case:

# block until a minute divisible by 10 is reached:
0 /5 1-14,20 * * sleep 120; echo "Scheduled time reached; current time is: $(date)"

Just like crontabs, dateblock supports minute, hour, day of month, month and day of week. In addition, dateblock support seconds too. dateblock accepts traditional crontab entries as well as arguments:

#     day of week (0 - 6) (Sunday=0) --+
#     month (1 - 12) ---------------+  |
#     day of month (1 - 31) -----+  |  |
#     hour (0 - 23) ----------+  |  |  |
#     min (0 - 59) --------+  |  |  |  |
#  ***sec (0 - 59) -----+  |  |  |  |  |
#                       |  |  |  |  |  |
#                       -  -  -  -  -  -
# Dateblock Cron Entry: *  *  *  *  *  *
# Cron Crontab Entry:      *  *  *  *  *

# Unblock on the specific hours of 0 and 12:
# ex: HH:00, HH:12
$> dateblock --cron="0 0 00,12"

You’ll notice in the above, I didn’t bother specifying the remaining cron fields… In this case they will assume the default of *. But you can feel free to specify a * for readability. The other thing to observe is the addition of the second column which isn’t present in a regular crontab entry. It’s rules are no different then what you’ve already learned from other fields.

Testing

Simply adding a –test (-t) switch to your dateblock entry will allow you to test the tool in a debugging mode to which it will present to you the current time followed by when it would have unblocked for had you not provided the –test (-t) switch. It’s a great way to calculate when the next processing time will be.

Python Extension

To handle scheduled processes for my websites, I created a python extension for dateblock. This allowed to extend it’s flexibility with other offline processing… consider the following example:

from dateblock import dateblock
while True:
    # /5 as first argument means unblock on the 5th second of each minute
    dateblock('/5')
    print 'begin processing ...'
    # your code here...
    # if you want, you can even report the next unblock time
    print 'Done processing; blocking until %s' % 
        dateblock('/5', block=False).strftime('%Y-%m-%d %H:%M:%S')

You can also also access the drift as such:

from dateblock import dateblock
print 'Unblock at %s' % 
    dateblock('/5', drift=120, block=False).strftime('%Y-%m-%d %H:%M:%S')

Finally the python extension allows you to pass in a datetime object as an argument for calculating a time based on another (and not the current time which is the default).

from dateblock import dateblock
from datetime import datetime
from datetime import timedelta

# 31 days ago
reftime = datetime.now() - timedelta(days=31)

print('Would blocking until %s' % 
    dateblock('/5', drift=120, block=False, )
      .strftime('%Y-%m-%d %H:%M:%S') + 
    " if time was %s" % reftime
      .strftime('%Y-%m-%d %H:%M:%S'))

Things to Consider

Just like sleep, dateblock uses SIGALARM to manage its wake up time. Therefore if your code relies heavily on SIGALARM for another purposes, dateblock may not be a solution for you since you could interrupt it’s reliability (though not likely). This really shouldn’t be a big concern because this exact same warning comes with the sleep libraries we’ve been using for years. But it does mean that sleep could interfere with dateblock just as dateblock could interfere with sleep if they were both used in separate threads.

Dateblock vs Sleep
Dateblock vs Sleep

Why would I use dateblock over sleep?

Scheduling is the key… If your program relies completely on sleep, then the only thing you’re accomplishing is cpu throttling (controlling unnecessary thrashing). This is approach is fine if you’re going to just retry connecting to an unresponsive server in ?? seconds. But what if timing becomes an important factor of your application? The dateblock tool ensures you only unblock at absolute times vs sleep which unblocks at relative times with respect to when it was called.

Dateblock also allows your program to chronologically work in turn with other applications that may be on their own processing cycle. Such as something delivering data at the top of every hour. You may wish to have your program wake up 5 min after the top of each hour to perform the processing regardless of when your program was started.

Datemath

There isn’t as much to be said about Datemath; I personally never found a Linux/Unix tool that would allow me to script date/time calculations from the command line. For that reason, this tool exists.
Here is an example of the tools function:

# what time is it now?
date +'%Y-%m-%d %H-%M-%S'
# The above output was '2013-10-26 09-42-21' at the time of this blog
# what time will it be 5 months and 3 days from now
datemath --months=5 --days=3 --format='%Y-%m-%d %H-%M-%S'
# the above output was '2014-03-29 09-42-21' at the time of this blog
# and this makes sense... this is the calculation we want.

The tool supports negative values for calculating into the past as well and will handle leap years in the calculations too.

# what time is it now?
date +'%Y-%m-%d %H-%M-%S'
# The above output was '2013-10-26 09-45-45' at the time of this blog
# What was the current date 753 days ago?
datemath --days=-753 --format='%Y-%m-%d %H-%M-%S'
# the above output was '2011-10-04 09-45-45' at the time of this blog
# and this makes sense... this is the calculation we want.

No Python Module For Datemath

There is no python module for datemath because Python’s datetime and timedelta libraries already provide a fantastic solution for the same problem datemath solves…

# Simple code example to show why it really isn't
# necessary to port datemath to Python:
from datetime import datetime
from datetime import timedelta
in_the_past = datetime.now() - timedelta(minutes=15)
print '15 minutes ago: %s' % in_the_past
                              .strftime('%Y-%m-%d %H:%M:%S')

Just give me the goods

No problem, below are the RPMS as well as their accompanied source packages:

Package Download Description
dateblock
el6.rpm
/
el7.rpm
The powerful (cron like) command line interface (CLI) tool.
python-dateblock
el6.rpm
/
el7.rpm
The python extension for dateblock.
datemath
el6.rpm
/
el7.rpm
The datemath command line interface tool.
datetools
el6.rpm
/
el7.rpm
An optional package which includes licensing and information.

Note: The source rpm can be obtained here which builds everything you see in the table above. It’s not required for the application to run, but might be useful for developers or those who want to inspect how I put the package together.

No way, I’m building this myself; I don’t trust you

That’s okay, I understand; here is how you can build it yourself:

# Install 'mock' into your environment if you don't have it already
# This step will require you to be the superuser (root) in your native
# environment.
yum install -y mock

# Grant your normal every day user account access to the mock group
# This step will also require you to be the root user.
usermod -a -G mock YourNonRootUsername

At this point it’s safe to change from the ‘root‘ user back to the user account you granted the mock group privileges to in the step above. We won’t need the root user again until the end of this tutorial when we install our built RPM.

# Create an environment we can work in
mkdir datetool-build

# Change into our temporary working directory
cd datetool-build

curl -L --output datetools-0.8.1.tar.gz \
   https://github.com/caronc/datetools/archive/v0.8.1.tar.gz

# Extract Spec File
tar xfz datetools-0.8.1.tar.gz \
   datetools-0.8.1/datetools.spec \
      --strip-components=1

# Initialize Mock Environment
mock -v -r epel-6-x86_64 --init

# Now install our dependencies
mock -v -r epel-6-x86_64 --install boost-devel libstdc++-devel 
          glib-devel python-devel autoconf automake libtool

# Copy in our downloaded content:
mock -v -r epel-6-x86_64 --copyin datetools-0.8.1.tar.gz
   /builddir/build/SOURCES
mock -v -r epel-6-x86_64 --copyin datetools.spec 
   /builddir/build/SPECS

# Shell into our environment
mock -v -r epel-6-x86_64 --shell

# Change to our build directory
cd builddir/build

# Build our RPMS
rpmbuild -ba SPECS/datetools.spec

# we're now done with our mock environment for now; Press Ctrl-D to exit
# or simply type exit on the command line of our virtual environment
exit

Future Considerations

This is totally up in the air, at the moment the tool does everything I needed at the time. However I could see the following becoming a useful feature in the future:

  • Pass in a different time into both programs (instead of always working with the current time) (You can already do this with the dateblock python extension).
  • Have dateblock additionally take in a program and arguments as input to have it automatically execute the call to it when the scheduled time is reached. In addition to this, it means the dateblock tool would daemonize itself and run in the background on reoccurring schedules.
  • Add a devel package and create a shared library for C++ linking; perhaps the binary tools and extensions could link here too. Right now the library is just so small it’s really nothing to just include it statically as it is now.
  • Got an idea of your own? Pass it along! You can also submit a pull request to me on GitHub here.

Credit

Please note that this information took me several days to put together and test thoroughly. I may not blog often; but I want to re-assure the stability and testing I put into everything I intend share.

If you like what you see and wish to copy and paste this information, please reference back to this blog post at the very least. It’s really all I ask.

Upgrading the MTP Support on CentOS 6

Introduction

The Media Transfer Protocol (MTP) provides us direct file access to devices such as our smart phones and other devices that choose to use the protocol. In my case, my Nexus 4 couldn’t correctly attach itself to my system. From my Google results; it seemed to be due to the version of libmtp I was running. You see, CentOS/RHEL ships with libmtp v1.0 and most of the support for cell phones that came out within the last 2 years isn’t available until v1.1. libmtp v1.1 additionally builds against most peoples FUSE implementations such as MTPfs, jmtpfs and simple-mtpfs. It’s through these FUSE/MTP implementations that we can easily mount and directly access the content on our smart phones and tablets that use the MTP protocol.

So upgrade libmtp already; why are you blogging about it?

Oh boy… well here are all the problems I faced:

  • VLC Media Player is built against this same library. Upgrading it means looking after this dependency too; or go without it (no, not going to happen).
  • Rhythmbox is also built against this same library (libmtp). Upgrading it means looking after this dependency too; or go without it (again… not going to happen).

Then the second problem… we’ve only tackled the upgrade of libmtp above if we go through with this… we still haven’t even tackled the FUSE implementation. Here are the results I had testing some of them:

  • MTPfs: At the time of this blog, the last update for this software was on February 4th, 2010. I got the software to compile by simply using the version that ships with Fedora 18 (1.1-0.3.svn20120510) but when I tried to use it, it just hung as a normal user. If I tried mounting the device as root it would fail (see below):
    $ mkdir media
    $ # As myself (non-root)
    $ mtpfs media/
    Listing raw device(s)
    Device 0 (VID=18d1 and PID=4ee1) is a Google Inc (for LG Electronics/Samsung) Nexus 4/10 (MTP).
       Found 1 device(s):
       Google Inc (for LG Electronics/Samsung): Nexus 4/10 (MTP) (18d1:4ee1) @ bus 2, dev 89
    Attempting to connect device
    libusb_open() failed!: Permission denied
    LIBMTP PANIC: Unable to initialize device
    Unable to open raw device 0
    
    $ # As root (superuser)
    $ sudo mtpfs media
    Listing raw device(s)
    Device 0 (VID=18d1 and PID=4ee2) is a Google Inc (for LG Electronics/Samsung) Nexus 4/10 (MTP+ADB).
       Found 1 device(s):
       Google Inc (for LG Electronics/Samsung): Nexus 4/10 (MTP+ADB) (18d1:4ee2) @ bus 2, dev 73
    Attempting to connect device
    Android device detected, assigning default bug flags
    Error 1: Get Storage information failed.
    Error 2: PTP Layer error 02fe: get_handles_recursively(): could not get object handles.
    Error 2: Error 02fe: PTP: Protocol error, data expected
    Listing File Information on Device with name: (NULL)
    LIBMTP_Get_Storage() failed:-1

    Now I don’t want to give MTPfs a bad name. It’s quite possible that it works really well but just not for my personal purposes. In fact it compiles perfectly in CentOS/RHEL 6 right out of the box without any modifications. Therefore just because it didn’t work for me doesn’t mean it won’t necessarily work for you. For this reason and the fact it was once maintained (up to Fedora 18) I’ll include my results with it here too.

  • jmtpfs worked the best for me. The only problems I ran into with this software was that it wasn’t already packaged for CentOS/RHEL. The only thing I had to work with for packaging it myself was a very long set of comments (see the link back to it’s official site to see what I’m talking about). That all said; the software works fantastic right out of the box. Kudos to the great work of the developer and his online support.
  • simple-mtpfs works well too but required some tweaking to make it possible. You see, simple-mtpfs relies on the new C++11 compiling features and standards (which truly are amazing BTW) but weren’t available during the release of CentOS/RHEL 6. Making these standards available by upgrading the glib core library to a newer version would be just crazy. In fact, you could seriously risk jeopardizing the stability of the OS itself since everything else would have been compiled against the old version. So, to work around this, I wrote a small patch file for portions of the simple-mtpfs code mimicking the old behavior prior to the C++11 standards.

Just hand over everything

Of course… and as always; I planned on it. Hopefully you’ll find the packages useful! Like my other blogs, the installation instructions will be near the bottom of this blog, along with the ‘do it yourself’ for those who don’t trust the sources.

First thing is first; this is what makes everything else tick:

Here are the dependencies you may or may not need to get out of the way. Note that I just went ahead and upgraded VLC to 2.0.6 (from the stock version 1.1). There may be other dependencies of libmtp you find that I didn’t simply because I didn’t use those packages. You can just assume that all of the binary packages listed below have been compiled against libmtp v1.1 (shared above).

After you’ve got libmtp correctly installed and your system appears to be right back in the state it was previously, we can now move on to adding one of the excellent FUSE/MTP packages some incredible developers have put together for us.

  • jmtpfs-0.4-1.el6.x86_64.rpm
    Author: Jason Ferrara (Website)
    Source: jmtpfs-0.4-1.el6.src.rpm
    Compilation Details: I made 1 small patch file (see here) introducing some of the nice idea’s his followers of his blog suggested for auto-mounting. I additionally had to make one small change so that his code could compile against the version of FUSE currently shipped with CentOS/RHEL 6. The only other thing I can take credit for is the rpm packaging itself; the rest of the work was that of the hard working developer who made this application possible. The spec file I wrote can be seen here.
  • simple-mtpfs-0.1-8.el6.x86_64.rpm
    Author: Peter Hatina (Website)
    Source: simple-mtpfs-0.1-8.el6.src.rpm
    Compilation Details: I created 2 separate patch files for this to properly compile in CentOS/RHEL. The first patch rolls back the C++11 coding styles to the older way of doing it (so it can compile against our existing libraries). The second patch introduces the auto-mounting for our environment (exactly how It was done in my patch for jmtpfs). To accommodate these patch files I needed to additionally update the spec file which you can view here.
  • mtpfs-1.1-0.3.svn20120510.el6.x86_64.rpm
    Author: Chris Debenham (Website)
    Source: mtpfs-1.1-0.3.svn20120510.el6.src.rpm
    Compilation Details: No magic went on here; I just grabbed the packaged code from a Fedora 18 release on pkgs.org and it compiled without any problems. Note: It appears the support for this package was dropped after Fedora 18 since it isn’t available for either Fedora 19 or 20.

Here are the source packages for those who are interested:

Show me what you did; I’m not using your stuff

In some cases you’ll need to use what I’ve done (patch wise); but to be as transparent as possible, I’ll show you how you can easily build everything and review all of what is going on before committing to using it. Now keep in mind also; there is A LOT of stuff covered here. There is libmtp itself, the dependencies issues and then finally the choice of FUSE/MTP implementation. I’ll cover the dependencies last since not everyone will have this issue.

First I set up a mock environment to work in; this allows us to do compiling outside of our native environment and means we don’t need to ever install any development libraries.

First prepare our development environment with mock if you haven’t already:

# Install 'mock' into your environment if you don't have it already
# This step will require you to be the superuser (root) in your native
# environment.
yum install -y mock

# Grant your normal every day user account access to the mock group
# This step will also require you to be the root user.
usermod -a -G mock YourNonRootUsername

At this point we can get away from the root user and build using our own user we created for our system.

Now we’ll cover libmtp v1.1 as everything revolves around this library.

# Fetch the latest copy (already packaged) of libmtp
wget http://dl.fedoraproject.org/pub/fedora/linux/development/20/source/SRPMS/l/libmtp-1.1.6-2.fc20.src.rpm
# As time goes on, this rpm may not be available, but
# you should be able to just get away with grabbing the latest
# copy of libmtp from http://pkgs.org

# Now we just rebuild it against our own environment
mock -v -r epel-6-x86_64 --resultdir=$(pwd)/results 
       --rebuild libmtp-1.1.6-2.fc20.src.rpm

# Now all our built RPMs will be in a directory
# entitled 'results' to make things easier, we'll just
# move it all into the directory we're working in now
find results -name '*.rpm' -exec mv {} . ;

# We can eliminate the results directory now
rm -rf results

Everything from this point forward assumes we have a built copy of libmtp to work with. Now we’ll focus on the FUSE/MTP options.

  • Option 1: simple-mtpfs
    # Fetch our source
    wget --output-document=simple-mtpfs-0.1-8.el6.src.rpm https://www.dropbox.com/sh/9dt7klam6ex1kpp/KEU68ZzLIv/20131015/mtp/simple-mtpfs-0.1-8.el6.src.rpm?dl=1
    
    # Initialize our Environment
    mock -v -r epel-6-x86_64 --init
    
    # Now install the nessisary dependencies for simple-mtpfs
    # which we must additionally include the libmtp package we
    # just built
    mock -v -r epel-6-x86_64 --install libmtp-1.1.6-2.el6.x86_64.rpm 
                                       libmtp-devel-1.1.6-2.el6.x86_64.rpm 
                                       fuse-devel
    
    # Now if you don't have any insecurities with source rpms you
    # can finish with the next command... or you can skip it and
    # extract all of it's content for inspection before building.
    # all of the results will appear in the 'results' directory.
    # Note: You'll want to skip this step and move to the next if
    # you have any insecurities at all.
    mock -v -r epel-6-x86_64 --no-clean 
                             --resultdir=$(pwd)/results 
                             --rebuild simple-mtpfs-0.1-8.el6.src.rpm
    
    # For those really insecure can keep reading and perform
    # the following steps instead of the single command above
    # Copy our source into the mock building environment
    mock -v -r epel-6-x86_64 --copyin simple-mtpfs-0.1-8.el6.src.rpm /builddir/build
    
    # Shell into our environment now
    mock -v -r epel-6-x86_64 --shell
    
    # Change to the build directory
    cd builddir/build
    
    # Install the source package (which will deploy within
    # the mock environment only)
    rpm -Uhi simple-mtpfs-0.1-8.el6.src.rpm
    
    # Now we can safely prepare the source for inspection.
    # - the spec file will be in the SPECS/* directory
    # - all patch files and source code will be in SOURCES/*
    #
    # Once your satisfied I'm not out to get you, you can build
    # the package:
    rpmbuild -ba SPECS/simple-mtpfs.spec
    
    # we're now done with our mock environment for now; Press Ctrl-D to
    # exit or simply type exit on the command line of our virtual
    # environment
    exit
    
    # We'll return to the directory we were previously in.  We can copy
    # out the packages we just built at this point. Ignore the warning
    # about SELinux if you get one because it doesn't impact our goals.
    mock -v -r epel-6-x86_64 --copyout /builddir/build/SRPMS/simple-mtpfs-0.1-8.el6.src.rpm .
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/simple-mtpfs-0.1-8.el6.x86_64.rpm .
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/simple-mtpfs-debuginfo-0.1-8.el6.x86_64.rpm .
    
  • Option 2: jmtpfs
    # Fetch our source
    wget --output-document=jmtpfs-0.4-1.el6.src.rpm https://www.dropbox.com/sh/9dt7klam6ex1kpp/KH6b9pmCRZ/20131015/mtp/jmtpfs-0.4-1.el6.src.rpm?dl=1
    
    # Initialize our Environment
    mock -v -r epel-6-x86_64 --init
    
    # Now install the necessary dependencies for jmtpfs
    # which we must additionally include the libmtp package we
    # just built
    mock -v -r epel-6-x86_64 --install libmtp-1.1.6-2.el6.x86_64.rpm 
                                       libmtp-devel-1.1.6-2.el6.x86_64.rpm 
                                       fuse-devel 
                                       file-devel
    
    # Now if you don't have any insecurities with source rpms you
    # can finish with the next command... or you can skip it and
    # extract all of it's content for inspection before building.
    # all of the results will appear in the 'results' directory.
    # Note: You'll want to skip this step and move to the next if
    # you have any insecurities at all.
    mock -v -r epel-6-x86_64 --no-clean 
                             --resultdir=$(pwd)/results 
                             --rebuild jmtpfs-0.4-1.el6.src.rpm
    
    # For those really insecure can keep reading and perform
    # the following steps instead of the single command above
    # Copy our source into the mock building environment
    mock -v -r epel-6-x86_64 --copyin jmtpfs-0.4-1.el6.src.rpm /builddir/build
    
    # Shell into our environment now
    mock -v -r epel-6-x86_64 --shell
    
    # Change to the build directory
    cd builddir/build
    
    # Install the source package (which will deploy within
    # the mock environment only)
    rpm -Uhi jmtpfs-0.4-1.el6.src.rpm
    
    # Now we can safely prepare the source for inspection.
    # - the spec file will be in the SPECS/* directory
    # - all patch files and source code will be in SOURCES/*
    #
    # Once your satisfied I'm not out to get you, you can build
    # the package:
    rpmbuild -ba SPECS/jmtpfs.spec
    
    # we're now done with our mock environment for now; Press Ctrl-D to
    # exit or simply type exit on the command line of our virtual
    # environment
    exit
    
    # We'll return to the directory we were previously in.  We can copy
    # out the packages we just built at this point. Ignore the warning
    # about SELinux if you get one because it doesn't impact our goals.
    mock -v -r epel-6-x86_64 --copyout /builddir/build/SRPMS/jmtpfs-0.4-1.el6.src.rpm .
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/jmtpfs-0.4-1.el6.x86_64.rpm .
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/jmtpfs-debuginfo-0.4-1.el6.x86_64.rpm .
    
  • Option 3: mtpfs
    # Fetch our source from pkgs.org
    wget http://dl.fedoraproject.org/pub/fedora/linux/releases/18/Everything/source/SRPMS/m/mtpfs-1.1-0.3.svn20120510.fc18.src.rpm
    
    # Initialize our Environment
    mock -v -r epel-6-x86_64 --init
    
    # Now install the necessary dependencies for mtpfs
    # which we must additionally include the libmtp package we
    # just built
    mock -v -r epel-6-x86_64 --install libmtp-1.1.6-2.el6.x86_64.rpm 
                                       libmtp-devel-1.1.6-2.el6.x86_64.rpm 
                                       autoconf automake libid3tag-devel 
                                       glib2-devel fuse-devel
    
    # If you have insecurities at this point, you'll need to
    # take them up with someone else since I didn't make any
    # modifications to this FUSE/MTP option at all. So we
    # just need to build from source at this point.
    mock -v -r epel-6-x86_64 --no-clean 
                             --resultdir=$(pwd)/result 
                             --rebuild mtpfs-1.1-0.3.svn20120510.fc18.src.rpm
    
    

Finally our libmtp dependency issues (if you have them).

  • Rhythmbox
    # Fetch our source from pkgs.org
    wget http://vault.centos.org/6.4/os/Source/SPackages/rhythmbox-0.12.8-1.el6.src.rpm
    
    # Initialize our Environment
    mock -v -r epel-6-x86_64 --init
    
    # Now install the necessary dependencies for Rhythmbox
    # which we must additionally include the libmtp package we
    # just built
    mock -v -r epel-6-x86_64 --install libmtp-1.1.6-2.el6.x86_64.rpm 
                                       libmtp-devel-1.1.6-2.el6.x86_64.rpm
    
    # Now we just rebuild from source using the new library
    mock -v -r epel-6-x86_64 --no-clean 
                             --resultdir=$(pwd)/results 
                             --rebuild rhythmbox-0.12.8-1.el6.src.rpm
    
    # Your rpm's will appear in the 'results' directory.
    
  • VLC
    # The catch with VLC is it's development isn't frozen with
    # the version of CentOS/RHEL like Rhythmbox does. This is because
    # if your running this, you've already chosen to link to other
    # repositories like RPMForge, ATrpms, LinuxTECH, etc.
    
    # Mock however (out of the box) does not link to these locations
    # at all. The quickest (and dirty) work around to this fix is to
    # simply install 'yum' into the mock environment which we can
    # use to fetch all of our dependencies (the best way is to actually)
    # create a new mock configuration that just includes them. But
    # everyone uses to many different alternatives; so we'll do it
    # my way :)
    
    # Fetch VLC source from the repository you use. You can do this
    # using yum if you have the --download-only rpm addon. Otherwise
    # just utilize pkgs.org and get the package:
    # http://pkgs.org/search/?keyword=vlc&search_on=name&distro=82&exact=1
    # At the time v2.0.6-1 was the version available to me.
    # You may need to adjust the below commands depending on which
    # package you get; I'll use a variable so you can set it to
    # whatever you get at the time.
    
    # If you are comfortable just using the version I have; you can
    # download it here:
    wget --output-document=vlc-2.0.6-1.el6.src.rpm https://www.dropbox.com/sh/9dt7klam6ex1kpp/pm7MCGHa9K/20131015/mtp/vlc-2.0.6-1.el6.src.rpm?dl=1
    
    # Set this variable to something else if you fetched a different
    # version
    VLCVER=2.0.6-1
    # Store our original source RPM Name; we do it this way
    # because some people might use different packages that
    # have the ending f20 (Fedora), atr (ATrpms), rf (RPMForge), etc
    # This way we're working with the same file regardless of
    # the version you picked
    VLCSRPM=$(find -name "vlc-$VLCVER.*.src.rpm" 2>/dev/null)
    
    # Make sure we found your rpm at this point
    # The following should output a file that exists
    echo $VLCSRPM
    
    # Initialize our Environment
    mock -v -r epel-6-x86_64 --init
    
    # Now install the nessisary dependencies for vlc which we must
    # additionally include the libmtp package we just built
    # including 'yum' so we'll fill all the dependencies
    mock -v -r epel-6-x86_64 --install libmtp-1.1.6-2.el6.x86_64.rpm 
                                       libmtp-devel-1.1.6-2.el6.x86_64.rpm 
                                       yum
    
    # Now this is the dirty trick I was telling you about; we'll
    # just install your current yum repositories into your mock
    # environment. This will make it really easy to fill our
    # dependency problems:
    find /etc/yum.repos.d/ -maxdepth 1 -mindepth 1 
                           -type f -name '*.repo' 
                           -exec mock -v -r epel-6-x86_64 
                                  --copyin {} /etc/yum.repos.d/ ;
    
    # Now we want to copy our VLC Source into the environment
    mock -v -r epel-6-x86_64 --copyin $VLCSRPM /builddir/build
    
    # Shell into our mock environment
    mock -v -r epel-6-x86_64 --shell
    
    # Change to our building directory
    cd /builddir/build
    
    # Install all our dependencies including our source
    yum install $(rpm -qp --requires vlc*.src.rpm | cut -d' ' -f1)
    
    # Install our source RPM
    rpm -Uhi vlc*.src.rpm
    
    # There is a chance this failed for you... if so
    # you can use the collection of rpms I built manually
    # to make it possible for me; just visit here:
    # https://www.dropbox.com/sh/9dt7klam6ex1kpp/_pczPoH-Wx/20131015/mtp/dep.vlc2.0
    # You can exit the shell (Ctrl-D) or type exit and by now
    # you should have enough info as to how you can install these
    # rpms into your mock environment using mock and --install
    
    # Now build your RPM (this can take some time):
    rpmbuild -ba SPECS/*.spec
    
    # We're now done with our mock environment for now; Press
    # Ctrl-D to exit or simply type exit on the command line
    # of our virtual environment
    exit
    
    Now just copy out our packages:
    mock -v -r epel-6-x86_64 --copyout /builddir/build/SRPMS/vlc-$VLCVER.el6.src.rpm .
    
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/vlc-$VLCVER.el6.x86_64.rpm .
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/vlc-core-$VLCVER.el6.x86_64.rpm .
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/vlc-extras-$VLCVER.el6.x86_64.rpm .
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/vlc-plugin-jack-$VLCVER.el6.x86_64.rpm
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/vlc-devel-$VLCVER.el6.x86_64.rpm
    mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/vlc-debuginfo-$VLCVER.el6.x86_64.rpm
    
    # You're done now! Phewf...
    

Installation

The installation will vary depending on your environment. If you are using either VLC or Rhythmbox, your life might be easier if you uninstall them now (knowing your going to re-install them again shortly).

yum remove rhythmbox vlc

Now you can install libmtp and your choice of FUSE/MTP implimentation (as root):

  • Option 1: simple-mtpfs
    yum localinstall libmtp-1.1.6-2.el6.x86_64.rpm simple-mtpfs-0.1-8.el6.x86_64.rpm
    
  • Option 2: jmtpfs
    yum localinstall libmtp-1.1.6-2.el6.x86_64.rpm jmtpfs-0.4-1.el6.x86_64.rpm
    
  • Option 3: mtpfs
    yum localinstall libmtp-1.1.6-2.el6.x86_64.rpm mtpfs-1.1-0.3.svn20120510.el6.x86_64.rpm
    

It’s important to note that you can go ahead and install all 3 options if you really wanted to (but it’s not really worth it). Pick one… if it doesn’t work well for you, try another.

Feel free to reinstall any other packaging you were using:

# VLC v2.0
yum localinstall vlc-core-2.0.6-1.el6.x86_64.rpm vlc-2.0.6-1.el6.x86_64.rpm

# Rythmbox v0.12.8
yum localinstall rhythmbox-upnp-0.12.8-1.el6.x86_64.rpm
Enable USB Debugging Mode
Enable USB Debugging Mode

It still isn’t mounting for me:

Some phones/tablets may require you to toggle the USB debugging mode of your phone and/or tablet for things to work smoothly for you. If you already have it enabled, then try disabling it. This is what I had to do with my Nexus 4 anyway; it worked beautifully for me after I toggled this option. I think this may be a bug with the current MTP v1.1 libraries that don’t seem to correctly handle the mounting of the device when being first connected. But whatever handshaking goes on, through the toggling of this option corrects the problem. I’m sure we will see this resolved in future builds.

Other MTP Alternatives

There are other alternatives; but this blog was going to be long enough just trying to share the ones I chose already.

  • FUSE Binding for Go is another alternative I didn’t test which seems to have a lot of active development on it as well. For those who aren’t familiar with The Go Programming Language, it is basically a fairly new language that is catching on with a lot of developers.
  • Gnome MTP is designed to be a simple MP3 player client for MTP based devices.

Credit

Please note that this information took me several days to put together and test thoroughly. I may not blog often; but I want to re-assure the stability and testing I put into everything I intend share.

If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

Sources:

Offline Blogging Solutions with CentOS 6

Introduction

BloGTK v2.0
BloGTK v2.0

Offline Blogging in Linux doesn’t offer us a wide selection of free (open source) choices. Of the choices we do have at our disposal each have their own pros and cons which are really just bias opinions we’ll all carry with each other. This blog isn’t going to tell you which product is better and which one isn’t. It will provide you some alternatives to what’s already available and allow you to choose on your own. I also make additionally options available to you here as well should you choose to try them.

Keep in mind I run CentOS 6 as my primary OS (currently), so I focus primarily on making these products work on this distribution. But this doesn’t mean that all of the source RPMs I provided won’t compile for you in another distribution.

Drivel v3.0.0 Login Screen
Drivel v3.0.0 Login Screen

Open Source Offline Blogging Choices

The below outline some of the choices I found to be worth of digging further in:

I’m not sure what the status is on all of these project themselves. At this current time, I have to assume that both Drivel and BloGTK are some what dead since the last update to either of them

Gnome Blog v0.9.2
Gnome Blog v0.9.2

was back in late 2009. Meanwhile the last update made to Gnome Blog was in early 2010.

It is said that beggars can’t be choosers. So rolling with that in mind and the Open Source solutions available to us, we’ll accept what is offered and move on.

Hand over your work

With pleasure; it really didn’t take any time at all to package these properly.

Drivel (v3.0.0) took the most time to package; but even that didn’t take much effort. Drop Line provided a spec file of their own which didn’t work out of the box. It also didn’t include all the necessary dependencies. For this reason I just spun my own version of it. Have a look here if you want to see the spec file I generated.

BlogGTK v2.0 didn’t take me hardly any time at all. They didn’t change the installation that much from v1.1. The fact that it’s python based; there really isn’t a whole lot too it. You can view the spec for yourself if you’re interested.

Alternatively you can just fetch bloGTK from Pkgs.org which does such a great job organizing packages other people have put together. It’ll probably be an older version (as it was for me). At the time I wrote this blog it was BloGTK v1.1 on Pkgs.org hosted by RPMForge. It might be different when you try.

Gnome Blog was another one that actually packaged it’s own spec file within the official packaging. But the file was drastically missing dependencies and would not work out of the box at all. I had to massage it quite a bit; you can view the spec file here if you feel the need.

I will never trust you; I’ll build it for myself

Still feeling that way? No problem; here is how you can do it:

First off, I’m not a big fan of compiling code as the root user on the system I work with daily.   I am however a big fan of a tool called ‘mock‘ which allows us to develop software as root except within a safe virtual environment instead of our native one. I am also a big fan of package management; whether its a .DEB (Debian Package) or .RPM (Red Hat Package) for obvious reasons. For this tutorial; I’ll stick with RPMs since it’s what CentOS uses. We’ll prepare the RPMs and preform all our compilations within the mock environment.

# Install 'mock' into your environment if you don't have it already
# This step will require you to be the superuser (root) in your native
# environment.
yum install -y mock

# Grant your normal every day user account access to the mock group
# This step will also require you to be the root user.
usermod -a -G mock YourNonRootUsername

At this point it’s safe to change from the ‘root‘ user back to the user account you granted the mock group privileges to in the step above.  We won’t need the root user again until the end of this tutorial when we install our built RPM.

# Optionally fetch bloGTK v2.0
wget https://launchpad.net/blogtk/2.0/2.0/+download/blogtk-2.0.tar.gz
wget --output-document=blogtk.spec https://www.dropbox.com/sh/9dt7klam6ex1kpp/GR0uXU6PaC/20131008/blogtk.spec?dl=1

# Optionally fetch Drivel 3.0.0
wget --output-document=drivel-3.0.0.tar.bz2 http://prdownloads.sourceforge.net/drivel/drivel-3.0.0.tar.bz2?download
wget --output-document=drivel.spec https://www.dropbox.com/sh/9dt7klam6ex1kpp/MKD34uuBMs/20131008/drivel.spec?dl=1

# Optionally fetch gnome-blog v0.9.2
wget http://ftp.gnome.org/pub/GNOME/sources/gnome-blog/0.9/gnome-blog-0.9.2.tar.gz
wget --output-document=gnome-blog.spec https://www.dropbox.com/sh/9dt7klam6ex1kpp/O9nJdxoJMZ/20131008/gnome-blog.spec?dl=1

# Initialize Mock Environment
mock -v -r epel-6-x86_64 --init

# bloGTK dependencies
mock -v -r epel-6-x86_64 --install 
  python pygtk2 pygtk2-libglade desktop-file-utils

# Drivel dependencies
mock -v -r epel-6-x86_64 --install 
  gnome-doc-utils intltool gtk2 gtkspell-devel 
  glib-devel gtk2-devel GConf2-devel 
  gnome-vfs2-devel gtksourceview2-devel 
  libsoup-devel libxml2-devel

# gnome-blog dependencies
mock -v -r epel-6-x86_64 --install 
  pygtk2-devel gettext intltool 
  desktop-file-utils GConf2-devel 
  python-devel

mock -v -r epel-6-x86_64 --copyin blogtk.spec /builddir/build/SPECS
mock -v -r epel-6-x86_64 --copyin gnome-blog.spec /builddir/build/SPECS
mock -v -r epel-6-x86_64 --copyin drivel.spec /builddir/build/SPECS

mock -v -r epel-6-x86_64 --copyin drivel-3.0.0.tar.bz2 /builddir/build/SOURCES
mock -v -r epel-6-x86_64 --copyin gnome-blog-0.9.2.tar.gz /builddir/build/SOURCES
mock -v -r epel-6-x86_64 --copyin blogtk-2.0.tar.gz /builddir/build/SOURCES
mock -v -r epel-6-x86_64 --shell

# Within Shell Environment, Build the Desired RPM
rpmbuild -ba builddir/build/SPECS/drivel.spec
rpmbuild -ba builddir/build/SPECS/blogtk.spec
rpmbuild -ba builddir/build/SPECS/gnome-blog.spec

# exit shell (or press Cntrl-D)
exit

# Copy out your blogger of interest
mock -v -r epel-6-x86_64 --copyout /builddir/build/SRPMS/drivel-3.0.0-1.el6.src.rpm .
mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/drivel-debuginfo-3.0.0-1.el6.x86_64.rpm .
mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/drivel-3.0.0-1.el6.x86_64.rpm .

mock -v -r epel-6-x86_64 --copyout /builddir/build/SRPMS/blogtk-2.0-1.el6.src.rpm .
mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/blogtk-2.0-1.el6.noarch.rpm .

mock -v -r epel-6-x86_64 --copyout /builddir/build/SRPMS/gnome-blog-0.9.2-1.src.rpm .
mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/gnome-blog-0.9.2-1.x86_64.rpm .
mock -v -r epel-6-x86_64 --copyout /builddir/build/RPMS/gnome-blog-debuginfo-0.9.2-1.x86_64.rpm .

# Install your blogger of choice; you'll need to be root or
# have sudoers permission to do this:
yum localinstall drivel-3.0.0-1.el6.x86_64.rpm
yum localinstall blogtk-2.0-1.el6.noarch.rpm
yum localinstall gnome-blog-0.9.2-1.x86_64.rpm

Drivel & WordPress

Drivel supports WordPress with a small with the following configuration:

  1. Configure your User/Pass as you normally would have
  2. Set the Movable Type to Journal type
  3. Set the Server Address field to be http://yourusername.wordpress.com/xmlrpc.php. For example I would have put http://nuxref.wordpress.com/xmlrpc.php for my own blog.

Another thing to note about Drivel is I was unable to retrieve a list of recent posts made to the WordPress server. However every other aspect of the tool appears to fine. People using different blog engines may not notice any problem at all.

Gnome-Blog & WordPress

  1. Set the Blog Type to Self-Run Other
  2. Set the Blog Protocol to MetaWeblog
  3. Set the XML-RPC URL field to be http://yourusername.wordpress.com/xmlrpc.php. For example I would have put http://nuxref.wordpress.com/xmlrpc.php for my own blog.
  4. Configure your User/Pass as you normally would have

Not Open Source, but other Free Alternatives:

  • ScribeFire:A plugin exists for Firefox & Chrome users called ScribeFire which also enables blogging functionality from within your browser. It’s worth noting as another alternative if you want it. It doesn’t involve extra packaging since it can be installed from within your browser.
  • Thingamablog: Another free solution; Thingamablog provides the binaries directly from their website here

Credit

If you like what you see and wish to copy and paste this HOWTO, please reference back to this blog post at the very least. It’s really all I ask.

If I forgot any (Open Source) Offline Bloggers that you know about; please let me know. I have no problem updating this blog to accommodate it.

Sources

I referenced the following resources to make this blog possible: