{"id":176,"date":"2013-10-26T15:53:34","date_gmt":"2013-10-26T19:53:34","guid":{"rendered":"http:\/\/nuxref.wordpress.com\/?p=176"},"modified":"2016-11-13T22:06:28","modified_gmt":"2016-11-14T03:06:28","slug":"datetools-date-manipulation-linux","status":"publish","type":"post","link":"http:\/\/nuxref.com\/2013\/10\/26\/datetools-date-manipulation-linux\/","title":{"rendered":"Datetools: Date Manipulation and Cron Alternative For Linux"},"content":{"rendered":"<h1>Introduction to Datetools<\/h1>\n<p>\nI wrote <a title=\"GitHub Datetool Source\" href=\"https:\/\/github.com\/caronc\/datetools\" target=\"_blank\">Datetools<\/a> (in C++) to allow the manipulation of date time from the command line.  It greatly simplified my life and maybe it will help yours out too!. It comprises of two core applications:<\/p>\n<ul>\n<li><a title=\"Readme on Dateblock\" href=\"https:\/\/github.com\/caronc\/datetools\/tree\/v0.8#dateblock\" target=\"_blank\"><strong>Dateblock<\/strong><\/a>: allows you to block <em>until<\/em> a scheduled period of time arrives unlike <em>sleep<\/em> which blocks <em>for<\/em> a set period of time. I found this so helpful, I ended up additionally building in an python extension for it.<\/li>\n<li><a title=\"Readme on Datemath\" href=\"https:\/\/github.com\/caronc\/datetools\/tree\/v0.8#datemath\" target=\"_blank\"><strong>Datemath<\/strong><\/a>: This application is just a simple way of preforming simple math on the system date.<\/li>\n<\/ul>\n<p>The source code can be found <a title=\"DateTools GitHub Source\" href=\"https:\/\/github.com\/caronc\/datetools\" target=\"_blank\">here<\/a> on GitHub if you&#8217;re interested in compiling it yourself. Or you can just scroll to the bottom of this blog where I provided the packaged goods.\n<\/p>\n<h1>Dateblock<\/h1>\n<p>\nThe tool works very similarly to <em>cron<\/em> and <em>sleep<\/em> (a combination of the two); you can pass it a crontab string if that&#8217;s what you&#8217;re used too, or you can simply pass it in variables as arguments as well (as all other commands work):<\/p>\n<p>Here&#8217;s is an example of what I mean:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# block until a minute divisible by 10 is reached:\r\n# ex: HH:00, HH:10, HH:20, HH:30, HH:40, and HH:50\r\ndateblock --minute=\/10\r\n# We will only reach this line when the above scheduled time has\r\n# been met.\r\necho &quot;Scheduled time reached; current time is: $(date)&quot;\r\n<\/pre>\n<p>An equivalent crontab entry would look like this:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# block until a minute divisible by 10 is reached:\r\n\/10 * * * * echo &quot;Scheduled time reached; current time is: $(date)&quot;\r\n<\/pre>\n<p>Dateblock can also do another cool feature called &#8216;drifting&#8217; which allows you to schedule processes on delayed cycles&#8230; Note that drifting is always specified in seconds. For example:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# Unblock on 5 minute cycles, but 2 minutes (120 seconds) into them:\r\n# ex: HH:02, HH:07, HH:12, HH:17, HH:22, etc..\r\ndateblock --minute=\/5 --drift=120\r\n# We will only reach this line when the above scheduled time has\r\n# been met.\r\necho &quot;Scheduled time reached; current time is: $(date)&quot;\r\n<\/pre>\n<p>An equivalent crontab entry would look like this:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# block until a minute divisible by 10 is reached:\r\n\/5 * * * * sleep 120; echo &quot;Scheduled time reached; current time is: $(date)&quot;\r\n<\/pre>\n<p>The complexity of the tool can be as powerful as you want it to be:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# Unblock only on hours divisible by 5 on the 1st through to the 14th\r\n# of every month (as well as the 20th). Unblock only when 30 seconds\r\n# of that minute has elapsed.\r\n dateblock -o \/5 -d 1-14,20 -s 30\r\n# We will only reach this line when the above scheduled time has\r\n# been met.\r\necho &quot;Scheduled time reached; current time is: $(date)&quot;\r\n<\/pre>\n<p>There is no way to reproduce this in a crontab unless the 30 second reference at the end is unnecessary&#8230; in that case:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# block until a minute divisible by 10 is reached:\r\n0 \/5 1-14,20 * * sleep 120; echo &quot;Scheduled time reached; current time is: $(date)&quot;\r\n<\/pre>\n<p>Just like crontabs, dateblock supports minute, hour, day of month, month and day of week. In addition, dateblock support seconds too. dateblock accepts traditional crontab entries as well as arguments:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n#     day of week (0 - 6) (Sunday=0) --+\r\n#     month (1 - 12) ---------------+  |\r\n#     day of month (1 - 31) -----+  |  |\r\n#     hour (0 - 23) ----------+  |  |  |\r\n#     min (0 - 59) --------+  |  |  |  |\r\n#  ***sec (0 - 59) -----+  |  |  |  |  |\r\n#                       |  |  |  |  |  |\r\n#                       -  -  -  -  -  -\r\n# Dateblock Cron Entry: *  *  *  *  *  *\r\n# Cron Crontab Entry:      *  *  *  *  *\r\n\r\n# Unblock on the specific hours of 0 and 12:\r\n# ex: HH:00, HH:12\r\n$&gt; dateblock --cron=&quot;0 0 00,12&quot;\r\n<\/pre>\n<p>You&#8217;ll notice in the above, I didn&#8217;t bother specifying the remaining cron fields&#8230; In this case they will assume the default of <strong>*<\/strong>. But you can feel free to specify a <em>*<\/em> for readability. The other thing to observe is the addition of the <em>second<\/em> column which isn&#8217;t present in a regular crontab entry. It&#8217;s rules are no different then what you&#8217;ve already learned from other fields.\n<\/p>\n<h2>Testing<\/h2>\n<p>\nSimply adding a <strong>&#8211;test<\/strong> (<strong>-t<\/strong>) switch to your dateblock entry will allow you to test the tool in a debugging mode to which it will present to you the current time followed by when it would have unblocked for had you not provided the <strong>&#8211;test<\/strong> (<strong>-t<\/strong>) switch. It&#8217;s a great way to calculate when the next processing time will be.\n<\/p>\n<h2>Python Extension<\/h2>\n<p>To handle scheduled processes for my websites, I created a python extension for dateblock. This allowed to extend it&#8217;s flexibility with other offline processing&#8230; consider the following example:<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\r\nfrom dateblock import dateblock\r\nwhile True:\r\n    # \/5 as first argument means unblock on the 5th second of each minute\r\n    dateblock('\/5')\r\n    print 'begin processing ...'\r\n    # your code here...\r\n    # if you want, you can even report the next unblock time\r\n    print 'Done processing; blocking until %s' % \r\n        dateblock('\/5', block=False).strftime('%Y-%m-%d %H:%M:%S')\r\n<\/pre>\n<p>You can also also access the drift as such:<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\r\nfrom dateblock import dateblock\r\nprint 'Unblock at %s' % \r\n    dateblock('\/5', drift=120, block=False).strftime('%Y-%m-%d %H:%M:%S')\r\n<\/pre>\n<p>Finally the python extension allows you to pass in a datetime object as an argument for calculating a time based on another (and not the current time which is the default).<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\r\nfrom dateblock import dateblock\r\nfrom datetime import datetime\r\nfrom datetime import timedelta\r\n\r\n# 31 days ago\r\nreftime = datetime.now() - timedelta(days=31)\r\n\r\nprint('Would blocking until %s' % \r\n    dateblock('\/5', drift=120, block=False, )\r\n      .strftime('%Y-%m-%d %H:%M:%S') + \r\n    &quot; if time was %s&quot; % reftime\r\n      .strftime('%Y-%m-%d %H:%M:%S'))\r\n<\/pre>\n<\/p>\n<h2>Things to Consider<\/h2>\n<p>\nJust like <em>sleep<\/em>, <em>dateblock<\/em> uses <em>SIGALARM<\/em> to manage its wake up time. Therefore if your code relies heavily on <em>SIGALARM<\/em> for another purposes, dateblock may not be a solution for you since you <em>could<\/em> interrupt it&#8217;s reliability (though not likely). This really shouldn&#8217;t be a big concern because this exact same warning comes with the <em>sleep<\/em> libraries we&#8217;ve been using for years. But it does mean that <em>sleep<\/em> could interfere with <em>dateblock<\/em> just as <em>dateblock<\/em> could interfere with <em>sleep<\/em> if they were both used in separate threads.<br \/>\n<figure id=\"attachment_1541\" aria-describedby=\"caption-attachment-1541\" style=\"width: 600px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/nuxref.com\/wp-content\/uploads\/2013\/10\/dateblock.executions.jpg\" alt=\"Dateblock vs Sleep\" width=\"600\" height=\"288\" class=\"size-full wp-image-1541\" srcset=\"http:\/\/nuxref.com\/wp-content\/uploads\/2013\/10\/dateblock.executions.jpg 600w, http:\/\/nuxref.com\/wp-content\/uploads\/2013\/10\/dateblock.executions-300x144.jpg 300w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><figcaption id=\"caption-attachment-1541\" class=\"wp-caption-text\">Dateblock vs Sleep<\/figcaption><\/figure>\n<\/p>\n<h2>Why would I use dateblock over sleep?<\/h2>\n<p>\nScheduling is the key&#8230; If your program relies completely on <em>sleep<\/em>, then the only thing you&#8217;re accomplishing is cpu throttling (controlling unnecessary thrashing). This is approach is fine if you&#8217;re going to just retry connecting to an unresponsive server in <em>??<\/em> seconds. But what if timing becomes an important factor of your application? The <em>dateblock<\/em> tool ensures you only unblock <em>at absolute times<\/em> vs <em>sleep<\/em> which unblocks at relative times with respect to when it was called.<\/p>\n<p>Dateblock also allows your program to chronologically work in turn with other applications that may be on their own processing cycle.  Such as something delivering data at the top of every hour.  You may wish to have your program wake up 5 min after the top of each hour to perform the processing regardless of when your program was started.\n<\/p>\n<h1>Datemath<\/h1>\n<p>\nThere isn&#8217;t as much to be said about Datemath; I personally never found a Linux\/Unix tool that would allow me to script date\/time calculations from the command line. For that reason, this tool exists.<br \/>\nHere is an example of the tools function:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# what time is it now?\r\ndate +'%Y-%m-%d %H-%M-%S'\r\n# The above output was '2013-10-26 09-42-21' at the time of this blog\r\n# what time will it be 5 months and 3 days from now\r\ndatemath --months=5 --days=3 --format='%Y-%m-%d %H-%M-%S'\r\n# the above output was '2014-03-29 09-42-21' at the time of this blog\r\n# and this makes sense... this is the calculation we want.\r\n<\/pre>\n<p>The tool supports negative values for calculating into the past as well and will handle leap years in the calculations too.<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# what time is it now?\r\ndate +'%Y-%m-%d %H-%M-%S'\r\n# The above output was '2013-10-26 09-45-45' at the time of this blog\r\n# What was the current date 753 days ago?\r\ndatemath --days=-753 --format='%Y-%m-%d %H-%M-%S'\r\n# the above output was '2011-10-04 09-45-45' at the time of this blog\r\n# and this makes sense... this is the calculation we want.\r\n<\/pre>\n<\/p>\n<h2>No Python Module For Datemath<\/h2>\n<p>\nThere is no python module for datemath because Python&#8217;s <em>datetime<\/em> and <em>timedelta<\/em> libraries already provide a fantastic solution for the same problem datemath solves&#8230;<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\r\n# Simple code example to show why it really isn't\r\n# necessary to port datemath to Python:\r\nfrom datetime import datetime\r\nfrom datetime import timedelta\r\nin_the_past = datetime.now() - timedelta(minutes=15)\r\nprint '15 minutes ago: %s' % in_the_past\r\n                              .strftime('%Y-%m-%d %H:%M:%S')\r\n<\/pre>\n<\/p>\n<h1>Just give me the goods<\/h1>\n<p>\nNo problem, below are the RPMS as well as their accompanied source packages:<\/p>\n<table>\n<tr bgcolor=\"#eee\">\n<th style=\"width:11.4em\">Package<\/th>\n<th>Download<\/th>\n<th>Description<\/th>\n<\/tr>\n<tr>\n<td style=\"font-weight: bold\">dateblock<\/td>\n<td><strong><br \/>\n          <a href=\"http:\/\/repo.nuxref.com\/centos\/6\/en\/x86_64\/custom\/repoview\/dateblock.html\" target=\"_blank\">el6.rpm<\/a><\/strong> \/<br \/>\n          <a href=\"http:\/\/repo.nuxref.com\/centos\/7\/en\/x86_64\/custom\/repoview\/dateblock.html\" target=\"_blank\">el7.rpm<\/a>\n      <\/td>\n<td>The powerful (cron like) command line interface (CLI) tool.<\/td>\n<\/tr>\n<tr>\n<td style=\"font-weight: bold\">python-dateblock<\/td>\n<td><strong><br \/>\n          <a href=\"http:\/\/repo.nuxref.com\/centos\/6\/en\/x86_64\/custom\/repoview\/python-dateblock.html\" target=\"_blank\">el6.rpm<\/a><\/strong> \/<br \/>\n          <a href=\"http:\/\/repo.nuxref.com\/centos\/7\/en\/x86_64\/custom\/repoview\/python-dateblock.html\" target=\"_blank\">el7.rpm<\/a>\n      <\/td>\n<td>The python extension for dateblock.<\/td>\n<\/tr>\n<tr>\n<td style=\"font-weight: bold\">datemath<\/td>\n<td><strong><br \/>\n          <a href=\"http:\/\/repo.nuxref.com\/centos\/6\/en\/x86_64\/custom\/repoview\/datemath.html\" target=\"_blank\">el6.rpm<\/a><\/strong> \/<br \/>\n          <a href=\"http:\/\/repo.nuxref.com\/centos\/7\/en\/x86_64\/custom\/repoview\/datemath.html\" target=\"_blank\">el7.rpm<\/a>\n      <\/td>\n<td>The datemath command line interface tool.<\/td>\n<\/tr>\n<tr>\n<td style=\"font-weight: bold\">datetools<\/td>\n<td><strong><br \/>\n          <a href=\"http:\/\/repo.nuxref.com\/centos\/6\/en\/x86_64\/custom\/repoview\/datetools.html\" target=\"_blank\">el6.rpm<\/a><\/strong> \/<br \/>\n          <a href=\"http:\/\/repo.nuxref.com\/centos\/7\/en\/x86_64\/custom\/repoview\/datetools.html\" target=\"_blank\">el7.rpm<\/a>\n      <\/td>\n<td>An optional package which includes licensing and information.<\/td>\n<\/tr>\n<\/table>\n<p><strong>Note<\/strong>: The source rpm can be obtained <a href=\"http:\/\/repo.nuxref.com\/centos\/7\/en\/x86_64\/source\/repoview\/datetools.html\" target=\"_blank\">here<\/a> which builds everything you see in the table above. It&#8217;s not required for the application to run, but might be useful for developers or those who want to inspect how I put the package together.<\/p>\n<h1>No way, I&#8217;m building this myself; I don&#8217;t trust you<\/h1>\n<p>That&#8217;s okay, I understand; here is how you can build it yourself:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# Install 'mock' into your environment if you don't have it already\r\n# This step will require you to be the superuser (root) in your native\r\n# environment.\r\nyum install -y mock\r\n\r\n# Grant your normal every day user account access to the mock group\r\n# This step will also require you to be the root user.\r\nusermod -a -G mock YourNonRootUsername\r\n<\/pre>\n<p>At this point it&#8217;s safe to change from the &#8216;<em>root<\/em>&#8216; user back to the user account you granted the<em> mock group<\/em> privileges to in the step above. We won&#8217;t need the <em>root<\/em> user again until the end of this tutorial when we install our built RPM.<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\n# Create an environment we can work in\r\nmkdir datetool-build\r\n\r\n# Change into our temporary working directory\r\ncd datetool-build\r\n\r\ncurl -L --output datetools-0.8.1.tar.gz \\\r\n   https:\/\/github.com\/caronc\/datetools\/archive\/v0.8.1.tar.gz\r\n\r\n# Extract Spec File\r\ntar xfz datetools-0.8.1.tar.gz \\\r\n   datetools-0.8.1\/datetools.spec \\\r\n      --strip-components=1\r\n\r\n# Initialize Mock Environment\r\nmock -v -r epel-6-x86_64 --init\r\n\r\n# Now install our dependencies\r\nmock -v -r epel-6-x86_64 --install boost-devel libstdc++-devel \r\n          glib-devel python-devel autoconf automake libtool\r\n\r\n# Copy in our downloaded content:\r\nmock -v -r epel-6-x86_64 --copyin datetools-0.8.1.tar.gz\r\n   \/builddir\/build\/SOURCES\r\nmock -v -r epel-6-x86_64 --copyin datetools.spec \r\n   \/builddir\/build\/SPECS\r\n\r\n# Shell into our environment\r\nmock -v -r epel-6-x86_64 --shell\r\n\r\n# Change to our build directory\r\ncd builddir\/build\r\n\r\n# Build our RPMS\r\nrpmbuild -ba SPECS\/datetools.spec\r\n\r\n# we're now done with our mock environment for now; Press Ctrl-D to exit\r\n# or simply type exit on the command line of our virtual environment\r\nexit\r\n<\/pre>\n<\/p>\n<h1>Future Considerations<\/h1>\n<p>\nThis is totally up in the air, at the moment the tool does everything I needed at the time. However I could see the following becoming a useful feature in the future:<\/p>\n<ul>\n<li>Pass in a different time into both programs (instead of always working with the current time) (You can already do this with the dateblock python extension).<\/li>\n<li>Have dateblock additionally take in a program and arguments as input to have it automatically execute the call to it when the scheduled time is reached. In addition to this, it means the dateblock tool would daemonize itself and run in the background on reoccurring schedules.<\/li>\n<li>Add a devel package and create a shared library for C++ linking; perhaps the binary tools and extensions could link here too.  Right now the library is just so small it&#8217;s really nothing to just include it statically as it is now.<\/li>\n<li>Got an idea of your own? Pass it along! You can also submit a pull request to me on GitHub <a title=\"GitHub Datetool Source\" href=\"https:\/\/github.com\/caronc\/datetools\" target=\"_blank\">here<\/a>.<\/li>\n<\/ul>\n<h1>Credit<\/h1>\n<p>\nPlease note that this information took me several days to put together and test thoroughly. I may not blog often; but I want to re-assure the stability and testing I put into everything I intend share.<\/p>\n<p>If you like what you see and wish to copy and paste this information, please reference back to this blog post at the very least. It\u2019s really all I ask.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction to Datetools I wrote Datetools (in C++) to allow the manipulation of date time from the command line. It greatly simplified my life and maybe it will help yours out too!. It comprises of two core applications: Dateblock: allows you to block until a scheduled period of time arrives unlike sleep which blocks for &hellip; <a href=\"http:\/\/nuxref.com\/2013\/10\/26\/datetools-date-manipulation-linux\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Datetools: Date Manipulation and Cron Alternative For Linux<\/span> <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":1549,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[26,31,32,34,37,38,39,54,135,136,143,150,158],"class_list":["post-176","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-linux-tools","tag-centos","tag-cluster","tag-clustering","tag-cron","tag-dateblock","tag-datemath","tag-datetools","tag-epel","tag-python","tag-python-extension","tag-rhel","tag-rpm","tag-sleep"],"_links":{"self":[{"href":"http:\/\/nuxref.com\/wp-json\/wp\/v2\/posts\/176","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/nuxref.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/nuxref.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/nuxref.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/nuxref.com\/wp-json\/wp\/v2\/comments?post=176"}],"version-history":[{"count":20,"href":"http:\/\/nuxref.com\/wp-json\/wp\/v2\/posts\/176\/revisions"}],"predecessor-version":[{"id":1663,"href":"http:\/\/nuxref.com\/wp-json\/wp\/v2\/posts\/176\/revisions\/1663"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/nuxref.com\/wp-json\/wp\/v2\/media\/1549"}],"wp:attachment":[{"href":"http:\/\/nuxref.com\/wp-json\/wp\/v2\/media?parent=176"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/nuxref.com\/wp-json\/wp\/v2\/categories?post=176"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/nuxref.com\/wp-json\/wp\/v2\/tags?post=176"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}