This was a problem that took more time than I wanted to solve. Hopefully posting this will help somebody find the answer faster than I could.

The launchd.plist documentation is pretty clear that StartCalendarInterval is way to approximate a crontab-like entry. The documentation is a little light on examples.

I wanted to do something like…

0,30 */4 * * * /bin/blah

All of the examples show single values for each of the five possible time arguments. The good news is that it’s possible to nominate more than one time. The bad news is that you can’t combine them into the same concise shorthand that cron used. Instead you define an array of dictionaries…

<key>StartCalendarInterval</key>
<array>
  <dict>
    <key>Hour</key>
    <integer>0</integer>
    <key>Minute</key>
    <integer>0</integer>
  </dict>
  <dict>
    <key>Hour</key>
    <integer>4</integer>
    <key>Minute</key>
    <integer>0</integer>
  </dict>
</pre>

After I got this answer from the Apple development forum, I also found this on Stack Overflow: http://stackoverflow.com/questions/2079130/run-a-cron-job-every-x-minutes-with-startcalendarinterval-launchd

I am no longer supporting or maintaining iTunes Movie Artwork.

It has been replaced by Movie Covers which is available on the Mac App Store.

Thankyou to everybody for their support and words of encouragement.

Want to Remove iTunes Movie Artwork?

The installer really only copies one applescript into the following location…

:Users:<username>:Library:iTunes:Scripts:iTunes Movie Artwork.scpt

By deleting that file you will remove it from the iTunes script menu. You may need to give your admin password because the user probably doesn’t have write permission to the file.

I do everything in VIM. I expect that quite a few of my posts will dwell on my love affair with that app. Anyway, sometimes I want to run a block of text (SQL) against my database, and have the results returned to the text file that i’m working on.

For this I knocked up three shell scripts; dbe, ebd and rr.

dbe and ebd are mirrors of one another. The dbe script will take something like this from stdin

select count(*) as c
from customer
where name like '%blah%';

…and will output this to stdout
/* BEGIN QUERY ================================================== */
select count(*) as c
from customer
where name like '%blah%';
/* BEGIN RESULT ------------------------------------------------- */
+-----+
| c   |
+-----+
| 862 | 
+-----+
/* END RESULT (0) =============================================== */

The reason that there are those comments in the results (with ‘BEGIN QUERY’ and ‘END RESULT’ in them) is so that ebd has a change to take all of that in from stdin and give the original back to you on stdout.
Where rr comes in is that you can pass that block of output (with the ‘BEGIN QUERY’ etc. in it) into rr along with the query command that you want to re-run. If you were to pass this…
/* BEGIN QUERY ================================================== */
select count(*) as c
from customer
where name like '%blah%';
/* BEGIN RESULT ------------------------------------------------- */
+-----+
| c   |
+-----+
| 862 | 
+-----+
/* END RESULT (0) =============================================== */

…into rr dbe as stdin, you’d get same output (unless you had changed the query or the data had changed in the meantime). If you wanted to rerun that query after adjusting the query, you could easily pass the whole block (query and result) into rr, and get roughly the same thing back, only with an updated answer…
/* BEGIN QUERY ================================================== */
select count(*) as c
from customer
where name like '%blah blah%';
/* BEGIN RESULT ------------------------------------------------- */
+-----+
| c   |
+-----+
| 101 | 
+-----+
/* END RESULT (0) =============================================== */

Right, if you’re still reading you must love VIM as much as me! Those three commands aren’t very useful by themselves. They become very useful in VIM when you use the ! operator. In VIM you can nominate a block of text and hand it out to a command and have that block of text replaced with the output.
Before you do any of this, you will want to issue the :set number command to show line numbers in your file. Then, in a file like this…
1 Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor
2 incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
3 exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute
4 irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla
5 pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui
6 officia deserunt mollit anim id est laborum.

…you can issue :2,4!wc
1 Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor
2       3      39     246
3 pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui
4 officia deserunt mollit anim id est laborum.

The 2,4 part nominated the three lines from the 2nd to the 4th. The ! (bang) operator is like a unix | (pipe) operator. In this example we’re passing those three lines to the wc (word-count) command. The answer is that those three lines contained three lines, thirty-nine words and 246 characters.
So, if you had dbe, ebd and rr in your PATH, you could take…
1
2 select count(*) as c
3 from customer
4 where name like '%blah%';
5

…and issue 2,4!dbe, and get…
 1
 2 /* BEGIN QUERY ================================================== */
 3 select count(*) as c
 4 from customer
 5 where name like '%blah%';
 6 /* BEGIN RESULT ------------------------------------------------- */
 7 +-----+
 8 | c   |
 9 +-----+
10 | 862 | 
11 +-----+
12 /* END RESULT (0) =============================================== */
13

You could then edit line 5 to look like this…
 5 where name like '%blah blah%';

…and then you can issue 2,12!rr dbe to see the new results…
 1
 2 /* BEGIN QUERY ================================================== */
 3 select count(*) as c
 4 from customer
 5 where name like '%blah blah%';
 6 /* BEGIN RESULT ------------------------------------------------- */
 7 +-----+
 8 | c   |
 9 +-----+
10 | 101 | 
11 +-----+
12 /* END RESULT (0) =============================================== */
13

In my environment I have named my script after the database that I want to query. I made it short and also made it easy to bash out with one hand so that I can be quick on my keyboard. Making ebd the reverse of dbe is important because it is how rr works out how to strip all of the markup out before re-querying.

I need one for Microsoft SQL Server soon, so I think i’ll have to write the equivalent of dbe and ebd in PHP.

dbe

#!/bin/bash

# Copyright (c) 2013, James Downie 
# 
# Permission to use, copy, modify, and/or distribute this
# software for any purpose with or without fee is hereby
# granted, provided that the above copyright notice and this
# permission notice appear in all copies.
# 
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
# THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

S="$1"
OPTS="-t"
if [ "$S" == "-v" ]; then
  OPTS="$OPTS -E"
fi

T="`mktemp --tmpdir=/tmp/ dbe.XXXXX.sql`"
cat - > "$T"
echo "/* BEGIN QUERY ================================================== */"
cat "$T"
echo "/* BEGIN RESULT ------------------------------------------------- */"
T0="`date +%s`"
mysql $OPTS -u user -psecret -e "source $T;" mydb 2>&1
T1="`date +%s`"
let TD=T1-T0;
echo "/* END RESULT ($TD) =============================================== */"
rm "$T"

ebd

#!/bin/bash

# Copyright (c) 2013, James Downie 
# 
# Permission to use, copy, modify, and/or distribute this
# software for any purpose with or without fee is hereby
# granted, provided that the above copyright notice and this
# permission notice appear in all copies.
# 
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
# THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

T="`mktemp --tmpdir=/tmp/ ebd.XXXXX`"
cat - > "$T"
BQ="`cat "$T" | nl -b a | grep "\/\* BEGIN QUERY " | head -n 1 | cut -f 1 | tr -d ' '`"
BR="`cat "$T" | nl -b a | grep "\/\* BEGIN RESULT " | head -n 1 | cut -f 1 | tr -d ' '`"
ER="`cat "$T" | nl -b a | grep "\/\* END RESULT " | head -n 1 | cut -f 1 | tr -d ' '`"
let h=BR-1
let t=BR-BQ-1
head -n $h "$T" | tail -n $t
rm "$T"

rr

#!/bin/bash

# Copyright (c) 2013, James Downie 
# 
# Permission to use, copy, modify, and/or distribute this
# software for any purpose with or without fee is hereby
# granted, provided that the above copyright notice and this
# permission notice appear in all copies.
# 
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
# THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

T="`mktemp --tmpdir=/tmp/ rr.XXXXX`"
cat - > "$T"
if [ "$1" != "" ]; then
  A="$1"
  B="`echo "$1" | rev`"
  cat "$T" | $B | $A
fi
rm "$T"

If you’ve ever had to write scripts that process large human-maintained filesystems, you’ll know what a pain “special characters” can be. It only takes one lousy single quote in a filename somewhere deep in a directory structure for your nightly jobs to start failing.

Fortunately, I was able to brute-force this problem in one particular environment by removing any undesirable characters. The following script is obviously pretty hacky but it got me a good result.

When it finds a directory entry that does not comply it tries to rename that entry to the compliant form. If the compliant form already exists it prefixes it with a timestamp so as to make it unique.

conv() contains the logic for converting a messy filename to a neat and tidy one, so if you want to be less tolerant than me with the characters that you permit in your filenames, that’s the place to make your change.

This is not my prettiest bit of code, but it did get me a good result… and that’s what we’re all about!

#!/usr/bin/python

# Copyright (c) 2013, James Downie <jdownie@gmail.com>
# 
# Permission to use, copy, modify, and/or distribute this
# software for any purpose with or without fee is hereby
# granted, provided that the above copyright notice and this
# permission notice appear in all copies.
# 
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
# THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

import os
import time
import shutil
import unicodedata
import re

def conv(str):
  str = list(str)
  ret = list()
  for c in str:
    if ord(c) < 128:
      ret.append(c)
    else:
      ret.append("?")
  ret = "".join(ret)
  ret = re.sub('\'', '?', ret)
  ret = re.sub('\`', '?', ret)
  ret = re.sub('\!', '?', ret)
  ret = re.sub('\:', '?', ret)
  ret = re.sub('\"', '?', ret)
  ret = re.sub('\%', '?', ret)
  ret = re.sub('\t', ' ', ret)
  ret = re.sub('\&', 'and', ret)
  ret = re.sub('#', 'and', ret)
  ret = ret.replace("\\", "")
  return ret

def walkDir(path):
  if os.path.isdir(path):
    for dirpath, dirnames, filenames in os.walk(path):
      for filename in filenames:
        walkDir("/".join([ dirpath, filename ]))
      for dirname in dirnames:
        converted = conv(dirname)
        src = "/".join([dirpath, dirname])
        dst = "/".join([dirpath, converted])
        e = dst
        if dirname != converted:
          if os.path.exists(dst):
            print "Collission:", dst
            converted = "-".join([ time.strftime("%Y%m%d%H%M%S"), converted ])
            dst = "/".join([dirpath, converted])
            shutil.move(src, dst)
            walkDir(dst)
          else:
            shutil.move(src, dst)
            walkDir(dst)
  else:
    elements = path.split("/")
    entry = elements.pop()
    converted = conv(entry)
    elements.append(converted)
    converted = "/".join(elements)
    if converted != path:
      if os.path.exists(converted):
        elements = converted.split("/")
        entry = elements.pop()
        converted = "-".join([ time.strftime("%Y%m%d%H%M%S"), entry ])
        elements.append(converted)
        converted = "/".join(elements)
        shutil.move(path, converted)
      else:
        shutil.move(path, converted)

walkDir(u"/Volumes/bigMess")

This post describes how I use Siri to create RTM task that have specific properties set (instead of being simple titled in “Inbox”). This is useful to note down tasks while you drive.
Bear with me on this one. This solution pretty complicated, but it was worth it for me so that I didn’t have a heap of tagging and date setting stuff left to do when I got back to a computer.

For the impatient, here’s what the end solution looks like…

  1. Tell Siri to send an email to “Groceries”. The subject of the email is the task description. The body can be “whatever”
  2. Siri has “Groceries” in your address book as “groceries@mydomain.com
  3. groceries@mydomain.com forwards to me@gmail.com
  4. me@gmail.com applies a filter that skips the inbox, marks it as read and puts the message in a Groceries folder
  5. IFTTT sees this new message in the Groceries folder and fires an email off to RTM with some extra commands that we’d want automatically applied to grocery tasks.

Let’s start at the finish line. This solution starts with Siri‘s email functionality. It is easy to send an email to one of your contacts, but the only functionality that RTM offers is a contact that will deliver emails as new tasks into your RTM Inbox. If you were typing the email manually, you could add “#Home #groceries ^Friday” to the subject line to better tailor the new task to your workflow. With Siri and RTM alone you are left to rework the task when you’re back at your desk.

This is where I added a few missing ingredients. You might be able to think of a simpler version, but i’ve been using this one and it has been working well. The missing ingredients were…

  • An email address that is used only for these tasks
  • IFTTT

For this explanation, let’s imagine that we want to be able to email “Groceries”, and have those tasks land in RTM with the following properties…

  • List: Home
  • Tag: groceries
  • Due: Friday

To accomplish this, we just need to add “#Home #groceries ^Friday” to the end of the subject before sending the email into RTM. IFTTT can do this very nicely. I happened to have a domain that permitted any number of email addresses, so I registered an email account at that domain for a user called “groceries” which forwarded to my GMail account. In hind-sight, I could have just had IFTTT monitor that email account directly with POP3 or IMAP or something. Anyway I didn’t.

Those emails, emailed to groceries@mydomain.com and forwarded to my GMail account, were marked in a way that would make it easy for IFTTT to single out.


Screen Shot 2013-06-19 at 3.59.56 PM

This keeps them out of my Inbox, they don’t appear as unread and they are all in a neat folder that I can check on later if I want to debug the process. Then, in IFTTT I set up the following recipe…


Screen Shot 2013-06-19 at 3.54.24 PM

…which means “If something happens in GMail, then do something in GMail“. The rule watches for new emails in the “RTM/Groceries” folder.


Screen Shot 2013-06-19 at 3.54.40 PM

…and is then initiates a new email, copying the subject and body (in plain text). The subject adds our special RTM commands.


IFTTT Groceries Recipe - THAT

The only thing left to do is to put that grocery@mydomain.com email address into my contacts as somebody called “Groceries” so that Siri knows who i’m talking about.

And there you have it. With this method, you can create special email addresses for each “profile” that you’d like to easily command tasks into using Siri.

 

I needed to move 7Tb of files from one host to another using a 4Tb external HDD drive. I want to preserve the file modification times and the directory structures so that I can reverse the procedure at the other end once the 4Tb HDD has shuttled the files to the other end. At the source end, source will split into destination/1, destination/2, destination/3. At the other end I want to be able to….

$ rsync -ra destination/1 reconstituted/
$ rsync -ra destination/2 reconstituted/
$ rsync -ra destination/3 reconstituted/

…so that source matches reconstituted. Incidentally, because my method only handles files, reconstituted will differ from source in that it will omit empty directories.
I started with the 7Tb of files in a directory called source. When i’m finished I want to see those files split into sub-directories within the destination directory. Those sub-directories cannot exceed the limit that I send. The size of those sub-directories is dependant on the mix of files that get put into each sub-directory.

./
./source/
./destination/

First I removed all of the special characters using the script I published here.

I then used a little shell script to generate an index of the files, the sizes and when they were last modified…

#!/bin/bash

# Copyright (c) 2013, James Downie 
# 
# Permission to use, copy, modify, and/or distribute this
# software for any purpose with or without fee is hereby
# granted, provided that the above copyright notice and this
# permission notice appear in all copies.
# 
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL 
# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
# THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
if [ -d "1" ]; then
  /usr/bin/find "$1"  -type f  -exec /usr/bin/stat -f "%z%t%m%t%N" "{}" \;
fi

So with that script index.sh in the same directory as source and destination I launch it like so…

$ ./index.sh source/ > index.txt

My working directory now looks like this…

./
./source/
./destination/
./index.sh
./index.txt

…and index.txt looks something like this…

24580 1366159671  fbhepr/ovtinhyg$/Pheerag/.QF_Fgber
21508 1339654491  fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/.QF_Fgber
15364 1335935650  fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/NOP Sbezngf/.QF_Fgber
47104 1275531397  fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/NOP Sbezngf/Oevrs/Nqqvgvbaf gb oevrs.qbp
1082834 1275628987 fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/NOP Sbezngf/Oevrs/NOP Oevrs.nv
1090494 1275540447 fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/NOP Sbezngf/Oevrs/Arj NOP nqqerff.nv
9971 1272846247 fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/NOP Sbezngf/Oevrs/Bhgre Wbo Thvqryvarf.kyfk
6148 1274414392 fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/NOP Sbezngf/THVQR SBE VZNTR CYNPRZRAG/.QF_Fgber
3176033 1273124458 fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/NOP Sbezngf/THVQR SBE VZNTR CYNPRZRAG/30883 NOP EVPUYNAQF Pnaq17RN7O.NV
1752558 1259881816 fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/NOP Sbezngf/THVQR SBE VZNTR CYNPRZRAG/NOP genl DYQ nccebirq 4 Qrp.cqs

The next step is to work through that list marking each file as part of a destination volume set. If the total volume of a destination volume set is reached, begin a new empty volume set. For this I used awk. This is split.awk

# Copyright (c) 2013, James Downie 
# 
# Permission to use, copy, modify, and/or distribute this
# software for any purpose with or without fee is hereby
# granted, provided that the above copyright notice and this
# permission notice appear in all copies.
# 
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL 
# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
# THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
BEGIN {
  FS="\t"
  OFS="\t"
  batch = 1;
  batch_size = 0;
  limit_m = 3500000
  limit = limit_m * 1024 * 1024
}
{
  size = $1;
  mtime = $2;
  path = $3;
  if (batch_size + size > limit) {
    batch_size = 0;
    batch++;
  }
  batch_size += size;
  printf "# %d %s\n", batch, path
  printf "./migrate_one.sh %d %d \"%s\"\n", batch, mtime, path
}
END {
}

Although my HDD is 4Tb, I left a bit of extra space ant set limit to 3.5Tb. For each line in index.txt, split.awk will output two lines; a comment indicating the batch number and the source file’s path, and a command that will run a script called migrate_one.sh on the file. We’ll run awk over index.txt and direct the output into migrate_all.sh

$ awk -f split.awk index.txt > migrate_all.sh

That runs pretty quickly, considering that index.txt contains 6.7 million lines. Before I run migrate_all.sh I should show you two other scripts. The first is migrate_one.sh

#!/bin/bash

# Copyright (c) 2013, James Downie 
# 
# Permission to use, copy, modify, and/or distribute this
# software for any purpose with or without fee is hereby
# granted, provided that the above copyright notice and this
# permission notice appear in all copies.
# 
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL 
# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
# THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

BATCH="$1"
MTIME="$2"
SOURCE="$3"

D="`dirname "$SOURCE"`"
TD="destination/$BATCH/$D"
T="destination/$BATCH/$SOURCE"
mkdir -v -p "$TD"
mv -v "$SOURCE" "$T";
./setmtime.py $MTIME "$T";

That script works out the deep pathname that we expect to move the file into, and then relies on mkdir‘s -p switch to make all of the parent directories leading to the deepest directory entry. Then a simple mv will relocate the volume of the file from the source tree into the destination tree.
The setmtime.py script is a little hack that sets a file’s mtime to the time nominated with the provided unix timestamp. We have all of the file’s unix timestamps in index.txt because stat determined them for us back when index.sh ran find across source. migrate_one.sh runs setmtime.py against the file once it as been moved to ensure that the file survived the move without losing it’s original “last modified time”.

#!/usr/bin/python

# Copyright (c) 2013, James Downie 
# 
# Permission to use, copy, modify, and/or distribute this
# software for any purpose with or without fee is hereby
# granted, provided that the above copyright notice and this
# permission notice appear in all copies.
# 
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL 
# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
# THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

import os, sys

mtime = int(sys.argv[1])
path = sys.argv[2]

if os.path.exists(path):
  os.utime(path, (mtime, mtime))
else:
  print "Where is '%s'" % path

My directory looked like this…

./
./source/
./destination/
./index.sh
./index.txt
./split.awk
./migrate_all.sh
./migrate_one.sh
./setmtime.py

…when I then ran…

$ sh migrate_all.sh

That took two and a half days to run. Well, actually it took weeks to get right because my script kept breaking as I discovered special characters in source. I made filenameCleanse.py aggressive enough to remove any characters that jeopardised any of these steps and finally it ran without error. The end result looks like this…

$ du -d 1 -h destination/
3.3T    destination/1
3.3T    destination/2
798G    destination/3
7.5T    destination