Scripts

I am no longer supporting or maintaining iTunes Movie Artwork.

It has been replaced by Movie Covers which is available on the Mac App Store.

Thankyou to everybody for their support and words of encouragement.

Want to Remove iTunes Movie Artwork?

The installer really only copies one applescript into the following location…

:Users:<username>:Library:iTunes:Scripts:iTunes Movie Artwork.scpt

By deleting that file you will remove it from the iTunes script menu. You may need to give your admin password because the user probably doesn’t have write permission to the file.

I do everything in VIM. I expect that quite a few of my posts will dwell on my love affair with that app. Anyway, sometimes I want to run a block of text (SQL) against my database, and have the results returned to the text file that i’m working on.

For this I knocked up three shell scripts; dbe, ebd and rr.

dbe and ebd are mirrors of one another. The dbe script will take something like this from stdin

select count(*) as c
from customer
where name like '%blah%';

…and will output this to stdout
/* BEGIN QUERY ================================================== */
select count(*) as c
from customer
where name like '%blah%';
/* BEGIN RESULT ------------------------------------------------- */
+-----+
| c   |
+-----+
| 862 |
+-----+
/* END RESULT (0) =============================================== */

The reason that there are those comments in the results (with ‘BEGIN QUERY’ and ‘END RESULT’ in them) is so that ebd has a change to take all of that in from stdin and give the original back to you on stdout.
Where rr comes in is that you can pass that block of output (with the ‘BEGIN QUERY’ etc. in it) into rr along with the query command that you want to re-run. If you were to pass this…
/* BEGIN QUERY ================================================== */
select count(*) as c
from customer
where name like '%blah%';
/* BEGIN RESULT ------------------------------------------------- */
+-----+
| c   |
+-----+
| 862 |
+-----+
/* END RESULT (0) =============================================== */

…into rr dbe as stdin, you’d get same output (unless you had changed the query or the data had changed in the meantime). If you wanted to rerun that query after adjusting the query, you could easily pass the whole block (query and result) into rr, and get roughly the same thing back, only with an updated answer…
/* BEGIN QUERY ================================================== */
select count(*) as c
from customer
where name like '%blah blah%';
/* BEGIN RESULT ------------------------------------------------- */
+-----+
| c   |
+-----+
| 101 |
+-----+
/* END RESULT (0) =============================================== */

Right, if you’re still reading you must love VIM as much as me! Those three commands aren’t very useful by themselves. They become very useful in VIM when you use the ! operator. In VIM you can nominate a block of text and hand it out to a command and have that block of text replaced with the output.
Before you do any of this, you will want to issue the :set number command to show line numbers in your file. Then, in a file like this…
1 Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor
2 incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
3 exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute
4 irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla
5 pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui
6 officia deserunt mollit anim id est laborum.

…you can issue :2,4!wc
1 Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor
2       3      39     246
3 pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui
4 officia deserunt mollit anim id est laborum.

The 2,4 part nominated the three lines from the 2nd to the 4th. The ! (bang) operator is like a unix | (pipe) operator. In this example we’re passing those three lines to the wc (word-count) command. The answer is that those three lines contained three lines, thirty-nine words and 246 characters.
So, if you had dbe, ebd and rr in your PATH, you could take…
1
2 select count(*) as c
3 from customer
4 where name like '%blah%';
5

…and issue 2,4!dbe, and get…
 1
 2 /* BEGIN QUERY ================================================== */
 3 select count(*) as c
 4 from customer
 5 where name like '%blah%';
 6 /* BEGIN RESULT ------------------------------------------------- */
 7 +-----+
 8 | c   |
 9 +-----+
10 | 862 |
11 +-----+
12 /* END RESULT (0) =============================================== */
13

You could then edit line 5 to look like this…
 5 where name like '%blah blah%';

…and then you can issue 2,12!rr dbe to see the new results…
 1
 2 /* BEGIN QUERY ================================================== */
 3 select count(*) as c
 4 from customer
 5 where name like '%blah blah%';
 6 /* BEGIN RESULT ------------------------------------------------- */
 7 +-----+
 8 | c   |
 9 +-----+
10 | 101 |
11 +-----+
12 /* END RESULT (0) =============================================== */
13

In my environment I have named my script after the database that I want to query. I made it short and also made it easy to bash out with one hand so that I can be quick on my keyboard. Making ebd the reverse of dbe is important because it is how rr works out how to strip all of the markup out before re-querying.

I need one for Microsoft SQL Server soon, so I think i’ll have to write the equivalent of dbe and ebd in PHP.

dbe

#!/bin/bash

# Copyright (c) 2013, James Downie 
# 
# Permission to use, copy, modify, and/or distribute this
# software for any purpose with or without fee is hereby
# granted, provided that the above copyright notice and this
# permission notice appear in all copies.
# 
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
# THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

S="$1"
OPTS="-t"
if [ "$S" == "-v" ]; then
  OPTS="$OPTS -E"
fi

T="`mktemp --tmpdir=/tmp/ dbe.XXXXX.sql`"
cat - > "$T"
echo "/* BEGIN QUERY ================================================== */"
cat "$T"
echo "/* BEGIN RESULT ------------------------------------------------- */"
T0="`date +%s`"
mysql $OPTS -u user -psecret -e "source $T;" mydb 2>&1
T1="`date +%s`"
let TD=T1-T0;
echo "/* END RESULT ($TD) =============================================== */"
rm "$T"

ebd

#!/bin/bash

# Copyright (c) 2013, James Downie 
# 
# Permission to use, copy, modify, and/or distribute this
# software for any purpose with or without fee is hereby
# granted, provided that the above copyright notice and this
# permission notice appear in all copies.
# 
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
# THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

T="`mktemp --tmpdir=/tmp/ ebd.XXXXX`"
cat - > "$T"
BQ="`cat "$T" | nl -b a | grep "\/\* BEGIN QUERY " | head -n 1 | cut -f 1 | tr -d ' '`"
BR="`cat "$T" | nl -b a | grep "\/\* BEGIN RESULT " | head -n 1 | cut -f 1 | tr -d ' '`"
ER="`cat "$T" | nl -b a | grep "\/\* END RESULT " | head -n 1 | cut -f 1 | tr -d ' '`"
let h=BR-1
let t=BR-BQ-1
head -n $h "$T" | tail -n $t
rm "$T"

rr

#!/bin/bash

# Copyright (c) 2013, James Downie 
# 
# Permission to use, copy, modify, and/or distribute this
# software for any purpose with or without fee is hereby
# granted, provided that the above copyright notice and this
# permission notice appear in all copies.
# 
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
# THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

T="`mktemp --tmpdir=/tmp/ rr.XXXXX`"
cat - > "$T"
if [ "$1" != "" ]; then
  A="$1"
  B="`echo "$1" | rev`"
  cat "$T" | $B | $A
fi
rm "$T"

If you’ve ever had to write scripts that process large human-maintained filesystems, you’ll know what a pain “special characters” can be. It only takes one lousy single quote in a filename somewhere deep in a directory structure for your nightly jobs to start failing.

Fortunately, I was able to brute-force this problem in one particular environment by removing any undesirable characters. The following script is obviously pretty hacky but it got me a good result.

When it finds a directory entry that does not comply it tries to rename that entry to the compliant form. If the compliant form already exists it prefixes it with a timestamp so as to make it unique.

conv() contains the logic for converting a messy filename to a neat and tidy one, so if you want to be less tolerant than me with the characters that you permit in your filenames, that’s the place to make your change.

This is not my prettiest bit of code, but it did get me a good result… and that’s what we’re all about!


#!/usr/bin/python

# Copyright (c) 2013, James Downie
#
# Permission to use, copy, modify, and/or distribute this
# software for any purpose with or without fee is hereby
# granted, provided that the above copyright notice and this
# permission notice appear in all copies.
#
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
# THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

import os
import time
import shutil
import unicodedata
import re

def conv(str):
str = list(str)
ret = list()
for c in str:
if ord(c) < 128: ret.append(c) else: ret.append("?") ret = "".join(ret) ret = re.sub('\'', '?', ret) ret = re.sub('\`', '?', ret) ret = re.sub('\!', '?', ret) ret = re.sub('\:', '?', ret) ret = re.sub('\"', '?', ret) ret = re.sub('\%', '?', ret) ret = re.sub('\t', ' ', ret) ret = re.sub('\&', 'and', ret) ret = re.sub('#', 'and', ret) ret = ret.replace("\\", "") return ret def walkDir(path): if os.path.isdir(path): for dirpath, dirnames, filenames in os.walk(path): for filename in filenames: walkDir("/".join([ dirpath, filename ])) for dirname in dirnames: converted = conv(dirname) src = "/".join([dirpath, dirname]) dst = "/".join([dirpath, converted]) e = dst if dirname != converted: if os.path.exists(dst): print "Collission:", dst converted = "-".join([ time.strftime("%Y%m%d%H%M%S"), converted ]) dst = "/".join([dirpath, converted]) shutil.move(src, dst) walkDir(dst) else: shutil.move(src, dst) walkDir(dst) else: elements = path.split("/") entry = elements.pop() converted = conv(entry) elements.append(converted) converted = "/".join(elements) if converted != path: if os.path.exists(converted): elements = converted.split("/") entry = elements.pop() converted = "-".join([ time.strftime("%Y%m%d%H%M%S"), entry ]) elements.append(converted) converted = "/".join(elements) shutil.move(path, converted) else: shutil.move(path, converted) walkDir(u"/Volumes/bigMess") [/python]

I needed to move 7Tb of files from one host to another using a 4Tb external HDD drive. I want to preserve the file modification times and the directory structures so that I can reverse the procedure at the other end once the 4Tb HDD has shuttled the files to the other end. At the source end, source will split into destination/1, destination/2, destination/3. At the other end I want to be able to….

$ rsync -ra destination/1 reconstituted/
$ rsync -ra destination/2 reconstituted/
$ rsync -ra destination/3 reconstituted/

…so that source matches reconstituted. Incidentally, because my method only handles files, reconstituted will differ from source in that it will omit empty directories.
I started with the 7Tb of files in a directory called source. When i’m finished I want to see those files split into sub-directories within the destination directory. Those sub-directories cannot exceed the limit that I send. The size of those sub-directories is dependant on the mix of files that get put into each sub-directory.

./
./source/
./destination/

First I removed all of the special characters using the script I published here.

I then used a little shell script to generate an index of the files, the sizes and when they were last modified…

#!/bin/bash

# Copyright (c) 2013, James Downie 
# 
# Permission to use, copy, modify, and/or distribute this
# software for any purpose with or without fee is hereby
# granted, provided that the above copyright notice and this
# permission notice appear in all copies.
# 
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL 
# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
# THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
if [ -d "1" ]; then
  /usr/bin/find "$1"  -type f  -exec /usr/bin/stat -f "%z%t%m%t%N" "{}" \;
fi

So with that script index.sh in the same directory as source and destination I launch it like so…

$ ./index.sh source/ > index.txt

My working directory now looks like this…

./
./source/
./destination/
./index.sh
./index.txt

…and index.txt looks something like this…

24580 1366159671  fbhepr/ovtinhyg$/Pheerag/.QF_Fgber
21508 1339654491  fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/.QF_Fgber
15364 1335935650  fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/NOP Sbezngf/.QF_Fgber
47104 1275531397  fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/NOP Sbezngf/Oevrs/Nqqvgvbaf gb oevrs.qbp
1082834 1275628987 fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/NOP Sbezngf/Oevrs/NOP Oevrs.nv
1090494 1275540447 fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/NOP Sbezngf/Oevrs/Arj NOP nqqerff.nv
9971 1272846247 fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/NOP Sbezngf/Oevrs/Bhgre Wbo Thvqryvarf.kyfk
6148 1274414392 fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/NOP Sbezngf/THVQR SBE VZNTR CYNPRZRAG/.QF_Fgber
3176033 1273124458 fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/NOP Sbezngf/THVQR SBE VZNTR CYNPRZRAG/30883 NOP EVPUYNAQF Pnaq17RN7O.NV
1752558 1259881816 fbhepr/ovtinhyg$/Pheerag/NOP Cebwrpg/NOP Sbezngf/THVQR SBE VZNTR CYNPRZRAG/NOP genl DYQ nccebirq 4 Qrp.cqs

The next step is to work through that list marking each file as part of a destination volume set. If the total volume of a destination volume set is reached, begin a new empty volume set. For this I used awk. This is split.awk

# Copyright (c) 2013, James Downie 
# 
# Permission to use, copy, modify, and/or distribute this
# software for any purpose with or without fee is hereby
# granted, provided that the above copyright notice and this
# permission notice appear in all copies.
# 
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL 
# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
# THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
BEGIN {
  FS="\t"
  OFS="\t"
  batch = 1;
  batch_size = 0;
  limit_m = 3500000
  limit = limit_m * 1024 * 1024
}
{
  size = $1;
  mtime = $2;
  path = $3;
  if (batch_size + size > limit) {
    batch_size = 0;
    batch++;
  }
  batch_size += size;
  printf "# %d %s\n", batch, path
  printf "./migrate_one.sh %d %d \"%s\"\n", batch, mtime, path
}
END {
}

Although my HDD is 4Tb, I left a bit of extra space ant set limit to 3.5Tb. For each line in index.txt, split.awk will output two lines; a comment indicating the batch number and the source file’s path, and a command that will run a script called migrate_one.sh on the file. We’ll run awk over index.txt and direct the output into migrate_all.sh

$ awk -f split.awk index.txt > migrate_all.sh

That runs pretty quickly, considering that index.txt contains 6.7 million lines. Before I run migrate_all.sh I should show you two other scripts. The first is migrate_one.sh

#!/bin/bash

# Copyright (c) 2013, James Downie 
# 
# Permission to use, copy, modify, and/or distribute this
# software for any purpose with or without fee is hereby
# granted, provided that the above copyright notice and this
# permission notice appear in all copies.
# 
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL 
# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
# THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

BATCH="$1"
MTIME="$2"
SOURCE="$3"

D="`dirname "$SOURCE"`"
TD="destination/$BATCH/$D"
T="destination/$BATCH/$SOURCE"
mkdir -v -p "$TD"
mv -v "$SOURCE" "$T";
./setmtime.py $MTIME "$T";

That script works out the deep pathname that we expect to move the file into, and then relies on mkdir‘s -p switch to make all of the parent directories leading to the deepest directory entry. Then a simple mv will relocate the volume of the file from the source tree into the destination tree.
The setmtime.py script is a little hack that sets a file’s mtime to the time nominated with the provided unix timestamp. We have all of the file’s unix timestamps in index.txt because stat determined them for us back when index.sh ran find across source. migrate_one.sh runs setmtime.py against the file once it as been moved to ensure that the file survived the move without losing it’s original “last modified time”.

#!/usr/bin/python

# Copyright (c) 2013, James Downie 
# 
# Permission to use, copy, modify, and/or distribute this
# software for any purpose with or without fee is hereby
# granted, provided that the above copyright notice and this
# permission notice appear in all copies.
# 
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL 
# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL
# THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
# FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

import os, sys

mtime = int(sys.argv[1])
path = sys.argv[2]

if os.path.exists(path):
  os.utime(path, (mtime, mtime))
else:
  print "Where is '%s'" % path

My directory looked like this…

./
./source/
./destination/
./index.sh
./index.txt
./split.awk
./migrate_all.sh
./migrate_one.sh
./setmtime.py

…when I then ran…

$ sh migrate_all.sh

That took two and a half days to run. Well, actually it took weeks to get right because my script kept breaking as I discovered special characters in source. I made filenameCleanse.py aggressive enough to remove any characters that jeopardised any of these steps and finally it ran without error. The end result looks like this…

$ du -d 1 -h destination/
3.3T    destination/1
3.3T    destination/2
798G    destination/3
7.5T    destination