Thursday, October 30, 2008

An embedded jar classloader in under 100 lines

To me one of the most annoying things about deploying a standalone java application is packaging. You can't just jar your code and dependencies into a single jar that's ready to run without jumping through hoops. All I want to be able to do is have jars live inside jars and have it all just work. As it turns out this is not an uncommon request, but it seems to be one that is also often dismissed, with few viable solutions. How hard can it be?

As it turns out, its not very hard at all, albeit with a few limitations. In just a couple of hours and using under one hundred lines of code I was able to write a classloader that would allow me to create my single jar deployment. The logic is fairly simple, walk the jar entries in the jar the app was loaded from and recursively extract any embeded jars and add them to the classpath. This sacrifices some temporary storage space at runtime, but it's an acceptable trade off in my eyes.

This solution was fine, but I wanted to be able to launch the app with the typical 'java -jar myapp.jar' convenience instead of overriding the system classloader on the command line. This requires a way to bootstrap the main application using this new classloader. Another fifty odd lines later and I have a bootstrapper that lets the magic happen with the addition of a single manifest entry.

Here is what the classloader looks like:


public class JarClassLoader extends URLClassLoader {

private static void close(Closeable closeable) {
if (closeable != null) {
try {
closeable.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}

private static boolean isJar(String fileName) {
return fileName != null && fileName.toLowerCase().endsWith(".jar");
}

private static File jarEntryAsFile(JarFile jarFile, JarEntry jarEntry) throws IOException {
InputStream input = null;
OutputStream output = null;
try {
String name = jarEntry.getName().replace('/', '_');
int i = name.lastIndexOf(".");
String extension = i > -1 ? name.substring(i) : "";
File file = File.createTempFile(name.substring(0, name.length() - extension.length()) + ".", extension);
file.deleteOnExit();
input = jarFile.getInputStream(jarEntry);
output = new FileOutputStream(file);
int readCount;
byte[] buffer = new byte[4096];
while ((readCount = input.read(buffer)) != -1) {
output.write(buffer, 0, readCount);
}
return file;
} finally {
close(input);
close(output);
}
}

public JarClassLoader(URL[] urls, ClassLoader parent) {
super(urls, parent);
try {
ProtectionDomain protectionDomain = getClass().getProtectionDomain();
CodeSource codeSource = protectionDomain.getCodeSource();
URL rootJarUrl = codeSource.getLocation();
String rootJarName = rootJarUrl.getFile();
if (isJar(rootJarName)) {
addJarResource(new File(rootJarUrl.getPath()));
}
} catch (IOException e) {
e.printStackTrace();
}
}

private void addJarResource(File file) throws IOException {
JarFile jarFile = new JarFile(file);
addURL(file.toURL());
Enumeration<JarEntry> jarEntries = jarFile.entries();
while (jarEntries.hasMoreElements()) {
JarEntry jarEntry = jarEntries.nextElement();
if (!jarEntry.isDirectory() && isJar(jarEntry.getName())) {
addJarResource(jarEntryAsFile(jarFile, jarEntry));
}
}
}

@Override
protected synchronized Class<?> loadClass(String name, boolean resolve)
throws ClassNotFoundException {
try {
Class<?> clazz = findLoadedClass(name);
if (clazz == null) {
clazz = findClass(name);
if (resolve)
resolveClass(clazz);
}
return clazz;
} catch (ClassNotFoundException e) {
return super.loadClass(name, resolve);
}
}
}

And with the help of some modified code lifted from a sun tutorial on jar classloading we have a bootstrapper that looks like this:

public class JarRunner {
private static final String BUNDLE_MAIN_CLASS = "Bundle-MainClass";

public static void main(String[] args) {
try {
invokeClass(getMainClassName(), args);
} catch (Throwable e) {
e.printStackTrace();
}
}

public static String getMainClassName() throws IOException {
ProtectionDomain protectionDomain = JarRunner.class.getProtectionDomain();
CodeSource codeSource = protectionDomain.getCodeSource();
URL rootUrl = codeSource.getLocation();
if (rootUrl.getFile().toLowerCase().endsWith(".jar")) {
URL rootJarUrl = new URL("jar", "", rootUrl + "!/");
JarURLConnection uc = (JarURLConnection) rootJarUrl.openConnection();
Attributes attr = uc.getMainAttributes();
return attr != null ? attr.getValue(BUNDLE_MAIN_CLASS) : null;
}
throw new IllegalStateException("JarRunner is not inside a jar file");
}

public static void invokeClass(String name, String[] args) throws ClassNotFoundException, NoSuchMethodException, InvocationTargetException {
final String mainMethodName = "main";
JarClassLoader loader = new JarClassLoader(((URLClassLoader) ClassLoader.getSystemClassLoader()).getURLs(), Thread.currentThread()
.getContextClassLoader());
Class clazz = loader.loadClass(name);
Method method = clazz.getMethod(mainMethodName, new Class[] { args.getClass() });
method.setAccessible(true);
int mods = method.getModifiers();
if (method.getReturnType() != void.class || !Modifier.isStatic(mods) || !Modifier.isPublic(mods)) {
throw new NoSuchMethodException(mainMethodName);
}
try {
method.invoke(null, new Object[] { args });
} catch (IllegalAccessException e) {
// This should not happen, as we have
// disabled access checks
}
}
}


Now all that is necessary to build an embedded jar application is to add these two classes to the application's jar bundle and add a couple of manifest entries that look like this:

Main-Class: JarRunner
Bundle-MainClass: your.app.MainClass


Simple as that, well sort of. There are a couple of limitations with this approach, specifically if you use things like JSR-232 scripting or certain JDK XML based functionality that expects the system classloader to have visibility of the required classes it won't be able to find them.

An alternative approach I could have possibly taken is to avoid using a custom classloader altogether and use the same recursive jar extraction logic from the JarClassLoader in the JarRunner and have it add the extracted jar paths to the system classloader using the URLClassLoader.addURL() method.

Friday, August 17, 2007

Automating torrent downloads with AppleTV + RSS

If you regularly subscribe to a few podcasts or video programs that are available via bittorrent with an RSS feed, and you watch them on your appletv, you might be wondering if you can automate the process so everything happens automatically without the need for any external devices (ie. no need to use iTunes), well you can.

Firstly, you can use Yahoo Pipes to aggregate your chosen subscription's RSS feeds into a single feed and only include the programming you want to download, then you setup your (hacked) AppleTV with atvTorrents, simple enough. Lastly you need something that will download the new items from the aggregate RSS feed and start them downloading.

I couldn't find anything that would do this last bit for the AppleTV, so I wrote something myself in perl, which is already installed on the AppleTV, after seeing a couple of simple examples in ruby and python.

Below is the code, is has no external dependencies other than /usr/bin/curl, which doesn't come with the AppleTV, but you can get it from the Weather plugin and copy it into /usr/bin

#! /usr/bin/perl

use warnings;
use strict;

# Fetch a URL with curl
sub fetchurl {
my $url = shift;
my $cmd = qq{/usr/bin/curl -s "$url"};
return `$cmd`;
}

# Return a list as an arrayref
sub as_arrayref {
my $arg0 = shift;
return \@_ if ! defined $arg0;
return [ @$arg0, @_ ] if ref $arg0 eq 'ARRAY';
return [ $arg0, @_ ];
}

# Convert feed XML data into a nested data structure
sub parse {
my ($markup) = @_;

# Split on matching start end tags
# - On match returns sets of 4 elements: [ '', tag, tag_values, content ]
# - On no match return 1 element: [ content ]
my @markup = split(/<([^\ >]+)\ *([^>]*)>(.*?)<\/\1>/, $markup);

my $result = ();

while ($#markup >= 0) {
if ($markup[0] eq '') {
my $element = {};
shift(@markup);
my $tag = shift(@markup);
my %values = split(/[\ =]/,shift(@markup));
$element->{$tag} = parse(shift(@markup));
$element->{$tag}->{_values} = \%values if %values;
foreach my $key (keys %$element) {
if (defined $result->{$key}) {
$result->{$key} = as_arrayref $result->{$key}, $element->{$key};
} else {
$result->{$key} = $element->{$key};
}
}
} else {
# This should only get called once on no matches from split
$result = shift(@markup);
}
}
return $result;
}

# Read RSS feed and return parsed data structure
sub rssread {
my $feedurl = shift;
my @result;
foreach my $item (split("\n", fetchurl($feedurl))) {
my $data = parse($item);
ref $data eq 'HASH' and push(@result, $data);
}
return @result;
}

my $downloadpath = "/Users/frontrow/Torrents/";
my $downloadlog = "/Users/frontrow/.tvdownloads";
my $feedurl = "http://pipes.yahoo.com/pipes/pipe.run?_id=SOMEUNIQUEID&_render=rss";

sub is_downloaded {
my $link = shift;
open(LOG, $downloadlog) or return 0;
while() { chomp(); return 1 if $_ eq $link }
close(LOG);
return 0;
}

sub write_log {
my $link = shift;
open(LOG, ">>$downloadlog");
printf LOG "%s\n", $link;
close(LOG);
}

sub get_tvtorrents {
my @result = rssread($feedurl);
foreach my $it (@result) {
my $items = as_arrayref $it->{rss}->{channel}->{item};
foreach my $item (@{$items}) {
if (! is_downloaded $item->{link} ) {
(my $torrentfile = $downloadpath . $item->{title} . '.torrent') =~ s/ /\_/g;
my $torrentdata = fetchurl($item->{link});

open(TORRENT, ">$torrentfile");
print TORRENT $torrentdata;
write_log($item->{link});
close(TORRENT);

print $torrentfile . "\n";
}
}
}
}

get_tvtorrents();


Once you have this working, you will need to have it run regularly. To do that we need to use launchd by creating a tvtorrents.plist file in /Users/frontrow/Library/LaunchAgents that looks something like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>appletv.tvtorrents</string>
<key>LowPriorityIO</key>
<true/>
<key>Nice</key>
<integer>1</integer>
<key>ProgramArguments</key>
<array>
<string>/Users/frontrow/bin/tvtorrents.pl</string>
</array>
<key>StartInterval</key>
<integer>300</integer>
</dict>
</plist>


This will run the script /Users/frontrow/bin/tvtorrents.pl every 5 minutes (300 seconds). And that's all there is to it.

Wednesday, August 08, 2007

Exporting to Excel from Numbers without TOC

Apple has finally announced iWork 08 and it includes Numbers, their take on the traditional spreadsheet. I have downloaded the trial and had a play with it, and while it might not satisfy the scientific or academic community, it does mostly what I need.

One annoyance however is that when you export your Numbers spreadsheet to Excel it will include a table of contents explaining how your sheets and tables were converted worksheets. While this might be helpful, it's not likely to be appreciated in a business environment.

So here is how you get Numbers to not include the table of contents when you export to an excel spreadsheet:

Simply edit your com.apple.iWork.Numbers.plist file in ~/Library/Preferences to include an additional key called "EEDropTableOfContents" and set it to TRUE.

From Terminal you can simply type
defaults write com.apple.iWork.Numbers EEDropTableOfContents
-bool TRUE

To edit it as an XML document you need to run
plutil -convert xml1 ~/Library/Preferences/com.apple.iWork.Numbers.plist
then add an entry that looks like this:
<key>EEDropTableOfContents</key>
<true/>

Or if you use Property List Editor included with the developer tools it will look like this: