Firefox extension development

Uhoh. Now that Firefix 3 is out, it looks like I have to update my extension Firegoose to work. The trick here is that Firegoose uses Java.

Briefly, Firegoose is a Firefox extension that integrates several bioinformatics web resources into the Gaggle integration framework. The Gaggle is based on passing messages of a few fundamental data types in the bioinformatics domain, including lists, matrices, and networks. The transport protocol used in Gaggle is (for better or worse) Java RMI, and that, of course, requires Java. Hence, the Firegoose's reliance on being able to crank up a working (and unrestricted) JVM from inside Firefox.

That was done in for Firefox 1.x and 2.x using an arcane and dirty trick from the fine folks at Simile project at MIT called javaFirefoxExtension. And the sad thing for me is that the trick is apparently broken in FireFox 3. [Note: this turns out not to be the case.]

Resources

• Extension Developer Center
• Mozilla Development Center reference for Java in Firefox Extensions.
• Javascript 1.5 LiveConnect overview.
• Setting up and extension development environment.
• MozillaZine Extension Development Forum.

Descent into the bowels of Firefox

Silent failure is the curse of the Firefox extension developer. Debugging in Firefox is painful, at best, and even more so when using the bridge between Java and javascript (aka LiveConnect).

In order to do anything useful with Java in a Firefox Extension, there are at least 2 nasty bits to overcome. First, you have to load classes from inside an XPI file. This is essentially a variant of the classpath problem. Second, you probably have to give yourself full permissions (by manipulating java.security.Policy).

As mentioned above, an arcane solution to these problems has been worked out by folks at the SIMILE lab at MIT. Their PiggyBank extension is the prototype for use of Java for heavy lifting inside a Firefox extension. They actually run a full app server inside the browser. Another really cool extension that uses the same technique is xquseme, which embeds the Saxon XQuery processor. You can then perform arbitrary XQueries against any document you can browse to. Using Java in an extension gives you the power to combine the wealth of libraries available in the Java universe with a fully featured browser. So how do we go about doing it?

Loading classes from your XPI file is possible using Firefox's capability to resolve chrome URLs to paths in the filesystem. The mapping between chrome URLs and files is defined in the chrome.manifest file in your XPI. Once we have paths (as file:/ URLs), we create our own java.net.URLClassLoader. Calling the constructor java.net.URLClassLoader(URL[] urls) requires a trick, because the Java bridge seems not to do a very good job of coercing javascript types to java types. To further muddy the waters, the way js-to-Java type coersion is handled in LiveConnect changed in Firefox 3. You'd expect that passing a javascript Array containing java.net.URL objects to work. But, try that and you'll get an error like this:

InternalError: Unable to convert JavaScript value [...blah blah...] to Java value of type java.net.URL[]

Code gleaned from the SIMILE lab (thanks!) solves this particular pain in the ass:

// from http://simile.mit.edu/repository/java-firefox-extension/firefox/chrome/content/scripts/browser-overlay.js
_toJavaUrlArray: function(a) {
 var urlArray = java.lang.reflect.Array.newInstance(java.net.URL, a.length);
 for (var i = 0; i < a.length; i++) {
  var url = a[i];
  java.lang.reflect.Array.set(
   urlArray,
   i,
   (typeof url == "string") ? new java.net.URL(url) : url
  );
 }
 return urlArray;
}

Truth and soul

Actually, problems with js-to-Java type conversion aren't limited to constructing classLoaders, but seem to be pervasive in FF3. Apparently, whenever you try to convert a js array to Java, LiveConnect screws it up. For example, passing a js array of strings, I get an array full of "true". Yes, it's true. There was an object there. Thanks a lot for that! (Happens on both Mac OS X and Windows, btw.)

Virtual Private Hosting vs Self-hosting

According to our IT people, a middle-of-the-road server w/ a 3 years service agreement costs about 5 grand. (or 5000/36 moths = 139/month). The server would be available outside the company, but would be located in a network DMZ. We can have sudo, but can't mount any network filesystems. Giving outsiders privileged access to the server is apparently possible through some complicated process.

Apparently binding a DNS entry for a subdomain to an external server is quick and easy. So, why not get a virtual private server somewhere and avoid jumping through management hoops?

Here are a few providers I'm considering:

• SilverRack
• Rimu
• Rack Force
• HostMySite
• modVPS
• liquidweb

On the Criteria to Be Used in Decomposing Systems into Modules

David L. Parnas's infuential 1972 paper On the Criteria to Be Used in Decomposing Systems into Modules (sited in Paul Shannon's Gaggle paper) described modularization by information-hiding, later developed into high cohesion and low coupling.

The paper points out that it takes deliberate attention to get good software structure and that limiting information is the key to good design. Where there are many modules, more structure is needed. It's useful to create a hierarchical modular structure using "part of" and arrange programs in a hierarchical structure based on "uses". (By programs, he seems to mean parts of a module?) He criticizes the idea of "layers of abstraction" as too vague in absence of any measure of what's more abstract than what.

The concept of information-hiding as a software design principle is widely accepted in academic circles. Many successful designs can be seen as successful applications of abstraction or information hiding. On the other hand, most industrial software developers do not apply the idea and many consider it unrealistic.

The principle is now clear. It is still hard to apply.

He also seems to foreshadow design patterns.

Researchers should be publishing proposed standard structures for classes of programs. Researchers should be publishing proposed standard designs.

This kind of design-centered approach to software engineering appeals to me a lot more than fluffy stuff like usability studies and requirements gathering.

General purpose programming on the GPU

The video-card manufacturers are at an interesting crossroads. nVidia is really pushing general purpose computing on graphics hardware.

Cool idea, but it can't really take off unless programs are written to detect the presence of capable hardware and load code specially compiled for that particular chipset. In other words, GPGPU code written for nVidia cards won't run on ATI cards. Bummer. Clearly nVidia understands the advantage to be had from a large body of code compiled for their instruction set. They want to be in the position of owning the GPU equivalent of the x86 instruction set. Nice move on their part.

But wouldn't it be much cooler for the rest of us if things were a little more open? Is a common instruction set for graphics cards at all plausable? Like ARM is for embedded devices? Or maybe a common intermediate layer and JIT for all types of GPU code? That would be cool. And something like that would induce a lot more developers to take the plunge into compiling to the GPU, which has got to be a fairly drastically different piece of code relative to a straightforward implementation.

What is a Ruby code block?

You can't believe everything you read on the internet. For example, you'll read that, in Ruby, use of the return keyword is optional. Like this:

def foo(a)
  return a*a
end

foo(9)
>>81

def bar(a)
  a*a
end

bar(8)
>>64

So far, so good. You'll also read that a code block in Ruby is an anonymous function. And what could be more natural than to return from a function? But try this:

[1,2,3].map {|a| a*a}
>> [1,4,9]

[4,5,6].map {|a| return a*a}
>> LocalJumpError: unexpected return
>> from (irb):14
>> from (irb):14:in `map'
>> from (irb):14
>> from :0

Well then, apparently one or both of our premises is wrong. Anyway, what is this LocalJump thing that is apparently in error?

My guess as to what's really going on is this: The code block is an attempt at creating a function-like artifact that matches the behavior of a code block in a non-functional language. That is, the code in the block doesn't execute in its own environment. Calling a code block doesn't cause a stack frame to be created. The reason the return causes problems is that there's no separate environment to return from.

It bothers me that I can't seem to google up an explanation of exactly what a Ruby code block (or Proc or lambda or method for that matter) is in terms of what does or doesn't happen to stack frames or environments. There's plenty of tutorials on how to use these constructs, but, little information on what they are, how they work, or why there are so many flavors of function-like things in Ruby.

Here's what I got from comp.lang.ruby.

Working with microarray data

State-of-the-art DNA microarrays contain between 1 million and 6 million features (different probes) on a single slide. Assuming a 32 bit floats, we need at 4 bytes per feature. If we include start and stop coordinates on the genome for each feature, we're up to 12 bytes per feature.

features	MB	MB w/ coords
1 million	4	24
6.5 million	24	75

In tiling arrays, probes target regularly spaced segments of the genome so that expression can be measured at every point along the genome. Our group has done some work with arrays containing 60bp probes tiled every 20 base-pairs along the genome of Halobacterium salinarium. The genome of H. salinarum is 2.6 million bases, so we are able to cover most of the genome with a little less than a quarter of a million probes. To cover the entire human genome at similar resolution would take ~307 million probes.

As a side note, AFFY sells a set of 14 arrays with a total of almost 90 million probes that covers the whole genome at 35 bp resolution. (90 million * 35bp = 3150 mpbs. Does their idea of a probe include both forward and reverse strand?)

These arrays are all custom made for an individual genome. I wonder if an array with all 4 million possible 11-mers would be useful? Since there's only 4 million of them, you could expect a fair number of collisions - where a probe is non-unique on the genome. How do you do that calculation?

64-bit Java

As far as I can tell, the capability to handle much bigger heap sizes is the main benefit from 64-bit Java. Primitive types don't change in size. An int is still 32 bits.

Java arrays are indexed by 32-bit integers (actually, signed integers, so 31 bits). Apparently this doesn't change in 64-bit Java. From the Java Language Spec:

Arrays must be indexed by int values; short, byte, or char values may also be used as index values because they are subjected to unary numeric promotion (§5.6.1) and become int values. An attempt to access an array component with a long index value results in a compile-time error.

This is a serious limitation, if you ask me. The main advantage of 64-bits is a simple programming model for large data objects. Apparently, Sun disagrees. Integer array indexes are baked fairly deeply into the language and into existing application code. Think of everywhere you refer to the length or an offset into an array as an int. Imagine how often that's done in the library code. Dunno if they'll ever change it.

Other side-effects of 64-bit

• Increased object size
• 64 bit pointers
• alignment to 8-byte boundaries
• The minimally required heap size is 51.1% larger
• More cache misses

source: 64-bit versus 32-bit Virtual Machines for Java

Finally, here's an intriguing tidbit: sun.misc.Unsafe