Cognitive Consonance

Python’s matplotlib rocks. It is, by itself, the fundamental reason I stay with Python (JVM-based JFreeChart is powerful but – as most JVM libraries – an over-engineered, complex piece of software).

Doing multiple charts in matplotlib is not trivial, though. Well, it is trivial, but the end result is not perfect. Here I will present a step-by-step guide to creating better subplots. “Better” is, of course, a matter of taste, therefore every change will be presented separately, so that you can pick and chose whichever you prefer.

We start with this (as with all charts presented here, you will need to click to enlarge):

OK, the space between charts is wasted. Also the outside border is too big. Outside borders are less of a problem with a single chart, but when you have lots of stuff to present, then all space that you can get is important. So, we add this:

fig = pylab.figure()
fig.subplots_adjust(hspace=0.0001, wspace=0.0001,
                    bottom=0.07, top=0.96,
                    left=0.04, right=0.96)

The fig variable might have been initialized before, so use your own. Notice that subplots_adjust allows you to control both external borders (left, right, bottom, top) and space between charts (hspace, wspace). The result is this:

Before we discuss what obviously looks bad, we need to discuss something that looks good but is, in fact, problematic: The fonts. The fonts look of a proper size and (at least some of them) appear properly positioned. This apparent “goodness” is dependent on figure size and resolution (check the parameters of figure). If you develop this on screen and then want to generate an eps version at the end (say, to publish a scientific paper) you might be going for a rough ride and just detect the problem way later (when you generate the final, production version)! Suggestion: Try adjusting the figure parameters from the start (use both pylab.show and pylab.savefig – png and eps – to test the results). Here I will not spend time with more details about this (the same applies to line thickness, by the way).

OK, irrespective of the point above, the figure clearly has other problems. I opt to remove all titles (just remove the title lines) and remove the unnecessary X and Y axis labels. Note that we have 2 Y axis (one on the left, and one on the right). So, we apply the following code:

#We assume that sp comes from
#sp = pylab.subplot()...
 
#You should have this line if you have an extra Y-axis
#Ignore this if you do not have one (most cases won't have one)
par1 = sp.twinx()
 
#Apply this line to all subplots NOT on the extreme left
pylab.setp(sp.get_yticklabels(), visible=False)
 
#Apply this line to all subplots NOT on the bottom
pylab.setp(sp.get_xticklabels(), visible=False)
 
#Apply this line to all subplots NOT on the extreme right
pylab.setp(par1, visible=False)

Remember that we also removed the titles, so the result is:

Before we re-instate the titles (and put some legends), notice that the Y-axis labels slightly overlap from one chart to the other (this also happens with the X-axis labels, but it is less visible). For this we will redo the labels:

1
2
3
4
5
6
7
8

#if left and not bottom:
    pylab.yticks((0,.2,.4,.6,.8,1),("","0.2","0.4","0.6", "0.8","1.0"))
...
#if right and not bottom:
    pylab.yticks((0,0.5,1,1.5,2,2.5,3,3.5,4,),
                 ("", "0.5", "1", "1.5", "2", "2.5" , "3", "3.5", "4"))
#if bottom and not left:
    pylab.xticks((0,20,40,60,80,100),("","20","40","60", "08","100"))

Now, you might be confused: Lines 2 and 5 call the same function. Well line 2 is called before the creation of the second axe, line 5 after. There might be a more general way of doing this, but I was lazy enough to try and discover it. Result:

OK, now titles and axis labels. I will do this in a non-standard way: By hard-coding everything! You can, if you prefer activate the calls to .set_xlabel, and .title in the right position. In any case, the following example might serve to illustrate fig.text:

1
2
3
4
5
6
7
8

fig.text(0.5, 0.01, "Generations", ha="center", va="bottom", size="medium")
 
fig.text(0.20, 0.99, "Full epistasis", ha="center", va="top", size="medium")
fig.text(0.52, 0.99, "Mixed mode", ha="center", va="top", size="medium")
fig.text(0.83, 0.99, "DGF", ha="center", va="top", size="medium")
 
fig.text(0.015, 0.52, "Fst / LD(r)", ha="right", va="center", size="medium", rotation="vertical")
fig.text(0.99, 0.52, "Ratio", ha="right", va="center", size="medium", rotation=270)

ha is horizontal alignment (I will leave up to you to discover what va is ). Note that the last 2 texts have different rotations. It is possible to put math notation in the text (LaTeX). Something like this: $\frac{1}{n}$. You need 2 things for this to work: add matplotlib.rc(‘text’, usetex = True) and have LaTex installed.

The end result is then:

Not fantastic, but you now have the techniques to alter at will. Just a final reminder that changing size and resolution will have a strong impact on font and line thickness. You might want to check your chart with all final media (screen, eps to print, …). For instance the example above would clearly benefit from thicker lines and larger fonts.

xtranormal is a company/service which allows you to produce movies just from text. Essentially you choose one or two characters, get a scenario and some music and then you create your script. The script is mostly the test spoken by the charater(s), but also camera changes, sounds, and character behaviour. All this wrapped up in a clever and easy web interface (they also have a desktop application by I did not use that). From a techy perspective the technology seems uber cool: text-to-speech (very good quality), 3D character animation and a nice interface on top. Adding to this their business model seems quite fair.

So that you can see what you can do with just one hour of work, here is an example that now sits in my home page (feel free to laugh at my expense, I do not mind )

Go there, and just try to make a video of your own to get the feel for the technology.

PS – no relationship with the company, just a delighted user here.

Nice reads mostly older:

• [jvm] YouDebug – A debug with a scriptable UI (not a GUI). Allowing for automated debugging. The main justification for this is e.g. on costumer sites where you cannot fire a graphical debugger. But an automated debugging framework can be the ground for some cool debugging tools. Furthermore it is scriptable in Groovy!
• [general]Are we there yet? – “Rich Hickey advocated for the reexamination of basic principles like state, identity, value, time, types, genericity, complexity, as they are used by OOP today, to be able to create the new constructs and languages to deal with the massive parallelism and concurrency of the future”. This is a 1 hour video (with synchronized slides). While I disagree with the basic premise that concurrency is fundamental in the future, this presentation is one of the most invaluable videos about the design of programming languages.
• [lisp] A Lisp User-group meetings calendar (Google based)
• [clojure] Be mindful of Clojure’s binding – Pitfalls of lazy sequences and thread bindings. The lazy sequence stuff seems more of an example of global variables are bad thing, as the example is dependent on an external symbol evaluation, but clearly one has to be mindful of lazy versus non-lazy evaluation.

Addendum: This requires biojava 1.8, no more, no less!

BioJava has a parser for the nexus file format. While Nexus supports quite a lot of information (like DNA sequences), the most complex part to process is the phylogenetic tree descriptions based on the Newick format. Below you can find a small tutorial on how to process these.

BioJava’s parser relies on JGraphT to create a representation of the “tree”. The tree is actually really more an acyclic graph than a tree, though some trees are rooted (and therefore trees in the proper sense). Manipulating the JGraphT weighted graph is the complicated part, not really the BioJava interface. Note that JGraphT objects can be easily rendered using the JGraph library (yeah, it is confusing: there is one lib with graph algorithms called JGraphT and another, for vizualisation, called JGraph).

In this small tutorial, we will only try to write a textual representation of a tree.

Imagine this simple nexus file:

#NEXUS

Begin TREES;
	tree test1 = (1,2);
	tree test4 = (1:0.1,(2:0.2,3:0.3):0.4);
End;

We just want to draw this:

coder@move-on:~/development/biobug/test$ java Test test1.nex test1
Will process file test1.nex tree test1
p0
  1: 1.0
  2: 1.0
coder@move-on:~/development/biobug/test$ java Test test1.nex test4
Will process file test1.nex tree test4
p0
  1: 0.1
  p1: 0.4
    2: 0.2
    3: 0.3

So, tree 1 is composed of nodes (leaves) 1 and 2 and the inner node which was named p1. Tree 2 has distances.

By the way, we will also want to know which trees are in the file.

Lets start!.

So, we start by loading and parsing the file:

import org.biojavax.bio.phylo.io.nexus.*;
 
[...]
        //file is a String with the name of the file to be processed
        NexusFileBuilder builder = new NexusFileBuilder();
        NexusFileFormat.parseFile(builder, new File(file));
        NexusFile nexus = builder.getNexusFile();

Nexus files have several blocks (Taxa, Data, Tree, Set). We are interested in getting the Tree block, lets do a function for that:

    TreesBlock getTreeNode(NexusFile nexus) {
        Iterator it = nexus.blockIterator();
        NexusBlock block;
        while(it.hasNext()) {
                block = (NexusBlock)it.next();
                if (block.getBlockName().equals("TREES")) {
                        return (TreesBlock)block;
                }
            }
            return null;
    }

We get the nexus block iterator and go through it until we find a block whose name is TREES, and return that block.

No that we have the TREES block, lets print the names of all trees:

    void printTrees(NexusFile nexus) {
            TreesBlock node = getTreeNode(nexus);
            Map trees = node.getTrees();
            Set keys = trees.keySet();
            System.out.println("Trees:");
            for (Object obj : keys) {
                System.out.println(obj);
            }
    }

We get a map, where the key is the name of the tree and the value would be the tree as, essentially, a String based (but with some annotated info) representation (not a graph). Now, given a certain name, lets get the graph:

import org.biojava.bio.seq.io.ParseException;
import org.jgrapht.*;
import org.jgrapht.graph.*;
 
[...]
 
    WeightedGraph<string , DefaultWeightedEdge> getTree(NexusFile nexus, String name)
    throws ParseException {
        String topNode;
        TreesBlock node = getTreeNode(nexus);
        WeightedGraph</string><string , DefaultWeightedEdge> graph = node.getTreeAsWeightedJGraphT(name);
        topNode = node.getTopNode();
        System.out.println("The top node is: " + topNode);
        return graph;
    }
</string>

Note that getTreeAsWeightedJGraphT will do some parsing, so ParsingException can be raised. Note also that the top node name can be retrieved (in the case of tree test1, that will be named p1). Some considerations: You can change the rules to create internal nodes; if there are clashes of names inner nodes will be renamed (not leaves!).

Regarding the top node, we call it top node and not root node. While from a data structure perspective the tree has a root, from a phylogenetic perspective the tree might be rooted or not (in which case being root has no meaning, and it is really just a simple weighted acyclic graph). How to know if the tree is rooted? Remember the function to get all trees (getTrees)? The value of the map has a method called getRootType. So, to know if is rooted, you need to use that function. Not the best design… but at least works.

Ok, now we just need to print a tree…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

    static void dump(WeightedGraph<string , DefaultWeightedEdge> graph,
            String parent, String node, String depth) {
        Set</string><string> verts = graph.vertexSet();
        String vertex = "";
        for (String candidate : verts) {
            if (candidate.equals(node)) {
                vertex = candidate;
                break;
            }
        }
        System.out.print (depth + vertex);
        if (parent != null) {
            System.out.print (": " + graph.getEdgeWeight(graph.getEdge(parent, vertex)));
        }
        System.out.println();
        for(DefaultWeightedEdge e: graph.edgesOf(vertex)){
            if (graph.getEdgeSource(e).equals(node)) {
                dump(graph, vertex, graph.getEdgeTarget(e), "  "+ depth);
            }
        }
    } 
</string>

Ok, this is the complicated part. Note the following:

• The complexity has to do with processing graphs
• dump is a recursive function
• Node is synonym with Vertex
• Notice the important bit, if you know that there is a node called “bla”, it is not enough to do graph.containsVertex(“bla”). The answer will probably be false. Remember that one thing is reference (which we have here, i.e. ==) and not content equality (.equals). See below, a remainder
• Finally we go through all edges referencing the current vertex and choose the ones that start on the current one. Again, if the tree is unrooted, the notion of direction does not apply, but it is still good to do a “tree” traversal

We end here.

Regarding the “equal” issue remember that:

        System.out.println("a" == new String("a"));
        System.out.println("a".equals(new String("a")));

Returns false, true. By this order. This is important when traversing the graph. If you know that the reference is equal (and it is when we getEdgeTarget) than one could use it. If you don’t know (like you pass a String that you have constructed yourself or got from some other place), then one needs to go through the vertex/node list and do a .equals to get the correct vertex.

A small example with all the above, is here, ready to use.

Having read Tim Bray’s tips for Clojure newbies from a newbie, I decided to write a version of my own.

First, I generally agree with Tim’s observations, so this post is more like minor extension. In fact I recommend reading Tim’s version first.

My background so that you know my point of view: Added to Java, C and Python experience I have lots of exposure to Prolog (which is, like Clojure, homoiconic) and some experience with OCaml (a functional language). I have had some practical contact with other “modern languages over JVM”: Scala and Groovy. No Lisp experience (other than basic elisp).

Reading suggestions: Other than Tim’s comments I would add: The clojure site information is really not very useful as a starting point, the reference part assumes that you already know quite a bit and it is though on newbies, the only thing that I tend to use is the API section. Like Tim, I also use Mark Volkmann’s introduction (It is my main documentation, and I would be a bit more positive than Tim about it. For an intro article it is great. I strongly recommend it as a starting point and as the reading anchor during the first weeks). Next week I plan to order Stuart Halloway’s book, so I still cannot comment on that.

On a more general note, while learning Clojure, I’ve found Paul Graham’s On Lisp (available for free), a gem and I would strongly recommend it. It is not an easy read, it probably takes months to digest the content. But it is really a great book.

Clojure is a fast moving target. Documentation of many modules and functions might not be up to date with the current code. I have noticed that sometimes the best “documentation” ends up being reading the code (clojure is hosted on github and so are many satellite projects – In fact getting familiar with github is probably another recommendation).

Some of the contrib stuff is a bit too green, and I would recommend inspection of some of the modules before using them. Just because it is accepted on clojure.contrib it does not mean it is production quality, has even bare functionality or the exposed API is stable. As an example the graph API is minimal and I question if the graph structure directed-graph is enough to represent a general directed graph (future changes to it will probably break existing code built on top of it). This is not a criticism, a new product is bound to be fast changing, and agile methods of development will entail lots of instability at the beginning. But a caveat should be added to the stability and completeness of parts of clojure-contrib.

Regarding editors and IDEs, I would probably recommend for you to stick with what you feel more comfortable with (vi, emacs, netbeans, eclipse, …). Note that Clojure, being a Lisp derivative has a big share of its user base on emacs. I mainly use Netbeans using the wonderful enclojure plug-in. I can only say that encloure is stable enough for usage and I recommend it if you are a Netbeans type of person.

I also second Tim’s comments on namespace hell. Uses/imports/requires can be quite confusing. Hell is too harsh of a word, but purgatory seems an accurate description . Also, as Tim says, the clojure mailing list and IRC channels are very, very helpful.

One of the most annoying kinds of bugs come from typing problems, things like this:

(defn hiddenBug [a b]
  (println a) ;lets do a println for debug purposes
  (println b) ;lets do a println for debug purposes
  (if-not (= a b) (println "they are different!"))
)

Now lets call this:

(hiddenBug 'x "x")
x
x
they are different!

Notice that, when you are debugging a and b will seem equal on a println, but they are not (one is a symbol, another a string)!

The biggest gotcha that I have been getting is the expectation (Java based) that stupid things raise exceptions, but sometimes they don’t. Here is an example:

user=> (contains? '(a b) 'a)
false
user=> (contains? 'blab 'a)
false
user=> (contains? (list 'a 'b) 'a)
false
user=> (contains? ['a 'b] 'a)
false
user=> (contains? '(a b) :a)
false
user=> (contains? '(:a b) :a)
false

In all these cases, the Java-expecting gnome inside me was hoping something of a throw (as the first argument is not of the type required by contains? ) . It is not clear to me if this is a design issue with contains? only, or it is something that is standard along the whole API. But I notice this from the API reference:

(even? n)
Returns true if n is even, throws an exception if n is not an integer

So it seems that the design is not homogeneous throughout the API as some functions throw exceptions. I would probably prefer a throw when the type of arguments is wrong, but people with lots of Lisp experience might have a different view on the issue. Anyway I would like to understand if the lack of homogeneity is a feature or a bug.

When doing Groovy scripts I sometimes have the need to redirect the output of a part of the computation that is going to standard out.. A possible solution would be to open a new Writer and change the code to write to it (i.e. replacing all prints with newStream.prints), this, of course requires changing all prints, which is cumbersome and boring. There is a lazy alternative, using method references:

s = new PrintStream(new FileOutputStream("/tmp/myOut"))
def print = s.&print
print "a"
print = System.out.&print
print "b"

In this case the a is written to /tmp/myOut and b gets back to the standard output again. The big gain: all those prints in a script (and printlns) don’t need to be changed! Lazy me is happy.

Caveat: I would be careful in using this strategy a lot, it is be very easy to loose track of what is happening to the output. But it can be quite an expedient way to redirect prints on simple scripts.