Cognitive Consonance

This is a post intended to help people trying to produce Python libraries and applications in 64-bit Windows (i.e. native 64-bit binaries, not just 32-bit applications running on 64-bit Windows). It reflects what I have learned when trying to compile and run Biopython. Corrections and comments are most welcome. Nothing here written is “rocket science”, but it took me quite a while to compile this information, therefore it might make your life easier also.

Self-imposed requirement: To be able to compile with free compilers (preferably free as in speech, but free as in beer will do).

Point 1: Python does not compile with the free 64-bit compiler, mingw-w64. I actually did not try this, but I tried to compile biopython with mingw-w64 and Python headers were problematic (plus, if you go to Python bug database you will find references there to not being able to use mingw-w64).

Point 2: Visual Studio Express (the free beer compiler from Microsoft) does not natively target 64-bit architectures. Fear not, if you download the Windows SDK, then you get a 64-bit compiler. As far I as can see, this is the best current solution. The easiest way to compile (at least from the command line) is by using the Windows SDK Command Prompt (available in the Microsoft Windows SDK menu). Have a look at setenv (setenv.cmd) which allows you to set quite a few things about the target architecture. I am using VS2010 Express. In theory distutils only supports up to 2008 (v9), but I had no problems other than to add a /MANIFEST flag to the linker.

Point 3: If the code you are compiling depends on external libraries, then you mileage might vary (a lot), there are 4 options: (i) either the dependencies already have 64-bit versions (like NumPy which I needed for Biopython); (ii) There are no 64-bit versions, but it is easy for you to compile; (iii) There are no 64-bit versions and it is complex work to generate one. If it is case 3, then, well, case lost, back to the 32-bit version. You might have noticed that I mention FOUR options, well (iv) the fantastic porting effort done by Christoph Gohlke, if you need any Python 64-bit library check his porting page with truckloads of stuff (Compiled using Visual Studio professional editions).

So: Use VS Express and add the MS Windows SDK. setenv is your friend (and the Windows SDK command prompt, which is cmd with setenv, really).

As a final note, while I do grok pragmatism, I find less than desirable (to be euphemistic) that Python does not support the existing FREE (as in speech) 64-bit compiler on Windows.

Returning to my pet Google plus problem, in this case with an example:

I wanted to share, with a circle the following video (spoken by Carl Sagan):

With Facebook it is trivial: In youtube there is a share button, I click it, a window opens where I can add some comments, I click “Share a link” and voila. It is done: 2 clicks, totally non-intrusive. Notice that a Google product (youtube) is trivial to use with FB.

With Google plus? Start a new browser tab/window, put plus.google.com in the address, copy the video URL from the youtube tab and paste it as a video on plus (choose the video posting button). One could almost make a user manual! Above all it is inconvenient: it requires too much effort for a thing that should be pleasurable.

I know, I know, the plus button is awsomeness. But it is not what I want (and I feel that I am not alone)! I want to be able to share with a group, to show something that I find interesting to my acquaintances/friends/colleagues/whatever. And have a cozy discussion with my pals about what I shared (or what they shared). That is what Facebook is all about to me. Google plus makes it hard.

I joined the new Google Scholar “My profile” stuff. It is, by the way, very welcome. Being dependent on a uber-closed system (ISI Web of Knowledge) is far from desirable. Furthermore Google scholar is much better at tracking references.

But, but, but… Check the picture below

According to Google Mail, Google Scholar is dangerous, non-trustable stuff: The email verification link ended up on my spam folder

Dear Google Mail, please take notice that Google Scholar is a decent citizen. A very welcome addition, please treat her accordingly.

Once again Google failed to understand the coolest feature of Facebook. The only difference this time is that they can still correct it.

Arguably the allure of Facebook is the social part of things. Part of it is a more private Twitter, actually a better twitter: you are not limited to 140 characters and the in-line response system is cool (i.e., interacting with your friends). Google+ has that, of course.

But there are other fundamental bits and pieces that G+ misses. Applications (creepy games included) are one. I understand Google’s option (and like it), but it loses a part of the crowd with that decision.

But the biggest problem is the concept of +1. +1 is a completely a-social thing. It goes onto the attic, in-a-place-nobody-cares-to-look! On the other hand Facebook (which also has the like stuff – broadly equivalent to +1) has the great “post to profile” button. There is a quantum leap of difference: You are on a page that you find interesting, post it to your profile, add a comment of your own and bang… your friends see it, they see your opinion, they can pop in and discuss it. Very SOCIAL! Yes, you can do this with G+, but it is very inconvenient: you have to open plus.google.com, and explicitly paste the link. Convenience here is key: Something as easy as the +1 button, but posting to the main G+ page (or circle).

I would venture that a clever tweeting system and the posting of pages justify most of FB allure (and the creepy games also, gnah). Also a more SOCIAL way of liking (+1) things: it is very easy to see people’s preferences and you can also like (and publicise) other FB profiles.

Also, in my profile I can say little about myself on G+: my movies, my religion, my footy club, my music, my political preferences are all out. It is a SOCIAL thing to do. So that people that share similarities to yourself can find you. Also, to make a statement!

And Circles, while being a GREAT idea, does not compensate for the lack of the other stuff (a “post to circle” feature would really be cool).

Of course, an API is also missing (external programmers could solve part of the above, if there was an API). And a few other things. People can say that this is an early release. But, in such a state, G+ will lose steam: it is very inconvenient to do some very simple SOCIAL things! And when these features eventually show up, several users will have lost interest.

I really hope Google puts is act up together. If I had to choose between both companies, undoubtedly I would prefer Google.

Google, get your act together!

Python’s matplotlib rocks. It is, by itself, the fundamental reason I stay with Python (JVM-based JFreeChart is powerful but – as most JVM libraries – an over-engineered, complex piece of software).

Doing multiple charts in matplotlib is not trivial, though. Well, it is trivial, but the end result is not perfect. Here I will present a step-by-step guide to creating better subplots. “Better” is, of course, a matter of taste, therefore every change will be presented separately, so that you can pick and chose whichever you prefer.

We start with this (as with all charts presented here, you will need to click to enlarge):

OK, the space between charts is wasted. Also the outside border is too big. Outside borders are less of a problem with a single chart, but when you have lots of stuff to present, then all space that you can get is important. So, we add this:

fig = pylab.figure()
fig.subplots_adjust(hspace=0.0001, wspace=0.0001,
                    bottom=0.07, top=0.96,
                    left=0.04, right=0.96)

The fig variable might have been initialized before, so use your own. Notice that subplots_adjust allows you to control both external borders (left, right, bottom, top) and space between charts (hspace, wspace). The result is this:

Before we discuss what obviously looks bad, we need to discuss something that looks good but is, in fact, problematic: The fonts. The fonts look of a proper size and (at least some of them) appear properly positioned. This apparent “goodness” is dependent on figure size and resolution (check the parameters of figure). If you develop this on screen and then want to generate an eps version at the end (say, to publish a scientific paper) you might be going for a rough ride and just detect the problem way later (when you generate the final, production version)! Suggestion: Try adjusting the figure parameters from the start (use both pylab.show and pylab.savefig – png and eps – to test the results). Here I will not spend time with more details about this (the same applies to line thickness, by the way).

OK, irrespective of the point above, the figure clearly has other problems. I opt to remove all titles (just remove the title lines) and remove the unnecessary X and Y axis labels. Note that we have 2 Y axis (one on the left, and one on the right). So, we apply the following code:

#We assume that sp comes from
#sp = pylab.subplot()...
 
#You should have this line if you have an extra Y-axis
#Ignore this if you do not have one (most cases won't have one)
par1 = sp.twinx()
 
#Apply this line to all subplots NOT on the extreme left
pylab.setp(sp.get_yticklabels(), visible=False)
 
#Apply this line to all subplots NOT on the bottom
pylab.setp(sp.get_xticklabels(), visible=False)
 
#Apply this line to all subplots NOT on the extreme right
pylab.setp(par1, visible=False)

Remember that we also removed the titles, so the result is:

Before we re-instate the titles (and put some legends), notice that the Y-axis labels slightly overlap from one chart to the other (this also happens with the X-axis labels, but it is less visible). For this we will redo the labels:

1
2
3
4
5
6
7
8

#if left and not bottom:
    pylab.yticks((0,.2,.4,.6,.8,1),("","0.2","0.4","0.6", "0.8","1.0"))
...
#if right and not bottom:
    pylab.yticks((0,0.5,1,1.5,2,2.5,3,3.5,4,),
                 ("", "0.5", "1", "1.5", "2", "2.5" , "3", "3.5", "4"))
#if bottom and not left:
    pylab.xticks((0,20,40,60,80,100),("","20","40","60", "08","100"))

Now, you might be confused: Lines 2 and 5 call the same function. Well line 2 is called before the creation of the second axe, line 5 after. There might be a more general way of doing this, but I was lazy enough to try and discover it. Result:

OK, now titles and axis labels. I will do this in a non-standard way: By hard-coding everything! You can, if you prefer activate the calls to .set_xlabel, and .title in the right position. In any case, the following example might serve to illustrate fig.text:

1
2
3
4
5
6
7
8

fig.text(0.5, 0.01, "Generations", ha="center", va="bottom", size="medium")
 
fig.text(0.20, 0.99, "Full epistasis", ha="center", va="top", size="medium")
fig.text(0.52, 0.99, "Mixed mode", ha="center", va="top", size="medium")
fig.text(0.83, 0.99, "DGF", ha="center", va="top", size="medium")
 
fig.text(0.015, 0.52, "Fst / LD(r)", ha="right", va="center", size="medium", rotation="vertical")
fig.text(0.99, 0.52, "Ratio", ha="right", va="center", size="medium", rotation=270)

ha is horizontal alignment (I will leave up to you to discover what va is ). Note that the last 2 texts have different rotations. It is possible to put math notation in the text (LaTeX). Something like this: $\frac{1}{n}$. You need 2 things for this to work: add matplotlib.rc(‘text’, usetex = True) and have LaTex installed.

The end result is then:

Not fantastic, but you now have the techniques to alter at will. Just a final reminder that changing size and resolution will have a strong impact on font and line thickness. You might want to check your chart with all final media (screen, eps to print, …). For instance the example above would clearly benefit from thicker lines and larger fonts.

Before we return to our regularly scheduled program (of technical and biological stuff), I want to advance the case that the financial markets are not only driven by “sentiment”, but also by sheer racism. The PIGS slur. Please look at the table below (source Eurostat):

Obviously the left column is much more important than the right column. From a debt/GDP ratio Spain is better than all the big other countries. Indeed Portugal is followed by… Germany in the list. But even if you think both columns are equally important, then please compare Spain with the UK in both benchmarks.

Now, why does everybody talk about Spain and not the UK? By the way, US debt (from wiki: “As of March 25, 2011, the Total Public Debt Outstanding of the United States of America was $14.26 trillion and was 97.3% of calendar year 2010 annual GDP”). Estimated deficit for 2010? 8.3%.

And while you are at it, you might want to check the external debt page at wikipedia (order by % of GDP).

Market sentiment (for a laugh):

My dear Finnish friends,

Recently you have been asked to bail us out. That you should use your hard earned money to help us.

If you are going to help us, I think you are entitled to make some hard questions. And to be quite frank, I do not have any good answers.

You could start by asking why do the Portuguese have 778 road motor vehicles per 1000 inhabitants while you have only 534 (OECD data for 2006). Indeed for 2006 we top the list (in front of the USA (775) known as the car nation). Or ask why we have 1835 Km of motorways (for 10 million inhabitants) while you have only 693 (for approximately half the population size).

If I were you, I would also ask why did we built 10 new pristine stadia for the Euro 2004. Insidiously some of these stadia are completely useless nowadays, being a massive financial burden in some of the municipalities where they were built. Indeed why did we organize the Euro alone when just in 2000, two much richer nations (The Netherlands and Belgium) decided to share the cost between them? Or, in 2008 Austria and Switzerland also decided to share costs.

I would also ask why did we build the longest European bridge (counting with viaducts), in the largest possible place in the Tagus river. Furthermore I would suggest you to investigate the financing strategy of that infrastructure. But you probably can guess: a variation of build now, pay later…

So, why should you bail us out? I do not know. If I were you, I would have my doubts.

But you know what is even more insidious? You are not bailing us out. Here this is going the way of a harsh recession (Greek style). Your money will be directed to pay for the loans that we made. Your money will be mostly saving the very same banks that lent money to us in a irresponsible fashion.

Do you want to know some of the “good” uses that your money will have? While the orders for the EU-ECB-IMF “troika” are still being prepared, a few things have been speculated:

• A reduction in VAT from 23% to 6% for… are you ready… Golf.
• That your money will be used to recapitalise our banks for the stress tests.

Exactly as you read: the exact same banks that be should have had an haircut on their debt holdings are instead getting some of your money!

In due honesty, I think you should protect your hard earned money. But if you are forced by the EU to send “us” (it is not really us, it is our creditors) money, here is a list of suggestions that you might consider impose. You will notice that they are not much different from some of the proposals that some of your skeptic parties make:

• Debt restructuring – Haircuts for the banks. Who lends money irresponsibly should pay a price.
• No caviar – Force people who want to buy a car or a house to make a hefty downpayment. Much of our external debt is private and related to superfluous spending.
• Transparency – There is too much secrecy with bank accounts. Too much tax evasion. Force things in the open!

When you make your rightful demands on your money I would nonetheless ask you to have one thing in consideration: Portugal is the most unequal country of Western Europe. So that you understand: The minimum wage here is 500€/month, but the president of the Bank of Portugal makes 243.000€/year (almost twice the chairman of the American Federal Reserve). I suggest you be vigilant. Actually be double vigilant: when our government proposed a cut on the salaries of Banco de Portugal, the ECB opposed. So, even when we try to correct our mistakes, the EU sometimes becomes the guarantor of the worse things.

I am not at all convinced that the EU+ECB+IMF “troika” will protect either our or your interests.

I look forward to learn more about your continued democratic discussion about this issue. It is clear that you have a rational and sensible approach to this problem. Whatever decision you take, it is deserving of respect, as are all democratic decisions. I respect and envy the quality of your democracy, of your republic. Especially because the first casualty of the EU+ECB+IMF “troika” was precisely our democratic sovereignty. I sincerely hope you never suffer such an humiliation.

With sincere respect and admiration,
Tiago
Update: This was proudly produced in the best Finnish product ever made: Linux

Lets continue the development agile DSL for music notation with Groovy. If you remember our fundamental concepts are Scores, Parts (for instruments), Phrases and Notes.

At a certain time we are typically working only on a Score, Part and Phrase (indeed we might work only on a single Score during a session). So, we would like to have a concept of default Score, Part and Phrase, and avoid referring to it (unless of, course, we want to change the default). For instance, instead of writing:

...
myScore = score(name:"Row Your Boat")
myPart = part(title: "Flute", instrument: FLUTE, channel: 0)
myPhrase = phrase(startTime: 0.0)
myPhrase.addNoteList pitchArray, rhythmArray

(pitchArray and rhythmArray are pre-defined before)
We want to write, the much simpler

1
2
3
4
5

...
score(name:"Row Your Boat")
part(title: "Flute", instrument: FLUTE, channel: 0)
phrase(startTime: 0.0)
addNoteList pitchArray, rhythmArray

All Score, Part and Phrase methods will implicitly refer to myScore, myPart and myPhrase. Note that you can still explicitly refer to them. Indeed this will be necessary has most scores.

In this first instalment (of 2) we will not deal with line 5 above. Part 1 is actually the bulk of the work. Breath deeply has this will be the tough part.

We will use Groovy ASTTransformations for this. The Groovy compiler allows us to attach code to it while it is working. We can manipulate the AST (Abstract Syntax Tree) of our code during most of the compilation stages. This means that we will need a separate program to attach to the compiler. So, if we step back we now have 3 artifacts:

1. The code to do the AST transformation (called during compilation)
2. The core DSL implementation (with all the other stuff except AST transforms)
3. Your music scripts with your score

So we need kind of a sub-project to handle this as Groovy requires a separate jar with the AST transformation code. This separate jar will have to have a descriptor file in the META-INF/services directory called

org.codehaus.groovy.transform.ASTTransformation

That is the name of the file (big one eh?). Inside it should have only one line: the fully qualified name for the class implementing the transformation (SimpleTransformation in our case).

OK, now we need to develop SimpleTransformation. This is not a trivial bit of code, I will splash it here and the it line by line (only dealing with Scores – Parts and Phrases are similar):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

@GroovyASTTransformation(phase=CompilePhase.CONVERSION)
public class SimpleTransformation implements ASTTransformation {
 
  public void visit(ASTNode[] astNodes, SourceUnit sourceUnit) {
    BlockStatement sblock = sourceUnit.getAST()?.getStatementBlock()
    List stmts = sblock.getStatements()
    int numStmts = stmts.size()
    for (int i=0;i < numStmts ;i ++) {
      Class cls = stmts.get(i).getClass()
      if (cls == ExpressionStatement) { 
        Expression es = stmts.get(i).expression
 
        if (es.getClass() == MethodCallExpression) {
          String method = es.method.text
          if (method.equals("score")) {
              Expression e = transformBinary("myScore", es)
              stmts[i].setExpression(e)
          }
        }
      }
    }
  }
...

So

• Lines 1-4 – Boilerplate of our class so that the Groovy compiler uses this. There is one important part here: the phase where the code will attach. For now I am attaching to the conversion phase. But this might change in the future (I would like to do some type analysis, but I do not even think that that is possible with Groovy. If it is possible, than analysis would have to be done at a later phase).
• 5-6 – We get the statements of the script that we are compiling
• 7 – Here we iterate through all statements. Note the for and not an each/closure. I do this because I might want to change the statement list (like adding stuff at the end – prints). That is not so easy with each/closures
• 10 – We get all expressions. This means we ignore fors, ifs, switches, function definitions, … We are not going deep, just changing methods at the top level of the code.
• 13-17 – If it is a Method Call, and it method name is called score then we apply our transformation (16) and replace the expression (line 17)

Our transformation is:

BinaryExpression transformBinary(String var, Expression expression) {
    BinaryExpression newExp = new BinaryExpression(
      new VariableExpression(var),new Token (100, "=", 1, 1), expression)
    return newExp
  }

OK, here the bulk of the work is done: We create a new BinaryExpression composed of a Variable (called myScore in our case - as per the code above), a Token and then we attach the old expression, as is. So score(name:"Row Your Boat") becomes myScore=score(name:"Row Your Boat").

Now, a confession. The 100 in the Token was a reverse engineering of an expression. I do not know where the table of options for token types is (If you know, please tell).

You will need a few imports to do the above, by the way

import org.codehaus.groovy.ast.*
import org.codehaus.groovy.ast.expr.*
import org.codehaus.groovy.ast.stmt.*
import org.codehaus.groovy.control.*
import org.codehaus.groovy.syntax.Token
import org.codehaus.groovy.transform.*

With all this you now create a jar that will have to be on the classpath of the Groovy compiler. So, this code will be used by the Groovy compiler to manipulate the AST.

Note that this code is pretty basic: It will not recurse through for/switch statements, will not go in closures, functions, etc. It will also only look at the first token in a method call expression. I will deal with this in time (not in the second part of this article). For now it is good for illustrative purposes and good for my personal needs.

Some final notes...

You can inspect an AST from groovyConsole (helps a lot), here is an example for sc=score(name:"Row Your Boat"):

AST viewing with groovyConsole

Another point if that these kind of transformations are a bit heavy in the theory and heavy in the approach. For instance, it was difficult, in netbeans to setup a project architecture that would allow easy build (an agile cycle of develop/build/test). This is of course because part of the code has to be hooked to the compiler and IDEs are not normally used to do that. It is a bit like compiling part of the compiler before going to the actual code. Well, I finally switched to emacs+gradle. Any excuse to stop using Oracle software (which netbeans nowadays is) is fair game for me.

In the second instalment we will trap method calls like addNoteList so that addNoteList listOfNotes becomes myPhrase.addNoteList listofNotes (like line 5 on the initial example above). In this case we will use some introspection to determine the method names of Score, Part and Phrase. The second part will be cooler as the bulk of the boilerplate work was done here.

You can find the code in launchpad. Note that this is still in early stages.

Comments and improvements will be most appreciated!