Perfect Storm

Preamble

This is an example of metaprogramming in Jython. I would really like to have a simpler example (either in just Python or Java), but this is directly taken from what I am doing. The idiom that I am using, Pythonwise, is a bit strange and old (I am using eval instead of __getattribute__), that is because of Jython’s limitations. This can be seen as a more advanced programming technique (If you are starting to learn programming, you might want to skip this for now, just to avoid excessive entropy in your learning process). Although this example is in Jython, it applies to many programming languages (Python, Java, Ruby, Prolog, …) but not C or C++ (or Caml, unfortunately).

The problem at hand

I am doing a selection detection workbench (to detect loci under selection). At certain points in time, I need to disallow the user to input data to a lot of entry fields, like these:

As you can see, they are all disabled.

How to do this? Option 1, go to all entry fields, one by one (more than 10, and changing) and call the method setEnabled(False). Lots of repeated code, and when there are changes I would have to add/remove a setEnabled.

Option 2. Do a piece of code to inspect my panel (a panel is what contains all the fields) object, check all object attributes that are entry fields and disable them. The point here is doing code that operates on the code itself. In this case, if one adds a new entry field to a panel, the code would automatically detect the field and disable it. How to code this?

1
2
3
4
5
6
7
8
9
10

import java.awt.Component
 
def disablePanel(panel):
    attrs = dir(panel)
    for attr in attrs:
        try:
            if eval('isinstance(panel.' + attr + ', Component)'):
                eval('panel.' + attr + '.setEnabled(False)')
        except TypeError: #Some attributes are write only
            pass

A small piece…
Line 4 (function dir) gets all attributes for the panel object.
Lines 7 and 8 do all the interesting work (eval, isinstance).
First, eval takes a string and executes it, so if you have

i = 1
i = eval('i+5')
print i

Will print 6. eval is very powerful (think about the possibilities of changing code in runtime). It is also quite dangerous, but I will not discuss that here…

isinstance checks to see if a certain object is an instance of a certain class, so

i = 1
print isinstance(i, int) # Will print True
print isinstance(i, str) # Will print False

So, back to our code
if eval(’isinstance(panel.’ + attr + ‘, Component)’):
is evaluating if panel.’attribute name’ is an instance of Component. For instance, my panel has a attribute, called core (storing the number of cores), which is a drop down list, so, when the code checks for isinstance(panel.core, Component), it will eval to True and execute the next line which is:

eval(’panel.’ + attr + ‘.setEnabled(False)’)
It will evaluate panel.’attribute name’.setEnabled(False), i.e., disable the field, in our previous example, it will do panel.core.setEnabled(False).

I will not explain the exception code as it is not important here.

So, a few lines now make it automatic to disable new entry fields, this without changing the code every time a field is added or removed (other than adding the field itself). Less code to maintain and less possibility of bugs.

I wanted just to illustrate the principle (the language used is not really important), but I need to stress out a fundamental point about this particular example in Python: Because of some Jython particularities I am using an old dialect to do this (Python gurus might be horrified), if you are using Python I recommend you to check __getattribute__ (to replace eval).

Greg Tyrelle made a very important comment regarding exploiting multiple cores and Python (which will surely be included in my next part on bioinformatics and multi-core computing).

First, my understanding of python threads is that they are not separate system level processes, but some kind of fake process that are python specific ? Trouble is that I see two separate process when I launch two Blast runs via threading ?

The other aspect of threading that I’m still not entirely clear about is how the global interpreter lock (GIL) fits into the picture. I get resource locking to prevent race conditions, but is the GIL also invoked each time an action that manipulates memory takes place in a thread ? I’ve heard this property of python makes it unsuitable for multi-core programming ?

I will trade formal correctness for clarity of explanation (namely I won’t discuss that much the difference between thread and process, as it would make this too techy and confusing).

Python uses real (i.e. native) threads. Ruby uses the so called green threads, those are “fake” (simulated). Ruby 2.0 will use native threads.

So, in theory, Python is OK in multi-core architectures. In practice there is a problem, a serious one, identified by Greg: the Global Interpreter Lock (GIL). The GIL makes it impossible for more than one thread to be executing Python code at a time. When you are dealing with Python code, even if you have many threads with many cores, only one thread can be executing Python code. This is not as serious as it looks, there are 4 ways to live with this:

• If you use a thread to start an external process, that process is not under the control of the GIL (it is a separate process), so it can run concurrently (think BLASTing something) as it is running outside Python, that is, it will be using a different core. So I think it covers one fundamental use case in bioinformatics: using external, computationally intensive, programs. In fact you can start as many instances of external programs as the number of cores you have (or even more, in case you think it will be advantageous). Note that the thread that calls the external application will block (well… depends, but for simplicity lets assume it), but your other Python threads can continue in concurrency with the application.
• This is subtle, but important: If you use CPython (the standard implementation), and you do your computationally intensive stuff in C (which makes sense - and is a common strategy - as Python is quite slow) then the C code, as long as it is not interacting with Python objects, can release the GIL and therefore make use of multiple cores. The Python code uses only one core, the C part might be using all the remaining available ones. This approach is not valid for Ruby because of the green threads issue (I am a simple Ruby newbie, so take my words with a grain of salt).
• Now… this GIL problem (or the green threads issue in Ruby) disappears if you use Jython or JRuby, as they use the JVM native concurrency mechanisms which have no notion of acquiring an exclusive lock for execution. By the way you can also use JVM based interpreters to call native (non-JVM) applications (think BLAST again, from inside Java). To put this point in another way: the GIL/green threads problem is not a language limitation, it is a limitation of the standard (C based) implementations that other implementations might not share (and the Java implementations, in fact, DO NOT).
• If you think about grids (and not multiple cores) then the problem disappears as we are then talking of different processes (even more, running on different hardware).

I am afraid of being too techy with this post (I am probably labeled as 100% computer nerd by now ), but I think Greg’s point is fundamental and required some discussion.

In my defense I would like to say that I am only writing too much about programming because I am in some sort of professional unclear phase, as soon as things get back on track I want to focus more on the biological part of things… Until there I will be writing of the issue that I know better, and that is, for better or worse, informatics…

Comments, especially constructive criticism, is, as always, welcome…

In yet another shameless promotion exercise I would like to present a easy to use interface in the area of coalescent simulation:

MODELER4SIMCOAL2

modeler4simcoal2 (m4s2) is a modeler for coalescent processes. It allows the modeling of both demographies and chromosomes (i.e., markers with linkage relationships in multiple chromosome blocks).

m4s2 is a Java Web Start application (requiring Java 1.4, available for Windows, Mac and Linux among others). It requires no installation and can be run directly from the web.

The purpose of m4s2 is to allow biologists to concentrate more on biology and the underlying models used on analysis (and less on having to learn a new computer simulation tools). We expect that m4s2 will lower the barrier for coalescent simulator use.

m4s2 was published on Bioinformatics.

Strings in Python (as in Java) are immutable. That means, that when you concatenate 2 strings what you are really doing is creating a new one from the 2 old ones. This can be very inefficient, how much? To the point of, in Jython, talking days just to prepare a String with around 400Kb, with several concats per line.

Solution? Can be found here
.

So, instead of doing str += ‘bla’, do something like str = “”.join([str, ‘bla’]).

I would also add an important tidbit: If you need to do lots of concatenations, append to a list and join at the end, don’t do join over join over join, ie, don’t do:

my_str = ''
for i in range(1000):
  my_str = ''.join([my_str, str(i)])

Do instead:

my_str_list = []
for i in range(1000):
  my_str_list.append(str(i))
my_str = "".join(my_str_list)

If you use the first dialect the result will be as bad as +=. If you use the second, things that took days will take less than a second.

Imagine that you need this kind of Java dialect in Jython:

  if (anObject instanceof aClass) {

I.e., to check if a certain object is an instance of a certain class (note, this will work if it is an instance of a subclass also)

This is quite easy to do in Jython:

 if isinstance(anObject, aClass):