DSLs involving existing classes in Groovy

Preamble: In order to understand this post you should know a little bit (a little is enough, that is how much I know) about ExpandoMetaClass and Categories in Groovy.

DSLs that involve existing classes might be a source of long term sorrow. Let me give an example: Imagine that you want to make a small DSL to handle equations, like

x = new Symbol("x")
(2 * x).differentiate(x) //Result is 2

The problem is that the * operator of Numbers doesn’t know how to handle Symbols, therefore an exception would be raised. The obvious solutions as discussed before on mailing lists and blog posts are:

Categories

Categories would solve the problem, but at the expense of polluting the source with things like

use (Something.Category) {
  //code here
}

Not a disaster, but not pretty too…

Talking about disasters…

Expando over Numbers

The idea here would be to change the behavior of Numbers to be able to handle Symbols. Code would be very clean, no need for uses…

As somebody said on the groovy mailing list: This is disaster in the making. The problem is that I change Numbers, then, for another valid reason you change Numbers, somebody else also changes Numbers… This is chaos. Or at least it would make code from different sources potentially not inter operable or exhibiting very strange, buggy, behavior. This is clearly akin to the “global variable” problem. I believe that in the long term and with big software projects, this approach is a dead end.

Enter Python

Python actually has a workaround (I will not call it a clear, beautiful solution) that might be somewhat useful here. Imagine that you do

1 + x

The default 1 (default class for number) is not able to handle the symbol. For python that is OK, it will try to call a “right add” method of x (Search for __radd__ in this page). So, the default behavior is not to raise an exception if the left object cannot handle the operator, but to try to call the “right” version on the right object (if it fails then raise).

Not perfect, but might be just enough to avoid Expando in anger.

I do believe that people still don’t appreciate the consequences of Expanding core classes and the interop disaster that that can entail.

Posted in DSL, groovy, python | 5 Comments »

Revisiting Groovy performance issues

I tried to drill down the Groovy performance issue that I had with what is in practice a text processing exercise.

The original code was written in Groovy (and then ported to Java, not the other way around), but as I was in a hurry it was written in idiomatic Java (I am too much of a Groovy newbie to be able to write in idiomatic Groovy if I am in a hurry). Ted Naleid left some great suggestions on how to be more groovyish.

Anyway, I took my original code and tried to understand what was going on, here are my findings.

Replacing duck typing with explicit typing took a minute out (from 4m to 3m).

Converting this

iCase.each {
    if (jCase.contains(it)) {
        isDifferent = false
    }
}

to this

if (jCase.contains(iCase[0]) || jCase.contains(iCase[1])) {
    isDifferent = false
}

took 1m10s (from 3m to 1m50s) - This is in a inner loop part.

8 seconds were gained by changing this:

for (int j=i+1; j<indivs .size(); j++) {

into

int iSize = indivs.size()
for (int j=i+1; j<isize ; j++) {

As inline comments you can find how much time each line took in the
following inner loop:

for (int j=i+1; j<isize ; j++) {
    int jPos = indivPos[indivs[j]] //~20s
    String jCase = lineTok[jPos] //~10s
    if (jCase.equals('NN')) continue //~8s
    boolean isDifferent = true //2s
    if (jCase.contains(iCase[0]) || jCase.contains(iCase[1])) {
        isDifferent = false //7s
    } //whole if is ~ 30s  - 23 condition, 7 assignment
    if (isDifferent) counts[indivs[i] + indivs[j]] += 1 //5 secs
}

The only stunning thing is the time lost at indexing String arrays (and maps, but that I can understand).

This text is being written as I was changing and trying things, I gained 20s from
minor changes of which I lost track . I am currently at 1m30s (down from the
original 4m and comparing with Java’s 4s).

Posted in groovy, java | 10 Comments »

Groovy performance speed

I think that language performance (from a speed point of view) is highly overrated. There are many factors that are more important. Well, on the time front alone, developer time is normally more important: The time spent groking code is normally more expensive than the running time of an application. Of course, there are many other important points, too much to enumerate (portability, declarativeness, readability, …). If performance was the fundamental variable, we would all be using assembler.

I am currently doing a bit of code to go over tens of millions of lines of text, while comparing separate columns.

I did a little piece of Groovy code to go over all those lines. The performance results were abysmal, so I decided to do a program in Java (copying the Groovy code to a Java file and converting in a very direct way). For 3000 lines of text here are the results (remember, this is to process hundreds of millions):

$ time java Do

real    0m4.427s
user    0m4.384s
sys     0m0.040s

$ time groovy do.groovy

real    2m53.303s
user    2m47.650s
sys     0m0.668s

4 seconds against 2mins 53 seconds. This is not serious as it is possible to write all Groovy intensive parts in Java. But, even so, it is too much.

The code? (Afterwards, some speculation and profiling)

Groovy:

...
while ((line = reader.readLine()) != null) {
    lineTok = line.tokenize()
    if (lineTok.size() == colIndiv.size() + 11) {
        for (int i=0; i<indivs .size()-1; i++) {
            int iPos = indivPos[indivs[i]]
            def iCase = lineTok[iPos]
            if (iCase.equals('NN')) continue
            for (int j=i+1; j<indivs.size(); j++) {
                int jPos = indivPos[indivs[j]]
                def jCase = lineTok[jPos]
                if (jCase.equals('NN')) continue
                def isDifferent = true
                iCase.each {
                    if (jCase.contains(it)) {
                        isDifferent = false
                    }
                }
                if (isDifferent) counts[indivs[i] + indivs[j]] += 1
            }
        }
    }
}

Java:

...
while ((line = reader.readLine()) != null) {
    String[] lineTok = line.split(" ");
    if (lineTok.length == colIndiv.size() + 11) {
        for (int i=0; i<indivs .size()-1; i++) {
            int iPos = indivPos.get(indivs.get(i));
            String iCase = lineTok[iPos];
            if (iCase.equals("NN")) continue;
            for (int j=i+1; j<indivs.size(); j++) {
                int jPos = indivPos.get(indivs.get(j));
                String jCase = lineTok[jPos];
                if (jCase.equals("NN")) continue;
                boolean isDifferent = true;
                if (jCase.indexOf(iCase.charAt(0)) > -1 ||
                    jCase.indexOf(iCase.charAt(1)) > -1 ) {
                    isDifferent = false;
                }
                if (isDifferent) counts.put(
                    indivs.get(i) + indivs.get(j),
                    indivs.get(i) + indivs.get(j) + 1);
            }
        }
    }
}

I cursorly run the Java profiler, though I did not spent much time on it, it seemed that (speculation alert!) Groovy was spending sometime on metaclassing/proxying parts. I wonder if the “defs” were making things much slower? Maybe if I had properly typed my loop variables (instead of being lazy and def duck typing) things would have ran smoother. If that is the case, then one more reason against duck typing (others being helping the IDEs and automated code tools and for debugging purposes)

Posted in groovy, java | 8 Comments »

Introducing Cognitive Consonance

Cognitive Consonance is a spin off of my original blog, Perfect Storm. Cognitive Consonance is about programming (for science Perfect Storm is still the place to go) from a declarative and agile perspective (high-level programming, domain specific languages, efficiency). I mainly develop under the JVM in Java and Groovy (though Scala and Ruby are also favorites). As I am the developer of the Biopython population genetics’ module, some Python content will appear.

Time and patience permitting I will be cross posting relevant old content from Perfect Storm. Future posts that are relevant to both blogs will also be cross posted.

Posted in meta | No Comments »

Groovy/Scala/Ruby/Python on JVM

There seems to be some competition in the field that can be vaguely defined as “The next Java”(TM).

I don’t know if there will be a “next Java” to start with. Things seem to shape up in way where the JVM is our common interoperability platform and on top of it we have a an ecology of JVM based languages.

I have used Jython quite a lot but have several doubts about it, not only on the current status of Jython (lags a bit behind CPython) but I also deslike Python (when compared with the other languages dicussed here). As such I decided to evaluate the other Scala, Ruby and Groovy.

I have done a couple of small projects in Scala (A prototype DSL for modeling malaria resistance is available here) and JRuby. I am now starting with Groovy, and I think I’ve found my new love. Here I will try to explain why, among Groovy, Scala and JRuby, I have chosen Groovy. To preempt any religious war idea, I would like to say I have full respect for Scala, Ruby, which are, with Caml and Prolog among my favorite languages (for a true crusade and flame ask me for my opinion about Perl or Visual Basic 6 ).

Steven Devijver suggests that Groovy is the language with more syntatic similarities with Java. I would say that, not only that, but on the semantics and everything, Groovy is the closest language to Java. And that is a good thing. The world (both in programming languages and all the rest) is never revolutionary. Revolutions, when they rarely happen, are either a disgrace or are not that much of big change below the surface. People normally prefer (for good and bad reasons) the path of least short term pain. Groovy delivers that: almost 0 cost in starting to code coming from a Java background. Most importantly Groovy does that but still delivers most of the new goodies. This is actually the cornerstone of my argument: path of least pain while delivering the good stuff (in some cases better than the competition, as we will see).

Let me start with the fundamental reasons why I dismiss JRuby (which is, nonetheless, my second option after Groovy). First, I would like to say, very honestly, that the work of the JRuby guys is nothing short of outstanding! But I have 3 problems:

1. One, by definition, JRuby is based on Ruby, a language from outside the JVM. That means semantic hurdles, coupling issues between the two worlds (think, e.g., libraries)
2. Most importantly (but connected with the first point): Typing. I am a bit far away from computing issues currently (I work with Malaria currently, so excuse me if I mess strong/explicit typing and such) but clearly the typing system of Ruby make like hard for IDEs (think IDEs to neded to tame those over engineered Java APIs) and automated tools around code. Debugging without explicit typing is also a pain in a big program (I actually suffered my first debug nightmare with typing systems with Caml, arguably the mother of Scala). Some might say that Scala type inference and Groovy duck typing also are problematic in this respect; while the argument might be correct both languages have mechanisms to support typical Java explicit/strong typing and as such profit from IDEs and automated analysis tools.
3. Ugly perlisms. Although I have read somewhere that those might be deprecated in the future.

Ah… Scala… Mats Henricson argues that Scala is the only option because of elegance regarding multicore computing. I fundamentally disagree with his point - multicore programming is fundamental but Scala is not really a good solution, but before we get there, lets talk about other Scala issues.

Type inference. I have some experience with the “mother” of Scala, Caml. Type inference in Caml is really elegant: I don’t remember a single case of it failing and requiring the programmers’ help in discovering the type of a parameter. That is not the case with Scala, several times the compiler seems to be “lost in translation”. Some might say that this is because of JVM imposed constraints, but if that is the case then it would raise the argument of bringing a language with a foreign semantics to the JVM and the ugliness attached to the process.

My biggest peeve? Metaprogramming. I won’t give you my opinion about it because it really doesn’t exist. It is on the Scala wiki in the section “future”. I am sorry, but a 21st century language where meta programming is absent can only be called in “beta stage”. As a side note, there seems to be something lost in the ML branch of functional programming from Lisp in this regard (no introspection and such), that is a shame (How is Haskell in that respect?).

Ok, multicore computing. This is an area where I have some experience in the JVM: [Shameless plug] I invite you to have a look at my Java Web Start, Jython based, multicore aware evolutionary biology workbench LOSITAN. Furthermore I have written tutorials for the multicore paradigm and bioinformatics:

Bioinformatics, multi-core CPUs and grid computing: Introduction (1/4)

Bioinformatics, multi-core CPUs and grid computing: User perspective (2/4)

Most importantly in this context: Bioinformatics, multi-core CPUs and grid computing: developer perspective (3/4)

Mats argues that Scala Actors and immutable data types provide a simple and elegant solution to the extremely complex problem (I am calling it extremely complex, because I think it really is) of concurrent programming. Immutable data types… Does anyone believe that the hordes of existing Java developers/programmers are ready and willing to do radical conceptual jump to immutable data types? The change from C++ to Java was minor in terms of semantics, even the change from C to C++ was much less radical that a change requiring to “get rid of all variables”. How do you think the majority of programmers will react when you say: “Forget variables”? More, as Scala allows for imperative type of programming, what do you think most programmers idiom wil be: Imperative or functional? To makes things worse, in Scala a immutable is called a “val” and the mutable a “var”. Am I the only only picturing hordes of developers, with tight deadlines just swapping L’s for R’s?

I speak for myself here: in spite of having probably more experience with “immutable” languages (Prolog a lot, Caml a bit) than most developers, when I wrote Scala code, my reasoning was so tainted by “real world” imperative languages that it was really hard to write in a functional dialect. I have the background, enough free time, and the motivation to write functional code, but it was hard to get back in that mindset.

Scala only apparently solves the multi core problem. Give it to a typical developer and he will write imperative code, unless you put a functional zealot behind him (and give the said zealot a strong, resistant whip).

How to address the multicore issue? Clearly we have a problem here. A few ideas:

• In many applications there is no big need to go multicore. In some cases lets not try to solve a problem that doesn’t exist in the first place.
• Many multicore applications can survive very well with simple concurrency management. Not all applications require a PhD in concurrent programming.
• Scala and the like. For those who can and are willing to go functional, why not? I have nothing against that. My only argument is that it won’t be mainstream.
• The way of PAIN. Most developers will continue to use old languages and paradigms and SUFFER with it. Only after much suffering there will be motivation to try out new things and, say, endure the pain of learning a new paradigm. That suffering still hasn’t happen, only after this becomes a big problem, there will be interest in accepting new solutions.
• A silver bullet that can be attached to the current programming paradigm. Sometimes it happens. Don’t misunderestimate (silly Bushism intended) the power of a “Black Swan” (A reference to Taleb’s book where he discusses the impact of the unexpected important events).

To finalize, I would like to say that I am not sticking with Groovy out of being conservative. Groovy seems to beat the competition in many areas (the biggest example is metaprogramming) and strikes a very good balance between being a “small evolutionary step” and delivering the goodies.

To really finalize, a caveat: my Groovy knowledge is still limited, one of these days you might read a post where I apologize for having written this

Originally posted on Perfect Storm

Posted in groovy, java, jvm, python, ruby, scala | No Comments »