Comparing apples with oranges: OCaml vs Java in teaching

Can we compare programming languages? Can we ever say that subject to some criteria one language is better than the alternatives? Very hard, in general. But this year I was in a position to do this comparison in a way which even though far from perfect is at least vaguely meaningful.

We teach our freshmen two programming languages simultaneously in Year 1 Term 1: Java and OCaml. We don't assume the incoming students have programming experience although, of course, some of them do. But in terms of backgrounds this is as clean and as homogeneous as you can hope for. In Week 5 and Week 6 two of the assignments are shared between OCaml and Java. In total this means 6 short algorithmic programming exercises, administered via HackerRank (and tested for correctness and efficiency). The assignments were completed by about 180 students each.

1. zipping three lists (arrays in Java).
2. unifying two strings with constants and variables.
3. finding the best trade in a historical sequence of prices.
4. running average of consecutive elements in a list (array for Java).
5. running average of k consecutive elements in a list (array for Java).
6. finding longest increasing sequence in a list. 

Here are the question-by-question statistics collected by HackerRank:

	median time		median score
assignment	OCaml	Java	OCaml	Java
1	1 hr 17 min	1 hr 3 min	3/3	5/5
2	1 hr 35 min	2 hrs 45 min	3/3	5/5
3	1 hr 29 min	0 hrs 32 min	0/3	5/5
4	0 hrs 44 min	0 hrs 22 min	4/4	5/5
5	0 hrs 23 min	0 hrs 34 min	3/3	5/5
6	2 hrs 13 min	1 hr 39 min	2/3	5/5

One very important caveat is that all students tried each assignment in OCaml first, then in Java in the following week. They were therefore familiar with the algorithmic solution when attempting it in Java, whereas in OCaml they saw the problem as new. However, some of the list-based algorithms are quite different than their array-based counterparts, so some element of novelty remains.

The only thing that stands out from this comparison is that questions (3) and (6) had efficient solutions in which keeping track of an array index was much easier than expressing the problem recursively using lists (try it!), which seems to have made the OCaml version harder than the Java.

Following this exercise in comparative programming (on simple algorithmic problems) I also gave students a survey to gage their perception of the comparative advantages and disadvantages of the two approaches, asking them to agree or disagree with the common claims made when comparing functional and imperative or OO programming. These claims are almost never, to my knowledge, backed by data but supported by arguments which, however thoughtful, are largely speculative.

I found that:

• only 19% of students were bothered by off-by-one array errors
• 54% of students found functional programs more concise and enjoyed that
• only 20% appreciated using the OCaml interpreter for rapid prototyping or quick testing
• 57% students prefered OO-style to OCaml-style polymorphism
• 40% of students considered that pattern matching in OCaml improved their productivity
• opinions were evenly divided on whether type inference and the stricter discipline of OCaml led to bug avoidance
• 35% thought OCaml's type system got in the way and reduced their productivity
• only 30% thought immutable data was helpful in avoiding programming errors
• whereas 40% thought immutable data was inconvenient to work with 
• 59% thought structural recursion (on a data structure, using pattern matching) was helpful
• only 14% thought OCaml programs are more readable than Java programs
• the opinions were evenly split on whether programming in OCaml was enjoyable 
• 78% liked programming in Java. 

**

This post is loosely related to my previous post, in which I was worrying about the fact that language comparisons, from the point of view of programmer productivity, are a rather neglected area of research. This is not a proper study, certainly, but there are some surprises in there for me. I personally much favour OCaml over Java, and the fact that the putative advantages of functional programming are not immediately apparent to the novice programmer is disappointing. In comparison with Java, the initial appeal seems questionable, for most students at least. On the other hand, certain features of functional programming (e.g. patterns) seem to be broadly liked. 

Maybe I will have time to conduct this study more thoroughly next year. Any suggestions for how to make it better would be warmly appreciated.