Alan's Ramblings

Why I quite like Scala's type system

Wed, 23 Nov 2011 16:42:03 GMT

My colleague Gary pointed me at a blog post by Stephen Colebourne that is critical of Scala, along with a related Reddit thread, and asked me what I thought. Some points are fair enough, for example the Scala community has an elitist Functional Programming cadre who clustered around the Scalaz library who tend to be particularly obnoxious. I've already experienced the ire of one of their members, and from the private emails I've received I know I'm not the only one to have been the butt-end of his displeasure. As a generalisation they seem to have come from a Haskell background and it appears that they won't be satisfied until Scala turns into Haskell. There's an obvious solution to that which I'll refrain from pointing out... ;-)

The blog post linked to above recycles common criticisms of two Scala features without actually considering why they might have been provided in the first place. The features in question are Higher-Kinded Types (AKA Constructor Polymorphism) and Implicit Parameters. I'd expect a more substantial argument than "It's too complicated and hard" - that just seems lazy. Yes, it's complicated, generic programming nearly always is. For example, Java Generics are commonly accepted as being A Good Thing In General yet I can't see how any language feature that has a 297-page FAQ is anything other than complicated.

There are many, many blog posts, mailing list discussions and academic papers out there which discuss Scala's use of Higher-Kinded Types and Implicit Parameters. Two of the best academic papers (both PDFs) are:

Generics of a Higher Kind Moors, Piessens & Odersky
Fighting Bit Rot with Types (Experience Report: Scala Collections) Odersky & Moors

While they are interesting and well worth a read they aren't exactly written in a way you'd use to describe the benefits of Scala to your mates in the pub. However, hidden away in the second paper is a little example that shows why the combination of Higher-Kinded Types and Implicit Parameters is so powerful, and is such an advance over Java.

Scala's Implicit Parameters are really just a house-trained version of C++ user-defined type conversion operators, nothing more. They are a way of allowing you to specify how one type of thing should be converted into another.

Higher-Kinded Types are a little more difficult to grasp. Java has types such as List which are 'kinded types', e.g. a List<Integer> is a 'kind of' list. However in Scala you can go beyond that and define types which are the equivalent of L<T> where both L and T are type parameters. Why is that useful? Well, because it allows you to write code that manipulates all sorts of list-like things irrespective of their actual types. Like Java Generics it's really of most use to the writers of libraries rather than day-to-day use, but in combination with Implicit Parameters it does allow things you just can't do in Java.

OK, on to the example. To show these two language features in action together, we'll use the Scala BitSet class which implements a set using bits in integers for compact storage:

scala> val b = BitSet(2,4,8)
b: scala.collection.immutable.BitSet = BitSet(2, 4, 8)

Straightforward so far. Now let's map that bitset by adding 1 to each element:

scala> b.map(v => v + 1)
res0: scala.collection.immutable.BitSet = BitSet(3, 5, 9)

OK, that's what we would expect - we get back a new BitSet. Now let's do something that would be impossible in Java's type system. Let's map over the bitset again, but this time the returned values will be something that can't be stored in bits. Let's map the bitset contents to String:

scala> b.map(v => "bit " + v)
res1: scala.collection.immutable.Set[java.lang.String] = Set(bit 2, bit 4, bit 8)

Now let's try mapping the contents to floating point numbers:

scala> b.map(v => v * 1.5)
res3: scala.collection.immutable.Set[Double] = Set(3.0, 6.0, 12.0)

So what we get back is not just a set of different values (and types) to the ones we started with, the container those types are stored in is different as well - BitSet becomes Set[Double]. That transformational magic is not a language-level feature, it's all done in at library level by making use of both the powerful (and yes, complex) type system and implicit parameters - for a full explanation, see the papers above. The important takeaway is that you get the appearance and behaviour you might expect of an untyped language such as Python (notice, the types of b and c are never declared) and yet Scala is actually strongly typed.

Here's the coup de grace. Let's write something that stores the return of the integer map operation in a variable, then let's assign the result of the floating point map operation to the same variable. I'm doing this in the interpreter's 'paste' mode so it's compiled all in on go:

scala> :paste
// Entering paste mode (ctrl-D to finish)

val b = BitSet(2,4,8)
var c = b.map(v => v + 1)
c = b.map(v => v * 1.5)
^D
// Exiting paste mode, now interpreting.

<console>:10: error: type mismatch;
 found   : scala.collection.Set[Double]
 required: scala.collection.BitSet
       c = b.map(v => v * 1.5)
                ^

Note that's a compile time error, not a run-time one, and the parts involved are all library classes, not fundamental language features. How does it work? Well, the type of c is inferred by the compiler so it knows that c is a BitSet. It can also work out at compile-time that the return type of the second map operation will be Set[Double], and that's not type-compatible with BitSet, so you get a compile-time error. Yet there's not an explicit type declaration in sight. Python, eat your heart out ;-)

I don't just think this is cool because I'm some sort of strict-typing bigot, one of my teensy claims to fame is that my name is in the Perl AUTHORS file so feel I've earned my dynamically-typed Scout's badge. However, given a choice between strong and weak typing I'd prefer strong typing just because it helps protect me from my own stupidity. Obviously the example above is not the only possible use of these Scala language features, but I do think it's a good, simple illustration of how they can be used. And no, this kind of power isn't easy to wield. As the second paper listed above says:

Getting this design right was very hard, however. It took us about a year to go from a first sketch to the final implementation. In doing this work, we also encountered some dead ends.

However saying that Scala shouldn't provide support for powerful concepts just because they are hard to get your head around seems completely wrong-headed. The difficulty is in the concepts more than it is in the implementation. The blog post that prompted this points to the Fantom language as a superior alternative to Scala, yet the Why Fantom page contains the following:

Currently Fantom takes a limited approach to generics. There is no support for user defined generics yet. However, three built-in classes List, Map, and Func can be parameterized using a special syntax. For example a list of Ints in Fantom is declared as Int[] using the familiar array type syntax of Java and C#. This trade-off seems to hit the sweet spot where generics make sense without complicating the overall type system.

So pretending the need for Generics doesn't exist seems to be their solution. Boggle.

Stupid BT router...

Wed, 23 Nov 2011 14:12:00 GMT

I have the misfortune to own a BT Business 2Wire router, and bleaklow.com sits behind that. A while back it threw a fit and reconfigured itself and when I beat it into submission I didn't notice that it had also lost the port mappings for bleaklow.com. As a result I've been off-air for quite some time. Bloody BT...

Hints on creating 256-colour themes for vim

Wed, 21 Sep 2011 20:06:00 GMT

Most existing vim colour themes are designed for the graphical version of vim, gvim. They don't work properly at all if you are running vim in a terminal, and there are very few themes available for vim in terminal-mode. You are pretty much obliged to make your own if one of the small number of available ones doesn't suit. Here's a list of hints and tips that I discovered in the process of creating one:

If your shell's setting for $TERM is xterm you won't get colour support. It needs to be xtermc or you need to add a set term=xtermc in your .vimrc.
Even with the terminal type set you may not get 256 colours if you are using gnome-terminal. To fix that, put set t_Co=256 in your .vimrc.
In vim, type :help highlight to get instructions on how to specify colours in vim.
To make the process of authoring a colour theme easier, install the Mkcolorscheme.vim script in your vim plugin directory, uncommenting the four commands at the top of the file. Read the instructions in the plugin, they give you hints on how to identify the colours currently being used by syntax elements.
Open a new subwindow in vim and type :so $VIMRUNTIME/syntax/hitest.vim. That will display a table of the current settings, which will be updated as you interactively change colours.
Open this colour chart in your browser, it gives you the colour numbers you'll need to set the colours.
Open up a source file of the type that you want to set the syntax colouring for, put the cursor over an element and use the :GetSyntax command to identify the element type, followed by the appropriate:highlight command to set the element colour.
When you are done, use the :Mkcolorscheme command to generate a set of commands to generate the colour scheme settings. Delete any lines containing cleared as they aren't valid.
Save the colour scheme to a file in your vim colors directory and load the scheme with a colorscheme command in your .vimrc.

Beautiful Scala

Tue, 23 Aug 2011 08:24:23 GMT

This is, of course, beautiful:

def msort[T](less: (T, T) => Boolean)
      (xs: List[T]): List[T] = {
  
    def merge(xs: List[T], ys: List[T]): List[T] =
      (xs, ys) match {
        case (Nil, _) => ys
        case (_, Nil) => xs
        case (x :: xs1, y :: ys1) =>
          if (less(x, y)) x :: merge(xs1, ys)
          else y :: merge(xs, ys1)
      }
  
    val n = xs.length / 2
    if (n == 0) xs
    else {
      val (ys, zs) = xs splitAt n
      merge(msort(less)(ys), msort(less)(zs))
    }
  }

scala> val intSort = msort((x: Int, y: Int) => x < y) _
intSort: (List[Int]) => List[Int] = <function1>
scala> intSort(List(3, 5, 1, 6, 2))
res3: List[Int] = List(1, 2, 3, 5, 6)

This is an example of a recursive merge sort implemented using a technique known as currying, expressed in the language Scala.

Some context: I was looking for a Java-based scripting language to act as 'glue' for my Java interface to OziExplorer which I'm using for a fenceline survey I'm doing for the Moors For The Future project. I initially looked at Groovy and came across an intriguing comment by James Strachan, the creator of Groovy:

I can honestly say if someone had shown me the Programming in Scala book by by Martin Odersky, Lex Spoon & Bill Venners back in 2003 I'd probably have never created Groovy.

That set me off on an exploration of the Scala website which has lots of good tutorials and links, including one to an online interpreter you can type examples into. Scala is a really intriguing language - it sits on top of the JVM so you get 'for free' Java's cross-platform support and access to the huge Java ecosystem, but Scala also blends together the features of some of the most influential languages that preceded it:

The 'everything is an object' nature of Smalltalk or Ruby
The imperative nature of languages that have C or Java heritage
Functional programming as per ML or Haskell
Strongly typed but with sophisticated type inferencing that makes it look & feel similar to Python or Groovy
Uniform access to methods and fields, like Eiffel
Actor-based concurrency similar to Erlang

Scala also adds new features of its own as well, such as abstract types, traits and extractors. It's also very easy to write domain-specific languages in Scala. James Strachan wrote a good summary of Scala, from which the above quote is taken.

I also ordered a copy of Programming in Scala, Second Edition which I'm enjoying reading and can thoroughly recommend. Unlike many programming books it's well written with a light but not overly-chatty style. It explicitly assumes you have programming experience, so it picks up pace pretty quickly. As I've been going through the book, most of my "Why did they do that?" questions have had good answers, which gives the impression that Scala has been very carefully thought through - always a good sign.

Now all I have to do is to think of a substantial project I can use Scala on :-)

Childhood autism spikes in geek heartlands

Fri, 24 Jun 2011 11:10:15 GMT

From the New Scientist:

Childhood autism is two to four times as common in Eindhoven, the centre of the Dutch information technology industry, as it is in two comparably sized Dutch cities with far fewer IT employees.

The result supports the suggestion that people who work in hi-tech engineering and computing industries, which demand the kinds of systemising and analytical skills often seen in people with autism, are more likely to have autistic children too.

New Scientist, 20 June 2011, Childhood autism spikes in geek heartlands

Who'd have thought it :-)