Why I'm ditching the Arduino software platform
I'm getting set up for my next project and decided to update my development environment. I've finally decided to entirely ditch the Arduino software environment and just use the boards. I stopped using the Arduino IDE some time ago, but now I'm going whole hog and ditching the Arduino library as well. Why? Well, it's simple:
Significant parts of it are pile of junk.
I know that's a pretty strong statement, so I better back it up with evidence. OK, let's start with the hardware serial IO code. Before version 1.0 of the Arduino platform, although reading from the serial ports was interrupt-driven, writing wasn't. Rather, the code went into a spin loop, polling the transmit status bit until the USART was idle before sending the next character. Why was that a problem? Well if you wrote a 80-character string at 9600 baud it would take (8 bits + 1 start bit + 1 stop bit) * 80 / 9600 = 0.083, i.e. 83 milliseconds. That's a huge amount of time for the CPU to be spending just to do some output. I found a number of posts where people were complaining that doing reasonable amounts of IO screwed up all the other bits of their sketches, and no wonder. Admittedly the Arduino 1.0 release notes say that's been changed so that output now uses interrupts as well, but that's not the end of the problems.
Let's take a peek at the HardwareSerial.cpp class. First thing to note is that two 64-byte buffers are allocated for each USART, even if it isn't used. That's 128 bytes on a Duemilanove and 512 bytes on a Mega, or 6% and 12% of the available SRAM respectively. On the Duemilanove that's reasonable as there's only 1 UART, but on the Mega it represents a significant waste of precious memory when only 1 USART is normally going to be in use.
OK, let's look at the new write() function that does interrupt-driven output:
size_t HardwareSerial::write(uint8_t c)
{
int i = (_tx_buffer->head + 1) % SERIAL_BUFFER_SIZE;
// If the output buffer is full, there's nothing for it other than to
// wait for the interrupt handler to empty it a bit
// ???: return 0 here instead?
while (i == _tx_buffer->tail)
;
_tx_buffer->buffer[_tx_buffer->head] = c;
_tx_buffer->head = i;
sbi(*_ucsrb, _udrie);
return 1;
}
Is there a problem? Let's look at the definition of _tx_buffer:
struct ring_buffer
{
unsigned char buffer[SERIAL_BUFFER_SIZE];
volatile unsigned int head;
volatile unsigned int tail;
};
Oh dear. head and tail are declared as int, i.e. 16 bits, 2 bytes. They are accessed by both the write routine and the interrupt service routine that actually transmits the data yet there's no locking in the write routine so the accesses aren't atomic. Why is that an issue? Well, the avr-libc documentation makes it clear:
A typical example that requires atomic access is a 16 (or more) bit variable that is shared between the main execution path and an ISR. While declaring such a variable as volatile ensures that the compiler will not optimize accesses to it away, it does not guarantee atomic access to it.
The documentation goes on to explain the sorts of symptoms you'll see if you ignore this, follow the link above if you want the full details. This is inexcusably shoddy code - the constraints on accessing variables that are shared between ISR and non-ISR code are well-known. What really concerns me is that people will use the Arduino code as an example of 'good' AVR code and it isn't, in many places it's frankly awful.
"So what?" you say, "That's only one chunk of code that's a bit naff." Unfortunately it's not an isolated instance. Let's move on now to look at one of the newer features that has been added to the Arduino platform, the re-implemented String class. Ok, let's build a minimal program that uses it:
#include "WString.h"
int main(void) {
String bloat = "hello world";
return 0;
}
And let's build it:
WString.cpp: In member function ‘int String::lastIndexOf(char, unsigned int) const’: WString.cpp:503:38: error: comparison of unsigned expression < 0 is always false [-Werror=type-limits] WString.cpp: In member function ‘int String::lastIndexOf(const String&, unsigned int) const’: WString.cpp:519:63: error: comparison of unsigned expression < 0 is always false [-Werror=type-limits]
Sigh. One would think that the Arduino developers would at least turn on warnings when they are compiling their code, but they don't. And in this case, the consequence is a bug. So, temporarily comment out the offending lines so we get a successful build, and:
/opt/avr-gcc/bin/avr-size build/test.elf text data bss dec hex filename 10194 20 5 10219 27eb build/test.elf
Can that really be right? 10K for a one-line program? Unfortunately it is. Any mention of String pulls in the entirety of the class, as well as all the other avr-libc routines it references. So on a Duemilanove that only has 32k to start with, a third of the available memory is gone before you start. At the time the class was being rewritten I expressed my opinion that it was probably a bad idea and that the Arduino developers really needed to target the platform they actually had and not the one they wished they had. And that's not the end of the issues with the String class - on a constrained-memory platform such as the AVR, providing a class like String that relies on malloc, creates lots of temporaries, fragments the (tiny) heap and has no real ability to deal with out-of-memory conditions is a recipe for problems, problems that will manifest themselves as random, mysterious and un-diagnosable run-time errors. And sure enough, a quick google shows that's exactly what tends to happen - just about the worst possible outcome for a platform that's targeted at neophytes.
That's just two examples - there are others as well, such as the well-known performance problems with pin access, which may be up to 50x slower that direct pin access. In fact the only two remaining parts of the Arduino libraries that I still use are the millisecond clock and the serial IO, and they are easy enough to replace, so that's what I'm doing.
While I applaud the aims of the Arduino project, the realities of the restricted hardware platform have to be taken into consideration. In addition, one of the aims of the project is to:
provide a well-designed, maintainable, and stable platform for the futureand despite its unquestionable success on many other fronts, on that one I feel the Arduino platform is less than entirely successful. I for one won't be using any of the software any more, it's just not what I consider to be acceptable quality.
Scala snippet of the day
The following is useful for benchmarking code from inside the REPL:
def timed(op: => Unit) = {
val start = System.nanoTime
op
(System.nanoTime - start) / 1e9
}
Then you can use it like this:
scala> timed { Thread.sleep(5000) }
res0: Double = 5.010083526
That's syntactic sugar for an anonymous function definition and a call of timed. If a function takes only a single parameter you can pass it inside { }, so if we wrote that out longhand it would be:
scala> def fn = Thread.sleep(5000) fn: Unit scala> timed(fn) res0: Double = 5.010088248
The other neat bit is the use of Scala's pass-by-name mechanism to pass the op parameter. That's the funny-looking op: => Unit argument list to timed. Normally Scala uses by-value parameters, i.e. the line timed(fn) would result in the evaluation of fn, and then passing the returned value (in this case, nothing) into timed. However by using pass-by-name, the evaluation of fn takes place inside timed, and by wrapping it inside timing instrumentation we can time how long op takes to execute.
Why I quite like Scala's type system
My colleague Gary pointed me at a blog post by Stephen Colebourne that is critical of Scala, along with a related Reddit thread, and asked me what I thought. Some points are fair enough, for example the Scala community has an elitist Functional Programming cadre who clustered around the Scalaz library who tend to be particularly obnoxious. I've already experienced the ire of one of their members, and from the private emails I've received I know I'm not the only one to have been the butt-end of his displeasure. As a generalisation they seem to have come from a Haskell background and it appears that they won't be satisfied until Scala turns into Haskell. There's an obvious solution to that which I'll refrain from pointing out... ;-)
The blog post linked to above recycles common criticisms of two Scala features without actually considering why they might have been provided in the first place. The features in question are Higher-Kinded Types (AKA Constructor Polymorphism) and Implicit Parameters. I'd expect a more substantial argument than "It's too complicated and hard" - that just seems lazy. Yes, it's complicated, generic programming nearly always is. For example, Java Generics are commonly accepted as being A Good Thing In General yet I can't see how any language feature that has a 297-page FAQ is anything other than complicated.
There are many, many blog posts, mailing list discussions and academic papers out there which discuss Scala's use of Higher-Kinded Types and Implicit Parameters. Two of the best academic papers (both PDFs) are:
- Generics of a Higher Kind Moors, Piessens & Odersky
- Fighting Bit Rot with Types (Experience Report: Scala Collections) Odersky & Moors
While they are interesting and well worth a read they aren't exactly written in a way you'd use to describe the benefits of Scala to your mates in the pub. However, hidden away in the second paper is a little example that shows why the combination of Higher-Kinded Types and Implicit Parameters is so powerful, and is such an advance over Java.
Scala's Implicit Parameters are really just a house-trained version of C++ user-defined type conversion operators, nothing more. They are a way of allowing you to specify how one type of thing should be converted into another.
Higher-Kinded Types are a little more difficult to grasp. Java has types such as ListList<Integer> is a 'kind of' list. However in Scala you can go beyond that and define types which are the equivalent of L<T> where both L and T are type parameters. Why is that useful? Well, because it allows you to write code that manipulates all sorts of list-like things irrespective of their actual types. Like Java Generics it's really of most use to the writers of libraries rather than day-to-day use, but in combination with Implicit Parameters it does allow things you just can't do in Java.
BitSet class which implements a set using bits in integers for compact storage:
scala> val b = BitSet(2,4,8) b: scala.collection.immutable.BitSet = BitSet(2, 4, 8)
Straightforward so far. Now let's map that bitset by adding 1 to each element:
scala> b.map(v => v + 1) res0: scala.collection.immutable.BitSet = BitSet(3, 5, 9)
OK, that's what we would expect - we get back a new BitSet. Now let's do something that would be impossible in Java's type system. Let's map over the bitset again, but this time the returned values will be something that can't be stored in bits. Let's map the bitset contents to String:
scala> b.map(v => "bit " + v) res1: scala.collection.immutable.Set[java.lang.String] = Set(bit 2, bit 4, bit 8)
Now let's try mapping the contents to floating point numbers:
scala> b.map(v => v * 1.5) res3: scala.collection.immutable.Set[Double] = Set(3.0, 6.0, 12.0)
So what we get back is not just a set of different values (and types) to the ones we started with, the container those types are stored in is different as well - BitSet becomes Set[Double]. That transformational magic is not a language-level feature, it's all done in at library level by making use of both the powerful (and yes, complex) type system and implicit parameters - for a full explanation, see the papers above. The important takeaway is that you get the appearance and behaviour you might expect of an untyped language such as Python (notice, the types of b and c are never declared) and yet Scala is actually strongly typed.
Here's the coup de grace. Let's write something that stores the return of the integer map operation in a variable, then let's assign the result of the floating point map operation to the same variable. I'm doing this in the interpreter's 'paste' mode so it's compiled all in on go:
scala> :paste
// Entering paste mode (ctrl-D to finish)
val b = BitSet(2,4,8)
var c = b.map(v => v + 1)
c = b.map(v => v * 1.5)
^D
// Exiting paste mode, now interpreting.
<console>:10: error: type mismatch;
found : scala.collection.Set[Double]
required: scala.collection.BitSet
c = b.map(v => v * 1.5)
^
Note that's a compile time error, not a run-time one, and the parts involved are all library classes, not fundamental language features. How does it work? Well, the type of c is inferred by the compiler so it knows that c is a BitSet. It can also work out at compile-time that the return type of the second map operation will be Set[Double], and that's not type-compatible with BitSet, so you get a compile-time error. Yet there's not an explicit type declaration in sight. Python, eat your heart out ;-)
I don't just think this is cool because I'm some sort of strict-typing bigot, one of my teensy claims to fame is that my name is in the Perl AUTHORS file so feel I've earned my dynamically-typed Scout's badge. However, given a choice between strong and weak typing I'd prefer strong typing just because it helps protect me from my own stupidity. Obviously the example above is not the only possible use of these Scala language features, but I do think it's a good, simple illustration of how they can be used. And no, this kind of power isn't easy to wield. As the second paper listed above says:
Getting this design right was very hard, however. It took us about a year to go from a first sketch to the final implementation. In doing this work, we also encountered some dead ends.
However saying that Scala shouldn't provide support for powerful concepts just because they are hard to get your head around seems completely wrong-headed. The difficulty is in the concepts more than it is in the implementation. The blog post that prompted this points to the Fantom language as a superior alternative to Scala, yet the Why Fantom page contains the following:
Currently Fantom takes a limited approach to generics. There is no support for user defined generics yet. However, three built-in classes List, Map, and Func can be parameterized using a special syntax. For example a list of Ints in Fantom is declared as Int[] using the familiar array type syntax of Java and C#. This trade-off seems to hit the sweet spot where generics make sense without complicating the overall type system.
So pretending the need for Generics doesn't exist seems to be their solution. Boggle.
Stupid BT router...
I have the misfortune to own a BT Business 2Wire router, and bleaklow.com sits behind that. A while back it threw a fit and reconfigured itself and when I beat it into submission I didn't notice that it had also lost the port mappings for bleaklow.com. As a result I've been off-air for quite some time. Bloody BT...
Hints on creating 256-colour themes for vim
Most existing vim colour themes are designed for the graphical version of vim, gvim. They don't work properly at all if you are running vim in a terminal, and there are very few themes available for vim in terminal-mode. You are pretty much obliged to make your own if one of the small number of available ones doesn't suit. Here's a list of hints and tips that I discovered in the process of creating one:
- If your shell's setting for
$TERMisxtermyou won't get colour support. It needs to bextermcor you need to add aset term=xtermcin your.vimrc. - Even with the terminal type set you may not get 256 colours if you are using gnome-terminal. To fix that, put
set t_Co=256in your.vimrc. - In vim, type
:help highlightto get instructions on how to specify colours in vim. - To make the process of authoring a colour theme easier, install the Mkcolorscheme.vim script in your vim plugin directory, uncommenting the four commands at the top of the file. Read the instructions in the plugin, they give you hints on how to identify the colours currently being used by syntax elements.
- Open a new subwindow in vim and type
:so $VIMRUNTIME/syntax/hitest.vim. That will display a table of the current settings, which will be updated as you interactively change colours. - Open this colour chart in your browser, it gives you the colour numbers you'll need to set the colours.
- Open up a source file of the type that you want to set the syntax colouring for, put the cursor over an element and use the
:GetSyntaxcommand to identify the element type, followed by the appropriate:highlightcommand to set the element colour. - When you are done, use the
:Mkcolorschemecommand to generate a set of commands to generate the colour scheme settings. Delete any lines containingclearedas they aren't valid. - Save the colour scheme to a file in your vim
colorsdirectory and load the scheme with acolorschemecommand in your.vimrc.
