cemerick

The Clojure community has come through, in a big way.

At 1:26PM, I published the blog post announcing a small, informal fundraiser to help fund a “scholarship” for Anthony Simpson (a.k.a. Raynes), enabling him to attend the first Clojure Conj next month.  Check out the post if you’ve not seen it already; it’s got the backstory.

At 2:38PM, we reached our goal of $1,000 (actually, we overshot it slightly; PayPal doesn’t allow you to cap donations as far as I could tell, so I had to time when I disabled the payment button).  This will enable Anthony and an adult relative of his to travel to the Conj.  I suspect he’ll take full advantage of this opportunity.  And, I’ll bet there’ll be pictures, too.

That’s 72 minutes to raise just over $1,000, donated by 29 different individuals (whose names are listed below), with an average donation of $37.  My original post hit hacker news, so there may be a few non-Clojure programmers mixed in that total that were inspired by the cause – but I recognize most of the names.  In any case, I’m floored.

I knew the Clojure community was phenomenal, but the reception to this effort has been more than even I expected.  I figured we’d make the goal, but 72 minutes? Damn.

Did you feel like you wanted to donate, but the opportunity had already passed?

Please don’t squander that impulse.  This particular effort was worthy, but it’s certainly not unique.  There is no shortage of people who could use a helping hand in one form or another.  Many of them are far away somewhere – and the internet can help in those cases as it has here – but surely there’s somebody right outside your front door, or down the street a little ways that could benefit mightily from that small measure of extra human connection that only you (and maybe a few friends of yours) could provide.

If there’s one thing that this episode has impressed upon me as no prior experience has before, it’s that just a few people (even just one!) can make a difference.  That may sound trite, and it might seem naïve, but I’m quite certain now that it’s true.

Thank you!

To those who donated, thank you so very much:

Craig Andera
Brenton Ashworth
Justin Balthrop
Lance Bradley
Brian Carper
Martin Clausen
Anonymous Donor
Chas Emerick
Mark Engelberg
Antoni Batchelli Estrada
Tom Faulhaber
Christina Gurga
Aria Haghighi
Nathan Hessler Hessler
Richard Hickey
Chris Jones
Zachary Kim
David Liebke
Darwin Lo
Mark McGranaghan
Hans-Christian Meier
William Morgan
Chris Petrilli
Robert Pitts
Luke Renn
Stuart Sierra
Sandeep Srinivasa
Levi Strope
Matt Tessar

If you’re a Clojure (or other FP language) programmer, you might know Anthony Simpson already.  Notable projects of his include sexpbot, irclj, and the guts behind try-clojure.org. He goes by IORayne on twitter, and Raynes in #clojure on freenode irc, where he’s been a positive, helpful presence.

It turns out that he’s 16 years old, is home-schooled in Eldridge, AL (remarkably, population 184!)…and he would like to attend the first Clojure Conj next month in Durham, NC, but he simply doesn’t have the funds to do so.

When I found out about this, I knew there had to be a way to “make it right”.  Now, I don’t know Anthony personally and I can’t “vouch” for him in any traditional sense of the word, but he seems like a fine fellow given my interactions with him on irc: bright, eager to learn, and extraordinarily capable for his age (or, to be fair, for any age). Fundamentally, it seems that helping to get him to the Conj is the right thing to do.  And, at least to some extent, I can relate to his situation – I remember times when I was young(er), strapped for cash, and wishing I could participate in things more typically reserved for older folk in more fashionable locales that were over-employed compared to myself.

So, I talked with the other distinguished invited speakers at the Conj about potentially standing up a fundraiser to help get Anthony to the Conj.  The response was uniformly positive, and Relevance, the conference host and cradle of the Clojure/core team, has graciously offered to take care of his hotel accommodations and admission to the conference (huge thanks to Alan, Justin, David, and anyone else at Relevance that had any hand in making that happen!).

All that’s left to do is raise enough money to pay for transportation (the 10-hour drive between Eldridge and Durham would seem to be unnecessarily epic) and miscellaneous travel expenses (food, cabs, the forgotten toothbrush, whatever).  Either Anthony’s mother or his brother will be traveling and staying with him (definitely necessary for legal and liability reasons), so I figure we’re looking at a $1,000 fundraising target, which will cover:

• 2x round-trip airline tickets
• 4 days of misc. travel expenses (the two days of the conference, plus the two traveling days before and afterwards)

It’s a sizable sum, but I think this is a worthy cause, and I suspect the Clojure community will concur.

I’ve set up a PayPal donation button below, along with a progress bar.  Some FYIs:

• I’ll be manually updating the progress bar, so don’t expect to see it go up immediately after donating!
• I’ll shut down the donation button once we hit the $1,000 target.
• I’ll add your name to this page as thanks, unless you notify me otherwise, either (preferably) in the comments associated with your PayPal transaction or by email/twitter/irc.

If you have any questions at all, leave a comment here, or don’t hesitate to contact me if it your question involves confidential details (payment issues, etc).

We did it! Details, numbers, and thanks you’s to all who donated.

I recently ran into a problem related to ahead of time-compiling (a.k.a. AOT) some Clojure code that took me a little while to figure out. Here, I’d like to leave behind some breadcrumbs for anyone else that happens to run into the same problem.

Consider a namespace at ./foo/myns.clj:

(ns foo.myns)
(defn hi [])
(defn HI [])

Let the fun begin. AOT-compile the namespace into the cwd:

[catapult:/tmp] chas% java -Dclojure.compile.path=. -cp clojure-1.2.0-RC3.jar:. clojure.lang.Compile foo.myns
Compiling foo.myns to .
[catapult:/tmp] chas% find foo
foo
foo/myns$hi.class
foo/myns$loading__4403__auto__.class
foo/myns.clj
foo/myns__init.class

Ut-oh.  There are two defined fns, hi and HI, but there’s only one classfile, ostensibly for hi (note that each Clojure function AOT-compiles down to its own classfile).

[catapult:/tmp] chas% java -cp clojure-1.2.0-RC3.jar:foo clojure.main
Clojure 1.2.0-RC3
user=> (require 'foo.myns)
java.lang.NoClassDefFoundError: foo/myns$hi (wrong name: foo/myns$HI) (NO_SOURCE_FILE:0)
user=> foo.myns$hi
java.lang.NoClassDefFoundError: foo/myns$hi (wrong name: foo/myns$HI) (NO_SOURCE_FILE:0)

What’s going on here?

It took me a while (probably a few hours) of false starts before things clicked:

• My disk is formatted as Mac OS Extended (aka HFS+), but not the case-sensitive variety (the fs is still case sensitive insofar as it retains case information in filenames [just for display purposes as far as I can tell], but that’s as far as it goes).
• When the compiler writes out the classnames for each fn in the namespace, the second one it writes out goes to the same location as the first, but has a different name as far as the classloader is concerned.
• The (require 'foo.myns) invocation uses the classloader to find a class named foo.myns, which goes on to load the classes associated with each of the functions defined in that namespace (or, .clj files are found and loaded, if they’re newer than any classfiles or if there are no matching classfiles).
• The classloader finds a classfile for the foo.myns/hi function, but the class’ internally-defined name (HI) doesn’t match the requested name (hi), so an exception is thrown.

As a sanity check, I mounted a ramdisk, formatted it with case-sensitive HFS+, and voilà, I could AOT-compile myns.clj and require it without a problem.  Filesystem case-sensitivity isn’t usually something one has to worry about with most languages: if you can’t name identical-except-for-case source files on disk, then you are saved from being in a position of potentially compiling to identical-except-for-case class or object files.  However, because of how Clojure maps code to classfiles (remember, one classfile per function, with source files [usually!] designating namespaces), it’s not until one attempts to load a Clojure namespace from AOT-compiled classfiles does one run up against any trouble.

Solutions

The easiest solution is to simply not AOT-compile your Clojure code.  Avoiding AOT carries no runtime performance penalty (though initialization of each namespace will be slower), and source distributables will always be smaller.  Of course, there are many nontechnical and some technical reasons why AOT-compilation is a necessity in various circumstances, so source distributions certainly won’t be right for everyone.

At some point while reading this, you’ve probably said under your breath, “Well, you shouldn’t have function names that differ only in case anyway!”  I agree wholeheartedly, and it must be said that the above situation is highly exceptional.

However, sometimes one isn’t necessarily choosing function names – this is often the case when when generating code from some dataset.  In my case, I was generating functions for handling PDF tokens, many of which differ only in case (e.g. Tj and TJ are one example).  The general solution when you’re in a circumstance like this is that your macro (or other code-generating facility) needs to take care to mangle the function names being emitted so that they are guaranteed to be unique.

A trickier question is whether the Clojure compiler should be doing something to prevent this scenario due to accidental function name collision.  Off the top of my head, I can imagine simply checking for existing/matching classfiles on disk that nonetheless have non-equal filenames would be sufficient cause to raise an exception when AOT-compiling.  That change, or something similar, may yet come to pass; until then, hopefully the above will be helpful.

Update: Christophe Grand has pointed out a flaw in the case+ implementation below that affects usage of Java enums and classes in code that is AOT-compiled.  My bad.  I’m working on an update to case+ that will address the problem for enums – classes will likely revert to being unsupported.

For those that aren’t familiar, Clojure’s case macro is an über-switch statement that allows one to dispatch on any compile-time literal in constant time. This includes essentially all scalars (strings, keywords, symbols, numbers) and “composites thereof” (as the docs put it): literal sets, maps, and vectors. The latter feature pushes case a fair ways into pattern-matching territory.

Brief factoid interlude

Clojure’s defrecord would not exist as we know it without case, which backs the implementation of defrecord‘s keyword lookup mechanism.  Without that, records would be without the congruence with regular maps that they currently enjoy: each slot access would either need to run through get (e.g. (get some-record :slot-name)) or direct field access would be required (which would rule out function composition with keyword “accessors”, the use of the -> and ->> families of threading macros, etc).

The only downside of case is that it doesn’t evaluate the dispatch values that are provided. Only compile-time constants are matched, so if one has values that are defined in a var, accessible as (almost always static) Java class fields, or Java enums, you can’t use them as case dispatch values:

user=> (case javax.swing.JFrame/NORMAL
         javax.swing.JFrame/NORMAL "normal!")
java.lang.IllegalArgumentException: No matching clause: 0 (repl:78)
user=> (case 'javax.swing.JFrame/NORMAL
         javax.swing.JFrame/NORMAL "normal!")
"normal!"

As you can see, the dispatch value referring to the NORMAL static int field is being matched as a symbol, rather than the field it names. The same thing happens when attempting to refer to Java enums, values held by vars, classes, and so on. This is a problem insofar as I really don’t want to fall back to the sequential dispatch of cond or condp, but I need to use dispatch values that are more than just compile-time literals. My current solution is the poorly named case+ macro:

(defmacro case+
  "Same as case, but evaluates dispatch values, needed for referring to
   class and def'ed constants as well as java.util.Enum instances."
  [value & clauses]
  (let [clauses (partition 2 2 nil clauses)
        default (when (-> clauses last count (== 1))
                  (last clauses))
        clauses (if default (drop-last clauses) clauses)
        eval-dispatch (fn [d]
                        (if (list? d)
                          (map eval d)
                          (eval d)))]
    `(case ~value
       ~@(concat (->> clauses
                   (map #(-> % first eval-dispatch (list (second %))))
                   (mapcat identity))
           default))))

With this available, the interop love flows like butter, and all of the dispatch literals already supported by case are unaffected because they all evaluate to themselves (FYI, Direction is a pre-existing enum class of mine):

user=> (case+ javax.swing.JFrame/NORMAL
         javax.swing.JFrame/NORMAL "normal!")
"normal!"
user=> (case Direction/East Direction/East "east!")
user=> java.lang.IllegalArgumentException: No matching clause: East (repl-1:88)
user=> (case+ Direction/East
         Direction/East "east!")
"east!"
user=> (case String
         String "it's a string class")
java.lang.IllegalArgumentException: No matching clause: class java.lang.String (repl-1:90)
user=> (case+ String
         String "it's a string class")
"it's a string class"
user=> (def some-constant 42)
#'user/some-constant
user=> (case 42
         some-constant "Towels for everyone!")
java.lang.IllegalArgumentException: No matching clause: 42 (repl-1:93)
user=> (case+ 42
         some-constant "Towels for everyone!")
"Towels for everyone!"

The big caveat is that the values used for dispatch in case+ are still resolved at compile-time – it is a macro, after all.  So, you can’t put case+ in a let form and dispatch on a let-bound value (attempting to do this will lead to an error, as locals cannot be eval‘ed).  Dispatch only on values available at compile-time, such as classes and class fields (which include enum instances).

I’ve used this extensively with good results (case+ accounts for perhaps a 50% of my total case usage), but I am just slightly concerned about my doing an end-around the key behaviour of case in not evaluating dispatch values. I didn’t happen to be paying attention in irc when the case design discussions were ongoing, and I don’t yet fully understand the motivation of the restriction. So, I offer this up somewhat tentatively for now – your mileage may vary. Do pipe up if you find a situation were case+ falls down.

Clojure’s datatypes are mana in a variety of ways.  In particular, for application-level programming, defrecord is the near-ideal construct — baked-in protocol support, Java interface interop, direct field storage of slots, fast keyword “accessors”, fast-as-Java method implementations…the list of goodness goes on and on.

Of course, there’s always room for improvement.  One big pain point for me was that I have a couple of records that, due to the domain, have a lot of slots, each of which needs to have a specific default value.  defrecord‘s generated constructor doesn’t provide for any kind of default value for slots.  The naive solution would be to write a factory function that aligned its arguments with some defined set of defaults, but that sounded painful from the start: slot definitions are then functionally spread over multiple forms and the handling of arguments into that factory function would either be a maintenance or performance nightmare.

It’s good to avoid macros as much as you can, but cases like this is where they shine:

(defmacro defrecord+defaults
  "Defines a new record, along with a new-RecordName factory function that
   returns an instance of the record initialized with the default values
   provided as part of the record's slot declarations.  e.g.
   (defrecord+ Foo [a 5 b \"hi\"])
   (new-Foo)
   => #user.Foo{:a 5, :b \"hi\"}"
  [name slots & etc]
  (let [fields (->> slots (partition 2) (map first) vec)
        defaults (->> slots (partition 2) (map second))]
    `(do
      (defrecord ~name
         ~fields
         ~@etc)
       (defn ~(symbol (str "new-" name))
         ~(str "A factory function returning a new instance of " name
            " initialized with the defaults specified in the corresponding defrecord+ form.")
         []
         (~(symbol (str name \.)) ~@defaults))
       ~name)))

This macro is breaking my general rules of thumb that no def form should define more than one entity (here, a class and a factory function), and that def forms shouldn’t create vars with generated names.  Unfortunately, I don’t think there’s any other better approach to be had given what I’m aiming to do.

There’s an example in the docstring for defrecord+defaults, but let’s take a look at some possible usage in a REPL interaction:

user=> (defrecord+defaults SomeClass
         [size 0
          root nil
          color java.awt.Color/black])
user.SomeClass
user=> (doc new-SomeClass)
-------------------------
user/new-SomeClass
([])
  A factory function returning a new instance of SomeClass initialized with the defaults specified in the corresponding defrecord+ form.
nil
user=> (new-SomeClass)
{:size 0,
 :root nil,
 :color
 #<Color java.awt.Color[r=0,g=0,b=0]>}
user=> (assoc (new-SomeClass)
         :size 1
         :root "foo")
{:size 1,
 :root "foo",
 :color
 #<Color java.awt.Color[r=0,g=0,b=0]>}

Yup, macros to the rescue (yet again!). What’s particularly handy about this is that the defined defaults can be any Clojure expression, which is evaluated each time the factory function is invoked. A neat future enhancement would be for the factory function to have a keyword-argument override that takes slot names and values that supersede the corresponding defaults provided in the defrecord specification.

I wouldn’t be surprised if something like this ends up in Clojure itself eventually, as a specialization of a future defrecord factory function facility.