Ambrose has received his Clojure “scholarship”, thanks to you.

The eve of Clojure Conj 2011 is upon us, so I feel compelled get off my butt and finally close the loop on this one.

Per usual, you, the Clojure community, have come through.  Ambrose Bonnaire-Sergeant will be attending the Conj and giving an introductory talk on logic programming, only made possible after $3,000 was raised through donations solicited via this blog and various pestering on Twitter, IRC, and the Clojure mailing list.

(As was mentioned from the start, the Conj organizers do provide a travel stipend, even a generous one for international speakers, but the costs of getting to North Carolina from Western Australia are far beyond that budget.  Thus the need for a fundraiser.)

Finally, I will soon be notifying the winners of the free copies of Clojure Programming from O’Reilly (signed by myself and hopefully my coauthors as well, Brian Carper and Christophe Grand) privately by email.  Your extraordinary generosity is a big part of how this effort was able to succeed.

Thank you!

To those that donated, thank you very, very much.  Your assistance is not taken for granted:

  • Anonymous x 3
  • Sam Aaron
  • Harold Ancell
  • Craig Andera
  • Justin Balthrop
  • Carlos Barboza
  • James Christopher Bare
  • Meikel Brandmeyer
  • David Cabana
  • Brian Cooley
  • Aaron Crow
  • Paul deGrandis
  • Manuel Angel Rivas Diaz
  • Alan Dipert
  • Cameron Dorrat
  • Kevin Downey
  • Jim Duey
  • Holger Durer
  • Antoni Batchelli Estrada
  • John Fingerhut
  • Louis Franco
  • Joseph Gallo
  • Baishampayan Ghose
  • Stephen Gilardi
  • Juan Manuel Gimeno
  • Christophe Grand
  • Anthony Grimes
  • Philip Hagelberg
  • Mark Henderson
  • Richard Hickey
  • William Hidden
  • Daniel Jomphe
  • Geoffrey Knauth
  • Vinod Kurup
  • David Liebke
  • Kent Mabee
  • Christopher Maier
  • Durgesh Mankekar
  • Roberto Mannai
  • Roberto Mannai
  • Nathan Marz
  • Carin Meier
  • Dave Newton
  • David Nolen
  • Mark Nutter
  • Alexey Ott
  • Donald Parish
  • Laurent Petit
  • Michael Ravits
  • Volker Schlecht
  • Ryan Senior
  • Patrick Shields
  • Stuart Sierra
  • Luo Tian
  • Benny Tsai
  • Moritz Ulrich
  • Craig Worrall
  • Koji Yusa
Posted in Clojure | 5 Comments

Enabling rich(er) interactions in the Clojure REPL

I love the Clojure REPL.  I’ve never been more productive in any other environment.  In particular, I happen to like the REPL and general development experience provided by Counterclockwise, the Eclipse Clojure plugin.

That said, the Clojure REPL is far from perfect.  A long time ago, I banged out this page describing what my ideal Clojure development environment would look like.  Much of what’s there is still an accurate reflection of my ideal end-state, and much of it is now available in Counterclockwise.  But, some things remain unfulfilled.

Like, when you are working with image data.  I do, all the time.  I’m parsing documents, producing rasterizations of them, performing all sorts of transformations and analysis on them, often in the REPL. Yet, I end up needing to have other tools around to do the simplest of things…like, oh, to look at an image.  Other fancier things should be possible, but let’s start with that and see where we end up.

This corresponds with this bullet in the aforementioned description of an ideal Clojure environment:

  • Configurable pretty-printing of output
    • This also means being able to “print” non-textual data, such as having images drawn inline into the REPL if so desired, etc.
    • See Factor’s REPL for an example

So, I implemented opt-in “rich content” Clojure REPL interactions in nREPL (a tool-agnostic network Clojure REPL server & client implementation), with a first spike of client support for such “rich content” in Counterclockwise.  The result is something that I think can be the basis for breaking a lot of new ground in how we use the [Clojure] REPL.

Video demo follows, with some discussion of the implementation and ways clients can add their own response types and such dynamically (direct link for HD video):

The code for this is currently on nREPL HEAD (the SNAPSHOT coordinates are [org.clojure/tools.nrepl "0.0.6-SNAPSHOT"]), and this branch of Counterclockwise (which, along with a couple of refinements, will hopefully be rolled into the next release). It’s all bleeding-edge stuff, so I’m sure there’ll be lots of change to come, but it’s a start.

Go poke and prod at my approach, let me know what you think — and, hey, it’d be awesome if someone knocked out rich content REPL  support for another tool, be it jark (which already uses nREPL), vimclojure, Enclojure (both of which are planned to use nREPL eventually AFAIK?), or others.

Posted in Clojure, lisp | 10 Comments

À la carte configuration in Clojure APIs

There are two dominant configuration patterns in Clojure libraries. The first is where configuration is provided explicitly via the first argument; here, in Rummage, accessing Amazon’s SimpleDB:

(require '[cemerick.rummage :as sdb])
(def config (sdb/create-client "aws id" "aws secret-key"))
(put-attrs config "demo"
  {::sdb/id "foo" :name "value" :key #{50 60 65}})

The other is where the configuration is defined implicitly, usually using binding and dynamic scope (and sometimes via a set!-style setter for REPL usage convenience); here, in Clutch, accessing CouchDB:

(require '[com.ashafa.clutch :as clutch])
(clutch/with-db "localhost"
  (clutch/put-document {:a 5 :b 6} :id "foo"))

The latter is arguably more common, especially in database libraries; in addition to Clutch, you can see the dynamic pattern in play in java.jdbc‘s with-connection and congomongo‘s with-mongo.

From the perspective of a user (they are us!), I sometimes prefer dynamic scope to avoid verbosity, yet I often like to be explicit about configuration (and therefore, usually the target of my code’s activity) at other times, especially when dynamic scope isn’t appropriate or downright dangerous.  My preferences vacillate depending on what I’m doing, where I’m doing it, and what tools I’m using.  In any case, each library that requires configuration almost always requires that you work with it the way its author intended, so I am left with no joy half the time.

As an author of and contributor to such libraries (including the two mentioned above), perhaps I’m in a position to resolve this dilemma.

Irreconcilable differences

Consider any function that needs to make use of configuration, be it a session, a database connection, etc.  As we’ve seen, there seem to be only two implementation strategies: either take the configuration as an explicit argument, or assume the configuration has been bound dynamically elsewhere (as e.g. with-db does in the Clutch example above):

(defn explicit-save
  [config data]
  ...do something with `data` in/at/with thing described by `config`...)

(def *config* nil)
(defn dynamic-save
  [data]
  ...do something with `data` in/at/with thing described by `*config*`...)

There is no way to unifying these two idioms. The only option is to manually provide additional arities of every single function in my API, delegating as necessary when it is suspected that dynamic scope is being used:

(def *config* nil)
(defn broken-save
  ([data] (broken-save *config* data))
  ([config data]
    ...do something with `data` in/at/with thing described by `config`...)))

In my opinion, this is a no-go from an implementor’s perspective: each additional library function implies extra maintenance for each additional arity, or the prospect of rigging up an alternative defn macro that adds the additional arity “automatically”…which doesn’t work if the API is to support rest or keyword args anywhere.

Not one to be disheartened, I’ve been working on an alternative that is either a proper solution, or a hack potentially even more ill-conceived than said defn macro.

Tasty brew: ns-publics + binding + partial

First, let’s implement our library as simply as possible, which means working with explicit arguments everywhere (you can build dynamic scope on top of simple functions, but it’s damn hard to make functions that depend on dynamic scope appear to do otherwise):

(defn save
  [config data]
  ...do something with `data` in/at/with thing described by `config`...)

Now, let’s think about what *config* really represents in prior examples: it’s an implicit indication of the scope of an operation.  We can get a similar effect using partial, which returns a new function that will use the provided arguments as the first arguments to the original function; using it, we can call (a derivative of) our save function with a single argument (our data), making the configuration “implicit” again:

((partial save {:some :configuration})
 {:some :data})

That’s hardly a syntactic improvement over explicitly passing our configuration value explicitly.  However, what if we had a with-config macro that performed this partial evaluation for us, supplying the configuration value to each of our library’s functions so that, within the with-config macro’s scope, each of those functions could be called sans configuration?  Well, we have macros and a reified dynamic environment, so let’s have at it:

(def public-api (vals (ns-publics *ns*)))

(defmacro with-config
  [config & body]
  `(with-bindings (into {} (for [var @#'your.library.ns/public-api]
                           [var (partial @var config)]))
     ~@body))

Explanation is surely in order.  First, we need define our public API; this is just a seq of the public vars in our library’s namespace (which need to be dynamic since we’re going to be rebinding all of them; make sure to use ^:dynamic metadata on them if you’re using Clojure 1.3.0+).

(It seems sane to me that this seq should be filtered based on other metadata to ensure that only those functions that take configuration as their first argument are included.  An example of this is below.)

Second, all our with-config macro does is set up a dynamic scope, binding to each of our library’s vars new functions with the provided configuration partially applied.  Within that scope, we can omit any further reference to the configuration value, even though the foundational implementations of our library’s functions require explicit configuration.

Here’s a complete example (which requires Clojure 1.3.0 because of the ^: metadata notation — porting to Clojure 1.2.0 is simple enough, and left as an exercise):

(ns example)
(defn ^:api ^:dynamic save
  [config data]
  (println (format "Saved %s with %s" data config)))

(def public-api (->> (ns-publics *ns*)
                  vals
                  (filter (comp :api meta))
                  doall))
(defmacro with-config
  [config & body]
  `(with-bindings (into {} (for [var @#'example/public-api]
                             [var (partial @var ~config)]))
     ~@body))

The save function takes configuration explicitly; also, I’ve added ^:api to its var’s metadata so our public-api seq of vars can be filtered of vars that shouldn’t be affected by with-config‘s dynamic scope.  Now our library can support both explicit and dynamic specification of configuration, yet we never really thought at all about the dynamic case when implementing the library:

=> (save {:some :config} {:a 5 :b 6})
Saved {:a 5, :b 6} with {:some :config}
nil
=> (with-config {:some :config}
     (save {:a 5 :b 6}))
Saved {:a 5, :b 6} with {:some :config}
nil

Fin?

I love the flexibility this approach affords the user (usually me!), with, in relative terms, minor effort on the part of the library author.  I’m enough of a fan of it that I’m using it in Clutch (hopefully to be released soon as part of v0.3.0).

However, I should say that I’m not yet entirely at ease:

  1. If your implementation of one public API function calls another, and the vars of both are being rebound by with-config(or its equivalent), then that intra-library function call is going to route through the var and get the function that already has the configuration value partially applied.My solution to this at the moment is to (ack!) use a defn-wrapping macro to define each public function in Clutch that pushes each definition into a closure containing all of the already-defined functions.  This keeps intra-library calls from getting mixed up in the dynamic scope that Clutch’s version of with-config might set up.
  2. Except for the results from one (likely errant) REPL session, I believe that self-calls never route back through the var named in function position.  However, if there is any case where that’s not true (i.e. if self-calls do route through the named var), then self-calls would have the same problem (but not the same — or any — solution) as (1).
  3. Using something like with-config, you’re looking at N partial invocations, N function instantiations, and N dynamic bindings for your library with N public functions, versus 0, 0, and 1 for dynamic configuration APIs that bind a single *config* var.  Insofar as all of the libraries that use this pattern that I know of are database and/or IO-related, this “overhead” can likely be discounted.  In any case, if you don’t want any overhead, with-config gives you the option of no dynamic scope at all.

If you have any comments, warnings, or rants about how this is evil, do share.

Posted in Clojure | 26 Comments

Writing CouchDB Views using ClojureScript

UPDATE [2012-05-07]: clutch-clojurescript is now deprecated, as its functionality has been rolled into Clutch proper in toto.  Documentation for the feature can be found here.

While I was in San Fransisco for JavaOne, I was lucky enough to be invited to speak at the Bay Area Clojure User Group (thanks, Sean and Toni!).  It was a great time, and gave me the kick in the pants I needed to finish hacking away at my first project involving ClojureScript: using it to write view functions for CouchDB.

The result is clutch-clojurescript, which naturally builds on top of the Clutch library that I’ve collaborated on with Tunde Ashafa for some time now.

My motivations for doing this were manifold:

  1. I quite enjoy using CouchDB, as its model and general philosophy meshes very naturally with my (and my tools’) disposition and the data I work with most often.
  2. The operational hassle associated with maintaining a Clojure view server (which Clutch provides) configuration alongside my CouchDB installs was always a hassle.
  3. I’ve been wanting to do more and more with Cloudant, but a Clojure view server is just not an option with a hosted database-as-a-service like that.
  4. I can’t stand writing JavaScript.  Give me the reach of JavaScript, but with sane abstractions, homoiconicity (macros!), and data structures that aren’t braindead? Sign me up.

Feel free to go check out clutch-clojurescript: beat on it some, and let me know if it breaks on you.  I would eventually like to fold it into Clutch proper.  Beware some limitations though — repeated here from the README in part to draw attention to them:

  • ClojureScript is not yet available as a proper library. This forces me to include some binary version of it in this git repo (a hefty 8.3MB!…which includes various google JavaScript UI bits that I’d hope would be broken out eventually), and bundle the necessary bits into the clutch-clojurescript jar. I would very much like to roll clutch-clojurescript’s functionality into Clutch proper, but I’ll not do so until the latter can rely upon a ClojureScript dependency.
  • ClojureScript / Google Closure produces a very large code footprint, even for the simplest of view functions. This is apparently an item of active development in ClojureScript.In any case, the code size of a view function string should have little to no impact on runtime performance of that view. The only penalty to be paid should be in view server initialization, which should be relatively infrequent. Further, the vast majority of view runtime is dominated by IO and actual document processing, not the loading of a handful of JavaScript functions.
  • To my surprise (and shock/horror), the version of Spidermonkey that is used by CouchDB (and Couchbase Single, and Cloudant) does not treat regular expression literals properly — they work fine as arguments, e.g. string.match(/foo/), but e.g. /foo/.exec("string") fails.  Using the RegExp() function with a string argument *does* work.  This was reported a long time ago, but has had little attention (though I’m trying to stir it up a bit).I’m hoping to get to the bottom of this sooner or later, but I wonder if it’d be worthwhile to change the ClojureScript reader to emit (js/RegExp "foo") calls instead of /foo/ literals (and hope that gClosure doesn’t optimize the former into the latter)?  After all, there’s lots of CouchDB deployments out there with apparently broken spidermonkey installs/configurations, and likely lots of other apps/servers/environments in similarly dire straits.

Finally, here are the slides from my talk at the BACUG (download/view PDF):

Posted in Clojure, couchdb | 1 Comment

2011 Clojure Scholarship: Help send Ambrose to the Conj!

Update 2011-11-08T01:41  We did it! Details and acknowledgements for all who donated now here!

About a year ago, I had the distinct privilege of having a front-row seat when the generosity of the Clojure community (and others!) helped Anthony Grimes (aka Raynes) attend the 2010 Clojure Conj.  Witnessing the success of that process made me certain that I would attempt to coordinate another “scholarship” fundraising drive for this year’s Clojure Conj.

Like Anthony last year, I wanted to help someone that:

  1. had already contributed significantly to the Clojure community,
  2. clearly has great potential, but
  3. without some financial assistance, would not be able to attend the Conj and benefit from the meeting-of-the-minds that such conferences uniquely enable.

Over the summer, it became clear to me that Ambrose Bonnaire-Sergeant was that someone.  A computer science undergraduate at the University of Western Australia in Perth, Ambrose has distinguished himself in the Clojure community this year.

His introduction to many was PragPub magazine publishing his Growing a DSL with Clojure article in July.  In the article, Ambrose led us through the building of a simplified shell-scripting DSL in Clojure modeled on the open source stevedore library that he’s contributed to.  More recently, Ambrose has been busy hacking away at significant contributions to the core.logic (logic programming) and core.match (optimized pattern matching and predicate dispatch) libraries led by David Nolen, producing content like his Logic-Starter documentation project that provides a compelling logic programming primer using core.logic as the raw materials, and helping out with jark tooling.  Finally, his has been a consistently helpful voice on the main Clojure mailing list.

Now, getting Ambrose to the Conj is going to be somewhat more challenging than last year’s effort: he’s in Australia after all, which makes both the travel and lodging expenses quite steep.  To make matters even more interesting, flights from Perth to Raleigh/Durham seem to fluctuate in price significantly — from $2,500 to $3,500, depending on when and where I check them.

Thankfully, Ambrose did submit a talk proposal to the Conj — an Introduction to Logic Programming with Clojure — which was accepted!  This means that the Conj organizers will be picking up his lodging during the conference, as well as generously providing $1,000 to help defray Ambrose’s travel expenses.  Thus, we (only?) need to raise enough to cover the remainder of his flight, as well as two extra hotel nights (so he can recuperate from jet lag enough to deliver his talk without experiencing a sudden bout of narcolepsy!).

To be safe, I am therefore setting the fundraising goal to $3,000.  As long as the gods of airline pricing are not against us, this will safely cover the remainder of Ambrose’s expenses, allow him to attend the Conj, and allow you to enjoy his sure-to-be-enlightening introductory logic programming talk.  Even moreso than last year, it’s a sizable sum; but, again, I think this is a worthy cause, and I suspect the Clojure community will rise to meet the challenge.

(If any funds remain, they will be used to provide discounted admission to the Conj for other students.)

I’ve set up a PayPal donation button below, along with a progress bar.  Some FYIs:

  • I’m just reusing Snowtide‘s PayPal account for this; so, you’ll be seeing its logo through the checkout process and “snowtide” on your credit card statement.
  • I’ll be manually updating the progress bar, so don’t expect to see it go up immediately after donating!
  • I’ll shut down the donation button once we hit the $3,000 target.
  • I’ll add your name to this page as thanks, unless you notify me otherwise, either (preferably) in the comments associated with your PayPal transaction or by email/twitter/irc.

Finally, as an added incentive: at the end of the fundraiser, the three people with the largest donations will each receive a copy of my forthcoming Clojure book, Clojure Programming from O’Reilly, signed by me (and probably my co-conspirators as well, Christophe Grand and Brian Carper).

If you have any questions at all, leave a comment here, or don’t hesitate to contact me if it your question involves confidential details (payment issues, etc).

We did it!

Details and acknowledgements for all who donated now here!

Posted in Clojure | 3 Comments

McCarthy on the W3C

I happened to listen to John McCarthy’s keynote at OOPSLA 2007.  There are some real gems in there for those interested in the history of Lisp, though I admit that I mostly blanked out through the exposition of his Elephant programming language — probably not something amenable to a purely audio delivery, at least for me.

Anyway, he delivers a great zinger at the end, a tangential response to a question about the aims of Semantic Web technologies as they relate to his work on Elephant:

When w3c decided to not use [s-expressions], but instead imitate SGML [for HTML], that showed a certain capacity to make mistakes — which, probably, they haven’t lost.

Nice.

Posted on by | 1 Comment

Clojure で適切な型定義を選択するためのフローチャート

A few weeks ago, I put together a flowchart to help choose the right type-definition form in Clojure in various circumstances.

Afterwards, OGINO Masanori contacted me, offering to produce a Japanese translation of the flowchart.  The result looks great…thank you, Masanori!  We both hope it is useful to all Japanese Clojure programmers:

Clojure で適切な型定義を選択するためのフローチャート

Posted in Clojure | 1 Comment

At what point does S3 break?

AWS’ Jeff Barr announced yesterday that their S3 service:

holds more than 449 billion objects and processes up to 290,000 requests per second for them at peak times

I’m a very happy user of S3 and much of the rest of AWS, but seeing figures like this forces me into risk-assessment mode: at what point does S3 (or similar services) break?

Yes, S3 has had outages (2008 was a bad year), but these were fundamentally minor.  In the scheme of things, the equivalent of someone tripping over the rack’s power cord.

What I’m wondering about is, where does S3 et al. start hitting fundamental limits that amp problems up from oops to ohhh shit?

S3 is presumably the largest system of its kind ever; it’s not clear to me that anyone would really know what its failure modes, thresholds, or weak links might be as it continues to grow. Anything from a breakdown in the Dynamo architecture to hard infrastructure limits to failures in operations and management strike me as plausible. What will we see first: data loss, increases in latency, repeated catastrophic outages, or “other”?

Posted in Amazon Web Services | 1 Comment

Scala isn’t complicated; it’s clever

I’ve been away from Scala for a long while now — a little more than three years.  v2.7.1 was the last rev I used significantly, if memory serves.  I enjoyed my time with it, but it just wasn’t the best fit for me.

Anyway, I’ve learned that sometime in that era, Scala added support for the use of Unicode arrows as optional alternatives to the usual ASCII approximations it’s always had.  For example, this:

for (x <- 1 to 10) { ... }

is equivalent to this:

for (x ← 1 to 10) { ... }

The second example uses a Unicode instead of the ASCII <-; other arrows (corresponding to -> and =>) also have Unicode equivalents in Scala.  There are various other Unicode characters that have been proposed to serve similar roles, generally with positive discussion around them, including , , , , , , and .  This syntactic flexibility isn’t defined in a library, or in private code, but in the language itself.

I’m aghast.  This blog now sports a “WTF” category.

An early Scala-specific keyboard design.

An early Scala-specific keyboard design.

In the ticket that suggested the addition of one of these equivalencies, various potential problems are pointed out, and presumably ignored.  Perhaps people yearn for APL; unfortunately, suitable keyboards are not widely available.

I’ve no issue with Unicode identifiers: I work with a language that imposes no restrictions at all on the characters used, so programmers that want to use Japanese, Cherokee, Sanskrit or Greek characters in their identifiers can have at it.  Being able to opt in to such things is unequivocally good.  However, that language wisely tries to avoid doing clever things, like aliasing the core anonymous function operator to the λ character for everyone1.

There’s been various discussions, controversy, and gnashing of teeth about whether Scala is a complicated programming language.  I won’t get into that; that’s an argument over semantics that I don’t think is framed properly to begin with.

Maybe, though, most of us can agree that, if Scala isn’t complicated, it is at least clever, and things like this are just the most facile examples of that nature.  We are left to our personal biases as to whether or not that is commendable, desirable, or good.


1Discussions like this must mention Fortess, and its duality of language representation: one writes Fortress in ASCII, but the language defines a sort of markup within that character set so as to produce quite elegant renderings of code suitable for publishing.  See an example here.

Posted in Scala, WTF | 14 Comments

Results of the 2011 State of Clojure survey

A few weeks ago, I opened the 2011 State of Clojure survey.  As with last year’s survey, my aim was to take a snapshot of the Clojure community — our origins in aggregate, how and where we are using Clojure, and what needs to improve to help the community grow and prosper.

First, some facts about the survey and how it was conducted. It was available for approximately 5 days (Wednesday night through Monday afternoon), during which time results were not available.  The survey was announced primarily via Twitter and two messages to the main Clojure mailing list.  670 responses were received — which, for sake of comparison, is ~13% of the ~5,000 subscribers to that mailing list.  As I’ve said before, I’m not a statistician, nor am I a professional when it comes to surveying, polling, or data collection of this sort in general, but this would seem to be a very fair sampling of the Clojure community.

With that, I’m going to run through each question in the survey, and offer my own perspective on each question’s results.  In contrast to last year’s results post, I’ll skip the Clojure cheerleading: there were certainly oodles of glowing comments of how fantastic Clojure is, but I think it’s safe to take that as a baseline at this point, and talk more about how things can be improved. :-)

See the link at the end to the raw data to satisfy all your statistical urges.

(Note: Any question whose results add up to more than 100% allowed respondents to select more than one response.)

Q: How long have you been using Clojure?

I think it’s safe to say that the Clojure community is growing (certainly more subscribers to the ML, more respondents to this survey than last year’s, more companies hiring Clojure programmers, etc), but trying to more precise than that would be folly.

One thing to note is that, nearly half of all respondents have been using Clojure for less than a year.  Last year had similar relative numbers of newcomers, but we aren’t seeing a big bulge of people reporting that they’ve been using Clojure for a year here.  This tell me that we’re doing a good job of attracting people to try Clojure, but they’re dropping out after some period of experimentation.  Surely some of that is natural, but some of that attrition can likely be attributed to the state of the resources provided to help people get started, and then get productive.  More about that later.

Q: How would you characterize your use of Clojure today?

Compared to last year’s survey results, there’s been an 8-point bump in the proportion of respondents that use Clojure at work, and an 8-point bump in the proportion that use it in school.  Though tinkering and hobby projects aren’t to be dismissed, this rise in “serious” use of Clojure is a very good sign — more people are feeling more comfortable using it the most important areas of their life.

Q: What is the status of Clojure in your workplace?

As was the case last year, 50% of respondents don’t use Clojure at work, but the rest of the numbers are pretty compelling: 31% use it at work in an “endorsed” capacity (up from 23% last year), and 22% of that usage is in production settings.  Again, signs that Clojure is being relied upon more, and perhaps with relatively little pushback from management given the low 9% rate of under-the-radar Clojure usage.

Q: In which domain(s) are you using Clojure?

This chart is nearly identical to the results from last year.  Web development and math/data analysis remain ubiquitous; lots of people are contributing to open source Clojure projects, and nonrelational databases remain preferred by Clojure programmers over RDBMS.

I enjoyed reading through the “other” domains people specified, including:

  • Unmanned Vehicle Control
  • AI / Natural Language Processing
  • Geoinformatics / GIS
  • semantic web
  • Rapid prototyping of ideas that are later ported to iOS. (Is this common?)
  • music via Overtone

The first two domains means that the robot invasion is coming, and they’ll all be running Clojure.

Q: Which development environment(s) do you use to work with Clojure?

This is also nearly identical to the results from last year, with mostly minor moves.  Emacs continues its dominance, though Vim + vimclojure did tick up to 21% from 18% last year.  All the other environments retained the same levels of support, except for the Netbeans + Enclojure combination, which dropped from 13% last year to 5% this year.

I remain baffled by the significant usage of command-line REPLs.  Just not my cup of tea, I suppose.

Honorable mentions in the “other” column include Sublime Text 2, JEdit, and Bluefish.  Also:

MCLIDE…should be a model for everyone.

Q: Clojure is primarily a JVM-hosted language. Which other platform(s) would you be interested in using Clojure on, given a mature implementation?

First, a couple of quotes from the “other” column:

None, JVM is enough for me.

and:

None. You have better things to do.

Compared to last year’s survey, CLR/.NET and Cocoa dropped quite a bit in relative terms, probably due to my adding a separate LLVM option.

Honorable mentions include Parrot, Erlang/BEAM, and Guile.

Those that want Clojure for the CLR should go check out ClojureCLR, tirelessly maintained by David Miller.  I honestly don’t know what its status is, although I have heard that some people are using it in serious applications these days.  If it is indeed “mature”, I apologize for implying that it’s not by including it in the question.

I’m inclined to drop this question from future surveys.  It’s sort of like asking, “What color unicorn would you like to own?” The implementation of other backends for Clojure is an incredibly difficult undertaking, and will likely be done very quietly by individuals or very small teams aiming to solve persistent problems of their own, not in response to an anonymous show of hands.

Q: What language did you use just prior to adopting Clojure — or, if Clojure is not your primary language now, what is that primary language?

As in last year’s survey, Clojure programmers indicate that they “come from” Java, Ruby, and Python, in that order and in roughly the same distribution.  Everything else is background noise.

Q: If Clojure disappeared tomorrow, what language(s) might you use as a “replacement”?

A quote from someone’s “other” response:

There is no replacement.  Please, do not disappear.

The winners here remain constant from last year.  Clojure programmers really like functional programming, and enough of them like the JVM enough to jump to Scala or JRuby if necessary, even though that’s not where they came from.

Honorable mentions include Go, Factor, F#, and Racket.

Q: What have been the biggest wins for you in using Clojure?

Various miscellaneous thoughts:

  • As much as some people gripe about the JVM, a far larger number are benefiting from it; ~60% noting it as a “big win” is a helluva endorsement of the choice of the JVM as a host platform.
  • People love their functional programming.  In fact, if you look at where Clojure programmers came from, and where they would go to if Clojure were to disappear (see the prior two questions), it seems clear that Clojure is a gateway drug for FP.
  • It seems that relatively few people are taking advantage of some of Clojure’s most sophisticated and unique features: metadata; protocols, records, and types; and multimethods.  These facilities are absolutely game-changers, at least IMO.  Either most domains have no use for them (I can’t believe that), or most people don’t know how to use them effectively, thus they are left unused.  Those of us that write about and teach Clojure, take note.

Q: What has been most frustrating for you in your use of Clojure; or, what has kept you from using Clojure more than you do now?

A lot of the most common and most frustrating problems were expanded upon in the free-form responses, so I talk about them a bit there (see below). Otherwise, just a couple thoughts:

  • Very few people have found the Clojure community to be a problem — just 2% report having unpleasant community interactions.  That’s both unsurprising and huge.  However, rather than simply bask in the awesomeness of the Clojure community, let’s ask: is it possible to make that number be 0% next year?  What can we do to minimize those negative interactions?  Are they caused by random misunderstandings, or are there just a few bad apples?
  • While there’s been a lot of third-party projects aiming to improve the Clojure documentation landscape (both ClojureDocs and Clojure Atlas received positive mentions), people remain unhappy with the state of Clojure’s documentation, tutorials, and other resources.  Last year’s survey didn’t ask this question, so we have no point of comparison, but my impression is that the number of complaints in this department has stayed steady.

Q: What do you think is Clojure’s most glaring weakness / blind spot / problem?

This is where most of the fun is: a free-form text entry field that respondents could use to vent their frustrations — and, of course, they did!  The topics named in the responses to this question ranged all over the place: from irritations about particular libraries to issues with outdated blog-documentation, people didn’t hesitate to express themselves.  If you’d like, go read what people had to say directly (see the raw data link below).  Here, I’ll run through the most common complaints I saw.

(FYI, the amount of attention I give to each issue is not proportional to the number of people that raised it; I just happen to find some issues more interesting to write about than others.)

The JVM

Some people love that Clojure uses the JVM; others, though, quite simply despise it.  Specific complaints range from its typical startup time (particularly painful for those wanting to use Clojure on the command line) to “leakage” of Java build and dependency management processes (i.e. all Clojure build tools reuse the Maven dependency model and many build on top of Maven itself) to difficulty using native libraries.  A vocal minority of Clojure programmers (at least within this survey) wish dearly to be able to use Clojure without the “baggage” of the JVM.

Personally, I am quite at home with the JVM, so this doesn’t resonate with me, but I can empathize.  The choice of the JVM was a practical one (discussed in the Languages and Platforms section here).  With the lone exception of the CLR/.NET, there’s really no other environment out there that you could reasonably build a language on top of and get a comparable mix of mature VM implementations, existing libraries, community and tool support, and so on…but, if you don’t like the VM options, the existing libraries, and have little use for such tool support, then the JVM is likely to feel like a 400-lb. high-maintenance gorilla you need to carry with you whereever you go.

That said, without its JVM underpinnings, I really doubt that we would see Clojure being accepted into commercial environments as we have been (note the figures above of people using Clojure at work), or supported by platforms like Heroku, Amazon Web Services, and so on.  The size of the sandbox you play in does matter — otherwise, we’d have seen broad third-party tool and platform support for Common Lisp and various Scheme implementations years ago.

No “Clojure Starter Kit”

People continue to ache for an easy way to get started with Clojure, a sane development environment, and the libraries they need.  The issues involved here are legion, and they’re all hairy: platform-native launchers, the pains of emacs/SLIME and/or the various IDEs, integrated documentation (with cross-referenced usage examples), tracking of library status (github forks + the deployment of personal artifacts to Clojars makes for much confusion), modern debugger support, and so on.

Some of these things have gotten better over the last year, some haven’t.  A couple of highlighted issues that are current pain points:

The default REPL sucks

I’ll agree with this one wholeheartedly.  Clojure’s default REPL, run via a plain-vanilla command line, is just really unpleasant.  It should ship with readline/jline out of the box, and some suggested (reasonably, IMO) that some form of tab completion would be most welcome.  This is something that other languages (JRuby and Groovy come to mind) have taken care of very nicely.

IDE desperation

As much as most people love their emacs, there is a significant number that, even if they use emacs currently, wish dearly for a “regular” IDE for their Clojure programming.  Eclipse is mentioned most often, followed by IntelliJ; both have plugins — Counterclockwise and La Clojure, respectively — but they are apparently not up to snuff for many respondents.

As a contributor to Counterclockwise, I can see how that could be the case.  Many enhancements to it have been made in the past year, to the point where I switched to it full-time sometime late last summer, but even my personal wishlist there is long.

In any case, it’s clear that more needs to be done in this area so that high-quality Clojure tooling is easily accessible to all.

Error messages

This is also a reprise from last year.  The Clojure stack is the JVM stack (a very good thing if you care about performance at all!), but that means that Clojure stack traces are fundamentally JVM stack traces.  This means that you’ll likely see a lot of intervening stack frames from Java code when you print a Clojure stack trace.  Many people find this confusing, especially beginners.

There are a number of libraries for helping with the readability of stack traces, including clj-stacktrace and stacktrace utilities included with Clojure in the clojure.stacktrace and clojure.repl namespaces. There is ongoing debate about how best to skin stacktraces to maximize their usefulness.

Separate from stack traces are the issue of compiler error messages, which was a big pain point last year.  I don’t see that same magnitude of complaints in this area; if I remember correctly, some effort was made to improve error messages emitted by Clojure’s compiler over the past year.  Perhaps those improvements were sufficient to alleviate the prior pains.

Debuggers

No matter what tooling you use with Clojure, there is no doubt that current debuggers need further improvement to meet peoples’ expectations.  While the excellent CDT is out there, there seems to be a fair bit of configuration pain associated with it.  And, I can’t be certain, but I suspect that requests for a capable debugger made via the survey may correllate strongly with responses wishing for better non-emacs development environments — making me think that “we need a better debugger” should really be taken to mean “we need a better point-and-click debugger”, with the sort of visually interactive data structure inspectors that Eclipse and Visual Studio have made de rigueur over the years.

“Parens still scare the crap out of people”

A number of respondents indicated that Lisp, as a basic concept (and in its typical parenthetical manifestation, i.e. “oatmeal with fingernail clippings mixed in”) remains an obstacle to Clojure’s continued adoption.

I don’t actually believe this.  Yes, I’m sure there’s plenty of people who react poorly to seeing Clojure (or any Lisp) at first, whether due to simple aversion to the unfamiliar, cargo-cultism, or lingering memories of pain associated with “old world” Lisps.  However, the first will be convinced by education, facts, and results (currently being delivered, based on the workplace usage stats above and recent Clojure-related business news), and the second will be convinced by the first.  I’m guessing the third group is small enough to not be a long-term problem.

Fundamentally, Clojure either is or is not a effective option for building compelling software.  If it is, then every successful, ideally, astonishing Clojure app that gets out in the world will convince some number that Clojure is something to consider for their efforts.  Everything else that matters flows from this.

Public relations

Where the prior topic might be considered the “public relations of the language”, this one is related to public relations for Clojure, the project.  This is an area that is ripe for bikeshedding, so it can be tough to separate the wheat from the chaff when it comes to suggestions and complaints, but I’ll take a stab at it (and do some bikeshedding of my own, I suppose!).

(I can hear the curmudgeon technologists’ eyes rolling already, but this stuff is actually important if you want Clojure to succeed broadly.  If you’d prefer Clojure to be a protected sanctuary for the 1337, by all means, let us know by arguing against efforts to promote Clojure.  Cheers.)

Project status and activity

I’ll quote one comment in its entirety:

Clojure is awesome.  So is the community.  The website should reflect a burgeoning vibrant community and today, it just doesn’t do that.  No news, no changes or updates, the copyright is from 2010.  The one job on the jobs page has been there for a while.  According to the Wayback Time Machine, the main page hasn’t been updated this year, and only twice (aug and dec) last year. That’s kinda depressing. Compare this with scala-lang.org, which is full of activity – case studies, news feeds and flare.  The Clojure.org website could have twitter feeds, google groups digests, a road map, some case studies… Even if the news was “Hey, we’re still here, super busy on 1.3…” it would be better than no updates.  Clojure.com has a lot going for it, perhaps Clojure.org should be a community link…

Showcasing some marquee projects, such as Cascalog and Incanter, interviews posted on the site, guest bloggers, tutorials, a forum would be great.

The scala-lang.org comparison is appropriate, but hardly unique; comparing clojure.org to any of erlang.org, jruby.org, or jython.org gives a similar impression to the passer-by of a site (and therefore, project?) that is less active.

Of course, this is all about perception, and instant, blink-of-an-eye perception at at that.  There’s tons of stuff going on that impacts the language itself, as one can see by checking out the development wiki’s dashboard stream, or the project’s JIRA activity stream, for example.  There’s no disputing the actual state of affairs with regard to activity.  Rather, the question is: why not surface those flashes of activity (perhaps along with e.g. feeds from Planet Clojure and #clojure tweets) so that people who wander into clojure.org can immediately have a positive impression, rather than requiring of them an analytical comprehension of Clojure’s minutiae?

The jobs page smarts in particular, especially since there are lots of Clojure jobs available. The jobs page notes that it is “provided as a courtesy to those companies who have supported Clojure“, but I wouldn’t bet on the average visitor taking note of that distinction before mashing the back button.  Indicators like this can hurt a new(ish) language looking to maximize its reach, doubly so when the indicator is false.

Insofar as Clojure’s web presence is supported entirely by disparate (hard-working, invaluable!) volunteers as they are willing and able, these sorts of issues are likely to persist.  My instinct is that they could be fixed pretty easily, but it wants for a single person with sufficient integrated technical, marketing, and design sense and the authority to drive the Clojure web presence effectively for all likely classes of visitors.

“What has Rich Hickey done for me today?”

That’s not a real quote, but my semi-exaggerated tongue-in-cheek summation of various comments: not just about Rich, but about all manner of Clojure specialists (I think one comment said “Clojure old-timers”!) who stood a post at the mailing list and in #clojure irc, seemingly constantly for a number of years doing their best to welcome newcomers, explain the smallest of details, defuse flamewars, and generally set the tone of the community.

The thing about those sorts of activities is that, as long as someone is participating in conversation, they’re not getting things done.  Over the past year, we’ve definitely seen some of the most senior Clojure programmers in the community “pull back” to some extent as they go about their business, putting Clojure to work building apps, building companies (and writing books!), and, yes, feverishly working on Clojure tools, libraries, and Clojure itself.

Aside from the fact that you can no longer pepper Rich with questions in irc whenever you feel like it (a facility that I’m sure I overused all on my own back in the day!), one concrete consequence of this is that the tenor of the community must maintain and evolve on its own to a larger extent than it did in years one, two, and three (it had to happen sometime!).  I think #clojure irc has done that, with a number of consistently friendly, helpful, knowledgeable people stepping up to help.  I personally think the mailing list hasn’t fared as well — it’s grown much, much larger than it was in the early days, and without a universally-accepted authoritative voice, it’s far easier for threads to go into the weeds than irc conversations.  Members of Clojure/core have been involved here to some extent (Aaron, Alan, and the Stuarts come to mind), but I think the medium is largely against them.

Another facet of this dynamic is that people had long used Rich’s statements in various public fora as a proxy for an official “roadmap”/status.  Thus, when Rich isn’t around to comment on direction, people start wondering about the fate of the entire endeavor.  Without the sort of public declarations/monitors of activity described in the prior section, this sort of speculation is probably unavoidable.

Raw Data

You can get the raw data here.  If you are particularly interested in the Clojure community, lead a significant open source project, or write about, teach, or promote Clojure, you would be well-served to browse around the raw results (especially through the free-form comment fields and what people noted for their “other” choices on certain questions).  Tip: don’t bother trying to read the long-form responses in the Google Docs UI, which seems to handle scrolling of spreadsheet rows containing lots of content very poorly — export to Excel or something.

Final Thoughts

Putting together this survey has again been fun, informative, and a good reminder of just how vibrant and promising the Clojure landscape is.  There’s been a lot of good news and good progress over the past year, and it’s clear that this has been accompanied by a lot of good growth, too.

The coming year will see the second Clojure Conj, the release of Clojure v1.3.0, and who knows what other advancements and surprises.  If you’re a Clojure programmer, I hope that reading this has inspired you to jump in to find a corner of the Clojure landscape that you can help grow and tend.

Posted in Clojure | 20 Comments